> For the complete documentation index, see [llms.txt](https://docs.pinot.apache.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pinot.apache.org/operate-pinot/troubleshooting/ingestion-faq/realtime-ingestion-stopped.md).

# Realtime Ingestion Stopped

## Symptons

When observed certain kafka partitioned stopped ingestion due to the segment commit failure.

Sample errrors:

```
2025/04/13 14:06:28.978 INFO [RealtimeSegmentDataManager_test_realtime_2__2__0__20250413T1148Z] [test_realtime_2__2__0__20250413T1148Z] Controller response {"buildTimeSec":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":true} for http://pinot-controller-2.pinot-controller-headless.pinot.svc.cluster.local:9000/segmentUpload?segmentSizeBytes=242027065&reason=rowLimit&buildTimeMillis=47403&streamPartitionMsgOffset=1393724252&instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&name=test_realtime_2__2__0__20250413T1148Z&rowCount=10000000&memoryUsedBytes=519779170
2025/04/13 14:06:28.978 WARN [RealtimeSegmentDataManager_test_realtime_2__2__0__20250413T1148Z] [test_realtime_2__2__0__20250413T1148Z] Controller response was FAILED and not COMMIT_SUCCESS
2025/04/13 14:06:28.978 INFO [RealtimeSegmentDataManager_test_realtime_2__2__0__20250413T1148Z] [test_realtime_2__2__0__20250413T1148Z] Could not commit segment. Retrying after hold
```

Usually here are the steps that a partition got stopped ingestion:

1. servers tell controller to commit,
2. controller ack and ask the lead server to commit
3. lead server failed to commit due to many reason( segment build time longer than the controller lease, server oom, etc)
4. the other server got permission to build and try to commit and also failed
5. you got a partition completely stopped

## Mitigations

To mitigate, we suggest below steps to ensure your setup is scalable and stable.

1. Ensure Pinot server directly save segments to deep store, avoid controller in the critical data path. Ref link: [Decoupling Controller from the Data Path](/operate-pinot/deployment/decoupling-controller-from-the-data-path.md) . This is the most critical fix as it will remove controller as the bottleneck for data commit:
   1. controller receive segment tarball from pinot server
   2. uncompress it
   3. extract segment metadata
   4. upload segment tarball to deep store
   5. update zookeeper segment metadata
   6. complete the protocol
2. For large realtime segment, suggest to use `DOWNLOAD` for `completionMode` so the other server replicas won't waste CPU cycles to build segments. Ref: [Realtime Table Config -> SegmentsConfig](/reference/configuration-reference/table.md#segments-config)
3. Limit the concurrent realtime segment build by configure Pinot servers:

```
realtime.max.parallel.segment.builds=2
```

This will reduce each segment build time to relief the segment commit timeout situation, as well the concurrent pressure on the controller side. Ref: [Server Config](/reference/configuration-reference/server.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.pinot.apache.org/operate-pinot/troubleshooting/ingestion-faq/realtime-ingestion-stopped.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.