> For the complete documentation index, see [llms.txt](https://docs.pinot.apache.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pinot.apache.org/operate-pinot/tuning/pause-ingestion-based-on-resource-utilization.md). # Pause Ingestion Based on Resource Utilization Pinot can pause real-time ingestion when disk utilization exceeds a configured threshold, and resume it when utilization returns within limits. The same resource-utilization status is also used to skip minion task generation for affected tables. ## How It Works The periodic task `ResourceUtilizationChecker` fetches disk-usage information from each Pinot server and stores it in the controller-side resource-utilization cache. Pinot then uses that cached status in two places: * `RealtimeSegmentValidationManager` pauses REALTIME tables when disk utilization is above the configured threshold. * `PinotTaskManager` skips minion task generation for tables whose servers are above the configured threshold. When the cached status returns to `PASS`, Pinot clears the resource-utilization pause state and allows ingestion to resume. If the status is `UNDETERMINED` and the table is already paused because of resource utilization, Pinot leaves it paused until fresh disk-usage data is available. ## How disk usage is collected The controller calls the server endpoint `GET /instance/diskUtilization` on each server instance. In the current implementation, that endpoint always computes disk usage for the server's `instanceDataDir`. The controller still sends the configured `controller.disk.utilization.path` header, but the server endpoint ignores that header today. ## Configuration The following controller configurations control this feature: | Config | Default Value | Description | | ---------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | controller.enable.resource.utilization.check | false | Master switch for enforcing resource-utilization checks during real-time ingestion validation and minion task generation. | | controller.enable.all.resource.utilization.checkers | true | Registers all resource-utilization checkers for backward compatibility. | | controller.enable.disk.utilization.checker | false | Registers the disk-utilization checker when `controller.enable.all.resource.utilization.checkers` is `false`. | | controller.resource.utilization.checker.frequency | 300 | Checker frequency in seconds. Setting this value to `-1` disables the periodic checker task. | | controller.resource.utilization.checker.initial.delay | 300 | Initial delay in seconds before the checker runs. If `controller.resource.utilization.checker.collect.usage.at.startup=true`, Pinot forces this value to `0`. | | controller.resource.utilization.checker.collect.usage.at.startup | false | When enabled, the controller waits for the checker cache to be populated before reporting itself healthy. If the startup collection times out, the controller still becomes healthy (fail-open). | | controller.disk.utilization.threshold | 0.95 | Disk-usage threshold, expressed as a fraction between 0 and 1. | | controller.disk.utilization.check.timeoutMs | 30000 | Timeout in milliseconds for collecting disk-usage responses from servers. | | controller.disk.utilization.path | /home/pinot/data | Compatibility config sent by the controller when requesting disk usage. The current server endpoint ignores this value and always measures `instanceDataDir`. | ## Metrics The gauge `pinot_controller_resourceUtilizationLimitExceeded_Value` is set to `1` when resource utilization is above the threshold for a table, and reset to `0` when the table is back within limits. ## FAQs ### Is controller restart required after changing any of the configuration properties? Yes. Update the property to the desired value and restart the controller(s). ### Does ResourceUtilizationChecker run only on the lead controller? The periodic task `ResourceUtilizationChecker` runs on all controllers. The controller periodic tasks `RealtimeSegmentValidationManager` and `PinotTaskManager` run only on the lead controller. ### How to identify the Pinot servers that are low on disk capacity? Search the controller logs for the message prefix `Disk utilization for server` to find the relevant instances. ## References * [Added support to pause and resume ingestion based on resource utilization](https://github.com/apache/pinot/pull/15008) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.pinot.apache.org/operate-pinot/tuning/pause-ingestion-based-on-resource-utilization.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.