# Pause ingestion based on resource utilization

A new capability has been added to Pinot to pause and un-pause ingestion based on resource utilization. This feature is designed to help users manage their Pinot clusters more effectively by pausing ingestion when resource utilization exceeds a specified threshold. Ingestion is un-paused when resource utilization falls below the threshold.

## How It Works

The periodic task `ResourceUtilizationChecker` runs periodically and computes the disk usage info of the Pinot server instances. The periodic task `RealTimeSegmentValidationManager` utilizes the disk usage info captured by the `ResourceUtilizationChecker` task and pauses consumption on REALTIME tables if disk utilization is above the threshold. The `RealTimeSegmentValidationManager` task would un-pause ingestion when disk utilization falls below the threshold. The periodic task `PinotTaskManager` utilizes the disk usage info and prevents minion based task generation if disk utilization is above threshold. The `PinotTaskManager` task would allow minion based task generation when disk utilization falls below the threshold.

## Configuration

The following configurations are available to control this feature:

<table><thead><tr><th width="530.7775334537681">Config</th><th>Default Value</th><th>Description</th></tr></thead><tbody><tr><td>controller.resource.utilization.checker.frequency</td><td>300</td><td>Value is in seconds. The disk utilization is computed for all Pinot servers in this frequency. Setting the value to -1 would disable the disk usage computation.</td></tr><tr><td>controller.disk.utilization.path</td><td>/home/pinot/data</td><td>Disk utilization is calculated for this path.</td></tr><tr><td>controller.disk.utilization.threshold</td><td>0.95</td><td>Value should be between 0 and 1.</td></tr><tr><td>controller.enable.resource.utilization.check</td><td>false</td><td>The feature is off by default.</td></tr></tbody></table>

## Metrics

The metric `pinot_controller_resourceUtilizationLimitExceeded_Value` would be set to `1` when disk utilization is above the threshold. The metric would be set to `0` when disk utilization is below the threshold.

## FAQs

### Is controller restart required after changing any of the configuration properties?

Yes, update the property to the desired value and restart the controller(s).

### Does ResourceUtilizationChecker run only on the lead controller?

The periodic task `ResourceUtilizationChecker` runs on all controllers. The controller periodic tasks `RealtimeSegmentValidationManager` and `PinotTaskManager` runs only on the lead controller.

### How to identify the Pinot servers that are low on disk capacity?

Grep for the keyword `Disk utilization for server` on any Pinot controller log to find the relevant servers.

## References

* [Added support to pause and resume ingestion based on resource utilization](https://github.com/apache/pinot/pull/15008)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.pinot.apache.org/release-1.4.0/for-operators/operating-pinot/pause-ingestion-based-on-resource-utilization.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.