Segment Operations Throttling
Segments undergo a series of operations such as download, untar, index rebuild, and much more before they are ready to serve data for queries. These operations can be resource intensive (CPU + memory) especially during server restarts and table rebalance. Throttling configurations can be added to control the concurrency of how many segments can undergo a specific operation at any given point of time to limit the resource utilization. Throttling is done today via the use of semaphores.
Each config added related to throttling come in two variants:
Before serving queries - the maximum parallel segments that can undergo the given operation before the server is marked as ready to serve queries. These configurations will include before.serving.queriesin the config name.
After serving queries - the maximum parallel segments that can undergo the given operation after the server is marked as ready to serve queries. These configurations will not include before.serving.queriesin the config name.
Two variants of each config were added due to the nature of requiring different characteristics depending on what the server is doing. During start up, servers are mostly trying to bring segments into a queryable state and using more resources to achieve this faster makes sense. Whereas once the server is ready to serve queries, more resources are required for query processing.
All throttling configurations are disabled by default by setting a very high parallelism of Integer.MAX_VALUE.
Config
Allowed Values
Default
Description
The above configurations can be updated via adding them as . No server restart is required for these configurations to take effect if updated in ZooKeeper. Pinot Server logs will provide information about the change in these configs. An example log of updated configs for all the segment operations throttling configs are:
An example of what cluster configurations overridden by setting them in ZooKeeper under /CONFIGS/CLUSTER/<PinotClusterName> looks like is shown below:
Some recommendations on how to choose values for these are:
The default was to Integer.MAX_VALUE, effectively disabling throttling by default. Lower throttling configurations if throttling is needed.
It is recommended to set the after serving queries variant to be <= the before serving queries variant, since once the server starts serving queries, resources are needed for query processing. If too many resources are used up for processing segments, queries can see higher latencies and even exhaust resources.
A related configuration controls the REST API invoked refresh (segment upload) and reload operations. This defaults to 1, but with the new throttling configurations mentioned in the table above, this can potentially be increased.
The following Gauge type metrics exist to monitor the segment operation threshold and count of number of segments undergoing a given operation:
Segment Operation
Scope
Threshold Metric
Count Metric
Relevant OSS PRs
Segment index rebuild across all index types:
StarTree segment index rebuild:
Segment download at server level:
