Broker
You can set broker properties in a configuration file. The file can be provided during startup time as follows -
bin/pinot-admin.sh StartBroker -configFileName /path/to/broker.confDuplicate Keys in Configuration File
Starting from Apache Pinot 1.3.0, duplicate keys in the configuration file will cause a ConfigurationException to be thrown during startup. Previously, duplicate keys would be silently merged into a list. If you encounter this error, ensure that each configuration property appears only once in your configuration file. The exception will include the exact file path, duplicate key name, and the line numbers where the duplicates were found.
Example error:
ConfigurationException: Duplicate key found in /path/to/broker.conf at line 10 and line 15: pinot.broker.timeoutMsbroker.conf can have the following properties. All properties are defined in this class.
pinot.broker.delayShutdownTimeMs
10 seconds
pinot.broker.enableTableLevelMetrics
true
pinot.broker.query.response.limit
Integer.MAX_VALUE
When config pinot.broker.enable.query.limit.overrideis enabled, reset limit for selection query if it exceeds this value.
pinot.broker.query.log.length
Integer.MAX_VALUE
pinot.broker.query.log.maxRatePerSecond
10000.0
Maximum queries to be logged per second. Queries with exceptions, or take longer than 1 second are always logged.
pinot.broker.query.log.sqlRedaction
none
Controls SQL text in broker query logs. Use none to log SQL unchanged, literal_values to replace literals with fingerprint placeholders while retaining query structure, or full to log REDACTED instead of SQL. literal_values automatically enables query fingerprinting. Invalid values default to full to avoid logging sensitive SQL.
pinot.broker.enable.query.fingerprinting
false
When true, the broker computes a normalized fingerprint and queryHash for queries handled by both the single-stage and multi-stage engines, then propagates queryHash to downstream servers or multi-stage workers for observability. This setting controls automatic fingerprinting during normal query execution.
pinot.query.multistage.explain.include.segment.plan
false
When true, EXPLAIN PLAN FOR on the multi-stage engine includes the segment plan by default instead of the logical plan. The per-query explainAskingServers query option overrides this setting.
pinot.broker.timeoutMs
10 seconds
Timeout for Broker Query in Milliseconds
pinot.broker.min.init.indexed.table.capacity
128
Minimum initial capacity of the broker-side IndexedTable Pinot uses while reducing group-by results. Increasing this value can reduce rehashing for large group-by result sets, but increases memory usage for smaller queries. This value can be overridden per query with SET minInitialIndexedTableCapacity = value.
pinot.broker.extraPassiveTimeoutMs
100
For multi-stage queries, extra time in milliseconds Pinot adds beyond timeoutMs for passive waits between stages, such as waiting for mailbox data from upstream workers. The per-query extraPassiveTimeoutMs query option overrides this broker default.
pinot.broker.startup.minResourcePercent
100
Configuration to consider the broker ServiceStatus as being STARTED if the percent of resources (tables) that are ONLINE for this this broker has crossed the threshold percentage of the total number of tables that it is expected to serve
pinot.broker.enable.query.limit.override
false
Configuration to enable Query LIMIT Override to protect Pinot Broker and Server from fetch too many records back.
pinot.broker.query.ignore.missing.segments
false
When true, the broker defaults the ignoreMissingSegments query option so queries can tolerate SERVER_SEGMENT_MISSING errors caused by short routing lag after segment deletion or movement. In the single-stage engine this auto-default only applies when the broker routes the query to a single server; in the multi-stage engine it applies unless the query already sets ignoreMissingSegments.
pinot.broker.use.mse.to.fill.empty.response.schema
false
When true, the broker defaults the useMSEToFillEmptyResponseSchema query option for single-stage queries that return zero rows. Pinot then uses the multi-stage engine compiler to try to fill a more accurate empty result schema. Query-level useMSEToFillEmptyResponseSchema overrides this setting. Enable it only if your workload does not rely on very large IN clauses, because the extra compile step can be expensive for those queries.
pinot.broker.mse.enable.group.trim
false
For multi-stage queries, default the aggregate hint is_enable_group_trim when the query does not set it explicitly. This enables group trimming wherever the plan supports it. Query-level aggregate hints still override the broker default. See Grouping Algorithm for the trim behavior and related limits.
pinot.broker.mse.streaming.group.by.flush.threshold
unset
For multi-stage GROUP BY queries, default the streamingGroupByFlushThreshold query option when the query does not set it explicitly. Set this to a positive integer to opt the cluster into the streaming group-by leaf-stage operator by default. If the query already sets streamingGroupByFlushThreshold, including 0 to disable streaming mode for that query, the query-level value wins.
pinot.broker.table.sampler.annotation.packages
Comma-separated list of Java packages to scan for classes annotated with @TableSamplerProvider. Use this when you want custom sampler aliases in table configs. Built-in samplers under org.apache.pinot.broker.routing.tablesampler are always available without setting this property.
pinot.broker.client.queryPort
8099
(Deprecated: use pinot.broker.client.access.protocols.http.port instead.) Legacy port to query broker via http.
pinot.broker.client.access.protocols
Ingress protocols to query broker (http or https or http,https)
pinot.broker.client.access.protocols.http.port
Port to query broker via http
pinot.broker.client.access.protocols.https.port
Port to query broker via https
pinot.broker.grpc.port
Port for the broker gRPC query listener. Set this to expose the plaintext broker gRPC API.
pinot.broker.grpc.tls.enabled
false
Enable TLS for the broker gRPC query listener. When true, start the secure listener on pinot.broker.grpc.tls.port.
pinot.broker.grpc.tls.port
Port for the secure broker gRPC query listener.
pinot.broker.netty.enabled
true
Enable unsecured netty connections to pinot-server
pinot.broker.nettytls.enabled
false
Enable secured netty connections to pinot-server
pinot.broker.tls.keystore.path
Path to broker TLS keystore
pinot.broker.tls.keystore.password
keystore password
pinot.broker.tls.truststore.path
Path to broker TLS truststore
pinot.broker.tls.truststore.password
truststore password
pinot.broker.tls.requires_client_auth
false
toggle for requiring TLS client auth
pinot.broker.http.server.thread.pool.corePoolSize
2 * cores
Config for the thread-pool used by pinot-broker's http-server.
pinot.broker.http.server.thread.pool.maxPoolSize
2 * cores
Config for the thread-pool used by pinot-broker's http-server.
pinot.broker.http.server.max.http.header.size
unset (Grizzly default)
Maximum total HTTP header size in bytes accepted by the broker's Grizzly HTTP/HTTPS listeners. Set this when upstream proxies, auth gateways, or clients send larger headers than the Grizzly default allows.
pinot.broker.http.server.max.request.headers
unset (Grizzly default)
Maximum number of HTTP headers accepted by the broker's Grizzly HTTP/HTTPS listeners.
pinot.broker.enable.bounded.jersey.threadpool.executor
false
Enable bounded Jersey thread-pool to handle async requests.
pinot.broker.jersey.threadpool.executor.max.pool.size
2 * cores
Config for the bounded Jersey thread-pool to handle async requests.
pinot.broker.jersey.threadpool.executor.core.pool.size
2 * cores
Config for the bounded Jersey thread-pool to handle async requests.
pinot.broker.jersey.threadpool.executor.queue.size
Integer.MAX_VALUE
Config for the bounded Jersey thread-pool to handle async requests.
pinot.broker.max.query.response.size.bytes
Long.MAX_VALUE
Config indicating the maximum serialized response size across all servers for a query. This value is // equally divided across all servers processing the query.
pinot.broker.max.server.response.size.bytes
Long.MAX_VALUE
Config indicating the maximum length of the serialized response per server for a query.
pinot.broker.event.listener.factory.className
org.apache.pinot.spi.eventlistener.query.NoOpBrokerQueryEventListener
Config indicating the event listener class that will be loaded during broker starter and the onQueryCompletion will be called post query completion. It is on the implementation of this method that user can improve query level observability in their system.
pinot.broker.event.listener.request.context.tracked.header.keys
Comma separated string values indicating the request-headers which will be tracked by the event listener class.
pinot.broker.new.segment.expiration.seconds
300 seconds (5 minutes)
Grace period before the broker treats a newly created segment as old. After this window, replica-group routing can start tagging replica-groups / instances as unavailable, and broker-side partition metadata / routing decisions can start including new realtime segments.
pinot.broker.query.enable.null.handling
false
Config that allows enabling advanced null handling support for all queries by default (see docs)
pinot.broker.multistage.sort.exchange.copy.threshold
10000
For multi-stage ORDER BY ... LIMIT queries, controls when Pinot pushes the sort-and-limit operation below the sort exchange so each upstream worker sends only its top rows. This sets the default value of the sortExchangeCopyThreshold query option.
pinot.broker.multistage.lite.mode.leaf.stage.limit
100000
For MSE Lite Mode, this controls the maximum number of records that a given Leaf Stage instance on a server is allowed to return. This controls the default value of the query option liteModeLeafStageLimit. Recommended value is 100k records or lower.
pinot.broker.multistage.lite.mode.leaf.stage.fanOutAdjustedLimit
-1
For MSE Lite Mode, if set to a positive value, Pinot divides this value by the number of workers assigned to the leaf stage and uses the quotient as the per-worker hard limit. This controls the default value of the query option liteModeLeafStageFanOutAdjustedLimit.
pinot.broker.multistage.use.lite.mode
false
Default value of the query option useLiteMode. Only takes effect when usePhysicalOptimizer=true is also set.
pinot.broker.multistage.spools
false
Default value of the query option useSpools for stage-level spooling in the multi-stage engine.
pinot.broker.multistage.run.in.broker
true
Whether to run the non-leaf stages in the broker by default. This controls the default value of the query option runInBroker. Only applicable for MSE Lite Mode.
pinot.broker.cursor.fetch.rows
10000
Default number of rows returned per cursor page when numRows is omitted or set to 0. Applies to both the initial cursor-backed query and later /responseStore/{requestId}/results fetches.
pinot.broker.cursor.response.store.type
file
Response-store implementation used for cursor-backed queries.
pinot.broker.cursor.response.store.expiration
1h
Retention window for cursor-backed query results before they expire and become eligible for controller cleanup.
pinot.broker.cursor.response.store.file.data.dir
${java.io.tmpdir}/broker/responseStore/data
Data directory for the default file-based response store.
pinot.broker.cursor.response.store.file.temp.dir
${java.io.tmpdir}/broker/responseStore/temp
Temporary directory used while writing cursor responses for the default file-based response store.
pinot.broker.cursor.response.store.file.extension
json
File extension used by the default file-based response store.
pinot.broker.default.query.limit
10
Default LIMIT applied to queries that don't specify one explicitly.
pinot.broker.enable.query.cancellation
true
Whether to allow queries to be cancelled via the cancel API.
pinot.broker.groupby.trim.threshold
1000000
Threshold for number of groups to trigger trimming at the broker level. Reducing this can lower memory usage at the cost of accuracy.
pinot.broker.min.group.trim.size
5000
Minimum number of groups kept at the broker level after trimming. The actual number kept is max(5 * LIMIT, minGroupTrimSize).
pinot.broker.adaptive.server.selector.type
NO_OP
Type of adaptive server selector for intelligent query routing. Options: NO_OP (round-robin), NUM_INFLIGHT_REQ (least in-flight requests), LATENCY (lowest latency), HYBRID (combines latency and in-flight).
pinot.broker.adaptive.server.selector.enable.stats.collection
false
Enable stats collection for the adaptive server selector. Must be true for LATENCY or HYBRID types, and it must already be enabled at broker startup before adaptive-routing metrics can be exported.
pinot.broker.adaptive.server.selector.ewma.alpha
0.666
EWMA (Exponentially Weighted Moving Average) alpha value for latency calculation. Higher values weight recent observations more heavily.
pinot.broker.adaptive.server.selector.hybrid.score.exponent
3
Exponent used in the hybrid scoring formula to balance latency and in-flight request count.
pinot.broker.adaptive.server.selector.hybrid.score.queue.size.floor
0
Value added to the estimated queue size in the HYBRID score before exponentiation. The default is 0 for backwards compatibility; set to 1 to keep latency in the score when all servers are idle.
pinot.broker.adaptive.server.selector.enable.stats.metric.export
false
Seed/default value for exporting adaptive-routing gauges as broker metrics. Once brokers are running with stats collection enabled, cluster config can override or unset this key at runtime without a restart; setting it to false removes the exported single-stage gauges.
pinot.broker.adaptive.server.selector.stats.metric.export.interval.ms
10000
Seed/default interval in milliseconds for periodic export of adaptive-routing gauges. Once brokers are running with stats collection enabled, cluster config can change this at runtime without a restart; non-numeric, zero, and negative live values are ignored, and unsetting the cluster key falls back to this value or the built-in default.
pinot.broker.multistage.use.physical.optimizer
false
Enable the physical optimizer for the multi-stage query engine.
pinot.broker.multistage.unnest.column.pruning
false
For multi-stage UNNEST queries on the logical planner path, default the unnestColumnPruning query option so Pinot prunes unused passthrough columns from the UNNEST output, including the source array when it is not referenced downstream. Keep this disabled until every server supports the smaller UNNEST output schema. Query-level unnestColumnPruning overrides this default, and Pinot ignores both on the usePhysicalOptimizer=true path.
pinot.broker.multistage.infer.partition.hint
false
Infer partition hints to optimize data shuffling in the multi-stage query engine.
pinot.broker.multistage.default.hash.function
absHashCode
Default hash function for data partitioning in the multi-stage engine.
pinot.broker.mse.planner.disabled.rules
Built-in disabled-rule set
Comma-separated list of MSE planner rule names that the broker should treat as disabled by default. Setting this replaces Pinot's built-in disabled-rule set documented in Default Disabled Rules. Queries can still re-enable names from this configured set with the usePlannerRules query option.
pinot.broker.mse.max.server.query.threads
-1
Broker-local concurrency throttle for multi-stage queries, expressed as estimated server query threads. When set to a positive value, this overrides the cluster fallback pinot.beta.multistage.engine.max.server.query.threads. When left at -1, the broker uses the cluster fallback value, or disables this throttle if the cluster fallback is also non-positive.
pinot.broker.mse.max.server.query.threads.exceed.strategy
WAIT
Behavior when a multi-stage query would exceed the broker-side concurrency throttle. Supported values: WAIT (block until capacity is available) and LOG (allow the query but log a warning).
pinot.broker.mse.stream.stats
false
For multi-stage queries, use the streaming SubmitWithStream stats path by default so workers push stage stats over a long-lived gRPC stream instead of attaching them to mailbox EOS. Query-level streamStats overrides this default. Enable it only after every server in the cluster supports the streaming RPC because mixed-version clusters fail rather than falling back.
pinot.broker.mse.stream.stats.drain.ms
50
Best-effort wait window, in milliseconds, that the broker spends draining trailing streamed stage stats after the query result mailbox has finished. Larger values can improve stats completeness when a stage reports late, but can also add up to this much latency after results are otherwise ready.
pinot.broker.request.handler.type
netty
Type of request handler for broker-to-server communication. Options: netty, grpc, multistage.
channelKeepAliveTimeSeconds
-1 (disabled)
Top-level broker.conf key for the broker-to-server gRPC query client keepalive interval, in seconds. Only applies when pinot.broker.request.handler.type=grpc. Set a positive value to enable keepalive pings.
channelKeepAliveTimeoutSeconds
20
Top-level broker.conf key for the keepalive ACK timeout, in seconds, for the broker-to-server gRPC query client. Only applies when keepalive is enabled and pinot.broker.request.handler.type=grpc.
channelKeepAliveWithoutCalls
true
Top-level broker.conf key controlling whether the broker-to-server gRPC query client sends keepalive pings while idle. Only applies when keepalive is enabled and pinot.broker.request.handler.type=grpc.
channelShutdownTimeoutSeconds
10
Top-level broker.conf key for how long the broker waits for broker-to-server gRPC query client channels to terminate during shutdown. Only applies when pinot.broker.request.handler.type=grpc.
pinot.broker.max.reduce.threads.per.query
min(10, cores/2)
Maximum number of threads used to reduce (merge) results from multiple servers for a single query.
pinot.broker.failure.detector.type
NO_OP
Type of failure detector for detecting unhealthy servers. Options: NO_OP, CONNECTION_BASED.
pinot.broker.use.fixed.replica
false
When true, routes queries to a fixed replica group for better cache locality.
pinot.broker.multistage.use.broker.pruning
true
For multi-stage queries on the physical optimizer path, default the useBrokerPruning query option so the broker prunes eligible servers and segments before dispatch. Query-level useBrokerPruning overrides this broker default. See Segment Pruning for supported pruning behavior.
pinot.broker.multistage.logical.planner.use.broker.pruning
true
For multi-stage queries on the logical planner path, default the useBrokerPruning query option for eligible non-partitioned leaves, partitioned leaves, and logical tables. Query-level useBrokerPruning overrides this broker default. Unsupported or pre-partitioned leaf shapes fall back to unpruned routing, and routing failures retry unpruned instead of failing the query. See Segment Pruning for the current scope and limitations of logical-planner pruning.
pinot.query.multistage.dispatch.channel.keep.alive.time.ms
300000
For MSE, the gRPC keep-alive interval in milliseconds for broker dispatch channels to intermediate-stage workers. If you tune this below the 300000 ms default, also set pinot.query.multistage.query.server.permit.keep.alive.time.ms to a value less than or equal to this client interval. Set to -1 to disable keep-alive on MSE dispatch channels.
pinot.query.multistage.dispatch.channel.keep.alive.timeout.ms
30000
For MSE, the gRPC keep-alive ACK timeout in milliseconds for broker dispatch channels. If a keep-alive ping does not receive an ACK within this interval, the channel is considered dead and will reconnect. Only applies when keep-alive is enabled.
pinot.query.multistage.dispatch.channel.keep.alive.without.calls
false
For MSE, whether broker dispatch channels send keep-alive pings even while idle (no active calls). Set this to true only when pinot.query.multistage.query.server.permit.keep.alive.without.calls=true. Otherwise QueryServer will close idle channels with GOAWAY(ENHANCE_YOUR_CALM).
Broker gRPC transport
Pinot brokers can participate in two different gRPC query paths:
Client to broker: the broker exposes the Broker gRPC API on
pinot.broker.grpc.portorpinot.broker.grpc.tls.port.Broker to server: the broker talks to Pinot servers over gRPC only when
pinot.broker.request.handler.type=grpc.
For the secure broker gRPC listener, Pinot currently reuses the TLS material configured under pinot.broker.tls.*. In other words, pinot.broker.grpc.tls.enabled and pinot.broker.grpc.tls.port select the secure listener, while the keystore and truststore still come from the normal broker TLS prefix.
Example:
If you use pinot.broker.request.handler.type=grpc, the broker's outbound gRPC channels to servers can be tuned with the keepalive and shutdown settings above. Keepalive remains disabled unless channelKeepAliveTimeSeconds is set to a positive value.
Example:
Last updated
Was this helpful?