# Broker You can set broker properties in a configuration file. The file can be provided during startup time as follows - ``` bin/pinot-admin.sh StartBroker -configFileName /path/to/broker.conf ``` {% hint style="warning" %} **Duplicate Keys in Configuration File** Starting from Apache Pinot 1.3.0, duplicate keys in the configuration file will cause a `ConfigurationException` to be thrown during startup. Previously, duplicate keys would be silently merged into a list. If you encounter this error, ensure that each configuration property appears only once in your configuration file. The exception will include the exact file path, duplicate key name, and the line numbers where the duplicates were found. Example error: ``` ConfigurationException: Duplicate key found in /path/to/broker.conf at line 10 and line 15: pinot.broker.timeoutMs ``` {% endhint %} `broker.conf` can have the following properties. All properties are defined in this class. | Property | Default | Description | | ---------------------------------------------------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pinot.broker.delayShutdownTimeMs | 10 seconds | | | pinot.broker.enableTableLevelMetrics | true | | | pinot.broker.query.response.limit | Integer.MAX\_VALUE | When config `pinot.broker.enable.query.limit.override`is enabled, reset limit for selection query if it exceeds this value. | | pinot.broker.query.log.length | Integer.MAX\_VALUE | | | pinot.broker.query.log.maxRatePerSecond | 10000.0 | Maximum queries to be logged per second. Queries with exceptions, or take longer than 1 second are always logged. | | pinot.broker.enable.query.fingerprinting | false | When `true`, the broker computes a normalized fingerprint and `queryHash` for queries handled by both the single-stage and multi-stage engines, then propagates `queryHash` to downstream servers or multi-stage workers for observability. This setting controls automatic fingerprinting during normal query execution. | | pinot.query.multistage.explain.include.segment.plan | false | When `true`, `EXPLAIN PLAN FOR` on the multi-stage engine includes the segment plan by default instead of the logical plan. The per-query `explainAskingServers` query option overrides this setting. | | pinot.broker.timeoutMs | 10 seconds | Timeout for Broker Query in Milliseconds | | pinot.broker.min.init.indexed.table.capacity | 128 | Minimum initial capacity of the broker-side `IndexedTable` Pinot uses while reducing group-by results. Increasing this value can reduce rehashing for large group-by result sets, but increases memory usage for smaller queries. This value can be overridden per query with `SET minInitialIndexedTableCapacity = value`. | | pinot.broker.extraPassiveTimeoutMs | 100 | For multi-stage queries, extra time in milliseconds Pinot adds beyond `timeoutMs` for passive waits between stages, such as waiting for mailbox data from upstream workers. The per-query `extraPassiveTimeoutMs` query option overrides this broker default. | | pinot.broker.startup.minResourcePercent | 100 | Configuration to consider the broker ServiceStatus as being STARTED if the percent of resources (tables) that are ONLINE for this this broker has crossed the threshold percentage of the total number of tables that it is expected to serve | | pinot.broker.enable.query.limit.override | false | Configuration to enable Query LIMIT Override to protect Pinot Broker and Server from fetch too many records back. | | pinot.broker.query.ignore.missing.segments | false | When `true`, the broker defaults the `ignoreMissingSegments` query option so queries can tolerate `SERVER_SEGMENT_MISSING` errors caused by short routing lag after segment deletion or movement. In the single-stage engine this auto-default only applies when the broker routes the query to a single server; in the multi-stage engine it applies unless the query already sets `ignoreMissingSegments`. | | pinot.broker.use.mse.to.fill.empty.response.schema | false | When `true`, the broker defaults the `useMSEToFillEmptyResponseSchema` query option for single-stage queries that return zero rows. Pinot then uses the multi-stage engine compiler to try to fill a more accurate empty result schema. Query-level `useMSEToFillEmptyResponseSchema` overrides this setting. Enable it only if your workload does not rely on very large `IN` clauses, because the extra compile step can be expensive for those queries. | | pinot.broker.mse.enable.group.trim | false | For multi-stage queries, default the aggregate hint `is_enable_group_trim` when the query does not set it explicitly. This enables group trimming wherever the plan supports it. Query-level aggregate hints still override the broker default. See [Grouping Algorithm](https://docs.pinot.apache.org/build-with-pinot/querying-and-sql/sql-syntax/grouping-algorithm) for the trim behavior and related limits. | | pinot.broker.table.sampler.annotation.packages | | Comma-separated list of Java packages to scan for classes annotated with `@TableSamplerProvider`. Use this when you want custom sampler aliases in table configs. Built-in samplers under `org.apache.pinot.broker.routing.tablesampler` are always available without setting this property. | | pinot.broker.client.queryPort | 8099 | **(Deprecated: use `pinot.broker.client.access.protocols.http.port` instead.)** Legacy port to query broker via http. | | pinot.broker.client.access.protocols | | Ingress protocols to query broker (http or https or http,https) | | pinot.broker.client.access.protocols.http.port | | Port to query broker via http | | pinot.broker.client.access.protocols.https.port | | Port to query broker via https | | pinot.broker.grpc.port | | Port for the broker gRPC query listener. Set this to expose the plaintext broker gRPC API. | | pinot.broker.grpc.tls.enabled | false | Enable TLS for the broker gRPC query listener. When `true`, start the secure listener on `pinot.broker.grpc.tls.port`. | | pinot.broker.grpc.tls.port | | Port for the secure broker gRPC query listener. | | pinot.broker.netty.enabled | true | Enable unsecured netty connections to pinot-server | | pinot.broker.nettytls.enabled | false | Enable secured netty connections to pinot-server | | pinot.broker.tls.keystore.path | | Path to broker TLS keystore | | pinot.broker.tls.keystore.password | | keystore password | | pinot.broker.tls.truststore.path | | Path to broker TLS truststore | | pinot.broker.tls.truststore.password | | truststore password | | pinot.broker.tls.requires\_client\_auth | false | toggle for requiring TLS client auth | | pinot.broker.http.server.thread.pool.corePoolSize | 2 \* cores | Config for the thread-pool used by pinot-broker's http-server. | | pinot.broker.http.server.thread.pool.maxPoolSize | 2 \* cores | Config for the thread-pool used by pinot-broker's http-server. | | pinot.broker.enable.bounded.jersey.threadpool.executor | false | Enable bounded Jersey thread-pool to handle async requests. | | pinot.broker.jersey.threadpool.executor.max.pool.size | 2 \* cores | Config for the bounded Jersey thread-pool to handle async requests. | | pinot.broker.jersey.threadpool.executor.core.pool.size | 2 \* cores | Config for the bounded Jersey thread-pool to handle async requests. | | pinot.broker.jersey.threadpool.executor.queue.size | Integer.MAX\_VALUE | Config for the bounded Jersey thread-pool to handle async requests. | | pinot.broker.max.query.response.size.bytes | Long.MAX\_VALUE | Config indicating the maximum serialized response size across all servers for a query. This value is // equally divided across all servers processing the query. | | pinot.broker.max.server.response.size.bytes | Long.MAX\_VALUE | Config indicating the maximum length of the serialized response per server for a query. | | pinot.broker.event.listener.factory.className | org.apache.pinot.spi.eventlistener.query.NoOpBrokerQueryEventListener | Config indicating the event listener class that will be loaded during broker starter and the `onQueryCompletion` will be called post query completion. It is on the implementation of this method that user can improve query level observability in their system. | | pinot.broker.event.listener.request.context.tracked.header.keys | | Comma separated string values indicating the request-headers which will be tracked by the event listener class. | | pinot.broker.new\.segment.expiration.seconds | 300 seconds (5 minutes) | Config for broker to consider a new segment as an old segment and start tagging replica-group / instances as unavailable after this time period. | | pinot.broker.query.enable.null.handling | false | Config that allows enabling advanced null handling support for all queries by default (see [docs](https://docs.pinot.apache.org/build-with-pinot/querying-and-sql/sql-syntax/null-value-support#advanced-null-handling-support)) | | pinot.broker.multistage.sort.exchange.copy.threshold | 10000 | For multi-stage `ORDER BY ... LIMIT` queries, controls when Pinot pushes the sort-and-limit operation below the sort exchange so each upstream worker sends only its top rows. This sets the default value of the `sortExchangeCopyThreshold` query option. | | pinot.broker.multistage.lite.mode.leaf.stage.limit | 100000 | For MSE Lite Mode, this controls the maximum number of records that a given Leaf Stage instance on a server is allowed to return. This controls the default value of the query option `liteModeLeafStageLimit`. Recommended value is 100k records or lower. | | pinot.broker.multistage.lite.mode.leaf.stage.fanOutAdjustedLimit | -1 | For MSE Lite Mode, if set to a positive value, Pinot divides this value by the number of workers assigned to the leaf stage and uses the quotient as the per-worker hard limit. This controls the default value of the query option `liteModeLeafStageFanOutAdjustedLimit`. | | pinot.broker.multistage.use.lite.mode | false | Default value of the query option `useLiteMode`. Only takes effect when `usePhysicalOptimizer=true` is also set. | | pinot.broker.multistage.spools | false | Default value of the query option `useSpools` for stage-level spooling in the multi-stage engine. | | pinot.broker.multistage.run.in.broker | true | Whether to run the non-leaf stages in the broker by default. This controls the default value of the query option `runInBroker`. Only applicable for MSE Lite Mode. | | pinot.broker.cursor.fetch.rows | 10000 | Default number of rows returned per cursor page when `numRows` is omitted or set to `0`. Applies to both the initial cursor-backed query and later `/responseStore/{requestId}/results` fetches. | | pinot.broker.cursor.response.store.type | file | Response-store implementation used for cursor-backed queries. | | pinot.broker.cursor.response.store.expiration | 1h | Retention window for cursor-backed query results before they expire and become eligible for controller cleanup. | | pinot.broker.cursor.response.store.file.data.dir | `${java.io.tmpdir}/broker/responseStore/data` | Data directory for the default file-based response store. | | pinot.broker.cursor.response.store.file.temp.dir | `${java.io.tmpdir}/broker/responseStore/temp` | Temporary directory used while writing cursor responses for the default file-based response store. | | pinot.broker.cursor.response.store.file.extension | json | File extension used by the default file-based response store. | | pinot.broker.default.query.limit | 10 | Default LIMIT applied to queries that don't specify one explicitly. | | pinot.broker.enable.query.cancellation | true | Whether to allow queries to be cancelled via the cancel API. | | pinot.broker.groupby.trim.threshold | 1000000 | Threshold for number of groups to trigger trimming at the broker level. Reducing this can lower memory usage at the cost of accuracy. | | pinot.broker.min.group.trim.size | 5000 | Minimum number of groups kept at the broker level after trimming. The actual number kept is `max(5 * LIMIT, minGroupTrimSize)`. | | pinot.broker.adaptive.server.selector.type | NO\_OP | Type of adaptive server selector for intelligent query routing. Options: `NO_OP` (round-robin), `NUM_INFLIGHT_REQ` (least in-flight requests), `LATENCY` (lowest latency), `HYBRID` (combines latency and in-flight). | | pinot.broker.adaptive.server.selector.enable.stats.collection | false | Enable stats collection for the adaptive server selector. Must be true for `LATENCY` or `HYBRID` types. | | pinot.broker.adaptive.server.selector.ewma.alpha | 0.666 | EWMA (Exponentially Weighted Moving Average) alpha value for latency calculation. Higher values weight recent observations more heavily. | | pinot.broker.adaptive.server.selector.hybrid.score.exponent | 3 | Exponent used in the hybrid scoring formula to balance latency and in-flight request count. | | pinot.broker.multistage.use.physical.optimizer | false | Enable the physical optimizer for the multi-stage query engine. | | pinot.broker.multistage.infer.partition.hint | false | Infer partition hints to optimize data shuffling in the multi-stage query engine. | | pinot.broker.multistage.default.hash.function | absHashCode | Default hash function for data partitioning in the multi-stage engine. | | pinot.broker.mse.planner.disabled.rules | Built-in disabled-rule set | Comma-separated list of MSE planner rule names that the broker should treat as disabled by default. Setting this replaces Pinot's built-in disabled-rule set documented in [Default Disabled Rules](https://docs.pinot.apache.org/build-with-pinot/querying-and-sql/query-execution-controls/default-disabled-rules). Queries can still re-enable names from this configured set with the `usePlannerRules` query option. | | pinot.broker.mse.max.server.query.threads | -1 | Broker-local concurrency throttle for multi-stage queries, expressed as estimated server query threads. When set to a positive value, this overrides the cluster fallback `pinot.beta.multistage.engine.max.server.query.threads`. When left at `-1`, the broker uses the cluster fallback value, or disables this throttle if the cluster fallback is also non-positive. | | pinot.broker.mse.max.server.query.threads.exceed.strategy | WAIT | Behavior when a multi-stage query would exceed the broker-side concurrency throttle. Supported values: `WAIT` (block until capacity is available) and `LOG` (allow the query but log a warning). | | pinot.broker.request.handler.type | netty | Type of request handler for broker-to-server communication. Options: `netty`, `grpc`, `multistage`. | | channelKeepAliveTimeSeconds | -1 (disabled) | Top-level `broker.conf` key for the broker-to-server gRPC query client keepalive interval, in seconds. Only applies when `pinot.broker.request.handler.type=grpc`. Set a positive value to enable keepalive pings. | | channelKeepAliveTimeoutSeconds | 20 | Top-level `broker.conf` key for the keepalive ACK timeout, in seconds, for the broker-to-server gRPC query client. Only applies when keepalive is enabled and `pinot.broker.request.handler.type=grpc`. | | channelKeepAliveWithoutCalls | true | Top-level `broker.conf` key controlling whether the broker-to-server gRPC query client sends keepalive pings while idle. Only applies when keepalive is enabled and `pinot.broker.request.handler.type=grpc`. | | channelShutdownTimeoutSeconds | 10 | Top-level `broker.conf` key for how long the broker waits for broker-to-server gRPC query client channels to terminate during shutdown. Only applies when `pinot.broker.request.handler.type=grpc`. | | pinot.broker.max.reduce.threads.per.query | min(10, cores/2) | Maximum number of threads used to reduce (merge) results from multiple servers for a single query. | | pinot.broker.failure.detector.type | NO\_OP | Type of failure detector for detecting unhealthy servers. Options: `NO_OP`, `CONNECTION_BASED`. | | pinot.broker.use.fixed.replica | false | When true, routes queries to a fixed replica group for better cache locality. | | pinot.broker.multistage.use.broker.pruning | true | For multi-stage queries, default the `useBrokerPruning` query option so the broker prunes eligible servers and segments before dispatch. Query-level `useBrokerPruning` overrides this broker default. See [Segment Pruning](https://docs.pinot.apache.org/operate-pinot/tuning/segment-pruning) for supported pruning behavior. | ## Broker gRPC transport Pinot brokers can participate in two different gRPC query paths: * **Client to broker**: the broker exposes the [Broker gRPC API](https://docs.pinot.apache.org/reference/api-reference/broker-grpc-api) on `pinot.broker.grpc.port` or `pinot.broker.grpc.tls.port`. * **Broker to server**: the broker talks to Pinot servers over gRPC only when `pinot.broker.request.handler.type=grpc`. For the secure broker gRPC listener, Pinot currently reuses the TLS material configured under `pinot.broker.tls.*`. In other words, `pinot.broker.grpc.tls.enabled` and `pinot.broker.grpc.tls.port` select the secure listener, while the keystore and truststore still come from the normal broker TLS prefix. Example: ```properties pinot.broker.grpc.port=8010 pinot.broker.grpc.tls.enabled=true pinot.broker.grpc.tls.port=8020 pinot.broker.tls.keystore.path=/path/to/broker-keystore.p12 pinot.broker.tls.keystore.password=changeit pinot.broker.tls.keystore.type=PKCS12 pinot.broker.tls.truststore.path=/path/to/broker-truststore.p12 pinot.broker.tls.truststore.password=changeit pinot.broker.tls.truststore.type=PKCS12 ``` If you use `pinot.broker.request.handler.type=grpc`, the broker's outbound gRPC channels to servers can be tuned with the keepalive and shutdown settings above. Keepalive remains disabled unless `channelKeepAliveTimeSeconds` is set to a positive value. Example: ```properties pinot.broker.request.handler.type=grpc channelKeepAliveTimeSeconds=60 channelKeepAliveTimeoutSeconds=20 channelKeepAliveWithoutCalls=true channelShutdownTimeoutSeconds=10 ```