LogoLogo
release-1.1.0
release-1.1.0
  • Introduction
  • Basics
    • Concepts
    • Architecture
    • Components
      • Cluster
        • Tenant
        • Server
        • Controller
        • Broker
        • Minion
      • Table
        • Segment
          • Deep Store
        • Schema
      • Pinot Data Explorer
    • Getting Started
      • Running Pinot locally
      • Running Pinot in Docker
      • Quick Start Examples
      • Running in Kubernetes
      • Running on public clouds
        • Running on Azure
        • Running on GCP
        • Running on AWS
      • Create and update a table configuration
      • Batch import example
      • Stream ingestion example
      • HDFS as Deep Storage
      • Troubleshooting Pinot
      • Frequently Asked Questions (FAQs)
        • General
        • Pinot On Kubernetes FAQ
        • Ingestion FAQ
        • Query FAQ
        • Operations FAQ
    • Import Data
      • From Query Console
      • Batch Ingestion
        • Spark
        • Flink
        • Hadoop
        • Backfill Data
        • Dimension table
      • Stream ingestion
        • Apache Kafka
        • Amazon Kinesis
        • Apache Pulsar
      • Stream Ingestion with Upsert
      • Segment compaction on upserts
      • Stream Ingestion with Dedup
      • Stream Ingestion with CLP
      • File Systems
        • Amazon S3
        • Azure Data Lake Storage
        • HDFS
        • Google Cloud Storage
      • Input formats
        • Complex Type (Array, Map) Handling
        • Ingest records with dynamic schemas
      • Reload a table segment
      • Upload a table segment
    • Indexing
      • Bloom filter
      • Dictionary index
      • Forward index
      • Geospatial
      • Inverted index
      • JSON index
      • Native text index
      • Range index
      • Star-tree index
      • Text search support
      • Timestamp index
    • Releases
      • 1.1.0
      • Apache Pinotâ„¢ 1.0.0 release notes
      • 0.12.1
      • 0.12.0
      • 0.11.0
      • 0.10.0
      • 0.9.3
      • 0.9.2
      • 0.9.1
      • 0.9.0
      • 0.8.0
      • 0.7.1
      • 0.6.0
      • 0.5.0
      • 0.4.0
      • 0.3.0
      • 0.2.0
      • 0.1.0
    • Recipes
      • Connect to Streamlit
      • Connect to Dash
      • Visualize data with Redash
      • GitHub Events Stream
  • For Users
    • Query
      • Querying Pinot
      • Query Syntax
        • Aggregation Functions
        • Cardinality Estimation
        • Explain Plan (Single-Stage)
        • Explain Plan (Multi-Stage)
        • Filtering with IdSet
        • GapFill Function For Time-Series Dataset
        • Grouping Algorithm
        • JOINs
        • Lookup UDF Join
        • Querying JSON data
        • Transformation Functions
        • Window aggregate
      • Query Options
      • User-Defined Functions (UDFs)
    • APIs
      • Broker Query API
        • Query Response Format
      • Controller Admin API
      • Controller API Reference
    • External Clients
      • JDBC
      • Java
      • Python
      • Golang
    • Tutorials
      • Use OSS as Deep Storage for Pinot
      • Ingest Parquet Files from S3 Using Spark
      • Creating Pinot Segments
      • Use S3 as Deep Storage for Pinot
      • Use S3 and Pinot in Docker
      • Batch Data Ingestion In Practice
      • Schema Evolution
  • For Developers
    • Basics
      • Extending Pinot
        • Writing Custom Aggregation Function
        • Segment Fetchers
      • Contribution Guidelines
      • Code Setup
      • Code Modules and Organization
      • Update documentation
    • Advanced
      • Data Ingestion Overview
      • Ingestion Aggregations
      • Ingestion Transformations
      • Null value support
      • Use the multi-stage query engine (v2)
      • Troubleshoot issues with the multi-stage query engine (v2)
      • Advanced Pinot Setup
    • Plugins
      • Write Custom Plugins
        • Input Format Plugin
        • Filesystem Plugin
        • Batch Segment Fetcher Plugin
        • Stream Ingestion Plugin
    • Design Documents
      • Segment Writer API
  • For Operators
    • Deployment and Monitoring
      • Set up cluster
      • Server Startup Status Checkers
      • Set up table
      • Set up ingestion
      • Decoupling Controller from the Data Path
      • Segment Assignment
      • Instance Assignment
      • Rebalance
        • Rebalance Servers
        • Rebalance Brokers
      • Separating data storage by age
        • Using multiple tenants
        • Using multiple directories
      • Pinot managed Offline flows
      • Minion merge rollup task
      • Consistent Push and Rollback
      • Access Control
      • Monitoring
      • Tuning
        • Real-time
        • Routing
        • Query Routing using Adaptive Server Selection
        • Query Scheduling
      • Upgrading Pinot with confidence
      • Managing Logs
      • OOM Protection Using Automatic Query Killing
    • Command-Line Interface (CLI)
    • Configuration Recommendation Engine
    • Tutorials
      • Authentication
        • Basic auth access control
        • ZkBasicAuthAccessControl
      • Configuring TLS/SSL
      • Build Docker Images
      • Running Pinot in Production
      • Kubernetes Deployment
      • Amazon EKS (Kafka)
      • Amazon MSK (Kafka)
      • Monitor Pinot using Prometheus and Grafana
      • Performance Optimization Configurations
  • Configuration Reference
    • Cluster
    • Controller
    • Broker
    • Server
    • Table
    • Ingestion
    • Schema
    • Ingestion Job Spec
    • Monitoring Metrics
    • Functions
      • ABS
      • ADD
      • ago
      • EXPR_MIN / EXPR_MAX
      • arrayConcatDouble
      • arrayConcatFloat
      • arrayConcatInt
      • arrayConcatLong
      • arrayConcatString
      • arrayContainsInt
      • arrayContainsString
      • arrayDistinctInt
      • arrayDistinctString
      • arrayIndexOfInt
      • arrayIndexOfString
      • ARRAYLENGTH
      • arrayRemoveInt
      • arrayRemoveString
      • arrayReverseInt
      • arrayReverseString
      • arraySliceInt
      • arraySliceString
      • arraySortInt
      • arraySortString
      • arrayUnionInt
      • arrayUnionString
      • AVGMV
      • Base64
      • caseWhen
      • ceil
      • CHR
      • codepoint
      • concat
      • count
      • COUNTMV
      • COVAR_POP
      • COVAR_SAMP
      • day
      • dayOfWeek
      • dayOfYear
      • DISTINCT
      • DISTINCTAVG
      • DISTINCTAVGMV
      • DISTINCTCOUNT
      • DISTINCTCOUNTBITMAP
      • DISTINCTCOUNTHLLMV
      • DISTINCTCOUNTHLL
      • DISTINCTCOUNTBITMAPMV
      • DISTINCTCOUNTMV
      • DISTINCTCOUNTRAWHLL
      • DISTINCTCOUNTRAWHLLMV
      • DISTINCTCOUNTRAWTHETASKETCH
      • DISTINCTCOUNTTHETASKETCH
      • DISTINCTSUM
      • DISTINCTSUMMV
      • DIV
      • DATETIMECONVERT
      • DATETRUNC
      • exp
      • FIRSTWITHTIME
      • FLOOR
      • FrequentLongsSketch
      • FrequentStringsSketch
      • FromDateTime
      • FromEpoch
      • FromEpochBucket
      • FUNNELCOUNT
      • Histogram
      • hour
      • isSubnetOf
      • JSONFORMAT
      • JSONPATH
      • JSONPATHARRAY
      • JSONPATHARRAYDEFAULTEMPTY
      • JSONPATHDOUBLE
      • JSONPATHLONG
      • JSONPATHSTRING
      • jsonextractkey
      • jsonextractscalar
      • LASTWITHTIME
      • length
      • ln
      • lower
      • lpad
      • ltrim
      • max
      • MAXMV
      • MD5
      • millisecond
      • min
      • minmaxrange
      • MINMAXRANGEMV
      • MINMV
      • minute
      • MOD
      • mode
      • month
      • mult
      • now
      • percentile
      • percentileest
      • percentileestmv
      • percentilemv
      • percentiletdigest
      • percentiletdigestmv
      • percentilekll
      • percentilerawkll
      • percentilekllmv
      • percentilerawkllmv
      • quarter
      • regexpExtract
      • regexpReplace
      • remove
      • replace
      • reverse
      • round
      • ROW_NUMBER
      • rpad
      • rtrim
      • second
      • SEGMENTPARTITIONEDDISTINCTCOUNT
      • sha
      • sha256
      • sha512
      • sqrt
      • startswith
      • ST_AsBinary
      • ST_AsText
      • ST_Contains
      • ST_Distance
      • ST_GeogFromText
      • ST_GeogFromWKB
      • ST_GeometryType
      • ST_GeomFromText
      • ST_GeomFromWKB
      • STPOINT
      • ST_Polygon
      • strpos
      • ST_Union
      • SUB
      • substr
      • sum
      • summv
      • TIMECONVERT
      • timezoneHour
      • timezoneMinute
      • ToDateTime
      • ToEpoch
      • ToEpochBucket
      • ToEpochRounded
      • TOJSONMAPSTR
      • toGeometry
      • toSphericalGeography
      • trim
      • upper
      • Url
      • UTF8
      • VALUEIN
      • week
      • year
      • yearOfWeek
      • Extract
    • Plugin Reference
      • Stream Ingestion Connectors
      • VAR_POP
      • VAR_SAMP
      • STDDEV_POP
      • STDDEV_SAMP
  • Reference
    • Single-stage query engine (v1)
    • Multi-stage query engine (v2)
  • RESOURCES
    • Community
    • Team
    • Blogs
    • Presentations
    • Videos
  • Integrations
    • Tableau
    • Trino
    • ThirdEye
    • Superset
    • Presto
    • Spark-Pinot Connector
  • Contributing
    • Contribute Pinot documentation
    • Style guide
Powered by GitBook
On this page

Was this helpful?

Export as PDF
  1. Configuration Reference

Server

Server configuration can be provided as part of the server startup parameters.

bin/pinot-admin.sh StartServer -configFileName /path/to/server.conf

server.conf can have the following properties

Property
Default
Description

pinot.server.netty.port

8098

Port to query Pinot Server

pinot.server.netty.host

Pinot server hostname

pinot.server.adminapi.port

8097

Port for Pinot Server Admin UI

pinot.server.instance.id

By default the server instance id used by Helix is Server_hostname_port where the hostname and port are configured through host and port config values above. This config overwrites the default setting. User can put server id independent of the server's hostname and port.

pinot.server.instance.dataDir

java.io.tmpdir + /PinotServer/index

Directory to hold all the data

pinot.server.instance.consumerDir

pinot.server.instance.segmentTarDir

java.io.tmpdir + /PinotServer/segmentTar

Directory to hold temporary segments downloaded from Controller or Deep Store

pinot.server.instance.readMode

mmap

pinot.server.instance.reload.consumingSegment

true

Specifies if the reload segment API should reload the consuming segments. This is useful when the corresponding schema is updated and we want the changes to be reflected in the consuming segment.

pinot.server.instance.data.manager.class

org.apache.pinot.server. starter.helix.HelixInstanceDataManager

pinot.query.scheduler.name

fcfs

Currently only FCFS (first-come-first-serve) is supported

pinot.query.scheduler.query_runner_threads

CPU cores

Main thread to execute the queries (one thread per query)

pinot.query.scheduler.query_worker_threads

2 * CPU cores

Worker thread to process the segments (multiple threads per query)

pinot.query.scheduler.query.log.maxRatePerSecond

unlimited

Maximum queries to be logged per second. Queries with exceptions and costly queries are always logged.

pinot.server.query.executor.class

org.apache.pinot.core.query. executor.ServerQueryExecutorV1Impl

pinot.server.query.executor.pruner.class

ValidSegmentPruner,DataSchemaSegmentPruner, ColumnValueSegmentPruner,SelectionQuerySegmentPruner

pinot.server.query.executor.timeout

15000

Timeout for Server to process Query in Milliseconds

pinot.server.query.executor.max.execution.threads

-1 (unlimited)

Maximum number of execution threads allowed for a query. Limiting this can prevent a single expensive query from occupying all the execution threads.

pinot.server.query.executor.max.init.group.holder.capacity

10000

Initial capacity of the group key holder. Increasing this value can reduce the resizing of the group key holder, but increase the heap usage for small group-by queries.

pinot.server.query.executor.num.groups.limit

100000

Maximum number of groups kept from each segment during query execution. Once this limit is reached, no more groups will be taken (will still aggregate on existing groups).

pinot.server.query.executor.min.segment.group.trim.size

-1 (do not trim)

Minimum number of groups kept at segment level during query execution. If there are enough groups found in a segment, pinot will trim the groups to max(5 * LIMIT, minSegmentgroupTrimSize) groups based on the order-by clause. Increasing this value can increase the accuracy of the results, but also increase the heap usage of the group-by queries.

pinot.server.query.executor.min.server.group.trim.size

5000

Minimum number of groups kept at server level during query execution. If there are enough groups found in a server, pinot will trim the groups to max(5 * LIMIT, minServergroupTrimSize) groups based on the order-by clause. Increasing this value can increase the accuracy of the results, but also increase the heap usage and data transfer cost of the group-by queries.

pinot.server.query.executor.groupby.trim.threshold

1000000

Threshold for number of groups to trigger the server group trimming. Increasing this value can reduce the times of trimming, but also increase the heap usage of the group-by queries.

pinot.server.requestHandlerFactory.class

org.apache.pinot.server. request.SimpleRequestHandlerFactory

pinot.server.instance.segment.format.version

pinot.server.instance.enable.split.commit

pinot.server.instance.enable.commitend.metadata

pinot.server.instance.max.parallel.refresh.threads

1

Number of simultaneous segments that can be refreshed on one server.

pinot.server.instance.realtime.max.parallel.segment.builds

0

Specifies how many parallel real-time segments can be built. Value of <= 0 indicates unlimited.

pinot.server.instance.realtime.alloc.offheap

false

Boolean value to control whether memory for real-time consuming segments should be allocated off-heap.

pinot.server.instance.realtime.alloc.offheap.direct

false

If 'real-time.alloc.offheap' is set to true, this boolean value controls whether the corresponding allocation should be direct or not (false indicate mmap allocation)

pinot.server.startup.minResourcePercent

100

The percentage of tables that need to be in an ONLINE state before the server is marked as STARTED to server queries

pinot.server.startup.timeoutMs

10 minutes

The total amount of time a server will wait for all status checks before server is marked as STARTED to server queries

pinot.server.starter.realtimeConsumptionCatchupWaitMs

0

On it's own, this is a static amount of time servers will wait for consuming segments before server is marked as STARTED to server queries When paired with pinot.server.starter.enableRealtimeOffsetBasedConsumptionStatusChecker or pinot.server.starter.enableRealtimeFreshnessBasedConsumptionStatusChecker, this is how long those status checkers will wait for segments to turn GOOD This should not be set to more than pinot.server.startup.timeoutMs as it will not be respected

pinot.server.starter.enableRealtimeOffsetBasedConsumptionStatusChecker

false

When true, the server will collect the current, latest offsets for all consuming segments and will mark those segments as GOOD once they have caught up to those offsets

pinot.server.starter.enableRealtimeFreshnessBasedConsumptionStatusChecker

false

When true, the server will not be marked as STARTED or serve queries until

1) the server has caught up to the latest available offset or the latest consumed event for each segment is at least within pinot.server.starter.realtimeMinFreshnessMs of the current time 2) pinot.server.startup.timeoutMs has elapsed

pinot.server.starter.realtimeMinFreshnessMs

10 seconds

When pinot.server.starter.enableRealtimeFreshnessBasedConsumptionStatusChecker=true, this configures the minimum freshness (now - last_event_time) for the server status checker to consider a consuming segment GOOD

This only applies when pinot.server.starter.enableRealtimeFreshnessBasedConsumptionStatusChecker=true

pinot.server.starter.realtimeFreshnessIdleTimeoutMs

0

When pinot.server.starter.enableRealtimeFreshnessBasedConsumptionStatusChecker=true, this will mark a segment as GOOD after this timeout if it has not consumed any events.

pinot.server.startup.exitOnServiceStatusCheckFailure

false

When true, the server will shutdown after pinot.server.shutdown.timeoutMs rather than be marked as STARTED if the status has not turned GOOD

pinot.server.startup.enableServiceStatusCheck

true

true - server will run all configured checks false - server will be marked as STARTED without performing an startup checks

pinot.server.startup.serviceStatusCheckIntervalMs

10 seconds

Interval at which server will perform service stats checks

pinot.server.shutdown.timeoutMs

10 minutes

pinot.server.shutdown.enableQueryCheck

true

pinot.server.shutdown.noQueryThresholdMs

15 seconds

pinot.server.shutdown.enableResourceCheck

false

pinot.server.shutdown.resourceCheckIntervalMs

10 seconds

pinot.server.admin.access.control.factory.class

org.apache.pinot.server. api.access.AllowAllAccessFactory

pinot.server.adminapi.access.protocols

http

Ingress protocols to access server admin api (http or https or http,https)

pinot.server.adminapi.access.protocols.http.port

Port to access server admin api via http

pinot.server.adminapi.broker.protocols.https.port

Port to access server admin api via https

pinot.server.tls.keystore.path

Path to server TLS keystore

pinot.server.tls.keystore.password

keystore password

pinot.server.tls.truststore.path

Path to server TLS truststore

pinot.server.tls.truststore.password

truststore password

inot.server.tls.client.auth

false

toggle for requiring TLS client auth

pinot.server.netty.enabled

true

toggle for enabling unsecured netty connections to server

pinot.server.netty.port

Port for accessing pinot server via unsecured netty

pinot.server.nettytls.enabled

pinot.server.nettytls.port

Port for accessing pinot server via TLS-secured netty

pinot.server.http.server.thread.pool.corePoolSize

2 * cores

Config for the thread-pool used by pinot-server's http-server.

pinot.server.http.server.thread.pool.maxPoolSize

2 * cores

Config for the thread-pool used by pinot-server's http-server.

pinot.server.segment.fetcher.http.client.maxConnTotal

Config for the http-client used by HttpSegmentFetcher for downloading segments

pinot.server.segment.fetcher.http.client.maxConnPerRoute

Config for the http-client used by HttpSegmentFetcher for downloading segments

pinot.server.instance.max.segment.preload.threads

0

Number of threads that should be created to preload the segments in an upsert table. Value should be greater than 0 otherwise preload is synchronous.

PreviousBrokerNextTable

Was this helpful?