1 of 1

Advanced Pinot Setup

Start Pinot components (scripts or docker images)

Set up Pinot by starting each component individually

Start Pinot Components using docker

Prerequisites

If running locally, ensure your docker cluster has enough resources, below is a sample config.

Pull Docker image

You can try out pre-built Pinot all-in-one Docker image.

(Optional) You can also follow the instructions to build your own images.

0. Create a network

Create an isolated bridge network in Docker.

1. Start Zookeeper

Start Zookeeper in daemon.

Start to browse Zookeeper data at .

2. Start Pinot Controller

Start Pinot Controller in daemon and connect to Zookeeper.

3. Start Pinot Broker

Start Pinot Broker in daemon and connect to Zookeeper.

4. Start Pinot Server

Start Pinot Server in daemon and connect to Zookeeper.

Now all Pinot related components are started as an empty cluster.

You can run below command to check container status.

Sample Console Output

Download Pinot Distribution from

Start Pinot components via launcher scripts

Start Zookeeper

Start Pinot Controller

See for more details .

Start Pinot Broker

Start Pinot Using Config Files

Often times we need to customized the setup of Pinot components. Hence user can compile a config file and use it to start Pinot components.

Below are the examples config files and sample command to start Pinot.

Pinot Controller

Below is a sample pinot-controller.conf used in HelmChart setup.

In order to run Pinot Controller, the command is:

Configure Controller

Below are some configurations you can set in Pinot Controller. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Pinot Broker

Below is a sample pinot-broker.conf used in HelmChart setup.

In order to run Pinot Broker, the command is:

Configure Broker

Below are some configurations you can set in Pinot Broker. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Pinot Server

Below is a sample pinot-server.conf used in HelmChart setup.

In order to run Pinot Server, the command is:

Configure Server

Below are some outstanding configurations you can set in Pinot Server. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Create and Configure table

A TABLE in regular database world is represented as <TABLE>_OFFLINE and/or <TABLE>_REALTIME in Pinot depending on the ingestion mode (batch, real-time, hybrid)

See for all possible batch/streaming tables.

Batch Table Creation

See for table configuration details and how to customize it.

Sample Console Output

Automatically add an inverted index to your batch table

By default, the inverted index type is the only type of index that isn't created automatically during segment generation. Instead, they are generated when the segments are loaded on the server. But, waiting to build indexes until load time increases the startup time and takes up resources with every new segment push, which increases the time for other operations such as rebalance.

To automatically create an inverted index during segment generation, add an entry to your in the table configuration file.

This setting works with .

When set to true, Pinot creates an inverted index for the columns that you specify in the invertedIndexColumns list in the table configuration.

This setting is false by default.

Set createInvertedIndexDuringSegmentGeneration to true in your table config, as follows:

When you update this setting in your table configuration, you must to apply the inverted index to all existing segments.

Streaming Table Creation

See for table configuration details and how to customize it.

Start Kafka

Create a Kafka Topic

Create a Streaming table

Sample output

Start Kafka-Zookeeper

Start Kafka

Use `sortedColumn` with streaming tables

For tables, you can use a sorted index with sortedColumn to sort data when generating segments as the segment is created. See for more information.

A sorted forward index can be used as an inverted index with better performance, but with the limitation that the search is only applied to one column per table. See to learn more.

Load Data

Now that the table is configured, let's load some data. Data can be loaded in batch mode or streaming mode. See page for details. Loading data involves generating pinot segments from raw data and pushing them to the pinot cluster.

Load Data in Batch

User can always generate and push segments to Pinot via standalone scripts or using frameworks such as Hadoop or Spark. See this for more details on setting up Data Ingestion Jobs.

Below example goes with the standalone mode.

Sample Console Output

JobSpec yaml file has all the information regarding data format, input data location and pinot cluster coordinates. Note that this assumes that the controller is RUNNING to fetch the table config and schema. If not, you will have to configure the spec to point at their location. See for more details.

Load Data in Streaming

Kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

Advanced Pinot Setup

Start Pinot components (scripts or docker images)

Set up Pinot by starting each component individually

Start Pinot Components using docker

Prerequisites

If running locally, ensure your docker cluster has enough resources, below is a sample config.

Pull Docker image

You can try out pre-built Pinot all-in-one Docker image.

(Optional) You can also follow the instructions to build your own images.

0. Create a network

Create an isolated bridge network in Docker.

1. Start Zookeeper

Start Zookeeper in daemon.

Start to browse Zookeeper data at .

2. Start Pinot Controller

Start Pinot Controller in daemon and connect to Zookeeper.

3. Start Pinot Broker

Start Pinot Broker in daemon and connect to Zookeeper.

4. Start Pinot Server

Start Pinot Server in daemon and connect to Zookeeper.

Now all Pinot related components are started as an empty cluster.

You can run below command to check container status.

Sample Console Output

Download Pinot Distribution from

Start Pinot components via launcher scripts

Start Zookeeper

Start Pinot Controller

See for more details .

Start Pinot Broker

Start Pinot Using Config Files

Often times we need to customized the setup of Pinot components. Hence user can compile a config file and use it to start Pinot components.

Below are the examples config files and sample command to start Pinot.

Pinot Controller

Below is a sample pinot-controller.conf used in HelmChart setup.

In order to run Pinot Controller, the command is:

Configure Controller

Below are some configurations you can set in Pinot Controller. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Pinot Broker

Below is a sample pinot-broker.conf used in HelmChart setup.

In order to run Pinot Broker, the command is:

Configure Broker

Below are some configurations you can set in Pinot Broker. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Pinot Server

Below is a sample pinot-server.conf used in HelmChart setup.

In order to run Pinot Server, the command is:

Configure Server

Below are some outstanding configurations you can set in Pinot Server. You can head over to for complete list of available configs.

Config Name

Description

Default Value

Create and Configure table

A TABLE in regular database world is represented as <TABLE>_OFFLINE and/or <TABLE>_REALTIME in Pinot depending on the ingestion mode (batch, real-time, hybrid)

See for all possible batch/streaming tables.

Batch Table Creation

See for table configuration details and how to customize it.

Sample Console Output

Automatically add an inverted index to your batch table

To automatically create an inverted index during segment generation, add an entry to your in the table configuration file.

This setting works with .

When set to true, Pinot creates an inverted index for the columns that you specify in the invertedIndexColumns list in the table configuration.

This setting is false by default.

Set createInvertedIndexDuringSegmentGeneration to true in your table config, as follows:

When you update this setting in your table configuration, you must to apply the inverted index to all existing segments.

Streaming Table Creation

See for table configuration details and how to customize it.

Start Kafka

Create a Kafka Topic

Create a Streaming table

Sample output

Start Kafka-Zookeeper

Start Kafka

Use `sortedColumn` with streaming tables

For tables, you can use a sorted index with sortedColumn to sort data when generating segments as the segment is created. See for more information.

A sorted forward index can be used as an inverted index with better performance, but with the limitation that the search is only applied to one column per table. See to learn more.

Load Data

Load Data in Batch

User can always generate and push segments to Pinot via standalone scripts or using frameworks such as Hadoop or Spark. See this for more details on setting up Data Ingestion Jobs.

Below example goes with the standalone mode.

Sample Console Output

Load Data in Streaming

Kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.avro
inputDirURI: examples/batch/airlineStats/rawdata
jobType: SegmentCreationAndTarPush
outputDirURI: examples/batch/airlineStats/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://pinot-controller:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader,
  configClassName: null, configs: null, dataFormat: avro}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema',
  tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 403 documents
Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389
Using fixed bytes value dictionary for column: Origin, size: 294
Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS
Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1
Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303
......
......
Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"}
Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"}

SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.avro
inputDirURI: examples/batch/airlineStats/rawdata
jobType: SegmentCreationAndTarPush
outputDirURI: examples/batch/airlineStats/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://pinot-controller:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader,
  configClassName: null, configs: null, dataFormat: avro}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema',
  tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 403 documents
Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389
Using fixed bytes value dictionary for column: Origin, size: 294
Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS
Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1
Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303
......
......
Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"}
Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"}

Advanced Pinot Setup

hashtagStart Pinot components (scripts or docker images)

hashtagStart Pinot Using Config Files

hashtagPinot Controller

hashtagConfigure Controller

hashtagPinot Broker

hashtagConfigure Broker

hashtagPinot Server

hashtagConfigure Server

hashtagCreate and Configure table

hashtagBatch Table Creation

hashtagAutomatically add an inverted index to your batch table

hashtagStreaming Table Creation

hashtagUse sortedColumn with streaming tables

hashtagLoad Data

hashtagLoad Data in Batch

hashtagLoad Data in Streaming

hashtagKafka

Advanced Pinot Setup

hashtagStart Pinot components (scripts or docker images)

hashtagStart Pinot Using Config Files

hashtagPinot Controller

hashtagConfigure Controller

hashtagPinot Broker

hashtagConfigure Broker

hashtagPinot Server

hashtagConfigure Server

hashtagCreate and Configure table

hashtagBatch Table Creation

hashtagAutomatically add an inverted index to your batch table

hashtagStreaming Table Creation

hashtagUse sortedColumn with streaming tables

hashtagLoad Data

hashtagLoad Data in Batch

hashtagLoad Data in Streaming

hashtagKafka

Start Pinot components (scripts or docker images)

Start Pinot Using Config Files

Pinot Controller

Configure Controller

Pinot Broker

Configure Broker

Pinot Server

Configure Server

Create and Configure table

Batch Table Creation

Automatically add an inverted index to your batch table

Streaming Table Creation

Use `sortedColumn` with streaming tables

Load Data

Load Data in Batch

Load Data in Streaming

Kafka

Start Pinot components (scripts or docker images)

Start Pinot Using Config Files

Pinot Controller

Configure Controller

Pinot Broker

Configure Broker

Pinot Server

Configure Server

Create and Configure table

Batch Table Creation

Automatically add an inverted index to your batch table

Streaming Table Creation

Use `sortedColumn` with streaming tables

Load Data

Load Data in Batch

Load Data in Streaming

Kafka