1 of 1

Advanced Pinot Setup

Start Pinot components (scripts or docker images)

Setup Pinot by starting each component individually

Start Pinot Components using docker

Pull docker image

You can try out pre-built Pinot all-in-one docker image.

(Optional) You can also follow the instructions to build your own images.

0. Create a Network

Create an isolated bridge network in docker

1. Start Zookeeper

Start Zookeeper in daemon.

Start to browse Zookeeper data at .

2. Start Pinot Controller

Start Pinot Controller in daemon and connect to Zookeeper.

3. Start Pinot Broker

Start Pinot Broker in daemon and connect to Zookeeper.

4. Start Pinot Server

Start Pinot Server in daemon and connect to Zookeeper.

Now all Pinot related components are started as an empty cluster.

You can run below command to check container status.

Sample Console Output

Download Pinot Distribution from

Start Pinot components via launcher scripts

Start Pinot Using Config Files

Often times we need to customized the setup of Pinot Components. Hence user can compile a config file and use it to start Pinot Components.

Below are the examples config files and sample command to start Pinot.

Pinot Controller

Below is a sample pinot-controller.conf used in HelmChart setup.

In order to run Pinot Controller, the command is:

Configure Controller

Below are some outstanding configurations you can set in Pinot Controller:

Pinot Broker

Below is a sample pinot-broker.conf used in HelmChart setup.

In order to run Pinot Broker, the command is:

Configure Broker

Below are some outstanding configurations you can set in Pinot Broker:

Pinot Server

Below is a sample pinot-server.conf used in HelmChart setup.

In order to run Pinot Server, the command is:

Configure Server

Below are some outstanding configurations you can set in Pinot Server:

Create and Configure table

A TABLE in regular database world is represented as <TABLE>_OFFLINE and/or <TABLE>_REALTIME in Pinot depending on the ingestion mode (batch, real-time, hybrid)

See for all possible batch/streaming tables.

Batch Table Creation

Please see for table configuration details and how to customize it.

Sample Console Output

Streaming Table Creation

Please see for table configuration details and how to customize it.

Start Kafka

Create a Kafka Topic

Create a Streaming table

Sample output

Start Kafka-Zookeeper

Start Kafka

Load Data

Now that the table is configured, let's load some data. Data can be loaded in batch mode or streaming mode. See page for details. Loading data involves generating pinot segments from raw data and pushing them to the pinot cluster.

Load Data in Batch

User can always generate and push segments to Pinot via standalone scripts or using frameworks such as Hadoop or Spark. See this for more details on setting up Data Ingestion Jobs.

Below example goes with the standalone mode.

Sample Console Output

JobSpec yaml file has all the information regarding data format, input data location and pinot cluster coordinates. Note that this assumes that the controller is RUNNING to fetch the table config and schema. If not, you will have to configure the spec to point at their location. See for more details.

Load Data in Streaming

Kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

Advanced Pinot Setup

Start Pinot components (scripts or docker images)

Setup Pinot by starting each component individually

Start Pinot Components using docker

Pull docker image

You can try out pre-built Pinot all-in-one docker image.

(Optional) You can also follow the instructions to build your own images.

0. Create a Network

Create an isolated bridge network in docker

1. Start Zookeeper

Start Zookeeper in daemon.

Start to browse Zookeeper data at .

2. Start Pinot Controller

Start Pinot Controller in daemon and connect to Zookeeper.

3. Start Pinot Broker

Start Pinot Broker in daemon and connect to Zookeeper.

4. Start Pinot Server

Start Pinot Server in daemon and connect to Zookeeper.

Now all Pinot related components are started as an empty cluster.

You can run below command to check container status.

Sample Console Output

Download Pinot Distribution from

Start Pinot components via launcher scripts

Start Pinot Using Config Files

Often times we need to customized the setup of Pinot Components. Hence user can compile a config file and use it to start Pinot Components.

Below are the examples config files and sample command to start Pinot.

Pinot Controller

Below is a sample pinot-controller.conf used in HelmChart setup.

In order to run Pinot Controller, the command is:

Configure Controller

Below are some outstanding configurations you can set in Pinot Controller:

Pinot Broker

Below is a sample pinot-broker.conf used in HelmChart setup.

In order to run Pinot Broker, the command is:

Configure Broker

Below are some outstanding configurations you can set in Pinot Broker:

Pinot Server

Below is a sample pinot-server.conf used in HelmChart setup.

In order to run Pinot Server, the command is:

Configure Server

Below are some outstanding configurations you can set in Pinot Server:

Create and Configure table

A TABLE in regular database world is represented as <TABLE>_OFFLINE and/or <TABLE>_REALTIME in Pinot depending on the ingestion mode (batch, real-time, hybrid)

See for all possible batch/streaming tables.

Batch Table Creation

Please see for table configuration details and how to customize it.

Sample Console Output

Streaming Table Creation

Please see for table configuration details and how to customize it.

Start Kafka

Create a Kafka Topic

Create a Streaming table

Sample output

Start Kafka-Zookeeper

Start Kafka

Load Data

Load Data in Batch

User can always generate and push segments to Pinot via standalone scripts or using frameworks such as Hadoop or Spark. See this for more details on setting up Data Ingestion Jobs.

Below example goes with the standalone mode.

Sample Console Output

Load Data in Streaming

Kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.avro
inputDirURI: examples/batch/airlineStats/rawdata
jobType: SegmentCreationAndTarPush
outputDirURI: examples/batch/airlineStats/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://pinot-controller:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader,
  configClassName: null, configs: null, dataFormat: avro}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema',
  tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 403 documents
Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389
Using fixed bytes value dictionary for column: Origin, size: 294
Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS
Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1
Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303
......
......
Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"}
Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"}

SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.avro
inputDirURI: examples/batch/airlineStats/rawdata
jobType: SegmentCreationAndTarPush
outputDirURI: examples/batch/airlineStats/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://pinot-controller:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader,
  configClassName: null, configs: null, dataFormat: avro}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema',
  tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 403 documents
Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389
Using fixed bytes value dictionary for column: Origin, size: 294
Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS
Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1
Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303
......
......
Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"}
Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"}

Advanced Pinot Setup

hashtagStart Pinot components (scripts or docker images)

hashtagStart Pinot Components using docker

hashtagPull docker image

hashtag0. Create a Network

hashtag1. Start Zookeeper

hashtag2. Start Pinot Controller

hashtag3. Start Pinot Broker

hashtag4. Start Pinot Server

hashtagStart Pinot components via launcher scripts

hashtag

hashtagStart Pinot Using Config Files

hashtagPinot Controller

hashtagConfigure Controller

hashtagPinot Broker

hashtagConfigure Broker

hashtagPinot Server

hashtagConfigure Server

hashtagCreate and Configure table

hashtagBatch Table Creation

hashtagStreaming Table Creation

hashtagLoad Data

hashtagLoad Data in Batch

hashtagLoad Data in Streaming

hashtagKafka

Advanced Pinot Setup

hashtagStart Pinot components (scripts or docker images)

hashtagStart Pinot Components using docker

hashtagPull docker image

hashtag0. Create a Network

hashtag1. Start Zookeeper

hashtag2. Start Pinot Controller

hashtag3. Start Pinot Broker

hashtag4. Start Pinot Server

hashtagStart Pinot components via launcher scripts

hashtag

hashtagStart Pinot Using Config Files

hashtagPinot Controller

hashtagConfigure Controller

hashtagPinot Broker

hashtagConfigure Broker

hashtagPinot Server

hashtagConfigure Server

hashtagCreate and Configure table

hashtagBatch Table Creation

hashtagStreaming Table Creation

hashtagLoad Data

hashtagLoad Data in Batch

hashtagLoad Data in Streaming

hashtagKafka

hashtagStart Pinot Controller

hashtagStart Pinot Broker

hashtagStart Pinot Controller

Start Pinot components (scripts or docker images)

Start Pinot Components using docker

Pull docker image

0. Create a Network

1. Start Zookeeper

2. Start Pinot Controller

3. Start Pinot Broker

4. Start Pinot Server

Start Pinot components via launcher scripts

Start Pinot Using Config Files

Pinot Controller

Configure Controller

Pinot Broker

Configure Broker

Pinot Server

Configure Server

Create and Configure table

Batch Table Creation

Streaming Table Creation

Load Data

Load Data in Batch

Load Data in Streaming

Kafka

Start Pinot components (scripts or docker images)

Start Pinot Components using docker

Pull docker image

0. Create a Network

1. Start Zookeeper

2. Start Pinot Controller

3. Start Pinot Broker

4. Start Pinot Server

Start Pinot components via launcher scripts

Start Pinot Using Config Files

Pinot Controller

Configure Controller

Pinot Broker

Configure Broker

Pinot Server

Configure Server

Create and Configure table

Batch Table Creation

Streaming Table Creation

Load Data

Load Data in Batch

Load Data in Streaming

Kafka

Start Pinot Controller

Start Pinot Broker

Start Pinot Controller