1 of 12

Getting started

This section contains quick start guides to help you get up and running with Pinot.

Running Pinot

We want your experience getting started with Pinot to be both low effort and high reward. Here you'll find a collection of quick start guides that contain starter distributions of the Pinot platform.

Frequent questions

This page has a collection of frequently asked questions with answers from the community.

This is a list of frequent questions most often asked in our troubleshooting channel on Slack. Please feel free to contribute your questions and answers here and make a pull request.

Ingestion

Running Pinot locally

This quick start guide will help you bootstrap a Pinot standalone instance on your local machine.

In this guide you'll learn how to download and install Apache Pinot as a standalone instance.

This is a quickstart guide that will show you how to quickly start an example recipe in a standalone instance and is meant for learning. To run Pinot in cluster mode, please take a look at Manual cluster setup.

Download Apache Pinot

First, let's download the Pinot distribution for this tutorial. You can either build the distribution from source or download a packaged release.

Prerequisites

Install JDK8 or higher.

Build from source or download the distribution

Follow these steps to checkout code from and build Pinot locally

Prerequisites

Install 3.6 or higher

Setting up a Pinot cluster

We'll be using a quick-start script, which does the following:

Sets up the Pinot cluster QuickStartCluster
Creates a sample table and loads sample data

There's 3 kinds of quick start

Batch

Batch quick start creates the pinot cluster, creates an offline table baseballStats and pushes sample offline data to the table.

That's it! We've spun up a Pinot cluster. You can continue playing with other types of quick start, or simply head on to to check out the data in the baseballStats table.

Streaming

Streaming quick start sets up a Kafka cluster and pushes sample data to a Kafka topic. Then, it creates the Pinot cluster and creates a realtime table meetupRSVP which ingests data from the Kafka topic.

We now have a Pinot cluster with a realtime table! You can head over to to check out the data in the meetupRSVP table.

Hybrid

Hybrid quick start sets up a Kafka cluster and pushes sample data to a Kafka topic. Then, it creates the Pinot cluster and creates a hybrid table airlineStats . The realtime table ingests data from the Kafka topic. Lastly, sample data is pushed into the offline table.

Let's head over to to check out the data we pushed to the airlineStats table.

Running Pinot in Docker

This quick start guide will show you how to run a Pinot cluster using Docker.

Prerequisites

Install Docker

You can also try Kubernetes quick start if you already have a local minikube cluster installed or Docker Kubernetes setup.

Create an isolated bridge network in docker

We'll be using our docker image apachepinot/pinot:latest to run this quick start, which does the following:

Sets up the Pinot cluster
Creates a sample table and loads sample data

There are 3 types of quick start examples.

Batch example
Streaming example
Hybrid example

Batch example

In this example we demonstrate how to do batch processing with Pinot.

Starts Pinot deployment by starting
- Apache Zookeeper
- Pinot Controller

Once the Docker container is running, you can view the logs by running the following command.

That's it! We've spun up a Pinot cluster.

It may take a while for all the Pinot components to start and for the sample data to be loaded.

Use the below command to check the status in the container logs.

Your cluster is ready once you see the cluster setup completion messages and sample queries, as demonstrated below.

You can head over to to check out the data in the baseballStats table.

Streaming example

In this example we demonstrate how to do stream processing with Pinot.

Starts Pinot deployment by starting
- Apache Kafka
- Apache Zookeeper

Once the cluster is up, you can head over to to check out the data in the meetupRSVPEvents table.

Hybrid example

In this example we demonstrate how to do hybrid stream and batch processing with Pinot.

Starts Pinot deployment by starting
- Apache Kafka
- Apache Zookeeper

Once the cluster is up, you can head over to to check out the data in the airlineStats table.

Public cloud examples

This page contains multiple quick start guides for deploying Pinot to a public cloud provider.

The following quick start guides will show you how to run an Apache Pinot cluster using Kubernetes on different public cloud providers.

Running on Azure

This starter guide provides a quick start for running Pinot on Microsoft Azure

This document provides the basic instruction to set up a Kubernetes Cluster on Azure Kubernetes Service (AKS)

1. Tooling Installation

1.1 Install Kubectl

Please follow this link () to install kubectl.

For Mac User

Please check kubectl version after installation.

QuickStart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12

1.2 Install Helm

Please follow this link () to install helm.

For Mac User

Please check helm version after installation.

This QuickStart provides helm supports for helm v3.0.0 and v2.12.1. Please pick the script based on your helm version.

1.3 Install Azure CLI

Please follow this link () to install Azure CLI.

For Mac User

Below script will open default browser to sign-in to your Azure Account.

3. (Optional) Create a Resource Group

Below script will create a resource group in location eastus.

4. (Optional) Create a Kubernetes cluster(AKS) in Azure

Below script will create a 3 nodes cluster named pinot-quickstart for demo purposes.

Please modify the parameters in the example command below:

Once the command is succeed, it's ready to be used.

5. Connect to an existing cluster

Simply run below command to get the credential for the cluster pinot-quickstart that you just created or your existing cluster.

To verify the connection, you can run:

6. Pinot Quickstart

Please follow this to deploy your Pinot Demo.

7. Delete a Kubernetes Cluster

Running on GCP

This starter provides a quick start for running Pinot on Google Cloud Platform (GCP)

This document provides the basic instruction to set up a Kubernetes Cluster on Google Kubernetes Engine(GKE)

1. Tooling Installation

1.1 Install Kubectl

Please follow this link () to install kubectl.

For Mac User

Please check kubectl version after installation.

QuickStart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12

1.2 Install Helm

Please follow this link () to install helm.

For Mac User

Please check helm version after installation.

This QuickStart provides helm supports for helm v3.0.0 and v2.12.1. Please pick the script based on your helm version.

1.3 Install Google Cloud SDK

Please follow this link () to install Google Cloud SDK.

1.3.1 For Mac User

Install Google Cloud SDK

Restart your shell

2. (Optional) Initialize Google Cloud Environment

3. (Optional) Create a Kubernetes cluster(GKE) in Google Cloud

Below script will create a 3 nodes cluster named pinot-quickstart in us-west1-b with n1-standard-2 machines for demo purposes.

Please modify the parameters in the example command below:

You can monitor cluster status by command:

Once the cluster is in RUNNING status, it's ready to be used.

4. Connect to an existing cluster

Simply run below command to get the credential for the cluster pinot-quickstart that you just created or your existing cluster.

To verify the connection, you can run:

5. Pinot Quickstart

Please follow this to deploy your Pinot Demo.

6. Delete a Kubernetes Cluster

Running on AWS

This guide provides a quick start for running Pinot on Amazon Web Services (AWS).

This document provides the basic instruction to set up a Kubernetes Cluster on

1. Tooling Installation

Manual cluster setup

This quick start guide will show you how to set up a Pinot cluster manually.

Start Pinot components (scripts or docker images)

Running Pinot locally

This quick start guide will help you bootstrap a Pinot standalone instance on your local machine.

In this guide you'll learn how to download and install Apache Pinot as a standalone instance.

Download Apache Pinot

First, let's download the Pinot distribution for this tutorial. You can either build the distribution from source or download a packaged release.

Prerequisites

Install JDK8 or higher.

Build from source or download the distribution

Follow these steps to checkout code from and build Pinot locally

Prerequisites

Install 3.6 or higher

Setting up a Pinot cluster

We'll be using a quick-start script, which does the following:

Sets up the Pinot cluster QuickStartCluster
Creates a sample table and loads sample data

There's 3 kinds of quick start

Batch

Batch quick start creates the pinot cluster, creates an offline table baseballStats and pushes sample offline data to the table.

That's it! We've spun up a Pinot cluster. You can continue playing with other types of quick start, or simply head on to to check out the data in the baseballStats table.

Streaming

We now have a Pinot cluster with a realtime table! You can head over to to check out the data in the meetupRSVP table.

Hybrid

Let's head over to to check out the data we pushed to the airlineStats table.

Running Pinot in Docker

This quick start guide will show you how to run a Pinot cluster using Docker.

Prerequisites

Install Docker

You can also try Kubernetes quick start if you already have a local minikube cluster installed or Docker Kubernetes setup.

Create an isolated bridge network in docker

We'll be using our docker image apachepinot/pinot:latest to run this quick start, which does the following:

Sets up the Pinot cluster
Creates a sample table and loads sample data

There are 3 types of quick start examples.

Batch example
Streaming example
Hybrid example

Batch example

In this example we demonstrate how to do batch processing with Pinot.

Starts Pinot deployment by starting
- Apache Zookeeper
- Pinot Controller

Once the Docker container is running, you can view the logs by running the following command.

That's it! We've spun up a Pinot cluster.

It may take a while for all the Pinot components to start and for the sample data to be loaded.

Use the below command to check the status in the container logs.

Your cluster is ready once you see the cluster setup completion messages and sample queries, as demonstrated below.

You can head over to to check out the data in the baseballStats table.

Streaming example

In this example we demonstrate how to do stream processing with Pinot.

Starts Pinot deployment by starting
- Apache Kafka
- Apache Zookeeper

Once the cluster is up, you can head over to to check out the data in the meetupRSVPEvents table.

Hybrid example

In this example we demonstrate how to do hybrid stream and batch processing with Pinot.

Starts Pinot deployment by starting
- Apache Kafka
- Apache Zookeeper

Once the cluster is up, you can head over to to check out the data in the airlineStats table.

Running Pinot in Kubernetes

Pinot quick start in Kubernetes

1. Prerequisites

This quick start assumes the existence of a Kubernetes cluster. Please follow the links below to setup your Kubernetes cluster.

2. Setting up a Pinot cluster in Kubernetes

Before continuing, please make sure that you've downloaded Apache Pinot. The scripts for the setup in this guide can be found in our open source project on GitHub.

The scripts can be found in the Pinot source at ./incubator-pinot/kubernetes/helm

2.1 Start Pinot with Helm

Pinot repo has pre-packaged HelmCharts for Pinot and Presto. Helm Repo index file is .

2.1.1 Update helm dependency

2.2 Check Pinot deployment status

3. Load data into Pinot using Kafka

3.1 Bring up a Kafka cluster for real-time data ingestion

3.2 Check Kafka deployment status

Ensure the Kafka deployment is ready before executing the scripts in the following next steps.

3.3 Create Kafka topics

The scripts below will create two Kafka topics for data ingestion:

3.4 Load data into Kafka and create Pinot schema/tables

The script below will deploy 3 batch jobs.

Ingest 19492 JSON messages to Kafka topic flights-realtime at a speed of 1 msg/sec
Ingest 19492 Avro messages to Kafka topic flights-realtime-avro at a speed of 1 msg/sec
Upload Pinot schema airlineStats

4. Query using Pinot Data Explorer

4.1 Pinot Data Explorer

Please use the script below to perform local port-forwarding, which will also open Pinot query console in your default web browser.

This script can be found in the Pinot source at ./incubator-pinot/kubernetes/helm

5. Using Superset to query Pinot

5.1 Bring up Superset

5.2 (First time) Set up Admin account

5.3 (First time) Init Superset

5.4 Load Demo data source

5.5 Access Superset UI

You can run below command to navigate superset in your browser with the previous admin credential.

You can open the imported dashboard by clicking Dashboards banner and then click on AirlineStats.

6. Access Pinot using Presto

6.1 Deploy Presto using Pinot plugin

You can run the command below to deploy a customized Presto with Pinot plugin installed.

6.2 Query Presto using Presto CLI

Once Presto is deployed, you can run the command below.

6.3 Sample queries to execute

List all catalogs

List All tables

Show schema

Count total documents

7. Deleting the Pinot cluster in Kubernetes

SegmentGenerationJobSpec: 
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**\/*.csv
inputDirURI: /tmp/pinot-quick-start/rawdata/
jobType: SegmentCreationAndTarPush
outputDirURI: /tmp/pinot-quick-start/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://localhost:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: null
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader,
  configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig,
  configs: null, dataFormat: csv}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://localhost:9000/tables/transcript/schema', tableConfigURI: 'http://localhost:9000/tables/transcript',
  tableName: transcript}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 4 documents
Using fixed bytes value dictionary for column: studentID, size: 9
Created dictionary for STRING column: studentID with cardinality: 3, max length in bytes: 3, range: 200 to 202
Using fixed bytes value dictionary for column: firstName, size: 12
Created dictionary for STRING column: firstName with cardinality: 3, max length in bytes: 4, range: Bob to Nick
Using fixed bytes value dictionary for column: lastName, size: 15
Created dictionary for STRING column: lastName with cardinality: 3, max length in bytes: 5, range: King to Young
Created dictionary for FLOAT column: score with cardinality: 4, range: 3.2 to 3.8
Using fixed bytes value dictionary for column: gender, size: 12
Created dictionary for STRING column: gender with cardinality: 2, max length in bytes: 6, range: Female to Male
Using fixed bytes value dictionary for column: subject, size: 21
Created dictionary for STRING column: subject with cardinality: 3, max length in bytes: 7, range: English to Physics
Created dictionary for LONG column: timestamp with cardinality: 4, range: 1570863600000 to 1572418800000
Start building IndexCreator!
Finished records indexing in IndexCreator!
Finished segment seal!
Converting segment: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0 to v3 format
v3 segment location for segment: transcript_OFFLINE_1570863600000_1572418800000_0 is /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3
Deleting files in v1 segment directory: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0
Starting building 1 star-trees with configs: [StarTreeV2BuilderConfig[splitOrder=[studentID, firstName],skipStarNodeCreation=[],functionColumnPairs=[org.apache.pinot.core.startree.v2.AggregationFunctionColumnPair@3a48efdc],maxLeafRecords=1]] using OFF_HEAP builder
Starting building star-tree with config: StarTreeV2BuilderConfig[splitOrder=[studentID, firstName],skipStarNodeCreation=[],functionColumnPairs=[org.apache.pinot.core.startree.v2.AggregationFunctionColumnPair@3a48efdc],maxLeafRecords=1]
Generated 3 star-tree records from 4 segment records
Finished constructing star-tree, got 9 tree nodes and 4 records under star-node
Finished creating aggregated documents, got 6 aggregated records
Finished building star-tree in 10ms
Finished building 1 star-trees in 27ms
Computed crc = 3454627653, based on files [/var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/columns.psf, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/index_map, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/metadata.properties, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/star_tree_index, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/star_tree_index_map]
Driver, record read time : 0
Driver, stats collector time : 0
Driver, indexing time : 0
Tarring segment from: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0 to: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0.tar.gz
Size for segment: transcript_OFFLINE_1570863600000_1572418800000_0, uncompressed: 6.73KB, compressed: 1.89KB
Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Start pushing segments: [/tmp/pinot-quick-start/segments/transcript_OFFLINE_1570863600000_1572418800000_0.tar.gz]... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@243c4f91] for table transcript
Pushing segment: transcript_OFFLINE_1570863600000_1572418800000_0 to location: http://localhost:9000 for table transcript
Sending request: http://localhost:9000/v2/segments?tableName=transcript to controller: nehas-mbp.hsd1.ca.comcast.net, version: Unknown
Response for pushing table transcript segment transcript_OFFLINE_1570863600000_1572418800000_0 to location http://localhost:9000 - 200: {"status":"Successfully uploaded segment: transcript_OFFLINE_1570863600000_1572418800000_0 of table: transcript"}

Getting started

hashtagRunning Pinot

hashtag

Frequent questions

hashtagIngestion

Running Pinot locally

hashtagDownload Apache Pinot

hashtagBuild from source or download the distribution

hashtagSetting up a Pinot cluster

hashtagBatch

hashtagStreaming

hashtagHybrid

Running Pinot in Docker

hashtagBatch example

hashtagStreaming example

hashtagHybrid example

Public cloud examples

Running on Azure

hashtag1. Tooling Installation

hashtag1.1 Install Kubectl

hashtag1.2 Install Helm

hashtag1.3 Install Azure CLI

hashtag2. (Optional) Login to your Azure account

hashtag3. (Optional) Create a Resource Group

hashtag4. (Optional) Create a Kubernetes cluster(AKS) in Azure

hashtag5. Connect to an existing cluster

hashtag6. Pinot Quickstart

hashtag7. Delete a Kubernetes Cluster

Running on GCP

hashtag1. Tooling Installation

hashtag1.1 Install Kubectl

hashtag1.2 Install Helm

hashtag1.3 Install Google Cloud SDK

hashtag1.3.1 For Mac User

hashtag2. (Optional) Initialize Google Cloud Environment

hashtag3. (Optional) Create a Kubernetes cluster(GKE) in Google Cloud

hashtag4. Connect to an existing cluster

hashtag5. Pinot Quickstart

hashtag6. Delete a Kubernetes Cluster

Running on AWS

hashtag1. Tooling Installation

hashtag

Manual cluster setup

hashtagStart Pinot components (scripts or docker images)

hashtag

Running Pinot locally

hashtagDownload Apache Pinot

hashtagBuild from source or download the distribution

hashtagSetting up a Pinot cluster

hashtagBatch

hashtagStreaming

hashtagHybrid

Getting started

hashtagRunning Pinot

hashtag

Frequent questions

hashtagIngestion

hashtagDeploy to a public cloud

hashtagHow to setup a Pinot cluster

hashtagData import examples

hashtagIndexing

hashtagHow to set inverted indexes?

hashtagHow to apply inverted index to existing setup?

hashtagHow to apply star tree index?

hashtagQuerying

hashtagWhat are all the fields in the Pinot query's JSON response?

hashtagSQL Query fails with "Encountered 'timestamp' was expecting one of..."

hashtagFiltering on STRING column WHERE column = "foo" does not work?

hashtagORDER BY using an alias doesn't work?

hashtagOperations

hashtagCan I change a column name in my table, without losing data?

hashtagHow to change number of replicas of a table?

hashtagHow to run a rebalance on a table?

hashtagTuning and Optimizations

hashtagDo replica groups work for real-time?

Public cloud examples

Running Pinot in Docker

hashtagBatch example

hashtagStreaming example

hashtagHybrid example

Running Pinot

Ingestion

Download Apache Pinot

Build from source or download the distribution

Setting up a Pinot cluster

Batch

Streaming

Hybrid

Batch example

Streaming example

Hybrid example

1. Tooling Installation

1.1 Install Kubectl

1.2 Install Helm

1.3 Install Azure CLI

2. (Optional) Login to your Azure account

3. (Optional) Create a Resource Group

4. (Optional) Create a Kubernetes cluster(AKS) in Azure

5. Connect to an existing cluster

6. Pinot Quickstart

7. Delete a Kubernetes Cluster

1. Tooling Installation

1.1 Install Kubectl

1.2 Install Helm

1.3 Install Google Cloud SDK

1.3.1 For Mac User

2. (Optional) Initialize Google Cloud Environment

3. (Optional) Create a Kubernetes cluster(GKE) in Google Cloud

4. Connect to an existing cluster

5. Pinot Quickstart

6. Delete a Kubernetes Cluster

1. Tooling Installation

Start Pinot components (scripts or docker images)

Download Apache Pinot

Build from source or download the distribution

Setting up a Pinot cluster

Batch

Streaming

Hybrid

Running Pinot

Ingestion

Deploy to a public cloud

How to setup a Pinot cluster

Data import examples

Indexing

How to set inverted indexes?

How to apply inverted index to existing setup?

How to apply star tree index?

Querying

What are all the fields in the Pinot query's JSON response?

SQL Query fails with "Encountered 'timestamp' was expecting one of..."

Filtering on STRING column WHERE column = "foo" does not work?

ORDER BY using an alias doesn't work?

Operations

Can I change a column name in my table, without losing data?

How to change number of replicas of a table?

How to run a rebalance on a table?

Tuning and Optimizations

Do replica groups work for real-time?

Batch example

Streaming example

Hybrid example

1. Tooling Installation

1.1 Install Kubectl

1.2 Install Helm

1.3 Install Google Cloud SDK

1.3.1 For Mac User

2. (Optional) Initialize Google Cloud Environment

3. (Optional) Create a Kubernetes cluster(GKE) in Google Cloud

4. Connect to an existing cluster

5. Pinot Quickstart

6. Delete a Kubernetes Cluster

1. Tooling Installation

1.1 Install Kubectl

1.2 Install Helm

1.3 Install Azure CLI

2. (Optional) Login to your Azure account

3. (Optional) Create a Resource Group

4. (Optional) Create a Kubernetes cluster(AKS) in Azure

5. Connect to an existing cluster