Running Pinot in Docker

Search…
This guide will show you to run a Pinot Cluster using Docker.
In this guide we will learn about running Pinot in Docker.
This guide assumes that you have installed Docker and have configured it with enough memory. A sample config is shown below:
Sample Docker resources
The latest Pinot Docker image is published at apachepinot/pinot:latest and you can see a list of all published tags on Docker Hub.
You can pull the Docker image onto your machine by running the following command:
1
docker pull apachepinot/pinot:latest
Copied!
Or if you want to use a specific version:
1
docker pull apachepinot/pinot:0.9.3
Copied!
Now that we've downloaded the Pinot Docker image, it's time to set up a cluster. There are two ways to do this:
Quick Start
Pinot comes with quick-start commands that launch instances of Pinot components in the same process and import pre-built datasets.
For example, the following quick-start launches Pinot with a baseball dataset pre-loaded:
1
docker run \
2
    -p 9000:9000 \
3
    apachepinot/pinot:0.9.3 QuickStart \
4
    -type batch
Copied!
For a list of all the available quick starts, see the Quick Start Examples.
Manual Cluster
The quick start scripts launch Pinot with minimal resources. If you want to play with bigger datasets (more than a few MB), you can launch each of the Pinot components individually.
Docker
Create a Network
Create an isolated bridge network in docker
1
docker network create -d bridge pinot-demo_default
Copied!
Start Zookeeper
Start Zookeeper in daemon mode. This is a single node zookeeper setup. Zookeeper is the central metadata store for Pinot and should be set up with replication for production use. For more information, see Running Replicated Zookeeper.
1
docker run \
2
    --network=pinot-demo_default \
3
    --name  manual-pinot-zookeeper \
4
    --restart always \
5
    -p 2181:2181 \
6
    -d zookeeper:3.5.6
Copied!
Start Pinot Controller
Start Pinot Controller in daemon and connect to Zookeeper.
The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo_default \
3
    --name manual-pinot-controller \
4
    -p 9000:9000 \
5
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" \
6
    -d ${PINOT_IMAGE} StartController \
7
    -zkAddress pinot-zookeeper:2181
Copied!
Start Pinot Broker
Start Pinot Broker in daemon and connect to Zookeeper.
The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo_default \
3
    --name manual-pinot-broker \
4
    -p 8099:8099 \
5
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" \
6
    -d ${PINOT_IMAGE} StartBroker \
7
    -zkAddress pinot-zookeeper:2181
Copied!
Start Pinot Server
Start Pinot Server in daemon and connect to Zookeeper.
The command below expects a 16GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo_default \
3
    --name manual-pinot-server \
4
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
5
    -d ${PINOT_IMAGE} StartServer \
6
    -zkAddress pinot-zookeeper:2181
Copied!
Start Kafka
Optionally, you can also start Kafka for setting up realtime streams. This brings up the Kafka broker on port 9092.
1
docker run --rm -ti \
2
    --network pinot-demo_default --name=kafka \
3
    -e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
4
    -e KAFKA_BROKER_ID=0 \
5
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
6
    -d wurstmeister/kafka:latest
Copied!
Now all Pinot related components are started as an empty cluster.
You can run the below command to check container status.
1
docker container ls -a
Copied!
Sample Console Output
1
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
2
9ec20e4463fa        wurstmeister/kafka:latest   "start-kafka.sh"         43 minutes ago      Up 43 minutes                                                              kafka
3
0775f5d8d6bf        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   44 minutes ago      Up 44 minutes       8096-8099/tcp, 9000/tcp                                pinot-server
4
64c6392b2e04        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   44 minutes ago      Up 44 minutes       8096-8099/tcp, 9000/tcp                                pinot-broker
5
b6d0f2bd26a3        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   45 minutes ago      Up 45 minutes       8096-8099/tcp, 0.0.0.0:9000->9000/tcp                  pinot-quickstart
6
570416fc530e        zookeeper:3.5.6             "/docker-entrypoint.…"   45 minutes ago      Up 45 minutes       2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   pinot-zookeeper
Copied!
Docker Compose
Create a file called docker-compose.yml that contains the following:
docker-compose.yml
1
version: '3.7'
2
services:
3
  zookeeper:
4
    image: zookeeper:3.5.6
5
    hostname: zookeeper
6
    container_name: manual-zookeeper
7
    ports:
8
      - "2181:2181"
9
    environment:
10
      ZOOKEEPER_CLIENT_PORT: 2181
11
      ZOOKEEPER_TICK_TIME: 2000
12
  pinot-controller:
13
    image: apachepinot/pinot:0.9.3
14
    command: "StartController -zkAddress manual-zookeeper:2181"
15
    container_name: "manual-pinot-controller"
16
    restart: unless-stopped
17
    ports:
18
      - "9000:9000"
19
    environment:
20
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
21
    depends_on:
22
      - zookeeper
23
  pinot-broker:
24
    image: apachepinot/pinot:0.9.3
25
    command: "StartBroker -zkAddress manual-zookeeper:2181"
26
    restart: unless-stopped
27
    container_name: "manual-pinot-broker"
28
    ports:
29
      - "8099:8099"
30
    environment:
31
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
32
    depends_on:
33
      - pinot-controller
34
  pinot-server:
35
    image: apachepinot/pinot:0.9.3
36
    command: "StartServer -zkAddress manual-zookeeper:2181"
37
    restart: unless-stopped
38
    container_name: "manual-pinot-server" 
39
    environment:
40
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
41
    depends_on:
42
      - pinot-broker
43
  
Copied!
Run the following command to launch all the components:
1
docker-compose --project-name pinot-demo up
Copied!
You can run the below command to check container status.
1
docker container ls 
Copied!
Sample Console Output
1
CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                                                     NAMES
2
ba5cb0868350   apachepinot/pinot:0.9.3   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8099/tcp, 9000/tcp                                                   manual-pinot-server
3
698f160852f9   apachepinot/pinot:0.9.3   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp        manual-pinot-broker
4
b1ba8cf60d69   apachepinot/pinot:0.9.3   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8099/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                  manual-pinot-controller
5
54e7e114cd53   zookeeper:3.5.6           "/docker-entrypoint.…"   About a minute ago   Up About a minute   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp   manual-zookeeper
Copied!
Once your cluster is up and running, you can head over to Exploring Pinot to learn how to run queries against the data.
If you have minikube or Docker Kubernetes installed, you could also try running the Kubernetes quick start.
Running Pinot locally
Quick Start Examples
Last modified 4mo ago
Copy link
Contents