This guide will show you to run a Pinot Cluster using Docker.
In this guide we will learn about running Pinot in Docker.
This guide assumes that you have installed Docker and have configured it with enough memory. A sample config is shown below:
The latest Pinot Docker image is published at apachepinot/pinot:latest and you can see a list of all published tags on Docker Hub .
You can pull the Docker image onto your machine by running the following command:
Copy docker pull apachepinot/pinot:latest Or if you want to use a specific version:
Copy docker pull apachepinot/pinot:0.9.3 Now that we've downloaded the Pinot Docker image, it's time to set up a cluster. There are two ways to do this:
Quick Start
Pinot comes with quick-start commands that launch instances of Pinot components in the same process and import pre-built datasets.
For example, the following quick-start launches Pinot with a baseball dataset pre-loaded:
Copy docker run \
-p 9000:9000 \
apachepinot/pinot:0.9.3 QuickStart \
-type batch For a list of all the available quick starts, see the Quick Start Examples .
Manual Cluster
The quick start scripts launch Pinot with minimal resources. If you want to play with bigger datasets (more than a few MB), you can launch each of the Pinot components individually.
Docker
Create a Network
Create an isolated bridge network in docker
Copy docker network create -d bridge pinot-demo_default Start Zookeeper
Start Zookeeper in daemon mode. This is a single node zookeeper setup. Zookeeper is the central metadata store for Pinot and should be set up with replication for production use. For more information, see Running Replicated Zookeeper .
Copy docker run \
--network=pinot-demo_default \
--name manual-pinot-zookeeper \
--restart always \
-p 2181:2181 \
-d zookeeper:3.5.6 Start Pinot Controller
Start Pinot Controller in daemon and connect to Zookeeper.
The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
Copy docker run --rm -ti \
--network=pinot-demo_default \
--name manual-pinot-controller \
-p 9000:9000 \
-e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" \
-d ${PINOT_IMAGE} StartController \
-zkAddress pinot-zookeeper:2181 Start Pinot Broker
Start Pinot Broker in daemon and connect to Zookeeper.
The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
Copy docker run --rm -ti \
--network=pinot-demo_default \
--name manual-pinot-broker \
-p 8099:8099 \
-e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" \
-d ${PINOT_IMAGE} StartBroker \
-zkAddress pinot-zookeeper:2181 Start Pinot Server
Start Pinot Server in daemon and connect to Zookeeper.
The command below expects a 16GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.
Copy docker run --rm -ti \
--network=pinot-demo_default \
--name manual-pinot-server \
-e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
-d ${PINOT_IMAGE} StartServer \
-zkAddress pinot-zookeeper:2181 Start Kafka
Optionally, you can also start Kafka for setting up realtime streams. This brings up the Kafka broker on port 9092.
Copy docker run --rm -ti \
--network pinot-demo_default --name=kafka \
-e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
-e KAFKA_BROKER_ID=0 \
-e KAFKA_ADVERTISED_HOST_NAME=kafka \
-d wurstmeister/kafka:latest Now all Pinot related components are started as an empty cluster.
You can run the below command to check container status.
Copy docker container ls -a Sample Console Output
Copy CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9ec20e4463fa wurstmeister/kafka:latest "start-kafka.sh" 43 minutes ago Up 43 minutes kafka
0775f5d8d6bf apachepinot/pinot:latest "./bin/pinot-admin.s…" 44 minutes ago Up 44 minutes 8096-8099/tcp, 9000/tcp pinot-server
64c6392b2e04 apachepinot/pinot:latest "./bin/pinot-admin.s…" 44 minutes ago Up 44 minutes 8096-8099/tcp, 9000/tcp pinot-broker
b6d0f2bd26a3 apachepinot/pinot:latest "./bin/pinot-admin.s…" 45 minutes ago Up 45 minutes 8096-8099/tcp, 0.0.0.0:9000->9000/tcp pinot-quickstart
570416fc530e zookeeper:3.5.6 "/docker-entrypoint.…" 45 minutes ago Up 45 minutes 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp pinot-zookeeper Docker Compose
Create a file called docker-compose.yml that contains the following:
Copy version : '3.7'
services :
zookeeper :
image : zookeeper:3.5.6
hostname : zookeeper
container_name : manual-zookeeper
ports :
- "2181:2181"
environment :
ZOOKEEPER_CLIENT_PORT : 2181
ZOOKEEPER_TICK_TIME : 2000
pinot-controller :
image : apachepinot/pinot:0.9.3
command : "StartController -zkAddress manual-zookeeper:2181"
container_name : "manual-pinot-controller"
restart : unless-stopped
ports :
- "9000:9000"
environment :
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
depends_on :
- zookeeper
pinot-broker :
image : apachepinot/pinot:0.9.3
command : "StartBroker -zkAddress manual-zookeeper:2181"
restart : unless-stopped
container_name : "manual-pinot-broker"
ports :
- "8099:8099"
environment :
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
depends_on :
- pinot-controller
pinot-server :
image : apachepinot/pinot:0.9.3
command : "StartServer -zkAddress manual-zookeeper:2181"
restart : unless-stopped
container_name : "manual-pinot-server"
environment :
JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
depends_on :
- pinot-broker
Run the following command to launch all the components:
Copy docker-compose --project-name pinot-demo up You can run the below command to check container status.
Sample Console Output
Copy CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ba5cb0868350 apachepinot/pinot:0.9.3 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8099/tcp, 9000/tcp manual-pinot-server
698f160852f9 apachepinot/pinot:0.9.3 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp manual-pinot-broker
b1ba8cf60d69 apachepinot/pinot:0.9.3 "./bin/pinot-admin.s…" About a minute ago Up About a minute 8096-8099/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp manual-pinot-controller
54e7e114cd53 zookeeper:3.5.6 "/docker-entrypoint.…" About a minute ago Up About a minute 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp manual-zookeeper Once your cluster is up and running, you can head over to Exploring Pinot to learn how to run queries against the data.
