Manual cluster setup

Search…
This quick start guide will show you how to set up a Pinot cluster manually.
Start Pinot components (scripts or docker images)
A manual cluster setup consists of the following components -
1. Zookeeper
2. Controller
3. Broker
4. Server
5. Kafka
We will run each of these components in separate containers
Using docker images
Using launcher scripts
Start Pinot Components using docker
Prerequisites
If running locally, please ensure your docker cluster has enough resources, below is a sample config.
Sample docker resources
Pull docker image
You can try out the pre-built Pinot all-in-one docker image.
1
export PINOT_VERSION=0.8.0
2
export PINOT_IMAGE=apachepinot/pinot:${PINOT_VERSION}
3
docker pull ${PINOT_IMAGE}
Copied!
(Optional) You can also follow the instructions here to build your own images.
0. Create a Network
Create an isolated bridge network in docker
1
docker network create -d bridge pinot-demo
Copied!
1. Start Zookeeper
Start Zookeeper in daemon mode. This is a single node zookeeper setup. Zookeeper is the central metadata store for Pinot and should be set up with replication for production use. See Running Replicated Zookeeper for more information.
1
docker run \
2
    --network=pinot-demo \
3
    --name  pinot-zookeeper \
4
    --restart always \
5
    -p 2181:2181 \
6
    -d zookeeper:3.5.6
Copied!
2. Start Pinot Controller
Start Pinot Controller in daemon and connect to Zookeeper.
Below command expects a 4GB memory container. Please tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo \
3
    --name pinot-controller \
4
    -p 9000:9000 \
5
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" \
6
    -d ${PINOT_IMAGE} StartController \
7
    -zkAddress pinot-zookeeper:2181
Copied!
3. Start Pinot Broker
Start Pinot Broker in daemon and connect to Zookeeper.
Below command expects a 4GB memory container. Please tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo \
3
    --name pinot-broker \
4
    -p 8099:8099 \
5
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" \
6
    -d ${PINOT_IMAGE} StartBroker \
7
    -zkAddress pinot-zookeeper:2181
Copied!
4. Start Pinot Server
Start Pinot Server in daemon and connect to Zookeeper.
Below command expects a 16GB memory container. Please tune-Xms and-Xmx if your machine doesn't have enough resources.
1
docker run --rm -ti \
2
    --network=pinot-demo \
3
    --name pinot-server \
4
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
5
    -d ${PINOT_IMAGE} StartServer \
6
    -zkAddress pinot-zookeeper:2181
Copied!
5. Start Kafka
Optionally, you can also start Kafka for setting up realtime streams. This brings up the Kafka broker on port 9092.
1
docker run --rm -ti \
2
    --network pinot-demo --name=kafka \
3
    -e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
4
    -e KAFKA_BROKER_ID=0 \
5
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
6
    -d wurstmeister/kafka:latest
Copied!
Now all Pinot related components are started as an empty cluster.
You can run the below command to check container status.
1
docker container ls -a
Copied!
Sample Console Output
1
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
2
9ec20e4463fa        wurstmeister/kafka:latest   "start-kafka.sh"         43 minutes ago      Up 43 minutes                                                              kafka
3
0775f5d8d6bf        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   44 minutes ago      Up 44 minutes       8096-8099/tcp, 9000/tcp                                pinot-server
4
64c6392b2e04        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   44 minutes ago      Up 44 minutes       8096-8099/tcp, 9000/tcp                                pinot-broker
5
b6d0f2bd26a3        apachepinot/pinot:latest    "./bin/pinot-admin.s…"   45 minutes ago      Up 45 minutes       8096-8099/tcp, 0.0.0.0:9000->9000/tcp                  pinot-quickstart
6
570416fc530e        zookeeper:3.5.6             "/docker-entrypoint.…"   45 minutes ago      Up 45 minutes       2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   pinot-zookeeper
Copied!
Prerequisites
Follow this instruction in Getting Pinot to get Pinot
Start Pinot components via launcher scripts
1. Start Zookeeper
1
cd apache-pinot-${PINOT_VERSION}-bin
2
bin/pinot-admin.sh StartZookeeper \
3
  -zkPort 2191
Copied!
You can use Zooinspector to browse the Zookeeper instance.
2. Start Pinot Controller
The examples below are for Java 8 users.
For Java 11+ users, please remove the GC settings insideJAVA_OPTS. So it looks like: export JAVA_OPTS="-Xms4G -Xmx8G"
1
export JAVA_OPTS="-Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
2
bin/pinot-admin.sh StartController \
3
    -zkAddress localhost:2191 \
4
    -controllerPort 9000
Copied!
3. Start Pinot Broker
1
export JAVA_OPTS="-Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
2
bin/pinot-admin.sh StartBroker \
3
    -zkAddress localhost:2191
Copied!
4. Start Pinot Server
1
export JAVA_OPTS="-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
2
bin/pinot-admin.sh StartServer \
3
    -zkAddress localhost:2191
Copied!
5. Start Kafka
1
bin/pinot-admin.sh  StartKafka \ 
2
  -zkAddress=localhost:2191/kafka \
3
  -port 19092
Copied!
Now all Pinot related components are started as an empty cluster.
Now it's time to start adding data to the cluster. Check out some of the Recipes or follow the Batch upload sample data and Stream sample data for instructions on loading your own data.
Hdfs as Deep Storage
Batch import example
Last modified 1mo ago
Copy link