Here you will find a collection of how-to guides for operators or developers tha
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
The scripts to build Pinot related docker images is located at here.
You can access those scripts by running below command to checkout Pinot repo:
You can find current supported 3 images in this directory:
Pinot: Pinot all-in-one distribution image
Pinot-Presto: Presto image with Presto-Pinot Connector built-in.
Pinot-Superset: Superset image with Pinot connector built-in.
This is a docker image of Apache Pinot.
There is a docker build script which will build a given Git repo/branch and tag the image.
Usage:
This script will check out Pinot Repo [Pinot Git URL]
on branch [Git Branch]
and build the docker image for that.
The docker image is tagged as [Docker Tag]
.
Docker Tag
: Name and tag your docker image. Default is pinot:latest
.
Git Branch
: The Pinot branch to build. Default is master
.
Pinot Git URL
: The Pinot Git Repo to build, users can set it to their own fork. Please note that, the URL is https://
based, not git://
. Default is the Apache Repo: https://github.com/apache/incubator-pinot.git
.
Example of building and tagging a snapshot on your own fork:
Example of building a release version:
Example of building current master branch as a snapshot:
Script docker-push.sh
publishes a given docker image to your docker registry.
In order to push to your own repo, the image needs to be explicitly tagged with the repo name.
Example of publishing an image to apachepinot/pinot dockerHub repo.
You can also tag a built image, then push it.
Script docker-build-and-push.sh
builds and publishes this docker image to your docker registry after build.
Example of building and publishing an image to apachepinot/pinot dockerHub repo.
Please refer to Kubernetes Quickstart for deployment examples.
Docker image for Presto with Pinot integration.
This docker build project is specialized for Pinot.
Usage:
This script will check out Presto Repo [Presto Git URL]
on branch [Git Branch]
and build the docker image for that.
The docker image is tagged as [Docker Tag]
.
Docker Tag
: Name and tag your docker image. Default is pinot-presto:latest
.
Git Branch
: The Presto branch to build. Default is master
.
Presto Git URL
: The Presto Git Repo to build, users can set it to their own fork. Please note that, the URL is https://
based, not git://
. Default is the Apache Repo: https://github.com/prestodb/presto.git
.
Follow the instructions provided by Presto for writing your own configuration files under etc
directory.
The image defines two data volumes: one for mounting configuration into the container, and one for data.
The configuration volume is located alternatively at /home/presto/etc
, which contains all the configuration and plugins.
The data volume is located at /home/presto/data
.
Please refer to presto-coordinator.yaml
as k8s deployment example.
Docker image for Superset with Pinot integration.
This docker build project is based on Project docker-superset and specialized for Pinot.
Please modify file Makefile
to change image
and superset_version
accordingly.
Below command will build docker image and tag it as superset_version
and latest
.
You can also build directly with docker build
command by setting arguments:
Follow the instructions provided by Apache Superset for writing your own superset_config.py
.
Place this file in a local directory and mount this directory to /etc/superset
inside the container. This location is included in the image's PYTHONPATH
. Mounting this file to a different location is possible, but it will need to be in the PYTHONPATH
.
The image defines two data volumes: one for mounting configuration into the container, and one for data (logs, SQLite DBs, &c).
The configuration volume is located alternatively at /etc/superset
or /home/superset
; either is acceptable. Both of these directories are included in the PYTHONPATH
of the image. Mount any configuration (specifically the superset_config.py
file) here to have it read by the app on startup.
The data volume is located at /var/lib/superset
and it is where you would mount your SQLite file (if you are using that as your backend), or a volume to collect any logs that are routed there. This location is used as the value of the SUPERSET_HOME
environmental variable.
Please refer to superset.yaml
as k8s deployment example.
General steps: update Kafka's advertised.listeners
and make sure Kafka is accessible (e.g. allow inputs on Security Groups).
You will probably face the following problems.
If you want to connect to Kafka outside of EKS, you will need to change advertised.listeners
. When a client connects to a single Kafka bootstrap server (like other brokers), a bootstrap server sends a list of addresses for all brokers to the client. If you want to connect to a EKS Kafka, these default values will not be correct. This post provides an excellent explanation of the field.
If you use Helm to deploy Kafka to AWS EKS, please review the chart's README. It describes multiple setups for communicating into EKS.
Running helm upgrade
on the Kafka chart does not always update the pods. The exact reason is unknown. It's probably an issue with the chart's implementation. You should run kubectl describe pod
and other commands to see the current status of the pods. During initial development, you can run helm uninstall
and then helm install
to force the values to update.
Pinot community has provided Helm based Kubernetes deployment template.
You can deploy it as simple as run a helm install
command.
However there are a few things to be noted before starting the benchmark/production.
We recommend to run Pinot with pre-defined resources for the container, and make requests and limits to be the same.
This will ensure the container won't be killed if there is a sudden bump of workload.
It will also be simpler to benchmark the system, e.g. get broker qps limit.
Below is an example for values to set in values.yaml
file. Default resources is not set.
JVM setting should be complaint with the container resources for Pinot Controller and Pinot Broker.
You can make JVM setting like below to make -Xmx
the same size as your container.
For Pinot Server, heap is majorly used for query processing, metadata management. It uses off-heap memory for data loading/persistence, memory mapped files page caching. So we recommend just keep minimal requirement for JVM, and leave the rest of the container for off-heap data operations.
E.g. Assuming data is 100 GB on disk, the container size is 4 CPU, 10GB Memory.
For JVM, limit -Xmx
to not exceed 50% container memory limit, so that the rest of the container could be leveraged by the off-heap operations.
Pinot uses remote storage as deep storage to backup segments.
Default deployment creates a mount disk(e.g Amazon EBS) as deep storage in controller.
You can configure your own S3/Azure DataLate/Google Cloud Storage following this link.
How to Connect Pinot with Amazon Managed Streaming for Apache Kafka (Amazon MSK)
This wiki documents how to connect Pinot deployed in Amazon EKS to Amazon Managed Kafka.
Please follow this AWS Quickstart Wiki to run Pinot on Amazon EKS.
Please go to MSK Landing Page to create a Kafka Cluster.
Note:
For demo simplicity, this MSK cluster reuses same VPC created by EKS cluster in the previous step. Otherwise a VPC Peering is required to ensure two VPCs could talk to each other.
Under Encryption section, chooseBoth TLS encrypted and plaintext traffic allowed
Below is a sample screenshot to create an Amazon MSK cluster.
After click on Create button, you can take a coffee break and come back.
Once the cluster is created, you can view it and click View client information
to see the Zookeeper and Kafka Broker list.
Sample Client Information
Until now, the MSK cluster is still not accessible, you can follow this Wiki to create an EC2 instance to connect to it for topic creation, run console producer and consumer.
In order to connect MSK to EKS, we need to allow the traffic could go through each other.
This is configured through Amazon VPC Page.
Record the Amazon MSK SecurityGroup
from the Cluster page, in the above demo, it's sg-01e7ab1320a77f1a9
.
Open Amazon VPC Page, click on SecurityGroups
on left bar. Find the EKS Security group: eksctl-${PINOT_EKS_CLUSTER}-cluster/ClusterSharedNodeSecurityGroup.
In SecurityGroup, click on MSK SecurityGroup (sg-01e7ab1320a77f1a9
), then Click on Edit Rules
, then add above ClusterSharedNodeSecurityGroup
(sg-0402b59d7e440f8d1
) to it.
Click EKS Security Group ClusterSharedNodeSecurityGroup
(sg-0402b59d7e440f8d1
), add In bound Rule for MSK Security Group (sg-01e7ab1320a77f1a9
).
Now, EKS cluster should be able to talk to Amazon MSK.
To run below commands, please ensure you set two environment variable with ZOOKEEPER_CONNECT_STRING
and BROKER_LIST_STRING
(Use plaintext) from Amazon MSK client information, and replace the Variables accordingly.
E.g.
You can log into one EKS node or container and run below command to create a topic.
E.g. Enter into Pinot controller container:
Then install wget
then download Kafka binary.
Create a Kafka topic:
Topic creation succeeds with below message:
Once topic is created, we can start a simple application to produce to it.
You can download below yaml file, then replace:
${ZOOKEEPER_CONNECT_STRING}
-> MSK Zookeeper String
${BROKER_LIST_STRING}
-> MSK Plaintext Broker String in the deployment
${GITHUB_PERSONAL_ACCESS_TOKEN}
-> A personal Github Personal Access Token generated from here, please grant all read permissions to it. Here is the source code to generate Github Events.
And apply the YAML file by.
Once the pod is up, you can verify by running a console consumer to read from it.
Try to run from the Pinot Controller container entered in above step.
This step is relatively easy.
Since we already put table creation request into the ConfigMap, we can just enter into pinot-github-events-data-into-msk-kafka
pod to execute the command.
Check if the pod is running:
Sample output:
Enter into the pod
Create Table
Sample output:
Then you can open Pinot Query Console to browse the data
You will need the following in order to run pinot in production:
Hardware for controller/broker/servers as per your load
Working installation of Zookeeper that Pinot can use. We recommend setting aside a path within zookpeer and including that path in pinot.controller.zkStr. Pinot will create its own cluster under this path (cluster name decided by pinot.controller.helixClusterName)
Shared storage mounted on controllers (if you plan to have multiple controllers for the same cluster). Alternatively, an implementation of PinotFS that the Pinot hosts have access to.
HTTP load balancers for spraying queries across brokers (or other mechanism to balance queries)
HTTP load balancers for spraying controller requests (e.g. segment push, or other controller APIs) or other mechanisms for distribution of these requests.
In general, when deploying Pinot services, it is best to adhere to a specific ordering in which the various components should be deployed. This deployment order is recommended in case of the scenario that there might be protocol or other significant differences, the deployments go out in a predictable order in which failure due to these changes can be avoided.
The ordering is as follows:
pinot-controller
pinot-broker
pinot-server
pinot-minion
Pinot provides a web-based management console and a command-line utility (pinot-admin.sh
) in order to help provision and manage pinot clusters.
The web based management console allows operations on tables, tenants, segments and schemas. You can access the console via http://controller-host:port/help
. The console also allows you to enter queries for interactive debugging. Here are some screen-shots from the console.
Listing all the schemas in the Pinot cluster:
Rebalancing segments of a table:
The command line utility (pinot-admin.sh
) can be generated by running mvn install package -DskipTests -Pbin-dist
in the directory in which you checked out Pinot.
Here is an example of invoking the command to create a pinot segment:
Here is an example of executing a query on a Pinot table:
Number of missing segments that the broker queried for (expected to be on the server) but the server didn’t have. This can be due to retention or stale routing table.
Total time to take from receiving to finishing executing the query.
The number of exception which might have occurred during query execution.
This gives a binary value based on whether low-level consumption is healthy (1) or unhealthy (0). It’s important to ensure at least a single replica of each partition is consuming.
The highest offset which has been consumed so far.
The rate which an individual broker is receiving queries. Units are in QPS.
These multiple metrics will indicate if a query is dropped, ie the processing of that query has been forfeited for some reason.
Indicates a count of partial responses. A partial response is when at least 1 of the requested servers fails to respond to the query.
Binary metric which will indicate when the configured QPS quota for a table is exceeded (1) or if there is capacity remaining (0).
Percentage of the configured QPS quota being utilized.
Many of the controller metrics include a table name and thus are dynamically generated in the code. The metrics below point to the classes which generate the corresponding metrics.
To get the real metric name, the easiest route is to spin up a controller instance, create a table with the desired name and look through the generated metrics.
Todo
Give a more detailed explanation of how metrics are generated, how to identify real metrics names and where to find them in the code.
Percentage of complete online replicas in external view as compared to replicas in ideal state.
Number of segments in an ERROR
state for a given table.
The time in hours since the last time an offline segment has been pushed to the controller.
Percentage of complete online replicas in external view as compared to replicas in ideal state.
Shows how much of the table’s storage quota is currently being used, metric will a percentage of a the entire quota.
In practice, we need to run Pinot data ingestion as a pipeline or a scheduled job.
Assuming pinot-distribution is already built, inside examples directory, you could find several sample table layouts.
Usually each table deserves its own directory, like airlineStats.
Inside the table directory, rawdata is created to put all the input data.
Typically, for data events with timestamp, we partition those data and store them into a daily folder. E.g. a typically layout would follow this pattern: rawdata/%yyyy%/%mm%/%dd%/[daily_input_files]
.
Create a batch ingestion job spec file to describe how to ingest the data.
Below is an example (also located at examples/batch/airlineStats/ingestionJobSpec.yaml
)
Below command will create example table into Pinot cluster.
Below command will kick off the ingestion job to generate Pinot segments and push them into the cluster.
After job finished, segments are stored in examples/batch/airlineStats/segments
following same layout of input directory layout.
Below example is running in a spark local mode. You can download spark distribution and start it by running:
Below command shows how to use spark-submit command to submit a spark job using pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar
.
Sample Spark ingestion job spec yaml, (also located at examples/batch/airlineStats/sparkIngestionJobSpec.yaml
):
Please ensure parameter PINOT_ROOT_DIR
and PINOT_VERSION
are set properly.
Please ensure you set
spark.driver.extraJavaOptions =>
-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins
Or put all the required plugins jars to CLASSPATH, then set -Dplugins.dir=${CLASSPATH}
spark.driver.extraClassPath =>
pinot-all-${PINOT_VERSION}-jar-with-depdencies.jar
Below command shows how to use Hadoop jar command to run a Hadoop job using pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar
.
Sample Hadoop ingestion job spec yaml(also located at examples/batch/airlineStats/hadoopIngestionJobSpec.yaml
):
Please ensure parameter PINOT_ROOT_DIR
and PINOT_VERSION
are set properly.
Pinot exposes several metrics to monitor the service and ensure that pinot users are not experiencing issues. In this section we discuss some of the key metrics that are useful to monitor. A full list of metrics is available in the section.
Missing Segments -
Query latency -
Query Execution Exceptions -
Realtime Consumption Status -
Realtime Highest Offset Consumed -
Incoming QPS (per broker) -
Dropped Requests - , ,
Partial Responses -
Table QPS quota exceeded -
Table QPS quota usage percent -
Percent Segments Available -
Segments in Error State -
Last push delay - Generated in the class.
Percent of replicas up -
Table storage quota usage percent -
Build latest Pinot Distribution following this .