Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment.
Kubernetes v1.16.5
HelmCharts v3.1.2
1. Configure jvmOpts:
Add JMX Prometheus Java Agent to controller.jvmOpts
/ broker.jvmOpts
/ server.jvmOpts
. Note that Pinot Docker image already packages jmx_prometheus_javaagent.jar
.
Below config will expose pinot metrics to port 8008 for Prometheus to scrape.
You can port forward port 8008 to local and access metrics though: http://localhost:8008/metrics
2. Configure service annotations:
Add Prometheus related annotations to enable Prometheus to scrape metrics.
controller.service.annotations
broker.service.annotations
server.service.annotations
controller.podAnnotations
broker.podAnnotations
server.podAnnotations
Once Pinot is deployed and running, we can start deploy Prometheus.
Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file:
Configure Prometheus
Please remember to check the configs:
server.persistentVolume: data storage location/size limit/storage class
server.retention: how long to keep the data (default is 15d)
Deploy Prometheus
Access Prometheus
Port forward Prometheus service to local and open the page on localhost:30080
Then we can query metrics Prometheus scrapped:
Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file:
Configure Grafana
Deploy Grafana
Get Password to access Grafana
Access Grafana dashboard
You can access it locally through port forwarding:
Once open the dashboard, you can login with credential:
admin
/[ PASSWORD GET FROM PREVIOUS STEP]
Add data source
Click on Prometheus and set HTTP URL to : http://prometheus-server.prometheus.svc.cluster.local
Configure Pinot Dashboard
Once data source is added, we can import a Pinot Dashboard:
A sample Pinot dashboard JSON is:
Now you can upload this file and select Prometheus as data source to finish the import
Then you can explore and make your own Pinot dashboard!