Monitor Pinot using Prometheus and Grafana
Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment.

Prerequisite

    Kubernetes v1.16.5
    HelmCharts v3.1.2

Deploy Pinot

Install Pinot helm repo

1
## Adding Pinot helm repo
2
helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm
3
## Extract all the configurable values of Pinot Helm into a config.
4
helm inspect values pinot/pinot > /tmp/pinot-values.yaml
Copied!

Configure Pinot Helm to enable Prometheus JMX Exporter

1. Configure jvmOpts:
Add JMX Prometheus Java Agent to controller.jvmOpts / broker.jvmOpts/ server.jvmOpts . Note that Pinot image already packages jmx_prometheus_javaagent-0.12.0.jar.
Below config will expose pinot metrics to port 8008 for Prometheus to scrape.
1
controller:
2
...
3
jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms256M -Xmx1G"
Copied!
You can port forward port 8008 to local and access metrics though: http://localhost:8008/metrics
Sample output of JMX metrics
2. Configure service annotations:
Add Prometheus related annotations to enable Prometheus to scrape metrics.
    controller.service.annotations
    broker.service.annotations
    server.service.annotations
    controller.podAnnotations
    broker.podAnnotations
    server.podAnnotations
1
controller:
2
...
3
service:
4
annotations:
5
"prometheus.io/scrape": "true"
6
"prometheus.io/port": "8008"
7
...
8
podAnnotations:
9
"prometheus.io/scrape": "true"
10
"prometheus.io/port": "8008"
Copied!

Deploy Pinot Helm

1
kubectl create ns pinot
2
helm install pinot pinot/pinot -n pinot --values /tmp/pinot-values.yaml
Copied!

Deploy Prometheus

Once Pinot is deployed and running, we can start deploy Prometheus.
Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file:
1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2
helm inspect values prometheus-community/prometheus > /tmp/prometheus-values.yaml
Copied!
Configure Prometheus
Please remember to check the configs:
    server.persistentVolume: data storage location/size limit/storage class
    server.retention: how long to keep the data (default is 15d)
Deploy Prometheus
1
kubectl create ns prometheus
2
helm install prometheus prometheus-community/prometheus -n prometheus --values /tmp/prometheus-values.yaml
Copied!
Access Prometheus
Port forward Prometheus service to local and open the page on localhost:30080
1
kubectl port-forward service/prometheus-server 30080:80 -n prometheus
Copied!
Then we can query metrics Prometheus scrapped:

Deploy Grafana

Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file:
1
helm repo add grafana https://grafana.github.io/helm-charts
2
helm inspect values grafana/grafana > /tmp/grafana-values.yaml
Copied!
    Configure Grafana
    Deploy Grafana
1
kubectl create ns grafana
2
helm install grafana grafana/grafana -n grafana --values /tmp/grafana-values.yaml
Copied!
    Get Password to access Grafana
1
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Copied!
    Access Grafana dashboard
You can access it locally through port forwarding:
1
kubectl port-forward service/grafana 20080:80 -n grafana
Copied!
Once open the dashboard, you can login with credential:
admin/[ PASSWORD GET FROM PREVIOUS STEP]
Grafana Dashboard
    Add data source
Click on Prometheus and set HTTP URL to : http://prometheus-server.prometheus.svc.cluster.local
Prometheus data source config
    Configure Pinot Dashboard
Once data source is added, we can import a Pinot Dashboard:
Grafana Import Button
A sample Pinot dashboard JSON is:
Pinot-1601334866100.json
59KB
Binary
sample-pinot-dashboard
Now you can upload this file and select Prometheus as data source to finish the import
Grafana Import Page
Then you can explore and make your own Pinot dashboard!
Last modified 2mo ago