Monitor Pinot using Prometheus and Grafana

Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment.

Prerequisite

  • Kubernetes v1.16.5

  • HelmCharts v3.1.2

Deploy Pinot

Install Pinot helm repo

## Adding Pinot helm repo
helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm
## Extract all the configurable values of Pinot Helm into a config.
helm inspect values pinot/pinot > /tmp/pinot-values.yaml

Configure Pinot Helm to enable Prometheus JMX Exporter

1. Configure jvmOpts:

Add JMX Prometheus Java Agent to controller.jvmOpts / broker.jvmOpts/ server.jvmOpts . Note that Pinot Docker image already packages jmx_prometheus_javaagent.jar.

Below config will expose pinot metrics to port 8008 for Prometheus to scrape.

controller:
  ...
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms256M -Xmx1G"

You can port forward port 8008 to local and access metrics though: http://localhost:8008/metrics

Sample output of JMX metrics

2. Configure service annotations:

Add Prometheus related annotations to enable Prometheus to scrape metrics.

  • controller.service.annotations

  • broker.service.annotations

  • server.service.annotations

  • controller.podAnnotations

  • broker.podAnnotations

  • server.podAnnotations

controller:
  ...
  service:
    annotations:
      "prometheus.io/scrape": "true"
      "prometheus.io/port": "8008"
  ...
  podAnnotations:
    "prometheus.io/scrape": "true"
    "prometheus.io/port": "8008"

Deploy Pinot Helm

kubectl create ns pinot
helm install pinot pinot/pinot -n pinot --values /tmp/pinot-values.yaml

Deploy Prometheus

Once Pinot is deployed and running, we can start deploy Prometheus.

Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm inspect values prometheus-community/prometheus > /tmp/prometheus-values.yaml

Configure Prometheus

Please remember to check the configs:

  • server.persistentVolume: data storage location/size limit/storage class

  • server.retention: how long to keep the data (default is 15d)

Deploy Prometheus

kubectl create ns prometheus
helm install prometheus prometheus-community/prometheus -n prometheus --values /tmp/prometheus-values.yaml

Access Prometheus

Port forward Prometheus service to local and open the page on localhost:30080

kubectl port-forward service/prometheus-server 30080:80 -n prometheus

Then we can query metrics Prometheus scrapped:

Deploy Grafana

Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file:

helm repo add grafana https://grafana.github.io/helm-charts
helm inspect values grafana/grafana > /tmp/grafana-values.yaml
  • Configure Grafana

  • Deploy Grafana

kubectl create ns grafana
helm install grafana grafana/grafana -n grafana --values /tmp/grafana-values.yaml
  • Get Password to access Grafana

kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
  • Access Grafana dashboard

You can access it locally through port forwarding:

kubectl port-forward service/grafana 20080:80 -n grafana

Once open the dashboard, you can login with credential:

admin/[ PASSWORD GET FROM PREVIOUS STEP]

Grafana Dashboard
  • Add data source

Click on Prometheus and set HTTP URL to : http://prometheus-server.prometheus.svc.cluster.local

Prometheus data source config
  • Configure Pinot Dashboard

Once data source is added, we can import a Pinot Dashboard:

Grafana Import Button

A sample Pinot dashboard JSON is:

sample-pinot-dashboard

Now you can upload this file and select Prometheus as data source to finish the import

Grafana Import Page

Then you can explore and make your own Pinot dashboard!

Last updated