Comment on page

Monitor Pinot using Prometheus and Grafana

Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment.
Prerequisite
Kubernetes v1.16.5
HelmCharts v3.1.2
Deploy Pinot
Install Pinot helm repo
## Adding Pinot helm repo
helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm
## Extract all the configurable values of Pinot Helm into a config.
helm inspect values pinot/pinot > /tmp/pinot-values.yaml
Configure Pinot Helm to enable Prometheus JMX Exporter
1. Configure jvmOpts:
Add JMX Prometheus Java Agent to controller.jvmOpts / broker.jvmOpts/ server.jvmOpts . Note that Pinot Docker image already packages jmx_prometheus_javaagent.jar.
Below config will expose pinot metrics to port 8008 for Prometheus to scrape.
controller:
  ...
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms256M -Xmx1G"
You can port forward port 8008 to local and access metrics though: http://localhost:8008/metrics​
Sample output of JMX metrics
2. Configure service annotations:
Add Prometheus related annotations to enable Prometheus to scrape metrics.
controller.service.annotations
broker.service.annotations
server.service.annotations
controller.podAnnotations
broker.podAnnotations
server.podAnnotations
controller:
  ...
  service:
    annotations:
      "prometheus.io/scrape": "true"
      "prometheus.io/port": "8008"
  ...
  podAnnotations:
    "prometheus.io/scrape": "true"
    "prometheus.io/port": "8008"
Deploy Pinot Helm
kubectl create ns pinot
helm install pinot pinot/pinot -n pinot --values /tmp/pinot-values.yaml
Deploy Prometheus
Once Pinot is deployed and running, we can start deploy Prometheus.
Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm inspect values prometheus-community/prometheus > /tmp/prometheus-values.yaml
Configure Prometheus
Remember to check the configs:
server.persistentVolume: data storage location/size limit/storage class
server.retention: how long to keep the data (default is 15d)
Deploy Prometheus
kubectl create ns prometheus
helm install prometheus prometheus-community/prometheus -n prometheus --values /tmp/prometheus-values.yaml
Access Prometheus
Port forward Prometheus service to local and open the page on localhost:30080
kubectl port-forward service/prometheus-server 30080:80 -n prometheus
Then we can query metrics Prometheus scrapped:
Deploy Grafana
Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file:
helm repo add grafana https://grafana.github.io/helm-charts
helm inspect values grafana/grafana > /tmp/grafana-values.yaml
Configure Grafana
Deploy Grafana
kubectl create ns grafana
helm install grafana grafana/grafana -n grafana --values /tmp/grafana-values.yaml
Get Password to access Grafana
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Access Grafana dashboard
You can access it locally through port forwarding:
kubectl port-forward service/grafana 20080:80 -n grafana
Once open the dashboard, you can login with credential:
admin/[ PASSWORD GET FROM PREVIOUS STEP]
Grafana Dashboard
Add data source
Click on Prometheus and set HTTP URL to : http://prometheus-server.prometheus.svc.cluster.local
Prometheus data source config
Configure Pinot Dashboard
Once data source is added, we can import a Pinot Dashboard:
Grafana Import Button
A sample Pinot dashboard JSON is:
Pinot-1601334866100.json
59KB
Code
sample-pinot-dashboard
Upload this file and select Prometheus as data source to finish the import:
Grafana Import Page
Then you can explore and make your own Pinot dashboard.

Amazon MSK (Kafka)

Performance Optimization Configurations

Last modified 1mo ago