# Monitor Pinot using Prometheus and Grafana Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment. ## Prerequisite * Kubernetes v1.16.5 * HelmCharts v3.1.2 ## Deploy Pinot ### Install Pinot helm repo ``` ## Adding Pinot helm repo helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm ## Extract all the configurable values of Pinot Helm into a config. helm inspect values pinot/pinot > /tmp/pinot-values.yaml ``` ### Configure Pinot Helm to enable Prometheus JMX Exporter 1\. Configure jvmOpts: Add [JMX Prometheus Java Agent](https://github.com/prometheus/jmx_exporter) to `controller.jvmOpts` / `broker.jvmOpts`/ `server.jvmOpts` . Note that Pinot image already packages `jmx_prometheus_javaagent-0.12.0.jar`. Below config will expose pinot metrics to port 8008 for Prometheus to scrape. ``` controller: ... jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms256M -Xmx1G" ``` You can port forward port 8008 to local and access metrics though: ![Sample output of JMX metrics](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MH8sczV1Wtx0v-M1IPt%2F-MH8sgzC_dXV7NIkoTd4%2Fimage.png?alt=media\&token=f86b4835-392e-4a00-b5fd-4c311c78ae80) 2\. Configure service annotations: Add Prometheus related annotations to enable Prometheus to scrape metrics. * `controller.service.annotations` * `broker.service.annotations` * `server.service.annotations` * `controller.podAnnotations` * `broker.podAnnotations` * `server.podAnnotations` ``` controller: ... service: annotations: "prometheus.io/scrape": "true" "prometheus.io/port": "8008" ... podAnnotations: "prometheus.io/scrape": "true" "prometheus.io/port": "8008" ``` ### Deploy Pinot Helm ``` kubectl create ns pinot helm install pinot pinot/pinot -n pinot --values /tmp/pinot-values.yaml ``` ## Deploy Prometheus Once Pinot is deployed and running, we can start deploy Prometheus. Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file: ``` helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm inspect values prometheus-community/prometheus > /tmp/prometheus-values.yaml ``` Configure Prometheus Please remember to check the configs: * server.persistentVolume: data storage location/size limit/storage class * server.retention: how long to keep the data (default is 15d) Deploy Prometheus ``` kubectl create ns prometheus helm install prometheus prometheus-community/prometheus -n prometheus --values /tmp/prometheus-values.yaml ``` Access Prometheus Port forward Prometheus service to local and open the page on `localhost:30080` ``` kubectl port-forward service/prometheus-server 30080:80 -n prometheus ``` Then we can query metrics Prometheus scrapped: ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGp25S25RI4ZpgNTQi3%2F-MGp33o3rbsSe8lwVekc%2Fimage.png?alt=media\&token=188d1824-784d-47c2-b667-637825387f8a) ## Deploy Grafana Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file: ``` helm repo add grafana https://grafana.github.io/helm-charts helm inspect values grafana/grafana > /tmp/grafana-values.yaml ``` * Configure Grafana * Deploy Grafana ``` kubectl create ns grafana helm install grafana grafana/grafana -n grafana --values /tmp/grafana-values.yaml ``` * Get Password to access Grafana ``` kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` * Access Grafana dashboard You can access it locally through port forwarding: ``` kubectl port-forward service/grafana 20080:80 -n grafana ``` Once open the dashboard, you can login with credential: `admin`/`[ PASSWORD GET FROM PREVIOUS STEP]` ![Grafana Dashboard](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGoq_KwsY_BcKxZoUa-%2F-MGouwJekkD-3bEfrX-D%2Fimage.png?alt=media\&token=08cd11ba-1090-419c-9714-e9c8d06c43f7) * Add data source ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGoq_KwsY_BcKxZoUa-%2F-MGovP9VBtbD9l5lch25%2Fimage.png?alt=media\&token=b3cb2c06-99ca-44af-99f3-f25dbaf146f0) Click on Prometheus and set HTTP URL to : `http://prometheus-server.prometheus.svc.cluster.local` ![Prometheus data source config](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGoq_KwsY_BcKxZoUa-%2F-MGovrJvB9hS9cw1Ms87%2Fimage.png?alt=media\&token=08e63f5e-8a1b-47d8-9eca-a9f9c2041c14) * Configure Pinot Dashboard Once data source is added, we can import a Pinot Dashboard: ![Grafana Import Button](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGoq_KwsY_BcKxZoUa-%2F-MGowAvY2mJDNIk74M7w%2Fimage.png?alt=media\&token=e8c61d1f-d873-488d-abc3-c95da25d72e8) A sample Pinot dashboard JSON is: {% file src="" %} sample-pinot-dashboard {% endfile %} Now you can upload this file and select Prometheus as data source to finish the import ![Grafana Import Page](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MGoq_KwsY_BcKxZoUa-%2F-MGp0M8gSATljKhYOKr5%2Fimage.png?alt=media\&token=1634b015-4972-4fce-8f98-31451bd36917) Then you can explore and make your own Pinot dashboard! ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MISwb3NVEPL4XdpjWxO%2F-MISwqdy1ofNMvbXp-wQ%2Fimage.png?alt=media\&token=25d80860-a0ce-407d-8061-627a37b7132e) ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MISvfL6s0Zroe81wJHN%2F-MISwI40KVNOrIiSRG6R%2Fimage.png?alt=media\&token=394bb028-726f-4dad-8211-82815a383ea0) ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MISvfL6s0Zroe81wJHN%2F-MISwNaWLEFQ3q5wFFPf%2Fimage.png?alt=media\&token=f9b33122-1ada-4029-912f-8711a086512e) ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MISvfL6s0Zroe81wJHN%2F-MISwTpQWDFqVd8S1QNI%2Fimage.png?alt=media\&token=ebd1dcfb-96ac-41a9-bd33-354c68a5c967) ![](https://2944557471-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LtH6nl58DdnZnelPdTc%2F-MISvfL6s0Zroe81wJHN%2F-MISwZY_9oGBObOg8kij%2Fimage.png?alt=media\&token=90b53ede-57a7-4d5a-954e-bdfab5cd5de5)