# Monitor Pinot using Prometheus and Grafana

Here we will introduce how to monitor Pinot with Prometheus and Grafana in Kubernetes environment.

## Prerequisite

* Kubernetes v1.16.5
* HelmCharts v3.1.2

## Deploy Pinot

### Install Pinot helm repo

```
## Adding Pinot helm repo
helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/kubernetes/helm
## Extract all the configurable values of Pinot Helm into a config.
helm inspect values pinot/pinot > /tmp/pinot-values.yaml
```

### Configure Pinot Helm to enable Prometheus JMX Exporter

1\. Configure jvmOpts:

Add [JMX Prometheus Java Agent](https://github.com/prometheus/jmx_exporter) to `controller.jvmOpts` / `broker.jvmOpts`/ `server.jvmOpts` . Note that Pinot Docker image already packages `jmx_prometheus_javaagent.jar`.

Below config will expose pinot metrics to port 8008 for Prometheus to scrape.

```
controller:
  ...
  jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms256M -Xmx1G"
```

You can port forward port 8008 to local and access metrics though: <http://localhost:8008/metrics>

![Sample output of JMX metrics](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/3ffsp4YoTos20zJDnvZ2/image.png)

2\. Configure service annotations:

Add Prometheus related annotations to enable Prometheus to scrape metrics.

* `controller.service.annotations`
* `broker.service.annotations`
* `server.service.annotations`
* `controller.podAnnotations`
* `broker.podAnnotations`
* `server.podAnnotations`

```
controller:
  ...
  service:
    annotations:
      "prometheus.io/scrape": "true"
      "prometheus.io/port": "8008"
  ...
  podAnnotations:
    "prometheus.io/scrape": "true"
    "prometheus.io/port": "8008"
```

### Deploy Pinot Helm

```
kubectl create ns pinot
helm install pinot pinot/pinot -n pinot --values /tmp/pinot-values.yaml
```

## Deploy Prometheus

Once Pinot is deployed and running, we can start deploy Prometheus.

Similar to Pinot Helm, we will have Prometheus Helm and its config yaml file:

```
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm inspect values prometheus-community/prometheus > /tmp/prometheus-values.yaml
```

Configure Prometheus

Remember to check the configs:

* server.persistentVolume: data storage location/size limit/storage class
* server.retention: how long to keep the data (default is 15d)

Deploy Prometheus

```
kubectl create ns prometheus
helm install prometheus prometheus-community/prometheus -n prometheus --values /tmp/prometheus-values.yaml
```

Access Prometheus

Port forward Prometheus service to local and open the page on `localhost:30080`

```
kubectl port-forward service/prometheus-server 30080:80 -n prometheus
```

Then we can query metrics Prometheus scrapped:

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/5O4MN1yagiH7nuToXyYd/image.png)

## Deploy Grafana

Similar to Pinot Helm, we will have Grafana Helm and it's config yaml file:

```
helm repo add grafana https://grafana.github.io/helm-charts
helm inspect values grafana/grafana > /tmp/grafana-values.yaml
```

* Configure Grafana
* Deploy Grafana

```
kubectl create ns grafana
helm install grafana grafana/grafana -n grafana --values /tmp/grafana-values.yaml
```

* Get Password to access Grafana

```
kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```

* Access Grafana dashboard

You can access it locally through port forwarding:

```
kubectl port-forward service/grafana 20080:80 -n grafana
```

Once open the dashboard, you can login with credential:

`admin`/`[ PASSWORD GET FROM PREVIOUS STEP]`

![Grafana Dashboard](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/gmcaAbsl98qv9XSk9JKF/image%20\(47\)%20\(1\).png)

* Add data source

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/eTvKeoemIUuclf86AHsF/image.png)

Click on Prometheus and set HTTP URL to : `http://prometheus-server.prometheus.svc.cluster.local`

![Prometheus data source config](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/MXbzk9qemfbsxVNPQGlr/image.png)

* Configure Pinot Dashboard

Once data source is added, we can import a Pinot Dashboard:

![Grafana Import Button](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/GQ82hfmfUc4egAhPDuEc/image.png)

A sample Pinot dashboard JSON is:

{% file src="<https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/8HIbfcH14Dys3LAhZTw9/Pinot-1601334866100.json>" %}
sample-pinot-dashboard
{% endfile %}

Now you can upload this file and select Prometheus as data source to finish the import

![Grafana Import Page](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/c7kES9JfhKALSXjiU3AM/image.png)

Then you can explore and make your own Pinot dashboard!

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/AD6wdbW8mC6FOicGsxL6/image.png)

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/IMh8SNhFiqGY1zRkzDj8/image.png)

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/VCooJ7tBGnVEzwFOagCu/image.png)

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/MM2RpuXtXo6ObT1ICfL7/image.png)

![](https://content.gitbook.com/content/gYpmZ5dnYUh7aQjLUJ9b/blobs/8pIrkOM4lauH0VIeAQjR/image.png)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.pinot.apache.org/release-1.0.0/for-operators/tutorials/monitor-pinot-using-prometheus-and-grafana.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.