Monitoring

Tutorial

If you are deploying using the helm chart with Kubernetes, see the tutorial on setting up Prometheus and Grafana to monitor Pinot.

https://docs.pinot.apache.org/users/tutorials/monitor-pinot-using-prometheus-and-grafana

Key Metrics to Watch

Please refer to key metrics documented in monitoring pinot.

Customizing Metrics

Pinot uses yammer MetricsRegistry to collect metrics within our application components. These metrics can be published to a metrics server with the help of MetricsRegistryRegistrationListener interface. By default, metrics are published to JMX using the JmxReporterMetricsRegistryRegistrationListener.

You can write a listener to publish metrics to another metrics server by implementing the MetricsRegistryRegistrationListener interface. This listener can be injected into the controller by setting the fully qualified name of the class in the controller configs for the property pinot.controller.metrics.metricsRegistryRegistrationListeners.

You would have to design your own systems to view and monitor these metrics. A list of all the metrics published for each component can be found in:

• ControllerMeter

• ControllerGauge

• BrokerMeter

• BrokerGauge

• ServerMeter

• ServerGauge

• MinionMeter

• MinionGauge

JMX to Prometheus

Metrics published to JMX could also be exposed to Prometheus through tooling like jmx_reporter.

To run as a javaagent, download jmx_prometheus_javaagent jar and pinot.yml run:

ALL_JAVA_OPTS="-javaagent:jmx_prometheus_javaagent-0.12.0.jar=8080:pinot.yml -Xms4G -Xmx4G -XX:MaxDirectMemorySize=30g -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -Dplugins.dir=$BASEDIR/plugins"
bin/pinot-admin.sh ....

This will expose a port at 8080 to dump metrics as Prometheus format for Prometheus scrapper to fetch.