| Metric Name | Description | Metric type |
|---|---|---|
| LLC-PARTITION-CONSUMING | This gives a binary value based on whether low-level consumption is healthy (1) or unhealthy (0). It’s important to ensure at least a single replica of each partition is consuming. | Gauge |
| HIGHEST-STREAM-OFFSET-CONSUMED | The highest offset which has been consumed so far | Gauge |
| DOCUMENT_COUNT | total number of records in table | Gauge |
| SEGMENT_COUNT | total number of segments in table | Gauge |
| UPSERT_PRIMARY_KEYS_COUNT | total unique primary keys in table | Gauge |
| LAST_REALTIME_SEGMENT_CREATION_DURATION_SECONDS | time in seconds it took for latest realtime segment to get created | Gauge |
| LAST_REALTIME_SEGMENT_CREATION_WAIT_TIME_SECONDS | time in seconds it took for segment creation to start (generally due to waiting for a lock to get acquired) | Gauge |
| LAST_REALTIME_SEGMENT_INITIAL_CONSUMPTION_DURATION_SECONDS | time in seconds spent consuming records for latest segment | Gauge |
| LAST_REALTIME_SEGMENT_CATCHUP_DURATION_SECONDS | time in seconds spent on catching up to the latest offset in metadata. This can happen when multiple servers are consuming from same partition. | Gauge |
| LAST_REALTIME_SEGMENT_COMPLETION_DURATION_SECONDS | time in seconds between when we stopped consuming records and when the segment gets committed | Gauge |
| REALTIME_OFFHEAP_MEMORY_USED | off heap memory in bytes current used by realtime segments | Gauge |
| REALTIME_SEGMENT_NUM_PARTITIONS | Number of partitions for a table | Gauge |
| LLC_SIMULTANEOUS_SEGMENT_BUILDS | Number of segments being built currently | Gauge |
| ROWS_WITH_ERRORS | number of rows that either didn't get transformed or didn't get indexed. | Meter |
| REALTIME_ROWS_CONSUMED | total number of records consumed from input | Meter |
| INVALID_REALTIME_ROWS_DROPPED | number of records that were filtered based on FilterConfig specified in table config | Meter |
| REALTIME_CONSUMPTION_EXCEPTIONS | number of rows that were not consumed because of some exception. It doesn't track exceptions during transformation and indexing. | Meter |
| RELOAD_FAILURES | Number of failures occurred while reloading segments | Meter |
| REFRESH_FAILURES | Number of failures occurred while refreshing segments | Meter |
| UNTAR_FAILURES | Number of failures occurred while uncompressing segments | Meter |
| SEGMENT_DOWNLOAD_FAILURES | Number of failures occurred while downloading segments from deep store to local | Meter |
| DELETED_SEGMENT_COUNT | Number of segments deleted either because of retention policies, explicit delete request etc. | Meter |
| QUERIES | Number of queries executed | Meter |
| QUERY_EXECUTION_EXCEPTIONS | Number of exceptions encountered during query execution | Meter |
| NUM-MISSING-SEGMENTS | Number of missing segments that the broker queried for (expected to be on the server) but the server didn’t have. This can be due to retention or stale routing table | Meter |
| NO_TABLE_ACCESS | number of query requests for which table access was denied either due to table not being present or access control restrictions. | Meter |
| HELIX_ZOOKEEPER_RECONNECTS | Number of times Server instance re-connected to zookeeper. | Meter |
| NETTY_CONNECTION_BYTES_RECEIVED | total bytes received by the server | Meter |
| NETTY_CONNECTION_BYTES_SENT | total bytes sent by the server | Meter |
| NETTY_CONNECTION_RESPONSES_SENT | total responses sent by the server | Meter |
| FRESHNESS_LAG_MS | time period between when the data was last updated in the table and the current time | Timer |
| NETTY_CONNECTION_SEND_RESPONSE_LATENCY | time spent in sending response to brokers after the results are available | Timer |
| EXECUTION_THREAD_CPU_TIME_NS | time spent by all threads processing query and results (doesn't includes time spent in system activities) | Timer |
| SYSTEM_ACTIVITIES_CPU_TIME_NS | time spent in nanoseconds processing query on the servers (only counts system acitivities such as GC, OS paging etc.) | Timer |
| RESPONSE_SER_CPU_TIME_NS | time spent in nanoseconds serializing query response on servers | Timer |
| TOTAL_CPU_TIME_NS | total time spent in nanoseconds processing query on the servers | Timer |
| Metric Name | Description | |
|---|---|---|
| REQUEST_DESERIALIZATION | Time spent in deserializing query request | Timer |
| SEGMENT_PRUNING | Time spent in Segment Pruning | Timer |
| BUILD_QUERY_PLAN | Time spent in building query plan | Timer |
| QUERY_PLAN_EXECUTION | Time spent in executing query plan | Timer |
| QUERY_PROCESSING | Total Time spent in processing the query request from receiving the parsed query to getting data. Doesn't include ser-de time. | Timer |
| SCHEDULER_WAIT | Time spent in the scheduler queue waiting for the query to be executed | Timer |
| RESPONSE_SERIALIZATION | Time spent in serializing query response | Timer |
| TOTAL_QUERY_TIME | Total time to take from receiving the query to returning the responde. | Timer |
| Metric Name | Description | Metric Type |
|---|---|---|
| UNHEALTHY_SERVERS | Number of unhealthy servers detected | Gauge |
| QUERY_QUOTA_CAPACITY_UTILIZATION_RATE | percentage of configured rate limit being used on each broker | Gauge |
| MAX_BURST_QPS | Gauge | |
| QUERY_RATE_LIMIT_DISABLED | 1 if rate limit is enabled on broker, 0 otherwise | Gauge |
| REQUEST_SIZE | Query String length on each broker | Gauge |
| RESIZE_TIME_MS | time spent in resizing results for the output. either because of LIMIT or maximum allowed group by keys or any other criteria | Gauge |
| QUERIES | The rate which an individual broker is receiving queries. Units are in QPS | Meter |
| REQUEST_COMPILATION_EXCEPTIONS | Number of queries which failed during compilation | Meter |
| RESOURCE_MISSING_EXCEPTIONS | Number of queries for which table doesn't exists | Meter |
| QUERY_VALIDATION_EXCEPTIONS | Number of invalid queries | Meter |
| UNKNOWN_COLUMN_EXCEPTIONS | Number of queries with unknown columns | Meter |
| NO_SERVER_FOUND_EXCEPTIONS | Number of queries for which no server was found to contain its data | Meter |
| REQUEST_TIMEOUT_BEFORE_SCATTERED_EXCEPTIONS | Number of times query timed out before even being sent to the servers | Meter |
| REQUEST_CHANNEL_LOCK_TIMEOUT_EXCEPTIONS | number of times query failes while trying to acquire lock to server connections | Meter |
| REQUEST_SEND_EXCEPTIONS | Number of queries failed while sending to server | Meter |
| RESPONSE_FETCH_EXCEPTIONS | Number of queries failed while handling response from servers | Meter |
| DATA_TABLE_DESERIALIZATION_EXCEPTIONS | Number of queries failed while deserializing response data from servers | Meter |
| RESPONSE_MERGE_EXCEPTIONS | Number of queries that failed while merging responses from multiple servers. This can be due to schema inconsitency or any other issues | Meter |
| BROKER_RESPONSES_WITH_PROCESSING_EXCEPTIONS | Number of queries where atleast one exception occured | Meter |
| BROKER_RESPONSES_WITH_PARTIAL_SERVERS_RESPONDED | Number of queries with incomplete results due to missing responses from servers | Meter |
| BROKER_RESPONSES_WITH_NUM_GROUPS_LIMIT_REACHED | Number of queries where total number of groups exceeded configured limit (default limit - 100K) | Meter |
| DOCUMENTS_SCANNED | Total number of documents read from segments in each query | Meter |
| ENTRIES_SCANNED_IN_FILTER | Meter | |
| ENTRIES_SCANNED_POST_FILTER | Meter | |
| NUM_RESIZES | Number of result resizes for queries | Meter |
| REQUEST_DROPPED_DUE_TO_ACCESS_ERROR | Number of queries dropped due to invalid access permissions on table | Meter |
| GROUP_BY_SIZE | Number of rows in group by queries | Meter |
| TOTAL_SERVER_RESPONSE_SIZE | Total number of bytes received from servers for queries | Meter |
| QUERY_QUOTA_EXCEEDED | Number of queries failed due to query rate limit being breached | Meter |
| NO_SERVING_HOST_FOR_SEGMENT | Number of segments per query for which no servers are available | Meter |
| SERVER_MISSING_FOR_ROUTING | Number of servers that could not be added to routing table for query | Meter |
| NETTY_CONNECTION_REQUESTS_SENT | total number of requests sent to servers | Meter |
| NETTY_CONNECTION_BYTES_SENT | total bytes sent to servers | Meter |
| NETTY_CONNECTION_BYTES_RECEIVED | total bytes received from servers | Meter |
| PROACTIVE_CLUSTER_CHANGE_CHECK | Number of requests raised to zookeeper to check the cluster state such as IDEAL STATES, EXTERNAL VIEW etc. | Meter |
| HELIX_ZOOKEEPER_RECONNECTS | Number of times broker instance re-connected to zookeeper. | Meter |
| CLUSTER_CHANGE_QUEUE_TIME | Time spent in milliseconds in queue for cluster change requests | Timer |
| FRESHNESS_LAG_MS | time period between when the data was last updated in the table and the current time | Timer |
| NETTY_CONNECTION_SEND_REQUEST_LATENCY | latency of sending the request from broker to server | Timer |
| OFFLINE_THREAD_CPU_TIME_NS | aggregated thread cpu time in nanoseconds for query processing from offline servers | Timer |
| REALTIME_THREAD_CPU_TIME_NS | aggregated thread cpu time in nanoseconds for query processing from realtime servers | Timer |
| OFFLINE_SYSTEM_ACTIVITIES_CPU_TIME_NS | aggregated system activities cpu time in nanoseconds for query processing from offline servers (e.g. GC, OS paging etc.) | Timer |
| REALTIME_SYSTEM_ACTIVITIES_CPU_TIME_NS | aggregated system activities cpu time in nanoseconds for query processing from realtime servers (e.g. GC, OS paging etc.) | Timer |
| OFFLINE_RESPONSE_SER_CPU_TIME_NS | aggregated response serialization cpu time in nanoseconds for query processing from offline servers | Timer |
| REALTIME_RESPONSE_SER_CPU_TIME_NS | aggregated response serialization cpu time in nanoseconds for query processing from realtime servers | Timer |
| OFFLINE_TOTAL_CPU_TIME_NS | aggregated total cpu time(thread + system activities + response serialization) in nanoseconds for query processing from offline servers | Timer |
| REALTIME_TOTAL_CPU_TIME_NS | time(thread + system activities + response serialization) in nanoseconds for query processing from realtime servers | Timer |
| Metric Name | Description | Metric Type |
|---|---|---|
| REQUEST_COMPILATION | Time spent in compiling SQL query | Timer |
| QUERY_EXECUTION | Total Time spent in query executiong | Timer |
| QUERY_ROUTING | Time spent in creating a routing table for segments | Timer |
| SCATTER_GATHER | Time spent in sending and collecting responses from servers. | Timer |
| REDUCE | Time spent in combining query results from multiple servers | Timer |
| AUTHORIZATION | Time spent checking table access after query compilation | Timer |
| Metric Name | Description | Metric Type |
|---|---|---|
| PERCENT_SEGMENTS_AVAILABLE | Percentage of complete online replicas in external view as compared to replicas in ideal state | Gauge |
| NUMBER_OF_REPLICAS | Total number of replicas available for table | Gauge |
| PERCENT_SEGMENTS_AVAILABLE | Percentage of complete online replicas in external view as compared to replicas in ideal state. | Gauge |
| SEGMENTS_IN_ERROR_STATE | Number of segments in an ERROR state for a given table. | Gauge |
| TABLE_STORAGE_QUOTA_UTILIZATION | Shows how much of the table’s storage quota is currently being used, metric will a percentage of a the entire quota. | Gauge |
| LAST_PUSH_TIME_DELAY_HOURS | The time in hours since the last time an offline segment has been pushed to the controller. | Gauge |
| HEALTHCHECK_OK_CALLS | Number of health check requests for which controller was healthy | Meter |
| HEALTHCHECK_BAD_CALLS | Number of health check requests for which controller was unhealthy | Meter |
| CONTROLLER_INSTANCE_POST_ERROR | Errors occurred while updating state for an instance (server and broker) | Meter |
| CONTROLLER_SEGMENT_UPLOAD_ERROR | Errors occurred while sending uploading segment request | Meter |
| CONTROLLER_SCHEMA_UPLOAD_ERROR | Errors occurred while uploading schema | Meter |
| CONTROLLER_TABLE_SCHEMA_UPDATE_ERROR | Errors occurred while updating schema | Meter |
| CONTROLLER_TABLE_ADD_ERROR | Errors occurred while adding table config | Meter |
| CONTROLLER_TABLE_UPDATE_ERROR | Errors occurred while updating table config | Meter |
| CONTROLLER_TABLE_TENANT_UPDATE_ERROR | Errors occurred while updating a Tenant | Meter |
| CONTROLLER_TABLE_TENANT_CREATE_ERROR | Errors occurred while creating a Tenant | Meter |
| CONTROLLER_TABLE_TENANT_DELETE_ERROR | Errors while deleting a Tenant | Meter |
| CONTROLLER_REALTIME_TABLE_SEGMENT_ASSIGNMENT_ERROR | Errors occurred while assigning a realtime segment to a server instance | Meter |
| CONTROLLER_LEADERSHIP_CHANGE_WITHOUT_CALLBACK | Number of times a controller loses/gains leadership without a callback from Helix | Meter |
| CONTROLLER_PERIODIC_TASK_RUN | Number of Periodic tasks running currently | Meter |
| CONTROLLER_PERIODIC_TASK_ERROR | Number of Periodic tasks that failed due to error | Meter |
| NUMBER_TIMES_SCHEDULE_TASKS_CALLED | Minion tasks schedule request sent to controller | Meter |
| NUMBER_TASKS_SUBMITTED | Number of minion tasks submitted to the controller. | Meter |
| NUMBER_SEGMENT_UPLOAD_TIMEOUT_EXCEEDED | Number of segments uploads failed due to timeout. Segments are re-uploaded in this case by controller itself. | Meter |
| CRON_SCHEDULER_JOB_TRIGGERED | Number of minion tasks triggered that use cron | Meter |
| NUMBER_ADHOC_TASKS_SUBMITTED | Number of minion ad-hoc tasks submitted | Meter |
| LLC_STATE_MACHINE_ABORTS | Number of times a realtime segment commit operation was aborted | Meter |
| LLC_ZOOKEEPER_FETCH_FAILURES | Number of Zookeeper metadata fetch requests failed | Meter |
| LLC_ZOOKEEPER_UPDATE_FAILURES | Number of Zookeeper metadata update requests failed | Meter |
| LLC_STREAM_DATA_LOSS | Indicates data loss for table either due to offsets available to consume from topic larger than the last stored offset in pinot or segment lost in CONSUMING state | Meter |
| HELIX_ZOOKEEPER_RECONNECTS | Number of times broker instance re-connected to zookeeper. | Meter |
| CRON_SCHEDULER_JOB_EXECUTION_TIME_MS | Time spent in scheduling cron jobs | Timer |
| Metric Name | Description | Metric Type |
|---|---|---|
| NUMBER_OF_TASKS | Number of tasks currently running | Gauge |
| NUMBER_TASKS_EXECUTED | Number of tasks triggered in Minion | Meter |
| NUMBER_TASKS_COMPLETED | Number of tasks completed successfully | Meter |
| NUMBER_TASKS_CANCELLED | Number of tasks that were cancelled | Meter |
| NUMBER_TASKS_FAILED | Number of tasks that failed | Meter |
| NUMBER_TASKS_FATAL_FAILED | Number of tasks that failed with unretryable exceptions | Meter |
| TASK_QUEUEING | Time spent by tasks in queue | Timer |
| TASK_EXECUTION | Time spent by tasks in execution | Timer |