Plugin reference

Dense reference for Pinot plugin families.

Plugin Families Reference

This section is the reference map for Pinot's built-in plugin families. The detailed family pages live here so the configuration tree can stay dense and the authoring guides can stay elsewhere.

Plugin Families

Family	Use it for	Page
Stream ingestion connectors	Kafka, Kinesis, and Pulsar consumer factories	Stream Ingestion Connectors
Stream connector version matrix	Compatibility between broker, connector, and Kafka major versions	Stream Connector Version Matrix
Metrics plugins	JMX metric backends and registry fan-out	Metrics Plugins
Environment provider	Cloud metadata discovery for instance placement	Environment Provider

What this page covered

• The plugin families that belong in the configuration reference.

• The detailed pages that live under this subtree.

• The areas where version compatibility matters most.

Next step

Open the plugin-family page for the integration you are changing, then verify supported versions before changing deployment settings.

Related pages

• Configuration Reference

• Stream Ingestion Connectors

• Metrics Plugins

---

description: >- Configuration and usage reference for every plugin family in Apache Pinot.

Plugin Reference

Apache Pinot has a plug-and-play architecture organized into ten plugin families. Each family targets a specific extensibility need — from reading data in different formats to exporting metrics to your monitoring stack.

This section covers the configuration side of each plugin family: which implementations ship with Pinot, what config keys they accept, and how to enable them. If you want to write your own plugin, see the Plugin Architecture section in the Developer Guide.

Plugin Families at a Glance

Plugin Family	What It Does	Config Reference	Authoring Guide
Stream Ingestion	Consume data from real-time streaming platforms (Kafka, Kinesis, Pulsar)	Stream Ingestion Connectors · Version Matrix	Stream Ingestion Plugin
Input Format	Read records from files or streams during ingestion (Avro, JSON, Parquet, ORC, CSV, …)	Input Formats	Input Format Plugin
Filesystem	Store and fetch segments from pluggable storage backends (S3, GCS, HDFS, ADLS)	Filesystem Plugins	Filesystem Plugin
Batch Ingestion	Run data ingestion jobs on different execution frameworks (Standalone, Hadoop, Spark)	Batch Ingestion	—
Metrics	Collect and expose internal JMX metrics via Dropwizard, Yammer, or a compound backend	Metrics Plugins	Metrics Plugin
Segment Writer	Programmatically build Pinot segments without a full batch ingestion job	—	Segment Writer Plugin
Segment Uploader	Upload completed segment tar files to the Pinot cluster	—	Segment Uploader Plugin
Minion Tasks	Run background processing tasks on Pinot Minion nodes (merge, purge, compaction, …)	Minion · Merge/Rollup Task	Minion Task Plugin
Environment	Discover cloud-specific instance metadata for failure-domain–aware placement	Environment Provider	—
Time Series Language	Support custom time series query languages (M3QL, PromQL)	—	Time Series Language Plugin

---

Stream Ingestion Connectors

Pinot ships connectors for Apache Kafka (3.x and 4.x), Amazon Kinesis, and Apache Pulsar. Each connector supplies a StreamConsumerFactory implementation.

Stream Ingestion ConnectorsStream Connector Version Matrix

Input Format

Input format plugins read data from files or streams during ingestion. Batch ingestion uses RecordReader implementations; real-time ingestion uses StreamMessageDecoder implementations. Pinot ships with readers for Avro, CSV, JSON, ORC, Parquet, Thrift, Protobuf, Arrow, CLP-Log, and Confluent Schema Registry variants.

Supported Data Formats

Filesystem

Filesystem plugins provide a PinotFS storage abstraction so that segments can live on different backends — S3, GCS, HDFS, or ADLS.

File Systems

Batch Ingestion

Batch ingestion plugins run ingestion jobs on different execution frameworks: Standalone, Hadoop, and Spark 3.

Batch Ingestion Guide

Metrics

Metrics plugins control which metrics library Pinot uses for internal JMX metrics. Pinot ships with Yammer (default), Dropwizard, and a Compound implementation that fans out to multiple registries.

Metrics Plugins

Segment Writer

The Segment Writer plugin provides an API for programmatically collecting GenericRow records and building Pinot segments without going through a full batch ingestion job. The built-in file-based implementation buffers rows as Avro records on local disk.

Segment Writer Plugin

Segment Uploader

The Segment Uploader plugin handles uploading completed segment tar files to the Pinot cluster. The default implementation supports all push modes configured via batchConfigMaps in the table config.

Segment Uploader Plugin

Minion Tasks

Minion task plugins define background processing tasks that run on Pinot Minion nodes. Built-in tasks include MergeRollup, Purge, RealtimeToOfflineSegments, SegmentGenerationAndPush, UpsertCompaction, UpsertCompactMerge, and RefreshSegment.

MinionMinion Merge Rollup Task

Environment Provider

Environment plugins allow Pinot to discover cloud-specific instance metadata at startup for failure-domain–aware data placement. The Azure provider is the only built-in implementation.

Environment Provider

Time Series Language

Time series language plugins let Pinot support custom time series query languages like M3QL and PromQL.

Time Series Language Plugin

---

Developing Custom Plugins

Plugins implement interfaces from pinot-spi. See the developer guide for the full plugin authoring workflow:

Write Custom Plugins

Legacy compatibility pages

• STDDEV_POP

• STDDEV_SAMP

• VAR_POP

• VAR_SAMP