# Plugin reference ## Plugin Families Reference This section is the reference map for Pinot's built-in plugin families. The detailed family pages live here so the configuration tree can stay dense and the authoring guides can stay elsewhere. ### Plugin Families | Family | Use it for | Page | | ------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | Stream ingestion connectors | Kafka, Kinesis, and Pulsar consumer factories | [Stream Ingestion Connectors](https://docs.pinot.apache.org/reference/plugin-reference/stream-ingestion-connectors) | | Stream connector version matrix | Compatibility between broker, connector, and Kafka major versions | [Stream Connector Version Matrix](https://docs.pinot.apache.org/reference/plugin-reference/stream-connector-matrix) | | Metrics plugins | JMX metric backends and registry fan-out | [Metrics Plugins](https://docs.pinot.apache.org/reference/plugin-reference/metrics-plugins) | | Environment provider | Cloud metadata discovery for instance placement | [Environment Provider](https://docs.pinot.apache.org/reference/plugin-reference/environment-provider) | ### What this page covered * The plugin families that belong in the configuration reference. * The detailed pages that live under this subtree. * The areas where version compatibility matters most. ### Next step Open the plugin-family page for the integration you are changing, then verify supported versions before changing deployment settings. ### Related pages * [Configuration Reference](https://docs.pinot.apache.org/reference/configuration-reference) * [Stream Ingestion Connectors](https://docs.pinot.apache.org/reference/plugin-reference/stream-ingestion-connectors) * [Metrics Plugins](https://docs.pinot.apache.org/reference/plugin-reference/metrics-plugins) *** ### description: >- Configuration and usage reference for every plugin family in Apache Pinot. ## Plugin Reference Apache Pinot has a plug-and-play architecture organized into **ten plugin families**. Each family targets a specific extensibility need — from reading data in different formats to exporting metrics to your monitoring stack. This section covers the **configuration** side of each plugin family: which implementations ship with Pinot, what config keys they accept, and how to enable them. If you want to **write your own plugin**, see the [Plugin Architecture](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture) section in the Developer Guide. ### Plugin Families at a Glance | Plugin Family | What It Does | Config Reference | Authoring Guide | | ------------------------ | -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Stream Ingestion** | Consume data from real-time streaming platforms (Kafka, Kinesis, Pulsar) | [Stream Ingestion Connectors](https://docs.pinot.apache.org/reference/plugin-reference/stream-ingestion-connectors) · [Version Matrix](https://docs.pinot.apache.org/reference/plugin-reference/stream-connector-matrix) | [Stream Ingestion Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/write-your-stream) | | **Input Format** | Read records from files or streams during ingestion (Avro, JSON, Parquet, ORC, CSV, …) | [Input Formats](https://docs.pinot.apache.org/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats) | [Input Format Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/record-reader) | | **Filesystem** | Store and fetch segments from pluggable storage backends (S3, GCS, HDFS, ADLS) | [Filesystem Plugins](https://docs.pinot.apache.org/build-with-pinot/ingestion/formats-filesystems/file-systems) | [Filesystem Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/pluggable-storage) | | **Batch Ingestion** | Run data ingestion jobs on different execution frameworks (Standalone, Hadoop, Spark) | [Batch Ingestion](https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) | — | | **Metrics** | Collect and expose internal JMX metrics via Dropwizard, Yammer, or a compound backend | [Metrics Plugins](https://docs.pinot.apache.org/reference/plugin-reference/metrics-plugins) | [Metrics Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/metrics-plugin) | | **Segment Writer** | Programmatically build Pinot segments without a full batch ingestion job | — | [Segment Writer Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/segment-writer-plugin) | | **Segment Uploader** | Upload completed segment tar files to the Pinot cluster | — | [Segment Uploader Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/segment-uploader-plugin) | | **Minion Tasks** | Run background processing tasks on Pinot Minion nodes (merge, purge, compaction, …) | [Minion](https://docs.pinot.apache.org/architecture-and-concepts/components/cluster/minion) · [Merge/Rollup Task](https://docs.pinot.apache.org/operate-pinot/segment-management/minion-merge-rollup-task) | [Minion Task Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/minion-task-plugin) | | **Environment** | Discover cloud-specific instance metadata for failure-domain–aware placement | [Environment Provider](https://docs.pinot.apache.org/reference/plugin-reference/environment-provider) | — | | **Time Series Language** | Support custom time series query languages (M3QL, PromQL) | — | [Time Series Language Plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/time-series-language-plugin) | *** ### Stream Ingestion Connectors Pinot ships connectors for Apache Kafka (3.x and 4.x), Amazon Kinesis, and Apache Pulsar. Each connector supplies a `StreamConsumerFactory` implementation. {% content-ref url="plugin-reference/stream-ingestion-connectors" %} [stream-ingestion-connectors](https://docs.pinot.apache.org/reference/plugin-reference/stream-ingestion-connectors) {% endcontent-ref %} {% content-ref url="plugin-reference/stream-connector-matrix" %} [stream-connector-matrix](https://docs.pinot.apache.org/reference/plugin-reference/stream-connector-matrix) {% endcontent-ref %} ### Input Format Input format plugins read data from files or streams during ingestion. Batch ingestion uses `RecordReader` implementations; real-time ingestion uses `StreamMessageDecoder` implementations. Pinot ships with readers for Avro, CSV, JSON, ORC, Parquet, Thrift, Protobuf, Arrow, CLP-Log, and Confluent Schema Registry variants. {% content-ref url="../build-with-pinot/ingestion/formats-filesystems/pinot-input-formats" %} [pinot-input-formats](https://docs.pinot.apache.org/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats) {% endcontent-ref %} ### Filesystem Filesystem plugins provide a `PinotFS` storage abstraction so that segments can live on different backends — S3, GCS, HDFS, or ADLS. {% content-ref url="../build-with-pinot/ingestion/formats-filesystems/file-systems" %} [file-systems](https://docs.pinot.apache.org/build-with-pinot/ingestion/formats-filesystems/file-systems) {% endcontent-ref %} ### Batch Ingestion Batch ingestion plugins run ingestion jobs on different execution frameworks: Standalone, Hadoop, and Spark 3. {% content-ref url="../build-with-pinot/ingestion/batch-ingestion/batch-ingestion" %} [batch-ingestion](https://docs.pinot.apache.org/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) {% endcontent-ref %} ### Metrics Metrics plugins control which metrics library Pinot uses for internal JMX metrics. Pinot ships with Yammer (default), Dropwizard, and a Compound implementation that fans out to multiple registries. {% content-ref url="plugin-reference/metrics-plugins" %} [metrics-plugins](https://docs.pinot.apache.org/reference/plugin-reference/metrics-plugins) {% endcontent-ref %} ### Segment Writer The Segment Writer plugin provides an API for programmatically collecting `GenericRow` records and building Pinot segments without going through a full batch ingestion job. The built-in file-based implementation buffers rows as Avro records on local disk. {% content-ref url="../develop-and-contribute/plugin-architecture/write-custom-plugins/segment-writer-plugin" %} [segment-writer-plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/segment-writer-plugin) {% endcontent-ref %} ### Segment Uploader The Segment Uploader plugin handles uploading completed segment tar files to the Pinot cluster. The default implementation supports all push modes configured via `batchConfigMaps` in the table config. {% content-ref url="../develop-and-contribute/plugin-architecture/write-custom-plugins/segment-uploader-plugin" %} [segment-uploader-plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/segment-uploader-plugin) {% endcontent-ref %} ### Minion Tasks Minion task plugins define background processing tasks that run on Pinot Minion nodes. Built-in tasks include MergeRollup, Purge, RealtimeToOfflineSegments, SegmentGenerationAndPush, UpsertCompaction, UpsertCompactMerge, and RefreshSegment. {% content-ref url="../architecture-and-concepts/components/cluster/minion" %} [minion](https://docs.pinot.apache.org/architecture-and-concepts/components/cluster/minion) {% endcontent-ref %} {% content-ref url="../operate-pinot/segment-management/minion-merge-rollup-task" %} [minion-merge-rollup-task](https://docs.pinot.apache.org/operate-pinot/segment-management/minion-merge-rollup-task) {% endcontent-ref %} ### Environment Provider Environment plugins allow Pinot to discover cloud-specific instance metadata at startup for failure-domain–aware data placement. The Azure provider is the only built-in implementation. {% content-ref url="plugin-reference/environment-provider" %} [environment-provider](https://docs.pinot.apache.org/reference/plugin-reference/environment-provider) {% endcontent-ref %} ### Time Series Language Time series language plugins let Pinot support custom time series query languages like M3QL and PromQL. {% content-ref url="../develop-and-contribute/plugin-architecture/write-custom-plugins/time-series-language-plugin" %} [time-series-language-plugin](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins/time-series-language-plugin) {% endcontent-ref %} *** ### Developing Custom Plugins Plugins implement interfaces from [pinot-spi](https://github.com/apache/pinot/tree/master/pinot-spi/src/main/java/org/apache/pinot/spi). See the developer guide for the full plugin authoring workflow: {% content-ref url="../develop-and-contribute/plugin-architecture/write-custom-plugins" %} [write-custom-plugins](https://docs.pinot.apache.org/develop-and-contribute/plugin-architecture/write-custom-plugins) {% endcontent-ref %} ### Legacy compatibility pages * [STDDEV\_POP](https://docs.pinot.apache.org/functions/statistical/stddevpop) * [STDDEV\_SAMP](https://docs.pinot.apache.org/functions/statistical/stddevsamp) * [VAR\_POP](https://docs.pinot.apache.org/functions/statistical/varpop) * [VAR\_SAMP](https://docs.pinot.apache.org/functions/statistical/varsamp)