Plugins

Starting from the 0.3.X release, Pinot supports a plug-and-play architecture. This means that starting from version 0.3.0, Pinot can be easily extended to support new tools, like streaming services, storage systems, input formats, and metrics providers.

Plugins are collected in folders, based on their purpose. Pinot organizes its plugins into eleven plugin families, each targeting a specific extensibility need. The table below summarizes every family, its SPI module, and the implementations that ship with Pinot.

Plugin Families at a Glance

Plugin Family	SPI Interface / Module	Built-in Implementations
Input Format	RecordReader / StreamMessageDecoder	Avro, CSV, JSON, ORC, Parquet, Thrift, Protobuf, Arrow, CLP-Log, Confluent Avro, Confluent JSON, Confluent Protobuf
Filesystem	PinotFS	S3, GCS, HDFS, ADLS
Stream Ingestion	StreamConsumerFactory	Kafka 3.0, Kafka 4.0, Kinesis, Pulsar
Batch Ingestion	IngestionJobRunner	Standalone, Hadoop, Spark 3
Metrics	PinotMetricsFactory	Dropwizard, Yammer, Compound
Segment Writer	SegmentWriter	File-based
Segment Uploader	SegmentUploader	Default
Minion Tasks	PinotTaskGenerator / PinotTaskExecutor	MergeRollup, Purge, RealtimeToOfflineSegments, SegmentGenerationAndPush, UpsertCompaction, UpsertCompactMerge, RefreshSegment
Environment	PinotEnv	Azure
Time Series Language	TimeSeriesLogicalPlanner	M3QL
OpChain Converter	OpChainConverter	Default

---

Input Format

Input format plugins read data from files or streams during data ingestion. Batch ingestion uses RecordReader implementations, while real-time ingestion uses StreamMessageDecoder implementations.

Supported Data Formats

Filesystem

Filesystem plugins provide a storage abstraction layer so that Pinot segments can be stored on and fetched from different storage backends.

File Systems

Stream Ingestion

Stream ingestion plugins allow Pinot to consume data from real-time streaming platforms.

Stream Ingestion Guide

Batch Ingestion

Batch ingestion plugins run data ingestion jobs on different execution frameworks.

Batch Ingestion Guide

Metrics

Metrics plugins control which metrics library Pinot uses to collect and expose internal metrics via JMX. Pinot ships with Dropwizard (default), Yammer, and a Compound implementation that can fan out to multiple registries simultaneously.

Metrics Plugin

Segment Writer

The Segment Writer plugin provides an API for programmatically collecting GenericRow records and building Pinot segments without going through a full batch ingestion job. The built-in file-based implementation buffers rows as Avro records on local disk.

Segment Writer Plugin

Segment Uploader

The Segment Uploader plugin handles uploading completed segment tar files to the Pinot cluster. The default implementation supports all push modes configured via batchConfigMaps in the table config.

Segment Uploader Plugin

Minion Tasks

Minion task plugins define background processing tasks that run on Pinot Minion nodes. Built-in tasks include segment merge/rollup, purge, real-time to offline conversion, upsert compaction, and more.

Minion

Environment

Environment plugins allow Pinot to integrate with cloud-specific features and configurations. The Azure environment plugin provides Azure-specific functionality.

Time Series Language

Time series language plugins allow Pinot to support custom time series query languages like PromQL or M3QL.

Time Series Language Plugin

OpChain Converter

OpChain Converter plugins provide custom implementations for converting logical query plans into executable OpChain objects in the multi-stage query engine. This enables alternative execution backends and plan-to-execution strategies.

Opchain Converter Plugin

---

Developing Plugins

Plugins can be developed with no restriction. There are some standards that have to be followed, though. The plugin has to implement the interfaces from pinot-spi.

Planner rule customizers

The multi-stage query engine also exposes an advanced planner SPI for broker-side Calcite rule customization. Implement org.apache.pinot.query.planner.spi.RuleSetCustomizer from pinot-query-planner-spi when you need to append, remove, reorder, or replace rules in Pinot's per-phase logical planning pipeline.

Discovery uses Java ServiceLoader. Pinot first loads RuleSetCustomizer implementations on the broker application classpath, then scans each loaded plugin classloader. A plugin JAR should include the standard Pinot plugin packaging plus a META-INF/services/org.apache.pinot.query.planner.spi.RuleSetCustomizer file listing the implementation class.

At rule match time, planner rules can read per-query planner options through the Calcite planner context. Pinot exposes PlannerContext from both planner variants, so rule code can unwrap it directly and inspect getOptions():

PlannerContext plannerContext =
    call.getPlanner().getContext().unwrap(PlannerContext.class);
String workerRuntime = plannerContext.getOptions().get("workerRuntime");

Use this path for query-scoped planner behavior. Rules that need broker-wide planner defaults can still unwrap QueryEnvironment.Config from the same Calcite context.

Initialization is one-time. PinotRuleSet.defaultInstance() builds the broker process-wide rule set lazily, and each discovered customizer runs once for every planner Phase before Pinot freezes those rule lists for the rest of the process. Load the plugin before broker startup, and restart the broker after adding or changing a planner-rules plugin.

This SPI is more upgrade-sensitive than the stable ingestion, filesystem, and metrics plugin families. Pinot keeps Phase append-only for binary compatibility, but new phases can be added and the built-in rule ordering can change between releases. Revalidate customizers on every Pinot upgrade, especially if they depend on a specific built-in rule name or order.

Materialized view DDL handlers

Controller-managed CREATE MATERIALIZED VIEW ... AS <query> also has an advanced extension point. Implement org.apache.pinot.sql.ddl.compile.MaterializedViewDdlHandler from pinot-sql-ddl when a downstream distribution needs a different materialized-view engine contract than the built-in single-source MaterializedViewTask path.

The handler owns three decisions:

• validateDefinedQuery(...) decides whether the AS <query> shape is valid for the target engine.

• supportsSchemaInference(...) decides whether Pinot may infer MV columns from the SELECT projection when the DDL omits an explicit column list.

• applyTaskConfig(...) routes the MV properties onto the TableConfigBuilder and returns the task type stamped onto the table.

Register the handler once at controller startup through DdlCompiler.setMaterializedViewDdlHandler(...). If no handler is registered, Pinot keeps the default behavior: JOINs are rejected, schema inference is allowed for the single-source path, and the MV runs under MaterializedViewTask.

If a custom handler stamps a task type other than the built-in MaterializedViewTask, it also owns that task type's runtime contract. In practice that means the custom task generator/executor, its validation rules, and any definition-metadata persistence or consistency tracking required by that alternate MV implementation.

For the default OSS materialized-view surface, see Materialized Views.

WorkerManager leaf-stage segment hooks

Advanced multi-stage routing customizations can also subclass org.apache.pinot.query.routing.WorkerManager. Pinot already lets a subclass influence worker placement through getCandidateServers(...) and getCandidateServersForReplicatedLeaf(...). Pinot 1.6.0 adds two later hooks that run after the leaf-stage segment assignment is built:

• filterLeafStageSegments(...) for the normal leaf-stage assignment path

• filterReplicatedLeafStageSegments(...) for replicated or broadcast leaf-stage assignments

These hooks receive the DispatchablePlanContext and DispatchablePlanMetadata, so a subclass can inspect or rewrite getWorkerIdToSegmentsMap() or getReplicatedSegments() and then write the adjusted assignment back with the existing setters. Query-scoped routing choices can still read request options from DispatchablePlanContext.getPlannerContext().getOptions().

Treat this as an upgrade-sensitive code-level extension, not as a stable standalone plugin family. Revalidate custom WorkerManager subclasses on every Pinot upgrade, especially if they depend on a specific leaf-stage planning path or DispatchablePlanMetadata shape.

Custom segment or index extensions that depend on pinot-segment-spi are a separate, more upgrade-sensitive path than the stable plugin families listed above. Revalidate these extensions on every Pinot upgrade. For example, Pinot 1.6.0 adds two required IndexType methods for custom index implementations: requiresDictionary(FieldSpec, C) and shouldInvalidateOnDictionaryChange(FieldSpec, C). See the upgrade notes before upgrading custom segment/index extensions.

The current latest branch also tightens the JsonIndexReader contract for custom JSON index readers and any code that calls the SPI directly: getMatchingDocIds(...) now returns ImmutableRoaringBitmap, and Pinot treats that result as read-only. A custom reader may return a bitmap that is borrowed from the index's underlying storage, so callers must not mutate it and should copy it with toMutableRoaringBitmap() before making in-place changes. Existing readers can still return MutableRoaringBitmap through the covariant override, but custom JSON index extensions should still be revalidated on every upgrade.

Write Custom Plugins