Segment Writer Plugin
Overview
The Segment Writer plugin provides an API for programmatically collecting GenericRow records and building Pinot segments without running a full batch ingestion job. This is particularly useful when you need to generate segments from application code, such as in a Minion task or a custom ingestion pipeline.
The built-in file-based implementation (FileBasedSegmentWriter) buffers incoming rows as Avro records on local disk and creates a Pinot segment when flush() is called.
SPI Interface
To write a custom segment writer, implement the SegmentWriter interface:
public interface SegmentWriter extends Closeable {
void init(TableConfig tableConfig, Schema schema) throws Exception;
void init(TableConfig tableConfig, Schema schema, Map<String, String> batchConfigOverride)
throws Exception;
void collect(GenericRow row) throws Exception;
default void collect(GenericRow[] rowBatch) throws Exception;
URI flush() throws Exception;
}Key Methods
init(TableConfig, Schema)
Initializes the writer with table config and Pinot schema.
init(TableConfig, Schema, Map)
Initializes with additional batch config overrides.
collect(GenericRow)
Buffers a single row. The row is not written to a segment until flush() is called.
collect(GenericRow[])
Buffers a batch of rows.
flush()
Builds a Pinot segment from buffered rows and returns the URI of the generated segment tar file. Resets the buffer on success.
close()
Releases resources.
File-Based Implementation
The FileBasedSegmentWriter works as follows:
Initialization -- Reads
batchConfigMapsfrom the table config. Requires exactly oneBatchConfigentry with anoutputDirURI.Buffering -- Each
collect()call applies the table's transform pipeline and appends the result as an Avro record to a local buffer file.Flushing -- The
flush()method builds a Pinot segment from the buffer file, compresses it as a.tar.gz, writes it to the configuredoutputDirURI, and resets the buffer. If flush fails, the buffer is preserved soflush()can be retried.Closing -- The
close()method releases resources and cleans up staging directories.
Configuration
The segment writer is configured through batchConfigMaps in the table config:
outputDirURI
Yes
Directory where generated segment tar files are written.
overwrite
No
Whether to overwrite segments with duplicate names. Defaults to false.
Usage Example
Writing a Custom Segment Writer
To implement a custom segment writer:
Create a class that implements
SegmentWriter.Package it as a Pinot plugin (see Write Custom Plugins).
Place the plugin JAR in the Pinot
/pluginsdirectory.
Custom implementations could use different buffering strategies (for example, in-memory buffering for smaller datasets) or write to remote storage directly.
Last updated
Was this helpful?

