Input Format Plugin
Pinot supports multiple input formats out of the box for batch ingestion. For realtime ingestion, currently only JSON is supported. However, due to pluggable architecture of pinot you can easily use any format by implementing standard interfaces.
To index the file into Pinot segment, simply implement the interface and plug it into the index engine - SegmentCreationDriverImpl. We use a 2-passes algorithm to index the file into Pinot segment, hence the rewind() method is required for the record reader.
GenericRow is the record abstraction which the index engine can read and index with. It is a map from column name (String) to column value (Object). For multi-valued column, the value should be an object array (Object).
There are several contracts for record readers that developers should follow when implementing their own record readers:
- The output GenericRow should follow the table schema provided, in the sense that:
- All the columns in the schema should be preserved (if column does not exist in the original record, put default value instead)
- Columns not in the schema should not be included
- Values for the column should follow the field spec from the schema (data type, single-valued/multi-valued)
- For the time column (refer to TimeFieldSpec), record reader should be able to read both incoming and outgoing time (we allow incoming time - time value from the original data to outgoing time - time value stored in Pinot conversion during index creation).
- If incoming and outgoing time column name are the same, use incoming time field spec
- If incoming and outgoing time column name are different, put both of them as time field spec
- We keep both incoming and outgoing time column to handle cases where the input file contains time values that are already converted
Pinot uses decoders to parse data available in realtime streams. Decoders are responsible for converting binary data in the streams to a GenericRow object.