Batch Ingestion
Choose batch ingestion when Pinot should load prebuilt data from files, warehouses, or distributed processing jobs.
Last updated
Was this helpful?
⌘Ctrlk
Choose batch ingestion when Pinot should load prebuilt data from files, warehouses, or distributed processing jobs.
Batch ingestion builds Pinot segments outside the cluster and pushes them into Pinot after the data is already shaped. Use it when the data changes in larger chunks, when you need deterministic backfills, or when the pipeline already produces files or segment artifacts.
The most important design choice is not the framework, but the output contract: what the schema looks like, what the table expects, and where the segments land.
Spark-based ingestion.
Hadoop-style distributed ingestion.
Backfill jobs for historical ranges. Pinot ships LaunchBackfillIngestionJob for the case where the backfilled input may have fewer files than the original ingestion — see Backfill Data.
Dimension tables and other specialized offline loads.
Decide on the file format, the deep-storage target, and the segment push workflow before you optimize the job itself. Most batch ingestion problems come from mismatched assumptions at those boundaries.
The original step-by-step batch docs live in Import Data and Data Ingestion Overview.
This page covered when to choose batch ingestion and the main design decisions that shape it.
Read Stream Ingestion if the source system is a live event stream.
Last updated
Was this helpful?
Was this helpful?