# Introduction Apache Pinot is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics, and perfect for user-facing analytical workloads. Apache Pinot™ is a real-time distributed online analytical processing (OLAP) datastore. Use Pinot to ingest and immediately query data from streaming or batch data sources (including, Apache Kafka, Amazon Kinesis, Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage). {% hint style="info" %} We'd love to hear from you! [Join us in our Slack channel](https://inviter.co/apache-pinot) to ask questions, troubleshoot, and share feedback. {% endhint %} Apache Pinot includes the following: * **Ultra low-latency analytics** even at extremely high throughput. * **Columnar data store** with several smart indexing and pre-aggregation techniques. * **Scaling up and out** with no upper bound. * **Consistent performance** based on the size of your cluster and an expected query per second (QPS) threshold. It's perfect for user-facing real-time analytics and other analytical use cases, including internal dashboards, anomaly detection, and ad hoc data exploration. {% embed url="" %} What is Apache Pinot? (and User-Facing Analytics) by Tim Berglund {% endembed %} ### User-facing real-time analytics User-facing analytics refers to the analytical tools exposed to the end users of your product. In a user-facing analytics application, all users receive personalized analytics on their devices, resulting in hundreds of thousands of queries per second. Queries triggered by apps may grow quickly in proportion to the number of active users on the app, as many as millions of events per second. Data generated in Pinot is immediately available for analytics in latencies under one second. User-facing real-time analytics requires the following: * **Fresh data.** The system needs to be able to ingest data in real time and make it available for querying, also in real time. * **Support for high-velocity, highly dimensional event data** from a wide range of actions and from multiple sources. * **Low latency.** Queries are triggered by end users interacting with apps, resulting in hundreds of thousands of queries per second with arbitrary patterns. * **Reliability and high availability.** * **Scalability.** * **Low cost to serve.** ## Why Pinot? Pinot is designed to execute OLAP queries with low latency. It works well where you need fast analytics, such as aggregations, on both mutable and immutable data. **User-facing, real-time analytics** Pinot was originally built at LinkedIn to power rich interactive real-time analytics applications, such as [Who Viewed Profile](https://www.linkedin.com/me/profile-views/urn:li:wvmp:summary/), [Company Analytics](https://www.linkedin.com/company/linkedin/insights/), [Talent Insights](https://business.linkedin.com/talent-solutions/talent-insights), and many more. [UberEats Restaurant Manager](https://eng.uber.com/restaurant-manager/) is another example of a user-facing analytics app built with Pinot. **Real-time dashboards for business metrics** Pinot can perform typical analytical operations such as slice and dice, drill down, roll up, and pivot on large scale multi-dimensional data. For instance, at LinkedIn, Pinot powers dashboards for thousands of business metrics. Connect various business intelligence (BI) tools such as [Superset](https://superset.apache.org/docs/intro/), [Tableau](https://www.tableau.com/resource/business-intelligence), or [PowerBI](https://powerbi.microsoft.com/en-us/) to visualize data in Pinot. **Enterprise business intelligence** For analysts and data scientists, Pinot works well as a highly-scalable data platform for business intelligence. Pinot converges big data platforms with the traditional role of a data warehouse, making it a suitable replacement for analysis and reporting. **Enterprise application development** For application developers, Pinot works well as an aggregate store that sources events from streaming data sources, such as Kafka, and makes it available for a query using SQL. You can also use Pinot to aggregate data across a microservice architecture into one easily queryable view of the domain. Pinot [tenants](/architecture-and-concepts/components/cluster/tenant) prevent any possibility of sharing ownership of database tables across microservice teams. Developers can create their own query models of data from multiple systems of record depending on their use case and needs. As with all aggregate stores, query models are eventually consistent. ## Get started If you're new to Pinot, take a look at our Getting Started guide: {% content-ref url="/pages/-LtH74Ib7K9iUoCsgDH4" %} [Start Here](/start-here/getting-started) {% endcontent-ref %} To start importing data into Pinot, see how to import batch and stream data: {% content-ref url="/pages/YqYKvoLQRBDeRhikxqKe" %} [Ingestion](/build-with-pinot/ingestion) {% endcontent-ref %} To start querying data in Pinot, check out our Query guide: {% content-ref url="/pages/ylJExJhigHcd3NiRoRFT" %} [Querying & SQL](/build-with-pinot/querying-and-sql) {% endcontent-ref %} ## Learn For a conceptual overview that explains how Pinot works, check out the Concepts guide: {% content-ref url="/pages/-M1SmDQWPB80r\_uM6nmZ" %} [Concepts](/architecture-and-concepts/concepts) {% endcontent-ref %} To understand the distributed systems architecture that explains Pinot's operating model, take a look at our basic architecture section: {% content-ref url="/pages/-M1SmG1cDvUFjSjD1tD8" %} [Architecture](/architecture-and-concepts/concepts/architecture) {% endcontent-ref %} # Start Here Start here to learn Apache Pinot and go from zero to running your first query. Follow the guided onboarding path or jump to the section that fits your experience level. Apache Pinot is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics. It ingests data from streaming and batch sources and makes it queryable in under a second. This guide walks you through everything you need to go from first contact to a working Pinot deployment. ## Onboarding path Follow these pages in order for a complete introduction: 1. [What is Pinot?](/start-here/what-is-pinot) -- Understand what Pinot does and whether it fits your use case. 2. [10-minute quickstart](/start-here/ten-minute-quickstart) -- Launch a local cluster and run your first query in minutes. 3. [Install / deploy](/start-here/install) -- Set up Pinot for local development, Docker, or Kubernetes. 4. [First table + schema](/start-here/first-table-and-schema) -- Define a schema and create your first table. 5. [First batch ingest](/start-here/first-batch-ingest) -- Load data from a file into Pinot. 6. [First stream ingest](/start-here/first-stream-ingest) -- Connect Pinot to a streaming source for real-time data. 7. [First query](/start-here/first-query) -- Write SQL queries against your Pinot tables. ## Choose your path ### Just exploring? Start with the conceptual overview, then try the quickstart to see Pinot in action with zero setup: {% content-ref url="/pages/l502VePnQCx6QraC3Qkh" %} [What is Pinot?](/start-here/what-is-pinot) {% endcontent-ref %} {% content-ref url="/pages/xORpTeBUSuN79G8JifJ7" %} [10-Minute Quickstart](/start-here/ten-minute-quickstart) {% endcontent-ref %} ### Ready to build? Jump straight to installation and follow the linear onboarding path from step 3 onward: {% content-ref url="/pages/JkPt4QP6Ryd3vkIhUNhA" %} [Install / Deploy](/start-here/install) {% endcontent-ref %} ## Next step {% content-ref url="/pages/l502VePnQCx6QraC3Qkh" %} [What is Pinot?](/start-here/what-is-pinot) {% endcontent-ref %} # What is Pinot? Learn what Apache Pinot is, what problems it solves, and whether it is the right tool for your use case. ## Outcome By the end of this page you will understand what Apache Pinot is, what problems it solves, and whether it is the right tool for your use case. ## Prerequisites None. This is the starting point of the onboarding path. ## What Apache Pinot does Apache Pinot is a real-time distributed online analytical processing (OLAP) datastore. It ingests data from streaming sources (such as Apache Kafka and Amazon Kinesis) and batch sources (such as Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage) and makes that data immediately available for analytic queries with sub-second latency. ### Key capabilities * **Ultra-low-latency analytics** -- Queries return in milliseconds, even at hundreds of thousands of queries per second. * **Columnar storage with smart indexing** -- Purpose-built storage format with inverted, sorted, range, text, and other indexes to accelerate query patterns. * **Horizontal scaling** -- Scale out by adding nodes with no upper bound on cluster size. * **Consistent performance** -- Latency stays predictable as data volume and query load grow, based on cluster sizing and expected throughput. * **Real-time ingestion** -- Data is available for querying within seconds of arriving at the streaming source. {% embed url="" %} What is Apache Pinot? (and User-Facing Analytics) by Tim Berglund {% endembed %} ## When to use Pinot ### User-facing real-time analytics Pinot was built at LinkedIn to power interactive analytics features such as Who Viewed Profile and Company Analytics. UberEats Restaurant Manager is another production example. These applications serve personalized analytics to every end user, generating hundreds of thousands of queries per second with strict latency requirements. ### Real-time dashboards Pinot supports slice-and-dice, drill-down, roll-up, and pivot operations on high-dimensional data. Connect business intelligence tools such as Apache Superset, Tableau, or PowerBI to Pinot to build live dashboards over streaming data. ### Enterprise analytics Pinot works well as a highly scalable platform for business intelligence. It converges the capabilities of a big data platform with the traditional role of a data warehouse, making it suitable for analysis and reporting at scale. ### Aggregate store for microservices Application developers can use Pinot as an aggregate store that consumes events from streaming sources and exposes them through SQL. This is useful for building a unified, queryable view across a microservice architecture. Query models are eventually consistent, as with all aggregate stores. ## When NOT to use Pinot {% hint style="warning" %} Pinot is not a general-purpose transactional database. It does not support row-level updates, deletes, or transactions in the way that PostgreSQL or MySQL do. If your workload requires ACID transactions or frequent single-row mutations, a relational database is a better fit. {% endhint %} {% hint style="info" %} If your dataset is small enough to fit comfortably in a single PostgreSQL or MySQL instance (a few million rows or less) and you do not need sub-second query latency at high concurrency, a traditional database will be simpler to operate and sufficient for your needs. {% endhint %} ## Verify You now know: * What Apache Pinot is and how it differs from transactional databases. * The four main categories of use cases where Pinot excels. * When a simpler tool would be a better choice. ## Next step Continue to the 10-minute quickstart to launch a local Pinot cluster and run your first query: {% content-ref url="/pages/xORpTeBUSuN79G8JifJ7" %} [10-Minute Quickstart](/start-here/ten-minute-quickstart) {% endcontent-ref %} # 10-Minute Quickstart Run a complete Pinot cluster with sample data in under 10 minutes. ## Outcome By the end of this guide you will have a fully functional Apache Pinot cluster running locally with sample data loaded, ready to query. ## Prerequisites * [Docker](https://docs.docker.com/get-docker/) installed and running * Recommended resources: 8 CPUs, 16 GB RAM ## Steps **1. Set the Pinot version** ```bash export PINOT_VERSION=1.5.1 ``` See the [Version reference](/start-here/pinot-versions) page for the current stable release. **2. Start Pinot with sample data** ```bash docker run \ -p 2123:2123 \ -p 9000:9000 \ -p 8000:8000 \ -p 7050:7050 \ -p 6000:6000 \ apachepinot/pinot:${PINOT_VERSION} QuickStart \ -type batch ``` This single command starts ZooKeeper, Controller, Broker, Server, and Minion, then loads a baseball statistics dataset. ## Verify 1. Open the Pinot Query Console at . 2. Run a sample query: ```sql SELECT playerName, sum(runs) AS totalRuns FROM baseballStats GROUP BY playerName ORDER BY totalRuns DESC LIMIT 10 ``` You should see results returned within milliseconds. ## Next step This quickstart bundles everything in a single process for convenience. For a list of all available quickstart types (batch, streaming, hybrid, and more), see [Quick Start Examples](/start-here/quick-start). Ready for a production-style setup? Continue to [Install / deploy](/start-here/install). # Quick Start Examples This section describes quick start commands that launch all Pinot components in a single process. Pinot ships with `QuickStart` commands that launch Pinot components in a single process and import pre-built datasets. These quick start examples are a good place if you're just getting started with Pinot. The examples begin with the [Batch Processing](#batch-processing) example, after the following notes: * **Prerequisites** You must have either [installed Pinot locally](/start-here/install/local) or [have Docker installed if you want to use the Pinot Docker image](/start-here/install/docker). The examples are available in each option and work the same. The decision of which to choose depends on your installation preference and how you generally like to work. If you don't know which to choose, using Docker will make your cleanup easier after you are done with the examples. * **Pinot versions in examples** The Docker-based examples on this page use `pinot:latest`, which instructs Docker to pull and use the most recent release of Apache Pinot. If you prefer to use a specific release instead, you can designate it by replacing `latest` with the release number, like this: `pinot:0.12.1`. The local install-based examples that are run using the launcher scripts will use the Apache Pinot version you installed. * **Stopping a running example** To stop a running example, enter `Ctrl+C` in the same terminal where you ran the `docker run` command to start the example. {% hint style="warning" %} **macOS Monterey Users** By default the Airplay receiver server runs on port 7000, which is also the port used by the Pinot Server in the Quick Start. You may see the following error when running these examples: ``` Failed to start a Pinot [SERVER] java.lang.RuntimeException: java.net.BindException: Address already in use at org.apache.pinot.core.transport.QueryServer.start(QueryServer.java:103) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906] at org.apache.pinot.server.starter.ServerInstance.start(ServerInstance.java:158) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906] at org.apache.helix.manager.zk.ParticipantManager.handleNewSession(ParticipantManager.java:110) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da2113 ``` If you disable the Airplay receiver server and try again, you shouldn't see this error message anymore. {% endhint %} {% hint style="info" %} **Use canonical quickstart types for new examples.** Use `BATCH` (or `OFFLINE`) for batch examples and `REALTIME` (or `STREAM`) for streaming examples. Each canonical quickstart initializes the samples that were previously started by separate feature-specific quickstarts and runs their sample queries. The following legacy types remain available as deprecated aliases, but emit a warning and are not listed in the command help: * Use `BATCH` instead of `MULTI_STAGE`, `JOIN`, `TIMESTAMP`, `BATCH_JSON_INDEX`, or `BATCH_COMPLEX_TYPE`. * Use `REALTIME` instead of `UPSERT`, `PARTIAL_UPSERT`, `UPSERT_JSON_INDEX`, `REALTIME_JSON_INDEX`, or `REALTIME_COMPLEX_TYPE`. {% endhint %} ## Command Options All QuickStart commands support the following optional parameters in addition to `-type`: | Option | Aliases | Description | | -------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-type` | | The quickstart type to run (see sections below). | | `-tmpDir` | `-quickstartDir`, `-dataDir` | Directory to store quickstart data. Use this to persist data across restarts so that tables and segments are reloaded from disk instead of being regenerated. | | `-bootstrapTableDir` | | A list of directories, each containing a table schema, table config, and raw data. Use this with `-type EMPTY` or `-type GENERIC` to load your own tables into the quickstart cluster. | | `-configFile` | `-configFilePath` | Path to a properties file that overrides default Pinot configuration values (controller, broker, server, etc.). | | `-zkAddress` | `-zkUrl`, `-zkExternalAddress` | URL for an external ZooKeeper instance (e.g. `localhost:2181`) instead of using the default embedded instance. | | `-kafkaBrokerList` | | Kafka broker list for streaming quickstarts (e.g. `localhost:9092`). Use this to connect to an external Kafka cluster instead of the embedded one. | **Example: Persist data across restarts** ```bash # First run: quickstart generates data in the specified directory ./bin/pinot-admin.sh QuickStart -type batch -dataDir /tmp/pinot-quick-start # Subsequent runs: quickstart reloads existing data from disk ./bin/pinot-admin.sh QuickStart -type batch -dataDir /tmp/pinot-quick-start ``` **Example: Use an external ZooKeeper and custom config** ```bash ./bin/pinot-admin.sh QuickStart -type batch \ -zkAddress localhost:2181 \ -configFile /path/to/pinot-quickstart.conf ``` **Example: Load custom tables into an empty cluster** ```bash ./bin/pinot-admin.sh QuickStart -type EMPTY \ -bootstrapTableDir /path/to/my-table-dir ``` ## Batch Processing This example demonstrates how to do batch processing with Pinot. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates the `baseballStats` table * Launches a standalone data ingestion job that builds one segment for a given CSV data file for the `baseballStats` table and pushes the segment to the Pinot Controller. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## Batch JSON This example demonstrates how to import and query JSON documents in Pinot. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates the `githubEvents` table * Launches a standalone data ingestion job that builds one segment for a given JSON data file for the `githubEvents` table and pushes the segment to the Pinot Controller. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## Batch with complex data types This example demonstrates how to do batch processing in Pinot where the the data items have complex fields that need to be unnested. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates the `githubEvents` table * Launches a standalone data ingestion job that builds one segment for a given JSON data file for the `githubEvents` table and pushes the segment to the Pinot Controller. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## Streaming This example demonstrates how to do stream processing with Pinot. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates `meetupRsvp` table * Launches a `meetup` stream * Publishes data to a Kafka topic `meetupRSVPEvents` that is subscribed to by Pinot. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type stream ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type stream ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type stream ``` {% endtab %} {% endtabs %} ## Streaming JSON This example demonstrates how to do stream processing with JSON documents in Pinot. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates `meetupRsvp` table * Launches a `meetup` stream * Publishes data to a Kafka topic `meetupRSVPEvents` that is subscribed to by Pinot * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime ``` {% endtab %} {% endtabs %} ## Streaming with minion cleanup This example demonstrates how to do stream processing in Pinot with RealtimeToOfflineSegmentsTask and MergeRollupTask minion tasks continuously optimizing segments as data gets ingested. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, Pinot Minion, and Pinot Server. * Creates `githubEvents` table * Launches a GitHub events stream * Publishes data to a Kafka topic `githubEvents` that is subscribed to by Pinot. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime_minion ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime_minion ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime_minion ``` {% endtab %} {% endtabs %} ## Streaming with complex data types This example demonstrates how to do stream processing in Pinot where the stream contains items that have complex fields that need to be unnested. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, Pinot Minion, and Pinot Server. * Creates `meetupRsvp` table * Launches a `meetup` stream * Publishes data to a Kafka topic `meetupRSVPEvents` that is subscribed to by Pinot. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime ``` {% endtab %} {% endtabs %} ## Upsert This example demonstrates how to do [stream processing with upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) with Pinot. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates `meetupRsvp` table * Launches a `meetup` stream * Publishes data to a Kafka topic `meetupRSVPEvents` that is subscribed to by Pinot * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime ``` {% endtab %} {% endtabs %} ## Upsert JSON This example demonstrates how to do [stream processing with upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) with JSON documents in Pinot. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates `meetupRsvp` table * Launches a `meetup` stream * Publishes data to a Kafka topic `meetupRSVPEvents` that is subscribed to by Pinot * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime ``` {% endtab %} {% endtabs %} ## Hybrid This example demonstrates how to do hybrid stream and batch processing with Pinot. The command: 1. Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. 2. Creates `airlineStats` table 3. Launches a standalone data ingestion job that builds segments under a given directory of Avro files for the `airlineStats` table and pushes the segments to the Pinot Controller. 4. Launches a stream of flights stats 5. Publishes data to a Kafka topic `airlineStatsEvents` that is subscribed to by Pinot. 6. Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type hybrid ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type hybrid ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type hybrid ``` {% endtab %} {% endtabs %} ## Join This example demonstrates how to do joins in Pinot using the [Lookup UDF](/build-with-pinot/querying-and-sql/sql-syntax/lookup-udf-join). The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server in the same container. * Creates the `baseballStats` table * Launches a data ingestion job that builds one segment for a given CSV data file for the `baseballStats` table and pushes the segment to the Pinot Controller. * Creates the `dimBaseballTeams` table * Launches a data ingestion job that builds one segment for a given CSV data file for the `dimBaseballStats` table and pushes the segment to the Pinot Controller. * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## Logical Table This example demonstrates how to use logical tables in Pinot, which provide a unified query interface over multiple physical tables. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, Pinot Server, and Pinot Minion. * Creates three physical tables (`ordersUS_OFFLINE`, `ordersEU_OFFLINE`, `ordersAPAC_OFFLINE`) representing regional order data * Creates a logical table (`orders`) that provides a unified view over all regional tables * Issues sample queries to both physical and logical tables {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type LOGICAL_TABLE ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type LOGICAL_TABLE ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type LOGICAL_TABLE ``` {% endtab %} {% endtabs %} For more details on logical tables, see [Logical Table](/architecture-and-concepts/components/table/logical-table). ## Empty This example starts a bare Pinot cluster with no tables or data loaded. Use this when you want to set up your own tables and schemas from scratch. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * No tables or data are created {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type EMPTY ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type EMPTY ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type EMPTY ``` {% endtab %} {% endtabs %} ## Multi-Stage Query Engine This example demonstrates the [multi-stage query engine](/build-with-pinot/querying-and-sql/multi-stage-query) with self-joins, dimension table joins, and vector distance queries. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates the `baseballStats` table and a fine food reviews table * Launches data ingestion jobs to build segments and push them to the Pinot Controller. * Issues sample multi-stage queries including joins and vector distance queries {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## Partial Upsert This example demonstrates how to do [stream processing with partial upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) in Pinot, where individual fields can be updated independently while preserving other column values. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates a realtime table with partial upsert enabled * Publishes data to a Kafka topic that is subscribed to by Pinot * Issues sample queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type realtime ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type realtime ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type realtime ``` {% endtab %} {% endtabs %} ## Geospatial This example demonstrates [geospatial indexing and query capabilities](/build-with-pinot/indexing/geospatial-support) in Pinot. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates a table with geospatial indexes * Launches a data ingestion job and pushes segments to the Pinot Controller. * Issues sample geospatial queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type GEOSPATIAL ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type GEOSPATIAL ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type GEOSPATIAL ``` {% endtab %} {% endtabs %} ## Null Handling This example demonstrates [null value handling](/build-with-pinot/querying-and-sql/sql-syntax/null-value-support) features in Pinot. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates a table containing null values * Launches a data ingestion job and pushes segments to the Pinot Controller. * Issues sample queries demonstrating IS NULL, IS NOT NULL, and aggregate behavior with nulls {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type NULL_HANDLING ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type NULL_HANDLING ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type NULL_HANDLING ``` {% endtab %} {% endtabs %} ## TPC-H This example loads the 8 TPC-H benchmark tables (customer, lineitem, nation, orders, part, partsupp, region, supplier) for multi-stage query testing. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates all 8 TPC-H tables * Launches data ingestion jobs to build segments for each table and pushes them to the Pinot Controller. * Issues sample TPC-H benchmark queries using the multi-stage query engine {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type TPCH ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type TPCH ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type TPCH ``` {% endtab %} {% endtabs %} ## Colocated Join This example demonstrates [colocated join](/build-with-pinot/querying-and-sql/multi-stage-query/join-strategies/colocated-join-strategy) operations using the multi-stage query engine with various partition configurations and parallelism hints. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates tables with matching partition configurations for colocated joins * Launches data ingestion jobs and pushes segments to the Pinot Controller. * Issues sample colocated join queries {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type COLOCATED_JOIN ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type COLOCATED_JOIN ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type COLOCATED_JOIN ``` {% endtab %} {% endtabs %} ## Lookup Join This example demonstrates the [lookup join strategy](/build-with-pinot/querying-and-sql/multi-stage-query/join-strategies/lookup-join-strategy) using dimension tables with the multi-stage query engine. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates fact and dimension tables * Launches data ingestion jobs and pushes segments to the Pinot Controller. * Issues sample lookup join queries {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type LOOKUP_JOIN ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type LOOKUP_JOIN ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type LOOKUP_JOIN ``` {% endtab %} {% endtabs %} ## Auth This example demonstrates how to run Pinot with [basic authentication](/operate-pinot/security/authentication/basic-auth-access-control) enabled. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server with basic auth configured. * Creates tables and loads data with authentication enabled * Issues sample authenticated queries to Pinot {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type AUTH ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type AUTH ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type AUTH ``` {% endtab %} {% endtabs %} ## Sorted Column This example demonstrates sorted column indexing in Pinot with a generated dataset containing sorted columns. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates a table with sorted column configuration * Generates a 100,000-row dataset and ingests it into Pinot * Issues sample queries demonstrating sorted index performance {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type SORTED ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type SORTED ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type SORTED ``` {% endtab %} {% endtabs %} ## Timestamp Index This example demonstrates [timestamp index](/build-with-pinot/indexing/timestamp-index) functionality, showing timestamp extraction at different granularities and dateTrunc bucketing. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates the `airlineStats` table with timestamp indexes * Launches a data ingestion job and pushes segments to the Pinot Controller. * Issues sample queries demonstrating timestamp extraction and bucketing {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type batch ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type batch ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type batch ``` {% endtab %} {% endtabs %} ## GitHub Events This example sets up a streaming demo using GitHub events data. The command: * Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server. * Creates a `pullRequestMergedEvents` realtime table * Publishes GitHub event data to a Kafka topic that is subscribed to by Pinot * Issues sample analytical queries on the GitHub event data {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type GITHUB_EVENTS ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type GITHUB_EVENTS ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type GITHUB_EVENTS ``` {% endtab %} {% endtabs %} ## Multi-Cluster This example demonstrates cross-cluster querying via [logical tables](/architecture-and-concepts/components/table/logical-table) by initializing two independent Pinot clusters. The command: * Starts two independent Pinot clusters, each with their own Zookeeper, Controller, Broker, and Server. * Creates physical tables in each cluster * Creates a logical table that spans both clusters * Issues sample cross-cluster queries {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type MULTI_CLUSTER ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type MULTI_CLUSTER ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type MULTI_CLUSTER ``` {% endtab %} {% endtabs %} ## Batch with Multi-Directory (Tiered Storage) This example demonstrates multi-directory (tiered storage) support with hot and cold tiers. The command: * Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server with tiered storage configured. * Creates the `airlineStats` table with hot and cold storage tiers * Launches a data ingestion job and pushes segments to the Pinot Controller. * Issues sample queries that run across storage tiers {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type BATCH_MULTIDIR ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type BATCH_MULTIDIR ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type BATCH_MULTIDIR ``` {% endtab %} {% endtabs %} ## Time Series {% hint style="info" %} For production use, you should ideally implement your own Time Series Language Plugin. The one included in the Pinot distribution is only for demonstration purposes. {% endhint %} This examples demonstrates Pinot's Time Series Engine, which supports running pluggable Time Series Query Languages via a Language Plugin architecture. The default Pinot binary includes a toy Time Series Query Language using the same name as Uber's language "m3ql". You can try the following query as an example: ``` fetch{table="meetupRsvp_REALTIME",filter="",ts_column="__metadata$recordTimestamp",ts_unit="MILLISECONDS",value="1"} | sum{rsvp_count} | transformNull{0} | keepLastValue{} ``` ![](/files/6FdoQas2Uy5ImfFxmV2W) \*\* {% tabs %} {% tab title="Docker" %} ``` docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type time_series ``` {% endtab %} {% tab title="Launcher scripts" %} ``` ./bin/pinot-admin.sh QuickStart -type time_series ``` {% endtab %} {% tab title="Brew" %} ``` pinot-admin QuickStart -type time_series ``` {% endtab %} {% endtabs %} # Install / Deploy Choose the deployment method that matches your environment. ## Outcome Select the right installation method for your use case and deploy a Pinot cluster. ## Deployment methods | Method | Best for | Time | Prerequisites | | ------------------------------------------------------------ | ---------------------- | ------ | ------------------------ | | [Local](/start-here/install/local) | Development, debugging | 10 min | JDK 25+ | | [Docker](/start-here/install/docker) | Quick evaluation, CI | 5 min | Docker | | [Kubernetes](/start-here/install/kubernetes) | Staging, production | 15 min | K8s cluster, Helm | | [Managed Kubernetes](/start-here/install/managed-kubernetes) | Production on cloud | 20 min | Cloud account, CLI tools | ## Next step Pick a method above, then continue to [First table and schema](/start-here/first-table-and-schema). # Local Start a Pinot cluster on your local machine. ## Outcome Start a multi-component Pinot cluster directly on your machine without containers. ## Prerequisites * JDK 25 or later (required for building and running Pinot services) * Apache Maven 3.6+ (only if building from source) {% hint style="info" %} **Note:** The SPI and Java/JDBC client modules (`pinot-spi`, `pinot-java-client`, `pinot-jdbc-client`) remain compatible with Java 11 for external JVM consumers. {% endhint %} ## Steps ### 1. Download or build Apache Pinot {% tabs %} {% tab title="Download release" %} ```bash export PINOT_VERSION=1.5.1 wget https://downloads.apache.org/pinot/apache-pinot-${PINOT_VERSION}/apache-pinot-${PINOT_VERSION}-bin.tar.gz ``` See the [Version reference](/start-here/pinot-versions) page for the current stable release. Extract and enter the directory: ```bash tar -zxvf apache-pinot-${PINOT_VERSION}-bin.tar.gz cd apache-pinot-${PINOT_VERSION}-bin ``` {% endtab %} {% tab title="Build from source" %} {% hint style="info" %} **Prerequisite:** Install [Apache Maven](https://maven.apache.org/install.html) 3.6 or higher. {% endhint %} ```bash git clone https://github.com/apache/pinot.git cd pinot mvn install package -DskipTests -Pbin-dist cd build ``` {% endtab %} {% tab title="Homebrew" %} ```bash brew install pinot ``` {% endtab %} {% endtabs %} ### 2. Start ZooKeeper ```bash ./bin/pinot-admin.sh StartZookeeper \ -zkPort 2181 ``` ### 3. Start Pinot Controller ```bash export JAVA_OPTS="-Xms4G -Xmx8G" ./bin/pinot-admin.sh StartController \ -zkAddress localhost:2181 \ -controllerPort 9000 ``` ### 4. Start Pinot Broker ```bash export JAVA_OPTS="-Xms4G -Xmx4G" ./bin/pinot-admin.sh StartBroker \ -zkAddress localhost:2181 ``` ### 5. Start Pinot Server ```bash export JAVA_OPTS="-Xms4G -Xmx16G" ./bin/pinot-admin.sh StartServer \ -zkAddress localhost:2181 ``` ### 6. Start Pinot Minion (optional) ```bash export JAVA_OPTS="-Xms4G -Xmx4G" ./bin/pinot-admin.sh StartMinion \ -zkAddress localhost:2181 ``` ### 7. Start Kafka (optional) Only needed if you plan to ingest real-time streaming data. ```bash ./bin/pinot-admin.sh StartKafka \ -zkAddress=localhost:2181/kafka \ -port 19092 ``` ## Verify Check that the Controller is healthy: ```bash curl localhost:9000/health ``` The response should return `OK`. You can also open the Pinot Query Console at . ## Next step Your cluster is running. Continue to [First table and schema](/start-here/first-table-and-schema) to load data. # Docker Start a Pinot cluster using Docker containers. ## Outcome Start a multi-component Pinot cluster using Docker, suitable for local evaluation and CI environments. ## Prerequisites * [Docker](https://docs.docker.com/get-docker/) installed and running * Recommended Docker resource settings: * CPUs: 8 * Memory: 16 GB * Swap: 4 GB * Disk image size: 60 GB ## Steps ### 1. Set the image versions ```bash export PINOT_VERSION=1.5.1 export PINOT_IMAGE=apachepinot/pinot:${PINOT_VERSION} export ZK_IMAGE=zookeeper:3.9.5 export KAFKA_IMAGE=apache/kafka:4.0.0 ``` See the [Version reference](/start-here/pinot-versions) page for the current stable release. ### 2. Pull the Pinot image ```bash docker pull apachepinot/pinot:${PINOT_VERSION} ``` View all available tags on [Docker Hub](https://hub.docker.com/r/apachepinot/pinot/tags). ### 3. Start the cluster {% tabs %} {% tab title="Docker Compose (recommended)" %} Create a file called `docker-compose.yml` with the following content: {% code title="docker-compose.yml" %} ```yaml version: '3.7' services: pinot-zookeeper: image: ${ZK_IMAGE:-zookeeper:3.9.5} container_name: "pinot-zookeeper" restart: unless-stopped ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 networks: - pinot-demo healthcheck: test: ["CMD", "zkServer.sh", "status"] interval: 30s timeout: 10s retries: 5 start_period: 10s pinot-controller: image: ${PINOT_IMAGE:-apachepinot/pinot:1.5.1} command: "StartController -zkAddress pinot-zookeeper:2181" container_name: "pinot-controller" restart: unless-stopped ports: - "9000:9000" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" depends_on: pinot-zookeeper: condition: service_healthy networks: - pinot-demo healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 10s pinot-broker: image: ${PINOT_IMAGE:-apachepinot/pinot:1.5.1} command: "StartBroker -zkAddress pinot-zookeeper:2181" container_name: "pinot-broker" restart: unless-stopped ports: - "8099:8099" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" depends_on: pinot-controller: condition: service_healthy networks: - pinot-demo healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:8099/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 10s pinot-server: image: ${PINOT_IMAGE:-apachepinot/pinot:1.5.1} command: "StartServer -zkAddress pinot-zookeeper:2181" container_name: "pinot-server" restart: unless-stopped ports: - "8098:8098" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" depends_on: pinot-broker: condition: service_healthy networks: - pinot-demo healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:8097/health/readiness || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 10s pinot-minion: image: ${PINOT_IMAGE:-apachepinot/pinot:1.5.1} command: "StartMinion -zkAddress pinot-zookeeper:2181" restart: unless-stopped container_name: "pinot-minion" ports: - "6000:6000" depends_on: - pinot-broker networks: - pinot-demo pinot-kafka: image: ${KAFKA_IMAGE:-apache/kafka:4.0.0} container_name: "kafka" restart: unless-stopped ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk networks: - pinot-demo healthcheck: test: ["CMD-SHELL", "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server kafka:9092"] interval: 30s timeout: 10s retries: 5 start_period: 10s deploy: replicas: ${KAFKA_REPLICAS:-0} networks: pinot-demo: name: pinot-demo driver: bridge ``` {% endcode %} Launch the cluster: ```bash docker compose --project-name pinot-demo up ``` To also start Kafka for real-time streaming: ```bash export KAFKA_REPLICAS=1 docker compose --project-name pinot-demo up ``` {% endtab %} {% tab title="Individual docker run commands" %} **Create a network** ```bash docker network create -d bridge pinot-demo ``` **Start ZooKeeper** ```bash docker run \ --network=pinot-demo \ --name pinot-zookeeper \ --restart always \ -p 2181:2181 \ -d ${ZK_IMAGE} ``` **Start Pinot Controller** ```bash docker run --rm -ti \ --network=pinot-demo \ --name pinot-controller \ -p 9000:9000 \ -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" \ -d ${PINOT_IMAGE} StartController \ -zkAddress pinot-zookeeper:2181 ``` **Start Pinot Broker** ```bash docker run --rm -ti \ --network=pinot-demo \ --name pinot-broker \ -p 8099:8099 \ -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" \ -d ${PINOT_IMAGE} StartBroker \ -zkAddress pinot-zookeeper:2181 ``` **Start Pinot Server** ```bash docker run --rm -ti \ --network=pinot-demo \ --name pinot-server \ -p 8098:8098 \ -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \ -d ${PINOT_IMAGE} StartServer \ -zkAddress pinot-zookeeper:2181 ``` **Start Pinot Minion (optional)** ```bash docker run --rm -ti \ --network=pinot-demo \ --name pinot-minion \ -p 6000:6000 \ -d ${PINOT_IMAGE} StartMinion \ -zkAddress pinot-zookeeper:2181 ``` **Start Kafka (optional)** Kafka 4.0 runs in KRaft mode and does not require ZooKeeper: ```bash docker run --rm -ti \ --network pinot-demo --name=kafka \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \ -p 9092:9092 \ -d ${KAFKA_IMAGE} ``` {% endtab %} {% endtabs %} ## Verify Check that all containers are running: ```bash docker container ls -a ``` You should see containers for ZooKeeper, Controller, Broker, Server, and Minion all in a healthy state. Open the Pinot Query Console at to confirm the cluster is ready. ## Docker image versions Pinot Docker images now target **JDK 25** runtimes. The `latest` tag promotes the `25-ms-openjdk` runtime, and `latest-25` plus versioned `-25` tags remain available for explicit pinning. Pinot no longer publishes `*-21-*` service image tags. If you are pinned to `apachepinot/pinot:latest-21-amazoncorretto`, `apachepinot/pinot:latest-21-ms-openjdk`, or similar versioned `*-21-*` tags, move to a `*-25-*` tag before upgrading. ## Next step Your cluster is running. Continue to [First table and schema](/start-here/first-table-and-schema) to load data. # Kubernetes Deploy a Pinot cluster on Kubernetes using Helm. ## Outcome Deploy a production-ready Pinot cluster on Kubernetes with Helm charts. {% hint style="warning" %} **Helm 3 Migration:** Pinot Helm charts have been upgraded to Helm 3 (apiVersion v2). If you are upgrading from older Helm charts (v1), please note: * **Minimum Helm version:** Helm 3.0+ is required * **Dependency management:** Use `helm dependency build` (not `helm dependency update`) * **Dependency file:** Charts now use `Chart.yaml` for dependency declarations (not `requirements.yaml`) * **Lock file:** Uses `Chart.lock` (not `requirements.lock`) * **Chart archives:** No longer checked into the repository; automatically fetched via `helm dependency build` * **Chart versions:** No longer use the `-SNAPSHOT` suffix For detailed upgrade instructions, see the [Upgrade Guide](#upgrading-from-helm-v1-charts) below. {% endhint %} {% hint style="info" %} The examples in this guide are sample configurations for reference. For production deployments, customize settings as needed -- especially security features like TLS and authentication. {% endhint %} ## Prerequisites * A running Kubernetes cluster. Options include: * [Docker Desktop with Kubernetes enabled](https://docs.docker.com/docker-for-mac/kubernetes/) * [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/) -- start with sufficient resources: `minikube start --vm=true --cpus=4 --memory=8g --disk-size=50g` * A managed cloud cluster -- see [Managed Kubernetes](/start-here/install/managed-kubernetes) for AWS, GCP, and Azure setup guides * **Helm 3** (apiVersion v2) -- Required for Pinot Helm charts. [Install Helm 3](https://helm.sh/docs/intro/install/) * [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) ## Steps ### 1. Add the Pinot Helm repository ```bash helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/helm ``` ### 2. Create a namespace ```bash kubectl create ns pinot-quickstart ``` ### 3. Install Pinot {% tabs %} {% tab title="Helm with pre-packaged chart" %} ```bash helm install pinot pinot/pinot \ -n pinot-quickstart \ --set cluster.name=pinot \ --set server.replicaCount=2 ``` {% hint style="info" %} **StorageClass:** Specify the StorageClass for your cloud vendor. Use block storage only -- do not mount blob stores (S3, GCS, AzureFile) as the data-serving file system. * AWS: `gp2` * GCP: `pd-ssd` or `standard` * Azure: `AzureDisk` * Docker Desktop: `hostpath` {% endhint %} {% endtab %} {% tab title="Helm from Git repo" %} ```bash git clone https://github.com/apache/pinot.git cd pinot/helm/pinot helm dependency build kubectl create ns pinot-quickstart helm install -n pinot-quickstart pinot ./pinot ``` {% hint style="info" %} **Helm 3 Migration (v2):** As of the recent migration to Helm 3 apiVersion v2, the Pinot charts now use `Chart.yaml` for dependency declarations (instead of `requirements.yaml`) and `Chart.lock` (instead of `requirements.lock`). The `helm dependency build` command fetches dependencies from the chart repository automatically. {% endhint %} {% endtab %} {% endtabs %} ## Verify Check the deployment status: ```bash kubectl get all -n pinot-quickstart ``` All pods should reach `Running` status. You can port-forward the Controller to access the UI: ```bash kubectl port-forward service/pinot-controller 9000:9000 -n pinot-quickstart ``` Then open . ## Loading data For stream ingestion on Kubernetes, see the [Kubernetes stream ingestion guide](/start-here/install/stream-ingestion). For batch data loading and table creation, continue with the onboarding path below. ## Deleting the cluster To remove Pinot from your cluster: ```bash kubectl delete ns pinot-quickstart ``` ## Next step Your cluster is running. Continue to [First table and schema](/start-here/first-table-and-schema) to load data. ## Upgrading from Helm v1 charts If you are upgrading from older Pinot Helm charts (apiVersion v1), follow these steps: ### 1. Update Helm repository ```bash helm repo update pinot ``` ### 2. Back up your configuration Save your current `values.yaml` in case you need to reference it: ```bash helm get values pinot -n pinot-quickstart > values-backup.yaml ``` ### 3. Uninstall old release (optional but recommended) If upgrading from v1 charts, it's safest to uninstall and reinstall: ```bash helm uninstall pinot -n pinot-quickstart ``` ### 4. Reinstall with new Helm v2 charts ```bash helm install pinot pinot/pinot \ -n pinot-quickstart \ --set cluster.name=pinot \ --set server.replicaCount=2 \ -f values-backup.yaml ``` ### Key changes in Helm v2 charts * **apiVersion:** Changed from `v1` to `v2` * **Chart type:** Added explicit `type: application` * **Dependencies:** Now declared in `Chart.yaml` instead of `requirements.yaml` * **Lock file:** Changed from `requirements.lock` to `Chart.lock` * **Dependency fetching:** Use `helm dependency build` instead of `helm dependency update` * **Chart versions:** No longer include `-SNAPSHOT` suffix (uses standard SemVer) * **Chart icons:** Now include official Apache Pinot icons ### If deploying from source When deploying directly from the Git repository, always run `helm dependency build` before installing: ```bash git clone https://github.com/apache/pinot.git cd pinot/helm/pinot helm dependency build # Required for Helm 3 v2 charts helm install pinot . -n pinot-quickstart ``` # Stream Ingestion on Kubernetes Load streaming data into Pinot on Kubernetes using Kafka This guide walks you through loading streaming data into a Pinot cluster running in Kubernetes. Make sure you have completed [Running in Kubernetes](https://github.com/pinot-contrib/pinot-docs/blob/latest/basics/getting-started/kubernetes/README.md) first. ## Load data into Pinot using Kafka ### **Bring up a Kafka cluster for real-time data ingestion** {% hint style="info" %} The Bitnami Kafka Helm chart deploys Kafka in **KRaft mode** (with a built-in controller quorum) by default, so a separate ZooKeeper deployment is not required for Kafka. {% endhint %} ```bash helm repo add kafka https://charts.bitnami.com/bitnami helm install -n pinot-quickstart kafka kafka/kafka \ --set replicas=1 \ --set listeners.client.protocol=PLAINTEXT ``` ### Check Kafka deployment status Ensure the Kafka deployment is ready before executing the scripts in the following steps. Run the following command: ```bash kubectl get all -n pinot-quickstart | grep kafka ``` Below is an example output showing the deployment is ready: ``` pod/kafka-controller-0 1/1 Running 0 2m pod/kafka-controller-1 1/1 Running 0 2m pod/kafka-controller-2 1/1 Running 0 2m ``` ### **Create Kafka topics** Run the scripts below to create two Kafka topics for data ingestion: ```bash kubectl -n pinot-quickstart exec kafka-controller-0 -- kafka-topics.sh --bootstrap-server kafka:9092 --topic flights-realtime --create --partitions 1 --replication-factor 1 kubectl -n pinot-quickstart exec kafka-controller-0 -- kafka-topics.sh --bootstrap-server kafka:9092 --topic flights-realtime-avro --create --partitions 1 --replication-factor 1 ``` ### **Load data into Kafka and create Pinot schema/tables** The script below does the following: * Ingests 19492 JSON messages to Kafka topic `flights-realtime` at a speed of 1 msg/sec * Ingests 19492 Avro messages to Kafka topic `flights-realtime-avro` at a speed of 1 msg/sec * Uploads Pinot schema `airlineStats` * Creates Pinot table `airlineStats` to ingest data from JSON encoded Kafka topic `flights-realtime` * Creates Pinot table `airlineStatsAvro` to ingest data from Avro encoded Kafka topic `flights-realtime-avro` ```bash kubectl apply -f pinot/helm/pinot/pinot-realtime-quickstart.yml ``` ## Query with the Pinot Data Explorer ### Pinot Data Explorer The following script (located at `./pinot/helm/pinot`) performs local port forwarding, and opens the Pinot query console in your default web browser. ```bash ./query-pinot-data.sh ``` # Managed Kubernetes Set up a Kubernetes cluster on your cloud provider. ## Outcome Provision a managed Kubernetes cluster on AWS, GCP, or Azure that is ready for a Pinot deployment. ## Overview These guides walk you through creating a managed Kubernetes cluster on your cloud provider. Once the cluster is running, you will use the [Kubernetes install](/start-here/install/kubernetes) page to deploy Pinot onto it. ## Cloud providers | Provider | Service | Guide | | --------------------- | ---------- | ----------------------------------------------------------- | | Amazon Web Services | Amazon EKS | [AWS setup](/start-here/install/managed-kubernetes/aws) | | Google Cloud Platform | Google GKE | [GCP setup](/start-here/install/managed-kubernetes/gcp) | | Microsoft Azure | Azure AKS | [Azure setup](/start-here/install/managed-kubernetes/azure) | ## Next step Once your cluster is ready, follow the [Kubernetes install guide](/start-here/install/kubernetes) to deploy Pinot. # AWS Provision a managed Kubernetes cluster on Amazon EKS ready for Pinot. ## Outcome Create an Amazon EKS cluster with the required tooling, ready to deploy Apache Pinot. ## Prerequisites * An AWS account * The following CLI tools installed (see steps below) ## Steps ### 1. Install tooling **kubectl** ```bash brew install kubernetes-cli ``` Verify: ```bash kubectl version ``` **Helm** ```bash brew install kubernetes-helm ``` Verify: ```bash helm version ``` **AWS CLI** Follow the [AWS CLI installation guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) or run: ```bash curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-macos.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install ``` **eksctl** ```bash brew tap weaveworks/tap brew install weaveworks/tap/eksctl ``` ### 2. Configure AWS credentials ```bash aws configure ``` {% hint style="info" %} Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` override credentials stored in `~/.aws/credentials`. {% endhint %} ### 3. Create an EKS cluster The following creates a single-node cluster named `pinot-quickstart` in `us-west-2` using `t3.xlarge` instances: ```bash EKS_CLUSTER_NAME=pinot-quickstart eksctl create cluster \ --name ${EKS_CLUSTER_NAME} \ --version 1.16 \ --region us-west-2 \ --nodegroup-name standard-workers \ --node-type t3.xlarge \ --nodes 1 \ --nodes-min 1 \ --nodes-max 1 ``` For Kubernetes 1.23+, enable the EBS CSI driver to allow persistent volume provisioning: ```bash eksctl utils associate-iam-oidc-provider --region=us-east-2 --cluster=pinot-quickstart --approve eksctl create iamserviceaccount \ --name ebs-csi-controller-sa \ --namespace kube-system \ --cluster pinot-quickstart \ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \ --approve \ --role-only \ --role-name AmazonEKS_EBS_CSI_DriverRole eksctl create addon --name aws-ebs-csi-driver --cluster pinot-quickstart \ --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_DriverRole --force ``` Monitor cluster status: ```bash EKS_CLUSTER_NAME=pinot-quickstart aws eks describe-cluster --name ${EKS_CLUSTER_NAME} --region us-west-2 ``` Wait until the cluster status is **ACTIVE**. ### 4. Connect to the cluster ```bash EKS_CLUSTER_NAME=pinot-quickstart aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME} ``` ## Verify ```bash kubectl get nodes ``` You should see your worker nodes listed and in `Ready` status. ## Cleaning up To delete the cluster when you are done: ```bash EKS_CLUSTER_NAME=pinot-quickstart aws eks delete-cluster --name ${EKS_CLUSTER_NAME} ``` ## Next step Your cluster is ready. Continue to [Kubernetes install](/start-here/install/kubernetes) to deploy Pinot. # GCP Provision a managed Kubernetes cluster on Google GKE ready for Pinot. ## Outcome Create a Google Kubernetes Engine cluster with the required tooling, ready to deploy Apache Pinot. ## Prerequisites * A Google Cloud account and project * The following CLI tools installed (see steps below) ## Steps ### 1. Install tooling **kubectl** ```bash brew install kubernetes-cli ``` Verify: ```bash kubectl version ``` **Helm** ```bash brew install kubernetes-helm ``` Verify: ```bash helm version ``` **Google Cloud SDK** Follow the [gcloud CLI installation guide](https://cloud.google.com/sdk/docs/install) or run: ```bash curl https://sdk.cloud.google.com | bash exec -l $SHELL ``` ### 2. Initialize Google Cloud ```bash gcloud init ``` ### 3. Create a GKE cluster The following creates a 3-node cluster named `pinot-quickstart` in `us-west1-b` using `n1-standard-2` machines: ```bash GCLOUD_PROJECT=[your gcloud project name] GCLOUD_ZONE=us-west1-b GCLOUD_CLUSTER=pinot-quickstart GCLOUD_MACHINE_TYPE=n1-standard-2 GCLOUD_NUM_NODES=3 gcloud container clusters create ${GCLOUD_CLUSTER} \ --num-nodes=${GCLOUD_NUM_NODES} \ --machine-type=${GCLOUD_MACHINE_TYPE} \ --zone=${GCLOUD_ZONE} \ --project=${GCLOUD_PROJECT} ``` Monitor cluster status: ```bash gcloud compute instances list ``` Wait until the cluster status is **RUNNING**. ### 4. Connect to the cluster ```bash GCLOUD_PROJECT=[your gcloud project name] GCLOUD_ZONE=us-west1-b GCLOUD_CLUSTER=pinot-quickstart gcloud container clusters get-credentials ${GCLOUD_CLUSTER} --zone ${GCLOUD_ZONE} --project ${GCLOUD_PROJECT} ``` ## Verify ```bash kubectl get nodes ``` You should see your worker nodes listed and in `Ready` status. ## Cleaning up To delete the cluster when you are done: ```bash GCLOUD_ZONE=us-west1-b gcloud container clusters delete pinot-quickstart --zone=${GCLOUD_ZONE} ``` ## Next step Your cluster is ready. Continue to [Kubernetes install](/start-here/install/kubernetes) to deploy Pinot. # Azure Provision a managed Kubernetes cluster on Azure AKS ready for Pinot. ## Outcome Create an Azure Kubernetes Service cluster with the required tooling, ready to deploy Apache Pinot. ## Prerequisites * An Azure account * The following CLI tools installed (see steps below) ## Steps ### 1. Install tooling **kubectl** ```bash brew install kubernetes-cli ``` Verify: ```bash kubectl version ``` **Helm** ```bash brew install kubernetes-helm ``` Verify: ```bash helm version ``` **Azure CLI** Follow the [Azure CLI installation guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) or run: ```bash brew update && brew install azure-cli ``` ### 2. Log in to Azure ```bash az login ``` ### 3. Create a resource group ```bash AKS_RESOURCE_GROUP=pinot-demo AKS_RESOURCE_GROUP_LOCATION=eastus az group create --name ${AKS_RESOURCE_GROUP} \ --location ${AKS_RESOURCE_GROUP_LOCATION} ``` ### 4. Create an AKS cluster The following creates a 3-node cluster named `pinot-quickstart`: ```bash AKS_RESOURCE_GROUP=pinot-demo AKS_CLUSTER_NAME=pinot-quickstart az aks create --resource-group ${AKS_RESOURCE_GROUP} \ --name ${AKS_CLUSTER_NAME} \ --node-count 3 ``` ### 5. Connect to the cluster ```bash AKS_RESOURCE_GROUP=pinot-demo AKS_CLUSTER_NAME=pinot-quickstart az aks get-credentials --resource-group ${AKS_RESOURCE_GROUP} \ --name ${AKS_CLUSTER_NAME} ``` ## Verify ```bash kubectl get nodes ``` You should see your worker nodes listed and in `Ready` status. ## Cleaning up To delete the cluster when you are done: ```bash AKS_RESOURCE_GROUP=pinot-demo AKS_CLUSTER_NAME=pinot-quickstart az aks delete --resource-group ${AKS_RESOURCE_GROUP} \ --name ${AKS_CLUSTER_NAME} ``` ## Next step Your cluster is ready. Continue to [Kubernetes install](/start-here/install/kubernetes) to deploy Pinot. # First Table + Schema Create your first Pinot schema and table, ready for data ingestion. ## Outcome By the end of this page you will have a Pinot schema and an offline table called `transcript` registered in your cluster, ready to receive data. ## Prerequisites * A running Pinot cluster. See the install guides for [Local](/start-here/install/local) or [Docker](/start-here/install/docker). * For Docker users: the cluster must be on the `pinot-demo` network. * Confirm your Pinot version. See the [Version reference](/start-here/pinot-versions) page and set the `PINOT_VERSION` environment variable: ```bash export PINOT_VERSION= ``` ## Steps ### 1. Understand schemas A Pinot schema defines every column in your table and assigns each one a column type. There are three column types: | Column type | Description | | ----------- | ----------------------------------------------------------------- | | Dimension | Used in filters and GROUP BY clauses for slicing and dicing data. | | Metric | Used in aggregations; represents quantitative measurements. | | DateTime | Represents the timestamp associated with each row. | Every table must have a schema before it can accept data. The schema tells Pinot how to interpret, index, and store each field. ### 2. Create the data directory ```bash mkdir -p /tmp/pinot-quick-start/rawdata ``` ### 3. Save the sample CSV data Create the file `/tmp/pinot-quick-start/rawdata/transcript.csv` with the following contents: {% code title="/tmp/pinot-quick-start/rawdata/transcript.csv" %} ``` studentID,firstName,lastName,gender,subject,score,timestampInEpoch 200,Lucy,Smith,Female,Maths,3.8,1570863600000 200,Lucy,Smith,Female,English,3.5,1571036400000 201,Bob,King,Male,Maths,3.2,1571900400000 202,Nick,Young,Male,Physics,3.6,1572418800000 ``` {% endcode %} In this dataset, `studentID`, `firstName`, `lastName`, `gender`, and `subject` are dimensions, `score` is a metric, and `timestampInEpoch` is the datetime column. ### 4. Save the schema Create the file `/tmp/pinot-quick-start/transcript-schema.json`: {% code title="/tmp/pinot-quick-start/transcript-schema.json" %} ```json { "schemaName": "transcript", "dimensionFieldSpecs": [ { "name": "studentID", "dataType": "INT" }, { "name": "firstName", "dataType": "STRING" }, { "name": "lastName", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "subject", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [{ "name": "timestampInEpoch", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` {% endcode %} ### 5. Understand table configs A table config tells Pinot how to manage the table at runtime -- which columns to index, how many replicas to keep, which tenants to assign, and whether the table is OFFLINE (batch) or REALTIME (streaming). You pair one table config with one schema. ### 6. Save the offline table config Create the file `/tmp/pinot-quick-start/transcript-table-offline.json`: {% code title="/tmp/pinot-quick-start/transcript-table-offline.json" %} ```json { "tableName": "transcript", "segmentsConfig": { "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "replication": "1", "schemaName": "transcript" }, "tableIndexConfig": { "invertedIndexColumns": [], "loadMode": "MMAP" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableType": "OFFLINE", "metadata": {} } ``` {% endcode %} ### 7. Upload the schema and table config {% tabs %} {% tab title="Local" %} ```bash bin/pinot-admin.sh AddTable \ -tableConfigFile /tmp/pinot-quick-start/transcript-table-offline.json \ -schemaFile /tmp/pinot-quick-start/transcript-schema.json \ -exec ``` {% endtab %} {% tab title="Docker" %} ```bash docker run --rm -ti \ --network=pinot-demo \ -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \ --name pinot-table-creation \ apachepinot/pinot:${PINOT_VERSION} AddTable \ -schemaFile /tmp/pinot-quick-start/transcript-schema.json \ -tableConfigFile /tmp/pinot-quick-start/transcript-table-offline.json \ -controllerHost pinot-controller \ -controllerPort 9000 \ -exec ``` {% hint style="info" %} Replace `pinot-controller` with the actual container name of your Pinot controller if you used a different name during setup. {% endhint %} {% endtab %} {% endtabs %} ## Verify 1. Open the Pinot Data Explorer at . 2. Navigate to the **Tables** tab. 3. Confirm you see `transcript_OFFLINE` listed. If the table appears, the schema and table config were registered successfully. ## Next step You now have an empty table. Continue to [First batch ingest](/start-here/first-batch-ingest) to import the CSV data into your `transcript` table. # First Batch Ingest Import your first batch of data into Pinot and see it appear in the query console. ## Outcome By the end of this page you will have imported CSV data into your `transcript` offline table and confirmed the rows are queryable. ## Prerequisites * Completed [First table and schema](/start-here/first-table-and-schema) -- the `transcript_OFFLINE` table must already exist. * The sample CSV file at `/tmp/pinot-quick-start/rawdata/transcript.csv` from the previous step. * For Docker users: set the `PINOT_VERSION` environment variable. See the [Version reference](/start-here/pinot-versions) page. ## Steps ### 1. Understand batch ingestion Batch ingestion reads data from files (CSV, JSON, Avro, Parquet, and others), converts them into Pinot segments, and pushes those segments to the cluster. A job specification YAML file tells Pinot where to find the input data, what format it is in, and where to send the finished segments. ### 2. Create the ingestion job spec {% tabs %} {% tab title="Local" %} Create the file `/tmp/pinot-quick-start/batch-job-spec.yml`: {% code title="/tmp/pinot-quick-start/batch-job-spec.yml" %} ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/tmp/pinot-quick-start/rawdata/' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/tmp/pinot-quick-start/segments/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'transcript' schemaURI: 'http://localhost:9000/tables/transcript/schema' tableConfigURI: 'http://localhost:9000/tables/transcript' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` {% endcode %} {% endtab %} {% tab title="Docker" %} When running inside Docker, the ingestion job container must reach the controller by its Docker network hostname, not `localhost`. Create the file `/tmp/pinot-quick-start/batch-job-spec.yml`: {% code title="/tmp/pinot-quick-start/batch-job-spec.yml" %} ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '/tmp/pinot-quick-start/rawdata/' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/tmp/pinot-quick-start/segments/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'transcript' schemaURI: 'http://pinot-controller:9000/tables/transcript/schema' tableConfigURI: 'http://pinot-controller:9000/tables/transcript' pinotClusterSpecs: - controllerURI: 'http://pinot-controller:9000' ``` {% endcode %} {% hint style="info" %} Replace `pinot-controller` with the actual container name of your Pinot controller if you used a different name during setup. {% endhint %} {% endtab %} {% endtabs %} ### 3. Run the ingestion job {% tabs %} {% tab title="Local" %} ```bash bin/pinot-admin.sh LaunchDataIngestionJob \ -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yml ``` {% endtab %} {% tab title="Docker" %} ```bash docker run --rm -ti \ --network=pinot-demo \ -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \ --name pinot-data-ingestion-job \ apachepinot/pinot:${PINOT_VERSION} LaunchDataIngestionJob \ -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yml ``` {% endtab %} {% endtabs %} The job reads the CSV file, builds a segment, and pushes it to the controller. You should see log output ending with a success message. ## Verify 1. Open the [Query Console](http://localhost:9000/query) in your browser. 2. Run the following query: ```sql SELECT * FROM transcript ``` 3. You should see **4 rows** returned, matching the CSV data you loaded: | studentID | firstName | lastName | gender | subject | score | timestampInEpoch | | --------- | --------- | -------- | ------ | ------- | ----- | ---------------- | | 200 | Lucy | Smith | Female | Maths | 3.8 | 1570863600000 | | 200 | Lucy | Smith | Female | English | 3.5 | 1571036400000 | | 201 | Bob | King | Male | Maths | 3.2 | 1571900400000 | | 202 | Nick | Young | Male | Physics | 3.6 | 1572418800000 | ## Next step Continue to [First stream ingest](/start-here/first-stream-ingest) to learn how to set up real-time ingestion from Kafka. # First Stream Ingest Set up real-time streaming ingestion from Kafka and watch data arrive in Pinot. {% hint style="info" %} For Kubernetes-specific streaming ingestion, see [Stream ingestion (Kubernetes)](/start-here/install/stream-ingestion). {% endhint %} ## Outcome By the end of this page you will have a realtime Pinot table consuming data from a Kafka topic, with 12 rows visible in the query console. ## Prerequisites * Completed [First table and schema](/start-here/first-table-and-schema) -- the `transcript` schema must already exist in the cluster. * A running Pinot cluster. See the install guides for [Local](/start-here/install/local) or [Docker](/start-here/install/docker). * For Docker users: set the `PINOT_VERSION` environment variable. See the [Version reference](/start-here/pinot-versions) page. ## Steps ### 1. Understand streaming ingestion Streaming ingestion lets Pinot consume data from a message queue in real time. As messages arrive in a Kafka topic, Pinot reads them and makes the rows queryable within seconds. The realtime table config specifies the Kafka broker, topic, and decoder so that Pinot knows how to connect and interpret incoming records. ### 2. Start Kafka {% tabs %} {% tab title="Local" %} Start Kafka on port `9876` using the same ZooKeeper from the Pinot quick-start: ```bash bin/pinot-admin.sh StartKafka -zkAddress=localhost:2123/kafka -port 9876 ``` {% endtab %} {% tab title="Docker" %} Kafka 4.0 runs in KRaft mode and does not require ZooKeeper: ```bash docker run \ --network pinot-demo --name=kafka \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \ -d apache/kafka:4.0.0 ``` {% endtab %} {% endtabs %} ### 3. Create a Kafka topic {% tabs %} {% tab title="Local" %} Download [Apache Kafka](https://kafka.apache.org/quickstart#quickstart_download) if you have not already, then create the topic: ```bash bin/kafka-topics.sh --create --bootstrap-server localhost:9876 \ --replication-factor 1 --partitions 1 --topic transcript-topic ``` {% endtab %} {% tab title="Docker" %} ```bash docker exec \ -t kafka \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server kafka:9092 \ --partitions=1 --replication-factor=1 \ --create --topic transcript-topic ``` {% endtab %} {% endtabs %} ### 4. Save the realtime table config Create the file `/tmp/pinot-quick-start/transcript-table-realtime.json`: {% tabs %} {% tab title="Local" %} {% code title="/tmp/pinot-quick-start/transcript-table-realtime.json" %} ```json { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9876", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` {% endcode %} {% endtab %} {% tab title="Docker" %} {% code title="/tmp/pinot-quick-start/transcript-table-realtime.json" %} ```json { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` {% endcode %} {% hint style="info" %} The Docker version uses `kafka:9092` as the broker address because both the Kafka and Pinot containers are on the same `pinot-demo` Docker network. {% endhint %} {% endtab %} {% endtabs %} ### 5. Upload the realtime table config As soon as the realtime table is created, Pinot begins consuming from the Kafka topic. {% tabs %} {% tab title="Local" %} ```bash bin/pinot-admin.sh AddTable \ -schemaFile /tmp/pinot-quick-start/transcript-schema.json \ -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \ -exec ``` {% hint style="info" %} If the `transcript` schema was already uploaded during [First table and schema](/start-here/first-table-and-schema), you can omit the `-schemaFile` flag. Including it is safe -- Pinot will skip re-creating an identical schema. {% endhint %} {% endtab %} {% tab title="Docker" %} ```bash docker run --rm -ti \ --network=pinot-demo \ -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \ --name pinot-streaming-table-creation \ apachepinot/pinot:${PINOT_VERSION} AddTable \ -schemaFile /tmp/pinot-quick-start/transcript-schema.json \ -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \ -controllerHost pinot-controller \ -controllerPort 9000 \ -exec ``` {% hint style="info" %} Replace `pinot-controller` with the actual container name of your Pinot controller if you used a different name during setup. {% endhint %} {% endtab %} {% endtabs %} ### 6. Save the sample streaming data Create the file `/tmp/pinot-quick-start/rawdata/transcript.json`: {% code title="/tmp/pinot-quick-start/rawdata/transcript.json" %} ```json {"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"Maths","score":3.8,"timestampInEpoch":1571900400000} {"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"History","score":3.5,"timestampInEpoch":1571900400000} {"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Maths","score":3.2,"timestampInEpoch":1571900400000} {"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Chemistry","score":3.6,"timestampInEpoch":1572418800000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Geography","score":3.8,"timestampInEpoch":1572505200000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"English","score":3.5,"timestampInEpoch":1572505200000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Maths","score":3.2,"timestampInEpoch":1572678000000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Physics","score":3.6,"timestampInEpoch":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"Maths","score":3.8,"timestampInEpoch":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"English","score":3.5,"timestampInEpoch":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"History","score":3.2,"timestampInEpoch":1572854400000} {"studentID":212,"firstName":"Nick","lastName":"Young","gender":"Male","subject":"History","score":3.6,"timestampInEpoch":1572854400000} ``` {% endcode %} ### 7. Push data into the Kafka topic {% tabs %} {% tab title="Local" %} ```bash bin/kafka-console-producer.sh \ --bootstrap-server localhost:9876 \ --topic transcript-topic < /tmp/pinot-quick-start/rawdata/transcript.json ``` {% endtab %} {% tab title="Docker" %} ```bash docker exec -t kafka /opt/kafka/bin/kafka-console-producer.sh \ --bootstrap-server localhost:9092 \ --topic transcript-topic < /tmp/pinot-quick-start/rawdata/transcript.json ``` {% endtab %} {% endtabs %} ## Verify 1. Open the [Query Console](http://localhost:9000/query) in your browser. 2. Run the following query: ```sql SELECT * FROM transcript ``` 3. You should see **12 rows** of streaming data. Pinot ingests from Kafka in real time, so the rows appear within seconds of being pushed to the topic. ## Next step Continue to [First query](/start-here/first-query) to learn how to write analytical queries against your Pinot tables. # First Query Run your first SQL queries against Pinot using the Query Console and REST API. ## Outcome Run your first SQL queries against Pinot and understand the query interface. ## Prerequisites * You have completed either [First batch ingest](/start-here/first-batch-ingest) or [First stream ingest](/start-here/first-stream-ingest). The `transcript` table exists and contains data. * The Pinot cluster is running (Controller on port 9000, Broker on port 8099). ## Steps ### 1. Open the Query Console Navigate to in your browser. Click **Query Console** in the left sidebar. You should see the `transcript` table listed in the table explorer on the left. ### 2. Run a simple SELECT Paste the following query into the query editor and click **Run Query**: ```sql SELECT * FROM transcript LIMIT 10 ``` The results panel shows all columns in the `transcript` table -- `studentID`, `firstName`, `lastName`, `gender`, `subject`, `score`, and `timestamp`. The rows returned come from whichever data you loaded (batch, stream, or both). `LIMIT 10` caps the result set so the response is fast. ### 3. Run an aggregation ```sql SELECT subject, AVG(score) AS avg_score FROM transcript GROUP BY subject ORDER BY avg_score DESC ``` This query calculates the average score per subject and sorts the results from highest to lowest. Pinot executes aggregations directly on each server's segment data and merges the results at the Broker, making GROUP BY queries fast even on large datasets. ### 4. Run a count ```sql SELECT COUNT(*) FROM transcript ``` This returns the total number of rows in the table. The exact count depends on which ingestion steps you completed: * Batch ingest only -- 4 rows * Stream ingest only -- the number of events you published (up to 12 in the tutorial) * Both -- the combined total ### 5. Run a filter ```sql SELECT firstName, lastName, score FROM transcript WHERE score > 3.5 ``` This filters rows to show only students with a score above 3.5. Pinot pushes filter predicates down to the servers so only matching rows are scanned and returned. ### 6. Try the REST API The Query Console UI is convenient for exploration, but production applications query Pinot through its REST API. Open a terminal and run: ```bash curl -X POST http://localhost:8099/query/sql \ -H 'Content-Type: application/json' \ -d '{"sql": "SELECT * FROM transcript LIMIT 5"}' ``` Port 8099 is the Broker, which handles all query requests. The Query Console UI uses the same API under the hood. The response is a JSON object containing the result rows, schema, and query execution metadata. ## Verify All five queries return results without errors. You have successfully completed the end-to-end onboarding flow: you set up a Pinot cluster, defined a schema and table, loaded data, and queried it through both the UI and the REST API. ## What's next You have finished the linear Start Here path. From here, explore the areas most relevant to your use case: * [Query Syntax](/build-with-pinot/querying-and-sql/sql-syntax/query-syntax-overview) -- the full SQL reference for Pinot's query language * [Multi-Stage Query Engine](/build-with-pinot/querying-and-sql/multi-stage-query) -- enable JOINs and complex queries across tables * [Architecture](/architecture-and-concepts/concepts/architecture) -- understand how queries flow from Broker to Server and back * [Stream Ingestion from Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) -- set up real-time ingestion for production workloads # Version Reference Current Apache Pinot release version and how to pin versions in examples. ## Outcome Know which Pinot version to use and how to pin versions in examples. {% hint style="warning" %} All code samples in the **Start Here** guide use `PINOT_VERSION=1.5.1`. If you are using a different version, set the variable accordingly before running any commands. {% endhint %} This page is the single source of truth for version information across the Start Here guide and the wider documentation. When following tutorials or code samples, make sure the version you use matches your installed release. ## Current stable release | Artifact | Version | | ---------------------- | ------------------------- | | Apache Pinot binary | **1.5.1** | | Docker image | `apachepinot/pinot:1.5.1` | | Maven / Gradle clients | `1.5.1` | ## Using PINOT\_VERSION in examples Most code samples in these docs set a `PINOT_VERSION` environment variable near the top of each snippet. Always verify that the value matches your installed version: ```bash export PINOT_VERSION=1.5.1 # Then use ${PINOT_VERSION} in commands: docker pull apachepinot/pinot:${PINOT_VERSION} ``` Once the variable is set, every command in the tutorial that references `${PINOT_VERSION}` will use the correct value automatically. {% hint style="info" %} Start Here pages never use the `latest` Docker tag. Always pin to a specific version for reproducibility. The `latest` tag can change without notice and may introduce breaking changes during a tutorial. {% endhint %} ## Compatibility notes | Requirement | Detail | | -------------------------- | --------------------------------------------------------------------------- | | **Build/Runtime baseline** | **JDK 25+** (required for Pinot services) | | **Client libraries** | JDK 11+ (SPI and Java/JDBC client artifacts remain compatible with Java 11) | | Pinot 1.0+ minimum | JDK 11 or higher required | | JDK 8 support | Pinot **0.12.1** is the last version that supports JDK 8 | **Starting with this release:** Pinot services require **JDK 25 or later** for building and runtime. The SPI and Java/JDBC client modules (`pinot-spi`, `pinot-java-client`, `pinot-jdbc-client`) are compiled to Java 11 bytecode for backward compatibility with external JVM consumers. ## Release links {% hint style="info" %} You can find all published releases on the [Release notes](/reference/release-notes/releases) page, and all Docker tags on [Docker Hub](https://hub.docker.com/r/apachepinot/pinot/tags). {% endhint %} ## Older versions Older Pinot binaries are archived at . # Concepts Explore the fundamental concepts of Apache Pinot™ as a distributed OLAP database. Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including: * Storing data in columnar form to support high-performance scanning * Sharding of data to scale both storage and computation * A distributed architecture designed to scale capacity linearly * A tabular data model read by SQL queries To learn about Pinot components, terminology, and gain a conceptual understanding of how data is stored in Pinot, review the following sections: * [Pinot storage model](/architecture-and-concepts/concepts/pinot-storage-model) * [Pinot architecture](/architecture-and-concepts/concepts/architecture) * [Pinot components](/architecture-and-concepts/components) # Pinot Storage Model Apache Pinot™ uses a variety of terms which can refer to either abstractions that model the storage of data or infrastructure components that drive the functionality of the system, including: * [Tables](#table) to store data * [Segments](#segment) to partition data * [Tenants](#tenant) to isolate data * [Clusters](#cluster) to manage data Pinot has a distributed systems architecture that scales horizontally. Pinot expects the size of a table to grow infinitely over time. To achieve this, all data needs to be distributed across multiple nodes. Pinot achieves this by breaking data into smaller chunks known as [segments](/architecture-and-concepts/components/table/segment) (similar to shards/partitions in HA relational databases). Segments can also be seen as time-based partitions. ## Table Similar to traditional databases, Pinot has the concept of a [table](/architecture-and-concepts/components/table)—a logical abstraction to refer to a collection of related data. As is the case with relational database management systems (RDBMS), a table is a construct that consists of columns and rows (documents) that are queried using SQL. A table is associated with a [schema](/architecture-and-concepts/components/table/schema), which defines the columns in a table as well as their data types. As opposed to RDBMS schemas, multiple tables can be created in Pinot (real-time or batch) that inherit a single schema definition. Tables are independently configured for concerns such as indexing strategies, partitioning, tenants, data sources, and replication. Pinot stores data in [tables](/architecture-and-concepts/components/table). A Pinot table is conceptually identical to a relational database table with rows and columns. Columns have the same name and data type, known as the table's [schema](/architecture-and-concepts/components/table/schema). Pinot schemas are defined in a JSON file. Because that schema definition is in its own file, multiple tables can share a single schema. Each table can have a unique name, indexing strategy, partitioning, data sources, and other metadata. Pinot table types include: * **real-time:** Ingests data from a streaming source like Apache Kafka® * **offline:** Loads data from a batch source * **hybrid:** Loads data from both a batch source and a streaming source ## Segment Pinot tables are stored in one or more independent shards called [segments](/architecture-and-concepts/components/table/segment). A small table may be contained by a single segment, but Pinot lets tables grow to an unlimited number of segments. There are different processes for creating segments (see [ingestion](/build-with-pinot/ingestion)). Segments have time-based partitions of table data, and are stored on Pinot [servers](/architecture-and-concepts/components/cluster/server) that scale horizontally as needed for both storage and computation. ## Tenant To support multi-tenancy, Pinot has first class support for tenants. A table is associated with a [tenant](/architecture-and-concepts/components/cluster/tenant). This allows all tables belonging to a particular logical namespace to be grouped under a single tenant name and isolated from other tenants. This isolation between tenants provides different namespaces for applications and teams to prevent sharing tables or schemas. Development teams building applications do not have to operate an independent deployment of Pinot. An organization can operate a single cluster and scale it out as new tenants increase the overall volume of queries. Developers can manage their own schemas and tables without being impacted by any other tenant on a cluster. Every table is associated with a [*tenant*](/architecture-and-concepts/components/cluster/tenant), or a logical namespace that restricts where the cluster processes queries on the table. A Pinot tenant takes the form of a text tag in the logical tenant namespace. Physical cluster hardware resources (i.e., [brokers](/architecture-and-concepts/components/cluster/broker) and [servers](/architecture-and-concepts/components/cluster/server)) are also associated with a tenant tag in the common tenant namespace. Tables of a particular tenant tag will only be scheduled for storage and query processing on hardware resources that belong to the same tenant tag. This lets Pinot cluster operators assign specified workloads to certain hardware resources, preventing data from separate workloads from being stored or processed on the same physical hardware. By default, all tables, brokers, and servers belong to a tenant called *DefaultTenant*, but you can configure multiple tenants in a Pinot cluster. ## Cluster A Pinot [*cluster*](/architecture-and-concepts/components/cluster) is a collection of the software processes and hardware resources required to ingest, store, and process data. For detail about Pinot cluster components, see [Physical architecture](#physical-architecture). ## Physical architecture ![](/files/HEFqT2CiwOHifbDPhdBt) \*\* A Pinot cluster consists of the following processes, which are typically deployed on separate hardware resources in production. In development, they can fit comfortably into Docker containers on a typical laptop. * **Controller**: Maintains cluster metadata and manages cluster resources. * **Zookeeper**: Manages the Pinot cluster on behalf of the controller. Provides fault-tolerant, persistent storage of metadata, including table configurations, schemas, segment metadata, and cluster state. * **Broker**: Accepts queries from client processes and forwards them to servers for processing. * **Server**: Provides storage for segment files and compute for query processing. * (Optional) **Minion**: Computes background tasks other than query processing, minimizing impact on query latency. Optimizes segments, and builds additional indexes to ensure performance (even if data is deleted). The simplest possible Pinot cluster consists of four components: a server, a broker, a controller, and a Zookeeper node. In production environments, these components typically run on separate server instances, and scale out as needed for data volume, load, availability, and latency. Pinot clusters in production range from fewer than ten total instances to more than 1,000. Pinot uses [Apache Zookeeper](https://zookeeper.apache.org/) as a distributed metadata store and and [Apache Helix](http://helix.apache.org/) for cluster management. Helix is a cluster management solution created by the authors of Pinot. Helix maintains a persistent, fault-tolerant map of the intended state of the Pinot cluster. It constantly monitors the cluster to ensure that the right hardware resources are allocated to implement the present configuration. When the configuration changes, Helix schedules or decommissions hardware resources to reflect the new configuration. When elements of the cluster change state catastrophically, Helix schedules hardware resources to keep the actual cluster consistent with the ideal represented in the metadata. From a physical perspective, Helix takes the form of a controller process plus agents running on servers and brokers. ## Controller A[ controller](/architecture-and-concepts/components/cluster/controller) is the core orchestrator that drives the consistency and routing in a Pinot cluster. Controllers are horizontally scaled as an independent component (container) and has visibility of the state of all other components in a cluster. The controller reacts and responds to state changes in the system and schedules the allocation of resources for tables, segments, or nodes. As mentioned earlier, Helix is embedded within the controller as an agent that is a participant responsible for observing and driving state changes that are subscribed to by other components. The Pinot [controller](/architecture-and-concepts/components/cluster/controller) schedules and re-schedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, it schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (e.g., ingest of [real-time tables](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion) and [offline tables](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion)). It can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration. The controller exposes a [REST API endpoint](/reference/api-reference/controller-api) for cluster-wide administrative operations as well as a web-based query console to execute interactive SQL queries and perform simple administrative tasks. ## Server [Servers](/architecture-and-concepts/components/cluster/server) host segments (shards) that are scheduled and allocated across multiple nodes and routed on an assignment to a tenant (there is a single tenant by default). Servers are independent containers that scale horizontally and are notified by Helix through state changes driven by the controller. A server can either be a real-time server or an offline server. A real-time and offline server have very different resource usage requirements, where real-time servers are continually consuming new messages from external systems (such as Kafka topics) that are ingested and allocated on segments of a tenant. Because of this, resource isolation can be used to prioritize high-throughput real-time data streams that are ingested and then made available for query through a broker. ## Broker Pinot [brokers](/architecture-and-concepts/components/cluster/broker) take query requests from client processes, scatter them to applicable servers, gather the results, and return them to the client. The controller shares cluster metadata with the brokers that allows the brokers to create a plan for executing the query involving a minimal subset of servers with the source data and, when required, other servers to shuffle and consolidate results. A production Pinot cluster contains many brokers. In general, the more brokers, the more concurrent queries a cluster can process, and the lower latency it can deliver on queries. ## Pinot minion Pinot minion is an optional component that can be used to run background tasks such as "purge" for GDPR (General Data Protection Regulation). As Pinot is an immutable aggregate store, records containing sensitive private data need to be purged on a request-by-request basis. Minion provides a solution for this purpose that complies with GDPR while optimizing Pinot segments and building additional indices that guarantees performance in the presence of the possibility of data deletion. One can also write a custom task that runs on a periodic basis. While it's possible to perform these tasks on the Pinot servers directly, having a separate process (Minion) lessens the overall degradation of query latency as segments are impacted by mutable writes. A Pinot [minion](/architecture-and-concepts/components/cluster/minion) is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing *minion tasks* as directed by the controller. Examples of minon tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically. Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function with or without minions, they are typically present to support routine tasks like batch data ingest. \ \\ # Architecture Understand how the components of Apache Pinot™ work together to create a scalable OLAP database that can deliver low-latency, high-concurrency queries at scale. Apache Pinot™ is a distributed OLAP database designed to serve real-time, user-facing use cases, which means handling large volumes of data and many concurrent queries with very low query latencies. Pinot supports the following requirements: * Ultra low-latency queries (as low as 10ms P95) * High query concurrency (as many as 100,000 queries per second) * High data freshness (streaming data available for query immediately upon ingestion) * Large data volume (up to petabytes) ## Distributed design principles To accommodate large data volumes with stringent latency and concurrency requirements, Pinot is designed as a distributed database that supports the following requirements: * **Highly available**: Pinot has no single point of failure. When tables are configured for replication, and a node goes down, the cluster is able to continue processing queries. * **Horizontally scalable**: Operators can scale a Pinot cluster by adding new nodes when the workload increases. There are even two node types ([servers](/architecture-and-concepts/components/cluster/server) and [brokers](/architecture-and-concepts/components/cluster/broker)) to scale query volume, query complexity, and data size independently. * **Immutable data**: Pinot assumes all stored data is immutable, which helps simplify the parts of the system that handle data storage and replication. However, Pinot still supports upserts on streaming entity data and background purges of data to comply with data privacy regulations. * **Dynamic configuration changes**: Operations like adding new tables, expanding a cluster, ingesting data, modifying an existing table, and adding indexes do not impact query availability or performance. ## Core components As described in the Pinot [Components](/architecture-and-concepts/components), Pinot has four node types: * [Controller](/architecture-and-concepts/components/cluster/controller) * [Broker](/architecture-and-concepts/components/cluster/broker) * [Server](/architecture-and-concepts/components/cluster/server) * [Minion](/architecture-and-concepts/components/cluster/minion) ![](/files/-M1pvo4yOL0qNSjSS5nc) ### Apache Helix and ZooKeeper Distributed systems do not maintain themselves, and in fact require sophisticated scheduling and resource management to function. Pinot uses [Apache Helix](http://helix.apache.org/) for this purpose. Helix exists as an independent project, but it was designed by the original creators of Pinot for Pinot's own cluster management purposes, so the architectures of the two systems are well-aligned. Helix takes the form of a process on the controller, plus embedded agents on the brokers and servers. It uses [Apache ZooKeeper](https://zookeeper.apache.org/) as a fault-tolerant, strongly consistent, durable state store. Helix maintains a picture of the intended state of the cluster, including the number of servers and brokers, the configuration and schema of all tables, connections to streaming ingest sources, currently executing batch ingestion jobs, the assignment of table segments to the servers in the cluster, and more. All of these configuration items are potentially mutable quantities, since operators routinely change table schemas, add or remove streaming ingest sources, begin new batch ingestion jobs, and so on. Additionally, physical cluster state may change as servers and brokers fail or suffer network partition. Helix works constantly to drive the actual state of the cluster to match the intended state, pushing configuration changes to brokers and servers as needed. There are three physical node types in a Helix cluster: * **Participant**: These nodes *do* things, like store data or perform computation. Participants host *resources*, which are Helix's fundamental storage abstraction. Because Pinot servers store segment data, they are participants. * **Spectator**: These nodes *see* things, observing the evolving state of the *participants* through events pushed to the spectator. Because Pinot brokers need to know which servers host which segments, they are spectators. * **Controller**: This node observes and manages the state of participant nodes. The controller is responsible for coordinating all state transitions in the cluster and ensures that state constraints are satisfied while maintaining cluster stability. In addition, Helix defines two logical components to express its storage abstraction: * **Partition**. A unit of data storage that lives on at least one participant. Partitions may be replicated across multiple participants. A Pinot segment is a partition. * **Resource**. A logical collection of partitions, providing a single view over a potentially large set of data stored across a distributed system. A Pinot table is a resource. In summary, the Pinot architecture maps onto Helix components as follows: | Pinot Component | Helix Component | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Segment | **Helix Partition** | | Table | **Helix Resource** | | Controller | **Helix Controller** or Helix agent that drives the overall state of the cluster | | Server | **Helix Participant** | | Broker | A **Helix Spectator** that observes the cluster for changes in the state of segments and servers. To support multi-tenancy, brokers are also modeled as **Helix Participants**. | | Minion | **Helix Participant** that performs computation rather than storing data | Helix uses ZooKeeper to maintain cluster state. ZooKeeper sends Helix spectators notifications of changes in cluster state (which correspond to changes in ZNodes). Zookeeper stores the following information about the cluster: | Resource | Stored Properties | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | Controller | - Controller that is assigned as the current leader | | Servers and Brokers | - List of servers and brokers - Configuration of all current servers and brokers - Health status of all current servers and brokers | | Tables | - List of tables - Table configurations - Table schema - List of the table's segments | | Segment | - Exact server locations of a segment - State of each segment (online/offline/error/consuming) - Metadata about each segment | Zookeeper, as a first-class citizen of a Pinot cluster, may use the well-known `ZNode` structure for operations and troubleshooting purposes. Be advised that this structure can change in future Pinot releases. ![Pinot's Zookeeper Browser UI](/files/-MEl-nabKI7SITFzkrrU) ### Controller The Pinot [controller](/architecture-and-concepts/components/cluster/controller) schedules and re-schedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, it schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (e.g., ingest of [real-time tables](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion) and [offline tables](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion)). It can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration. #### Fault tolerance Only one controller can be active at a time, so when multiple controllers are present in a cluster, they elect a leader. When that controller instance becomes unavailable, the remaining instances automatically elect a new leader. Leader election is achieved using Apache Helix. A Pinot cluster can serve queries without an active controller, but it can't perform any metadata-modifying operations, like adding a table or consuming a new segment. #### Controller REST interface The controller provides a REST interface that allows read and write access to all logical storage resources (e.g., servers, brokers, tables, and segments). See [Pinot Data Explorer](/architecture-and-concepts/components/exploring-pinot) for more information on the web-based admin tool. ### Broker The [broker's](/architecture-and-concepts/components/cluster/broker) responsibility is to route queries to the appropriate [server](/architecture-and-concepts/components/cluster/server) instances, or in the case of multi-stage queries, to compute a complete query plan and distribute it to the servers required to execute it. The broker collects and merges the responses from all servers into a final result, then sends the result back to the requesting client. The broker exposes an HTTP endpoint that accepts SQL queries in JSON format and returns the response in JSON. Each broker maintains a query routing table. The routing table maps segments to the servers that store them. (When replication is configured on a table, each segment is stored on more than one server.) The broker computes multiple routing tables depending on the configured [routing](/operate-pinot/tuning/routing) strategy for a table. The default strategy is to balance the query load across all available servers. {% hint style="info" %} Advanced routing strategies are available, such as replica-aware routing, partition-based routing, and minimal server selection routing. {% endhint %} ```javascript //This is an example ZNode config for EXTERNAL VIEW in Helix { "id" : "baseballStats_OFFLINE", "simpleFields" : { ... }, "mapFields" : { "baseballStats_OFFLINE_0" : { "Server_10.1.10.82_7000" : "ONLINE" } }, ... } ``` #### Query processing Every query processed by a broker uses the single-stage engine or the [multi-stage engine](/build-with-pinot/querying-and-sql/sse-vs-mse). For single-stage queries, the broker does the following: * Computes query routes based on the routing strategy defined in the [table](/architecture-and-concepts/components/table) configuration. * Computes the list of segments to query on each [server](/architecture-and-concepts/components/cluster/server). (See [routing](/operate-pinot/tuning/routing) for further details on this process.) * Sends the query to each of those servers for local execution against their segments. * Receives the results from each server and merges them. * Sends the query result to the client. ```javascript // Query: select count(*) from baseballStats limit 10 // RESPONSE // ======== { "resultTable": { "dataSchema": { "columnDataTypes": ["LONG"], "columnNames": ["count(*)"] }, "rows": [ [97889] ] }, "exceptions": [], "numServersQueried": 1, "numServersResponded": 1, "numSegmentsQueried": 1, "numSegmentsProcessed": 1, "numSegmentsMatched": 1, "numConsumingSegmentsQueried": 0, "numDocsScanned": 97889, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 0, "numGroupsLimitReached": false, "totalDocs": 97889, "timeUsedMs": 5, "segmentStatistics": [], "traceInfo": {}, "minConsumingFreshnessTimeMs": 0 } ``` For multi-stage queries, the broker performs the following: * Computes a query plan that runs on multiple sets of servers. The servers selected for the first stage are selected based on the segments required to execute the query, which are determined in a process similar to single-stage queries. * Sends the relevant portions of the query plan to one or more servers in the cluster for each stage of the query plan. * The servers that received query plans each execute their part of the query. For more details on this process, read about the [multi-stage engine](/build-with-pinot/querying-and-sql/sse-vs-mse). * The broker receives a complete result set from the final stage of the query, which is always a single server. * The broker sends the query result to the client. ### Server [Servers](/architecture-and-concepts/components/cluster/server) host [segments](/architecture-and-concepts/components/table/segment) on locally attached storage and process queries on those segments. By convention, operators speak of "real-time" and "offline" servers, although there is no difference in the server process itself or even its configuration that distinguishes between the two. This is merely a convention reflected in the [table](/architecture-and-concepts/components/table) assignment strategy to confine the two different kinds of workloads to two groups of physical instances, since the performance-limiting factors differ between the two kinds of workloads. For example, offline servers might optimize for larger storage capacity, whereas real-time servers might optimize for memory and CPU cores. #### Offline servers Offline servers host segments created by ingesting batch data. The controller writes these segments to the offline server according to the table's replication factor and segment assignment strategy. Typically, the controller writes new segments to the [deep store](/architecture-and-concepts/components/table/segment/deep-store), and affected servers download the segment from deep store. The controller then notifies brokers that a new segment exists, and is available to participate in queries. Because offline tables tend to have long retention periods, offline servers tend to scale based on the size of the data they store. ![](/files/-M1cBoDyJPidtPVN1ZEA) #### Real-time servers Real-time servers ingest data from streaming sources, like Apache Kafka®, Apache Pulsar®, or AWS Kinesis. Streaming data ends up in conventional segment files just like batch data, but is first accumulated in an in-memory data structure known as a consuming segment. Each message consumed from a streaming source is written immediately to the relevant consuming segment, and is available for query processing from the consuming segment immediately, since consuming segments participate in query processing as first-class citizens. Consuming segments get flushed to disk periodically based on a completion threshold, which can be calculated by row count, ingestion time, or segment size. A flushed segment on a real-time table is called a *completed* segment, and is functionally equivalent to a segment created during offline ingest. Real-time servers tend to be scaled based on the rate at which they ingest streaming data. ![](/files/-M1poAtLWkGycoWFwCEx) ### Minion A Pinot [minion](/architecture-and-concepts/components/cluster/minion) is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing *minion tasks* as directed by the controller. Examples of minion tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically. Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function without minions, they are typically present to support routine tasks like ingesting batch data. ## Data ingestion overview Pinot [tables](/architecture-and-concepts/components/table) exist in two varieties: offline (or batch) and real-time. Offline tables contain data from batch sources like CSV, Avro, or Parquet files, and real-time tables contain data from streaming sources like like Apache Kafka®, Apache Pulsar®, or AWS Kinesis. ### Offline (batch) ingest ![](/files/-M1cBoDyJPidtPVN1ZEA) Pinot ingests batch data using an [ingestion job](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion), which follows a process like this: 1. The job transforms a raw data source (such as a CSV file) into [segments](/architecture-and-concepts/components/table/segment). This is a potentially complex process resulting in a file that is typically several hundred megabytes in size. 2. The job then transfers the file to the cluster's [deep store](/architecture-and-concepts/components/table/segment/deep-store) and notifies the [controller](/architecture-and-concepts/components/cluster/controller) that a new segment exists. 3. The controller (in its capacity as a Helix controller) updates the ideal state of the cluster in its cluster metadata map. 4. The controller then assigns the segment to one or more "offline" [servers](/architecture-and-concepts/components/cluster/server) (depending on replication factor) and notifies them that new segments are available. 5. The servers then download the newly created segments directly from the deep store. 6. The cluster's brokers, which watch for state changes as Helix spectators, detect the new segments and update their segment routing tables accordingly. The cluster is now able to query the new offline segments. ### Real-time ingest Ingestion is established at the time a real-time table is created, and continues as long as the table exists. When the controller receives the metadata update to create a new real-time table, the table configuration specifies the source of the streaming input data—often a topic in a Kafka cluster. This kicks off a process like this: 1. The controller picks one or more servers to act as direct consumers of the streaming input source. 2. The controller creates consuming segments for the new table. It does this by creating an entry in the global metadata map for a new consuming segment for each of the real-time servers selected in step 1. 3. Through Helix functionality on the controller and the relevant servers, the servers proceed to create consuming segments in memory and establish a connection to the streaming input source. When this input source is Kafka, each server acts as a Kafka consumer directly, with no other components involved in the integration. 4. Through Helix functionality on the controller and all of the cluster's brokers, the brokers become aware of the consuming segments, and begin including them in query routing immediately. 5. The consuming servers simultaneously begin consuming messages from the streaming input source, storing them in the consuming segment. 6. When a server decides its consuming segment is *complete*, it commits the in-memory consuming segment to a conventional segment file, uploads it to the deep store, and notifies the controller. 7. The controller and the server create a new consuming segment to continue real-time ingestion. 8. The controller marks the newly committed segment as online. Brokers then discover the new segment through the Helix notification mechanism, allowing them to route queries to it in the usual fashion. ![](/files/-M1poAtLWkGycoWFwCEx) # Components Discover the core components of Apache Pinot, enabling efficient data processing and analytics. Unleash the power of Pinot's building blocks for high-performance data-driven applications. Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including: * Storing data in columnar form to support high-performance scanning * Sharding of data to scale both storage and computation * A distributed architecture designed to scale capacity linearly * A tabular data model read by SQL queries ## Components Learn about the major components and logical abstractions used in Pinot. ![](/files/-M60c8gIYuj9VugbZGLf) #### Operator reference {% content-ref url="/pages/-M1SwMeAkncc4yvIhVWP" %} [Cluster](/architecture-and-concepts/components/cluster) {% endcontent-ref %} {% content-ref url="/pages/-M1SwP0Pe1UROmJ0Jawj" %} [Controller](/architecture-and-concepts/components/cluster/controller) {% endcontent-ref %} {% content-ref url="/pages/-M1SwVH1GROxiqA8J209" %} [Broker](/architecture-and-concepts/components/cluster/broker) {% endcontent-ref %} {% content-ref url="/pages/-M1SwSfgDA6hYjodT63z" %} [Server](/architecture-and-concepts/components/cluster/server) {% endcontent-ref %} {% content-ref url="/pages/-M1Swkf2kSXi8fRpwfqz" %} [Minion](/architecture-and-concepts/components/cluster/minion) {% endcontent-ref %} {% content-ref url="/pages/-M1SwmissUqIdwJb5Sql" %} [Tenant](/architecture-and-concepts/components/cluster/tenant) {% endcontent-ref %} #### Developer reference {% content-ref url="/pages/-M1Swq2JuBYmEGT8APA4" %} [Table](/architecture-and-concepts/components/table) {% endcontent-ref %} {% content-ref url="/pages/-MF9wogbFfW875Hxu7aG" %} [Schema](/architecture-and-concepts/components/table/schema) {% endcontent-ref %} {% content-ref url="/pages/-M1WNbG9kTML5cje9fSN" %} [Segment](/architecture-and-concepts/components/table/segment) {% endcontent-ref %} # Cluster Learn to build and manage Apache Pinot clusters, uncovering key components for efficient data processing and optimized analysis. A Pinot cluster is a collection of the software processes and hardware resources required to ingest, store, and process data. For detail about Pinot cluster components, see [Physical architecture](/architecture-and-concepts/concepts/pinot-storage-model#physical-architecture). A Pinot cluster consists of the following processes, which are typically deployed on separate hardware resources in production. In development, they can fit comfortably into Docker containers on a typical laptop: * **Controller**: Maintains cluster metadata and manages cluster resources. * **Zookeeper**: Manages the Pinot cluster on behalf of the controller. Provides fault-tolerant, persistent storage of metadata, including table configurations, schemas, segment metadata, and cluster state. * **Broker**: Accepts queries from client processes and forwards them to servers for processing. * **Server**: Provides storage for segment files and compute for query processing. * (Optional) **Minion**: Computes background tasks other than query processing, minimizing impact on query latency. Optimizes segments, and builds additional indexes to ensure performance (even if data is deleted). The simplest possible Pinot cluster consists of four components: a server, a broker, a controller, and a Zookeeper node. In production environments, these components typically run on separate server instances, and scale out as needed for data volume, load, availability, and latency. Pinot clusters in production range from fewer than ten total instances to more than 1,000. Pinot uses [Apache Zookeeper](https://zookeeper.apache.org/) as a distributed metadata store and [Apache Helix](http://helix.apache.org/) for cluster management. Helix is a cluster management solution that maintains a persistent, fault-tolerant map of the intended state of the Pinot cluster. Helix constantly monitors the cluster to ensure that the right hardware resources are allocated for the present configuration. When the configuration changes, Helix schedules or decommissions hardware resources to reflect the new configuration. When elements of the cluster change state catastrophically, Helix schedules hardware resources to keep the actual cluster consistent with the ideal represented in the metadata. From a physical perspective, Helix takes the form of a controller process plus agents running on servers and brokers. ## Cluster configuration For details of cluster configuration settings, see [Cluster configuration reference](/reference/configuration-reference/cluster). ## Cluster components Helix divides nodes into logical components based on their responsibilities: ### Participant Participants are the nodes that host distributed, partitioned resources Pinot servers are modeled as participants. For details about server nodes, see [Server](/architecture-and-concepts/components/cluster/server). ### Spectator Spectators are the nodes that observe the current state of each participant and use that information to access the resources. Spectators are notified of state changes in the cluster (state of a participant, or that of a partition in a participant). Pinot brokers are modeled as spectators. For details about broker nodes, see [Broker](/architecture-and-concepts/components/cluster/broker). ### Controller The node that observes and controls the Participant nodes. It is responsible for coordinating all transitions in the cluster and ensuring that state constraints are satisfied while maintaining cluster stability. Pinot controllers are modeled as controllers. For details about controller nodes, see [Controller](/architecture-and-concepts/components/cluster/controller). ## Logical view Another way to visualize the cluster is a logical view, where: * A cluster contains [tenants](/architecture-and-concepts/components/cluster/tenant) * Tenants contain [tables](/architecture-and-concepts/components/table) * Tables contain [segments](/architecture-and-concepts/components/table/segment) ![](/files/-M1cHtSieWXlok7s8_HW) ## Set up a Pinot cluster Typically, there is only one cluster per environment/data center. There is no need to create multiple Pinot clusters because Pinot supports [tenants](/architecture-and-concepts/components/cluster/tenant). To set up a cluster, see one of the following guides: * [Running Pinot in Docker](/start-here/install/docker) * [Running Pinot locally](/start-here/install/local) # Tenant Discover the tenant component of Apache Pinot, which facilitates efficient data isolation and resource management within Pinot clusters. Every table is associated with a *tenant*, or a logical namespace that restricts where the cluster processes queries on the table. A Pinot tenant takes the form of a text tag in the logical tenant namespace. Physical cluster hardware resources (i.e., [brokers](/architecture-and-concepts/components/cluster/broker) and [servers](/architecture-and-concepts/components/cluster/server)) are also associated with a tenant tag in the common tenant namespace. Tables of a particular tenant tag will only be scheduled for storage and query processing on hardware resources that belong to the same tenant tag. This lets Pinot cluster operators assign specified workloads to certain hardware resources, preventing data in separate workloads from being stored or processed on the same physical hardware. By default, all tables, brokers, and servers belong to a tenant called *DefaultTenant*, but you can configure multiple tenants in a Pinot cluster. If the cluster is planned to have multiple tenants, consider setting `cluster.tenant.isolation.enable=false` so that servers and brokers won't be tagged with *DefaultTenant* automatically while added into the cluster. To support multi-tenancy, Pinot has first-class support for tenants. Every table is associated with a server tenant and a broker tenant, which controls the nodes used by the table as servers and brokers. Multi-tenancy lets Pinot group all tables belonging to a particular use case under a single tenant name. The concept of tenants is very important when the multiple use cases are using Pinot and there is a need to provide quotas or some sort of isolation across tenants. For example, consider we have two tables `Table A` and `Table B` in the same Pinot cluster. ![Defining tenants for tables](/files/-M1cK7krxr6ZraoKsRgf) We can configure `Table A` with server tenant `Tenant A` and `Table B` with server tenant `Tenant B`. We can tag some of the server nodes for `Tenant A` and some for `Tenant B`. This will ensure that segments of `Table A` only reside on servers tagged with `Tenant A`, and segment of `Table B` only reside on servers tagged with `Tenant B`. The same isolation can be achieved at the broker level, by configuring broker tenants to the tables. ![Table isolation using tenants](/files/-M1cKrY5Sp3RxtA003-U) No need to create separate clusters for every table or use case! ## Tenant configuration This tenant is defined in the [tenants](/architecture-and-concepts/components/table#tenants) section of the table config. This section contains two main fields `broker` and `server` , which decide the tenants used for the broker and server components of this table. ```javascript "tenants": { "broker": "brokerTenantName", "server": "serverTenantName" } ``` In the above example: * The table will be served by brokers that have been tagged as `brokerTenantName_BROKER` in Helix. * If this were an offline table, the offline segments for the table will be hosted in Pinot servers tagged in Helix as `serverTenantName_OFFLINE` * If this were a real-time table, the real-time segments (both consuming as well as completed ones) will be hosted in pinot servers tagged in Helix as `serverTenantName_REALTIME`. ## Create a tenant `POST /tenants` creates a tenant by tagging **currently untagged** broker or server instances. The JSON body maps to the `Tenant` config: | Field | Broker | Server | Notes | | ------------------- | ------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `tenantRole` | required (`BROKER`) | required (`SERVER`) | Role selects which instance pool is tagged. | | `tenantName` | required | required | Logical tenant name (Helix tags use `_BROKER`, `_OFFLINE`, `_REALTIME` suffixes). | | `numberOfInstances` | required | required | Total untagged instances to allocate for this tenant. Defaults to `0` if omitted, which causes server create to fail with errors such as `Cannot request more offline instances ... than total instances: 0`. | | `offlineInstances` | n/a | required for server tenants | How many of the allocated servers receive the `{tenant}_OFFLINE` tag. | | `realtimeInstances` | n/a | required for server tenants | How many of the allocated servers receive the `{tenant}_REALTIME` tag. | For **server** tenants, Pinot validates: * `numberOfInstances >= offlineInstances` * `numberOfInstances >= realtimeInstances` Those checks are **per tag type**, not against `offlineInstances + realtimeInstances`. When `offlineInstances + realtimeInstances > numberOfInstances` but each count still fits in `numberOfInstances`, Pinot co-locates tags: the same physical server can receive both `{tenant}_OFFLINE` and `{tenant}_REALTIME`. Creation still needs at least `numberOfInstances` **untagged** online servers (or brokers for broker tenants). Already-tagged instances are not reassigned by `POST /tenants`. ### Broker tenant Here's a sample broker tenant config. This creates broker tenant `sampleBrokerTenant` by tagging three untagged broker nodes as `sampleBrokerTenant_BROKER`. {% code title="sample-broker-tenant.json" %} ```javascript { "tenantRole": "BROKER", "tenantName": "sampleBrokerTenant", "numberOfInstances": 3 } ``` {% endcode %} Creation fails if the number of untagged broker nodes is less than `numberOfInstances`. {% tabs %} {% tab title="pinot-admin.sh" %} Follow instructions in [Getting Pinot](/start-here/install/local#1-download-or-build-apache-pinot) to get Pinot locally, and then: ```bash bin/pinot-admin.sh AddTenant \ -name sampleBrokerTenant \ -role BROKER \ -instanceCount 3 \ -exec ``` {% endtab %} {% tab title="curl" %} ```bash curl -i -X POST -H 'Content-Type: application/json' \ -d @sample-broker-tenant.json \ http://localhost:9000/tenants ``` {% endtab %} {% endtabs %} Check out the tenants list in the [Rest API](http://localhost:9000/help#!/Tenant/getAllTenants) to make sure the tenant was created. ### Server tenant Here's a sample server tenant config. With `numberOfInstances: 2`, this tags one untagged server as `sampleServerTenant_OFFLINE` and another as `sampleServerTenant_REALTIME`. {% code title="sample-server-tenant.json" %} ```javascript { "tenantRole": "SERVER", "tenantName": "sampleServerTenant", "numberOfInstances": 2, "offlineInstances": 1, "realtimeInstances": 1 } ``` {% endcode %} **Co-located offline and realtime on one server** (one untagged server receives both tags): {% code title="sample-server-tenant-colocated.json" %} ```javascript { "tenantRole": "SERVER", "tenantName": "sampleServerTenant", "numberOfInstances": 1, "offlineInstances": 1, "realtimeInstances": 1 } ``` {% endcode %} Creation fails if there are fewer than `numberOfInstances` untagged server nodes, or if `offlineInstances` or `realtimeInstances` is greater than `numberOfInstances`. {% tabs %} {% tab title="pinot-admin.sh" %} Follow instructions in [Getting Pinot](/start-here/install/local#1-download-or-build-apache-pinot) to get Pinot locally, and then: ```bash bin/pinot-admin.sh AddTenant \ -name sampleServerTenant \ -role SERVER \ -instanceCount 2 \ -offlineInstanceCount 1 \ -realTimeInstanceCount 1 \ -exec ``` `-instanceCount` is required for every `AddTenant` call. For `SERVER` role, `-offlineInstanceCount` and `-realTimeInstanceCount` are also required (note the capital `T` in `-realTimeInstanceCount`). {% endtab %} {% tab title="curl" %} ```bash curl -i -X POST -H 'Content-Type: application/json' \ -d @sample-server-tenant.json \ http://localhost:9000/tenants ``` {% endtab %} {% endtabs %} Check out the tenants list in the [Rest API](http://localhost:9000/help#!/Tenant/getAllTenants) to make sure the tenant was created. ### Tagging instances without untagged capacity `POST /tenants` only consumes the untagged broker/server pools. If servers are already tagged (for example with `DefaultTenant_OFFLINE`) and you want to move or add tenant tags on those instances, update the instance tags directly instead of expecting `POST /tenants` to re-tag them: ```bash curl -i -X PUT \ "http://localhost:9000/instances/Server_host1_8098/updateTags?tags=sampleServerTenant_OFFLINE,sampleServerTenant_REALTIME" ``` You can also set tags when adding or updating an instance via the Instances APIs. A server may hold more than one tag (for example both `_OFFLINE` and `_REALTIME`, or tags for more than one tenant) when your isolation model allows it. To grow or shrink an existing tenant after creation, use `PUT /tenants` with the same `Tenant` payload shape (including `numberOfInstances` for both roles, and offline/realtime counts for servers). # Server Uncover the efficient data processing and storage capabilities of Apache Pinot's server component, optimizing performance for data-driven applications. Pinot servers provide the primary storage for [segments](/architecture-and-concepts/components/table/segment) and perform the computation required to execute queries. A production Pinot cluster contains many servers. In general, the more servers, the more data the cluster can retain in tables, the lower latency the cluster can deliver on queries, and the more concurrent queries the cluster can process. Servers are typically segregated into real-time and offline workloads, with "real-time" servers hosting only real-time tables, and "offline" servers hosting only offline tables. This is a ubiquitous operational convention, not a difference or an explicit configuration in the server process itself. There are two types of servers: ## Offline Offline servers are responsible for downloading segments from the segment store, to host and serve queries off. When a new segment is uploaded to the controller, the controller decides the servers (as many as replication) that will host the new segment and notifies them to download the segment from the segment store. On receiving this notification, the servers download the segment file and load the segment onto the server, to server queries off them. ![](/files/-M1cBoDyJPidtPVN1ZEA) ## Real-time Real-time servers directly ingest from a real-time stream (such as Kafka or EventHubs). Periodically, they make segments of the in-memory ingested data, based on certain thresholds. This segment is then persisted onto the segment store. ![](/files/-M1cD4trRQ2yIC5LAmvJ) Pinot servers are modeled as Helix participants, hosting Pinot tables (referred to as *resources* in Helix terminology). Segments of a table are modeled as Helix partitions (of a resource). Thus, a Pinot server hosts one or more Helix partitions of one or more helix resources (*i.e.* one or more segments of one or more tables). ## Starting a server Make sure you've [set up Zookeeper](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). If you're using Docker, make sure to [pull the Pinot Docker image](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). To start a server: ``` Usage: StartServer -serverHost : Host name for controller. (required=false) -serverPort : Port number to start the server at. (required=false) -serverAdminPort : Port number to serve the server admin API at. (required=false) -dataDir : Path to directory containing data. (required=false) -segmentDir : Path to directory containing segments. (required=false) -zkAddress : Http address of Zookeeper. (required=false) -clusterName : Pinot cluster name. (required=false) -configFileName : Broker Starter Config file. (required=false) -help : Print this message. (required=false) ``` {% tabs %} {% tab title="Docker Image" %} ``` docker run \ --network=pinot-demo \ --name pinot-server \ -d ${PINOT_IMAGE} StartServer \ -zkAddress pinot-zookeeper:2181 ``` {% endtab %} {% tab title="Launcher Scripts" %} ``` bin/pinot-admin.sh StartServer \ -zkAddress localhost:2181 ``` {% endtab %} {% endtabs %} # Controller Discover the controller component of Apache Pinot, enabling efficient data and query management. The Pinot controller schedules and reschedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, the Pinot controller schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (for example, ingest of [real-time tables](/reference/configuration-reference/table#real-time-table-config) and [offline tables](/reference/configuration-reference/table#offline-table)). The Pinot controller can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration. The controller exposes a [REST API endpoint](/reference/api-reference/controller-api) for cluster-wide administrative operations as well as a web-based query console for interactive SQL queries, time-series queries, and simple administrative tasks. The Pinot controller is responsible for the following: * Maintaining **global metadata** (e.g., configs and schemas) of the system with the help of Zookeeper which is used as the persistent metadata store. * Hosting the **Helix Controller** and managing other Pinot components (brokers, servers, minions) * Maintaining the **mapping of which servers are responsible for which segments**. This mapping is used by the servers to download the portion of the segments that they are responsible for. This mapping is also used by the broker to decide which servers to route the queries to. * Serving **admin endpoints** for viewing, creating, updating, and deleting configs, which are used to manage and operate the cluster. * Serving endpoints for **segment uploads**, which are used in offline data pushes. They are responsible for initializing **real-time consumption** and coordination of persisting real-time segments into the segment store periodically. * Undertaking other **management activities** such as managing retention of segments, validations. For redundancy, there can be multiple instances of Pinot controllers. Pinot expects that all controllers are configured with the same back-end storage system so that they have a common view of the segments (*e.g.* NFS). Pinot can use other storage systems such as HDFS or [ADLS](https://azure.microsoft.com/en-us/services/storage/data-lake-storage/). ## Running the periodic task manually The controller runs several periodic tasks in the background, to perform activities such as management and validation. Each periodic task has [its own configuration](/reference/configuration-reference/controller#periodic-task-configuration) to define either a fixed `frequencyPeriod` or, for supported tasks, a Quartz `cronExpression`. If both are set for a task, Pinot uses the cron schedule. See the configuration reference for cron syntax and copyable examples. Each task can also be triggered manually if needed. The task runs on the lead controller for each table. For period task configuration details, see [Controller configuration reference](/reference/configuration-reference/controller#periodic-task-configuration). Use the `GET /periodictask/names` API to fetch the names of all the periodic tasks running on your Pinot cluster. ``` curl -X GET "http://localhost:9000/periodictask/names" -H "accept: application/json" [ "RetentionManager", "OfflineSegmentIntervalChecker", "RealtimeSegmentValidationManager", "BrokerResourceValidationManager", "SegmentStatusChecker", "SegmentRelocator", "StaleInstancesCleanupTask", "TaskMetricsEmitter" ] ``` To manually run a named periodic task, use the `GET /periodictask/run` API: ``` curl -X GET "http://localhost:9000/periodictask/run?taskname=SegmentStatusChecker&tableName=jsontypetable&type=OFFLINE" -H "accept: application/json" { "Log Request Id": "api-09630c07", "Controllers notified": true } ``` The `Log Request Id` (`api-09630c07`) can be used to search through pinot-controller log file to see log entries related to execution of the Periodic task that was manually run. If `tableName` (and its type `OFFLINE` or `REALTIME`) is not provided, the task will run against all tables. ## Starting a controller Make sure you've [set up Zookeeper](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). If you're using Docker, make sure to [pull the Pinot Docker image](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). To start a controller: {% tabs %} {% tab title="Docker Image" %} ``` docker run \ --network=pinot-demo \ --name pinot-controller \ -p 9000:9000 \ -d ${PINOT_IMAGE} StartController \ -zkAddress pinot-zookeeper:2181 ``` {% endtab %} {% tab title="Launcher Scripts" %} ``` bin/pinot-admin.sh StartController \ -zkAddress localhost:2181 \ -clusterName PinotCluster \ -controllerPort 9000 ``` {% endtab %} {% endtabs %} # Broker Discover how Apache Pinot's broker component optimizes query processing, data retrieval, and enhances data-driven applications. Pinot brokers take query requests from client processes, scatter them to applicable servers, gather the results, and return results to the client. The controller shares cluster metadata with the brokers, which allows the brokers to create a plan for executing the query involving a minimal subset of servers with the source data and, when required, other servers to shuffle and consolidate results. A production Pinot cluster contains many brokers. In general, the more brokers, the more concurrent queries a cluster can process, and the lower latency it can deliver on queries. ![Broker interaction with other components](/files/-M1c97qmI9TI8SSD0-5a) Pinot brokers are modeled as Helix **spectators**. They need to know the location of each segment of a table (and each replica of the segments) and route requests to the appropriate server that hosts the segments of the table being queried. The broker ensures that all the rows of the table are queried exactly once so as to return correct, consistent results for a query. The brokers may optimize to **prune some of the segments** as long as accuracy is not sacrificed. Helix provides the framework by which spectators can learn the location in which each partition of a resource (*i.e.* participant) resides. The brokers use this mechanism to learn the servers that host specific segments of a table. In the case of hybrid tables, the brokers ensure that the overlap between real-time and offline segment data is queried exactly once, by performing **offline and real-time federation**. Let's take this example, we have real-time data for five days - March 23 to March 27, and offline data has been pushed until Mar 25, which is two days behind real-time. The brokers maintain this time boundary. ![](/files/-M1Y6WPgBfIM-iC7cHq3) Suppose, we get a query to this table : `select sum(metric) from table`. The broker will split the query into 2 queries based on this time boundary – one for offline and one for real-time. This query becomes `select sum(metric) from table_REALTIME where date >= Mar 25`\ and `select sum(metric) from table_OFFLINE where date < Mar 25` \ The broker merges results from both these queries before returning the result to the client. ## Starting a broker Make sure you've [set up Zookeeper](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). If you're using Docker, make sure to [pull the ](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster)[Pinot Docker image](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). To start a broker: {% tabs %} {% tab title="Docker Image" %} ``` docker run \ --network=pinot-demo \ --name pinot-broker \ -d ${PINOT_IMAGE} StartBroker \ -zkAddress pinot-zookeeper:2181 ``` {% endtab %} {% tab title="Launcher Script" %} ``` bin/pinot-admin.sh StartBroker \ -zkAddress localhost:2181 \ -clusterName PinotCluster \ -brokerPort 7000 ``` {% endtab %} {% endtabs %} # Minion Explore the minion component in Apache Pinot, empowering efficient data movement and segment generation within Pinot clusters. A Pinot minion is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing *minion tasks* as directed by the controller. Examples of minon tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically. Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function with or without minions, they are typically present to support routine tasks like batch data ingest. ## Starting a minion Make sure you've [set up Zookeeper](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). If you're using Docker, make sure to [pull the Pinot Docker image](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster). To start a minion: ``` Usage: StartMinion -help : Print this message. (required=false) -minionHost : Host name for minion. (required=false) -minionPort : Port number to start the minion at. (required=false) -zkAddress : HTTP address of Zookeeper. (required=false) -clusterName : Pinot cluster name. (required=false) -configFileName : Minion Starter Config file. (required=false) ``` {% tabs %} {% tab title="Docker Image" %} ``` docker run \ --network=pinot-demo \ --name pinot-minion \ -d ${PINOT_IMAGE} StartMinion \ -zkAddress pinot-zookeeper:2181 ``` {% endtab %} {% tab title="Launcher Scripts" %} ``` bin/pinot-admin.sh StartMinion \ -zkAddress localhost:2181 ``` {% endtab %} {% endtabs %} ## Interfaces ![](/files/-Maelat1Ve1MbniPgah6) ### Pinot task generator The Pinot task generator interface defines the APIs for the controller to generate tasks for minions to execute. ```java {% hint style="warning" %} **Duplicate Keys in Configuration File** Starting from Apache Pinot 1.3.0, duplicate keys in the minion configuration file will cause a `ConfigurationException` to be thrown during startup. Previously, duplicate keys would be silently merged into a list. If you encounter this error, ensure that each configuration property appears only once in your configuration file. The exception will include the exact file path, duplicate key name, and the line numbers where the duplicates were found. Example error: ``` ConfigurationException: Duplicate key found in /path/to/minion.conf at line 10 and line 15: pinot.minion.task.allow\.download.from.server ``` {% endhint %} public interface PinotTaskGenerator { /** * Initializes the task generator. */ void init(ClusterInfoAccessor clusterInfoAccessor); /** * Returns the task type of the generator. */ String getTaskType(); /** * Generates a list of tasks to schedule based on the given table configs. */ List generateTasks(List tableConfigs); /** * Returns the timeout in milliseconds for each task, 3600000 (1 hour) by default. */ default long getTaskTimeoutMs() { return JobConfig.DEFAULT_TIMEOUT_PER_TASK; } /** * Returns the maximum number of concurrent tasks allowed per instance, 1 by default. */ default int getNumConcurrentTasksPerInstance() { return JobConfig.DEFAULT_NUM_CONCURRENT_TASKS_PER_INSTANCE; } /** * Performs necessary cleanups (e.g. remove metrics) when the controller leadership changes. */ default void nonLeaderCleanUp() { } } ``` ### PinotTaskExecutorFactory Factory for `PinotTaskExecutor` which defines the APIs for Minion to execute the tasks. ```java public interface PinotTaskExecutorFactory { /** * Initializes the task executor factory. */ void init(MinionTaskZkMetadataManager zkMetadataManager); /** * Returns the task type of the executor. */ String getTaskType(); /** * Creates a new task executor. */ PinotTaskExecutor create(); } ``` ```java public interface PinotTaskExecutor { /** * Executes the task based on the given task config and returns the execution result. */ Object executeTask(PinotTaskConfig pinotTaskConfig) throws Exception; /** * Tries to cancel the task. */ void cancel(); } ``` ### MinionEventObserverFactory Factory for `MinionEventObserver` which defines the APIs for task event callbacks on minion. ```java public interface MinionEventObserverFactory { /** * Initializes the task executor factory. */ void init(MinionTaskZkMetadataManager zkMetadataManager); /** * Returns the task type of the event observer. */ String getTaskType(); /** * Creates a new task event observer. */ MinionEventObserver create(); } ``` ```java public interface MinionEventObserver { /** * Invoked when a minion task starts. * * @param pinotTaskConfig Pinot task config */ void notifyTaskStart(PinotTaskConfig pinotTaskConfig); /** * Invoked when a minion task succeeds. * * @param pinotTaskConfig Pinot task config * @param executionResult Execution result */ void notifyTaskSuccess(PinotTaskConfig pinotTaskConfig, @Nullable Object executionResult); /** * Invoked when a minion task gets cancelled. * * @param pinotTaskConfig Pinot task config */ void notifyTaskCancelled(PinotTaskConfig pinotTaskConfig); /** * Invoked when a minion task encounters exception. * * @param pinotTaskConfig Pinot task config * @param exception Exception encountered during execution */ void notifyTaskError(PinotTaskConfig pinotTaskConfig, Exception exception); } ``` ## Built-in tasks Pinot ships with the following built-in Minion tasks: | Task | Purpose | Table Types | | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------- | | [SegmentGenerationAndPushTask](/operate-pinot/segment-management/segment-generation-and-push-task) | Batch ingestion: reads raw data files and converts them into Pinot segments | OFFLINE | | [RealtimeToOfflineSegmentsTask](/operate-pinot/segment-management/pinot-managed-offline-flows) | Converts completed real-time segments into optimized offline segments | REALTIME to OFFLINE | | [MergeRollupTask](/operate-pinot/segment-management/minion-merge-rollup-task) | Merges small segments into larger ones and optionally rolls up data at coarser granularity | OFFLINE, REALTIME (without upsert/dedup) | | [PurgeTask](/operate-pinot/segment-management/purge-task) | Removes or modifies records for data retention and compliance (e.g., GDPR) | OFFLINE, REALTIME | | [RefreshSegmentTask](/operate-pinot/segment-management/refresh-segment-task) | Reprocesses segments after table config or schema changes (new indexes, columns, data types) | OFFLINE, REALTIME | | [UpsertCompactionTask](/operate-pinot/segment-management/upsert-compaction-task) | Compacts individual upsert segments by removing invalidated records | REALTIME (upsert only) | | [UpsertCompactMergeTask](/operate-pinot/segment-management/upsert-compact-merge-task) | Merges multiple small upsert segments into larger ones to reduce segment count | REALTIME (upsert only) | {% hint style="info" %} `PurgeTask`, `RefreshSegmentTask`, and `UpsertCompactionTask` all rebuild a single segment and upload the replacement segment. If that upload fails, Pinot marks the task attempt as failed instead of reporting success, so the Minion task framework can retry it. {% endhint %} ### SegmentGenerationAndPushTask The SegmentGenerationAndPushTask can fetch files from an input folder (e.g. from an S3 bucket) and convert them into segments. It converts one file into one segment and keeps the file name in segment metadata to avoid duplicate ingestion. See [SegmentGenerationAndPushTask runbook](/operate-pinot/segment-management/segment-generation-and-push-task) for full configuration details. Below is an example task config to put in TableConfig to enable this task. The task is scheduled every 10min to keep ingesting remaining files, with 10 parallel task at max and 1 file per task. NOTE: You may want to simply omit "tableMaxNumTasks" due to this caveat: the task generates one segment per file, and derives segment name based on the time column of the file. If two files happen to have same time range and are ingested by tasks from different schedules, there might be segment name conflict. To overcome this issue for now, you can omit “tableMaxNumTasks” and by default it’s Integer.MAX\_VALUE, meaning to schedule as many tasks as possible to ingest all input files in a single batch. Within one batch, a sequence number suffix is used to ensure no segment name conflict. Because the sequence number suffix is scoped within one batch, tasks from different batches might encounter segment name conflict issue said above. {% hint style="info" %} When performing ingestion at scale remember that Pinot will list all of the files contained in the \`inputDirURI\` every time a \`SegmentGenerationAndPushTask\` job gets scheduled. This could become a bottleneck when fetching files from a cloud bucket like GCS. To prevent this make \`inputDirURI\` point to the least number of files possible. {% endhint %} ``` "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY", "batchConfigMaps": [ { "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "us-west-2", "input.fs.prop.secretKey": "....", "input.fs.prop.accessKey": "....", "inputDirURI": "s3://my.s3.bucket/batch/airlineStats/rawdata/", "includeFileNamePattern": "glob:**/*.avro", "excludeFileNamePattern": "glob:**/*.tmp", "inputFormat": "avro" } ] } }, "task": { "taskTypeConfigsMap": { "SegmentGenerationAndPushTask": { "schedule": "0 */10 * * * ?", "tableMaxNumTasks": "10" } } } ``` ### RealtimeToOfflineSegmentsTask See [Pinot managed Offline flows](/operate-pinot/segment-management/pinot-managed-offline-flows) for details. ### MergeRollupTask See [Minion merge rollup task](/operate-pinot/segment-management/minion-merge-rollup-task) for details. ### PurgeTask See [PurgeTask runbook](/operate-pinot/segment-management/purge-task) for details. ### RefreshSegmentTask See [RefreshSegmentTask runbook](/operate-pinot/segment-management/refresh-segment-task) for details. ### UpsertCompactionTask See [UpsertCompactionTask runbook](/operate-pinot/segment-management/upsert-compaction-task) for details. ### UpsertCompactMergeTask See [UpsertCompactMergeTask runbook](/operate-pinot/segment-management/upsert-compact-merge-task) for details. ## Enable tasks Tasks are enabled on a per-table basis. To enable a certain task type (e.g. `myTask`) on a table, update the table config to include the task type: ```javascript { ... "task": { "taskTypeConfigsMap": { "myTask": { "myProperty1": "value1", "myProperty2": "value2" } } } } ``` Under each enable task type, custom properties can be configured for the task type. You can also override how Pinot schedules task generation for a table by setting `concurrentSchedulingEnabled` in the same `task` block: ```javascript { ... "task": { "concurrentSchedulingEnabled": true, "taskTypeConfigsMap": { "myTask": { "myProperty1": "value1" } } } } ``` Use `concurrentSchedulingEnabled` as follows: * `null` or omitted: inherit the cluster default from `controller.task.concurrentSchedulingEnabled` * `true`: opt this table into concurrent task scheduling * `false`: force the legacy serialized scheduling path for this table, even if the cluster default is concurrent There are also two task configs to be set as part of cluster configs like below. One controls task's overall timeout (1hr by default) and one for how many tasks to run on a single minion worker (1 by default). ``` Using "POST /cluster/configs" API on CLUSTER tab in Swagger, with this payload { "RealtimeToOfflineSegmentsTask.timeoutMs": "600000", "RealtimeToOfflineSegmentsTask.numConcurrentTasksPerInstance": "4" } ``` ## Schedule tasks ### Auto-schedule There are 2 ways to enable task scheduling: #### Controller level schedule for all minion tasks Tasks can be scheduled periodically for all task types on all enabled tables. Enable auto task scheduling by configuring the schedule frequency in the controller config with the key `controller.task.frequencyPeriod`. This takes period strings as values, e.g. 2h, 30m, 1d. To let PinotTaskManager generate tasks for different tables in parallel, enable distributed locking first and then enable concurrent scheduling: ```properties controller.task.enableDistributedLocking=true controller.task.concurrentSchedulingEnabled=true ``` If you want to keep the cluster default serialized, leave `controller.task.concurrentSchedulingEnabled=false` and opt individual tables in with `task.concurrentSchedulingEnabled=true`. Pinot uses the concurrent path only when every table targeted by a scheduling request resolves to concurrent scheduling. #### Per table and task level schedule Tasks can also be scheduled based on cron expressions. The cron expression is set in the `schedule` config for each task type separately. This config in the controller config, `controller.task.scheduler.enabled` should be set to `true` to enable cron scheduling. As shown below, the RealtimeToOfflineSegmentsTask will be scheduled at the first second of every minute (following the syntax [defined here](http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html)). ``` "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "1h", "bufferTimePeriod": "1h", "schedule": "0 * * * * ?" } } }, ``` ### Manual schedule Tasks can be manually scheduled using the following controller rest APIs: | Rest API | Description | | -------------------------------------------------------------------- | ------------------------------------------------------------ | | **POST /tasks/schedule** | Schedule tasks for all task types on all enabled tables | | **POST /tasks/schedule?taskType=myTask** | Schedule tasks for the given task type on all enabled tables | | **POST /tasks/schedule?tableName=myTable\_OFFLINE** | Schedule tasks for all task types on the given table | | **POST /tasks/schedule?taskType=myTask\&tableName=myTable\_OFFLINE** | Schedule tasks for the given task type on the given table | ### Schedule task on specific instances Tasks can be scheduled on specific instances using the following config at task level: ``` "task": { "taskTypeConfigsMap": { "RealtimeToOfflineSegmentsTask": { "bucketTimePeriod": "1h", "bufferTimePeriod": "1h", "schedule": "0 * * * * ?", "minionInstanceTag": "tag1_MINION" } } }, ``` By default, the value is `minion_untagged` to have backward-compatibility. This will allow users to schedule tasks on specific nodes and isolate tasks among tables / task-types. | Rest API | Description | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | **POST /tasks/schedule?taskType=myTask\&tableName=myTable\_OFFLINE\&minionInstanceTag=tag1\_MINION** | Schedule tasks for the given task type of the given table on the minion nodes tagged as tag1\_MINION. | ## Task level advanced configs ### allowDownloadFromServer When a task is executed on a segment, the minion node fetches the segment from deepstore. If the deepstore is not accessible, the minion node can download the segment from the server node. This is controlled by the `allowDownloadFromServer` config in the task config. By default, this is set to `false`. We can also set this config at a minion instance level `pinot.minion.task.allow.download.from.server` (default is `false`). This instance level config helps in enforcing this behaviour if the number of tables / tasks is pretty high and we want to enable for all. Note: task-level config will override instance-level config value. ## Plug-in custom tasks To plug in a custom task, implement `PinotTaskGenerator`, `PinotTaskExecutorFactory` and `MinionEventObserverFactory` (optional) for the task type (all of them should return the same string for `getTaskType()`), and annotate them with the following annotations: | Implementation | Annotation | | -------------------------- | --------------------- | | PinotTaskGenerator | @TaskGenerator | | PinotTaskExecutorFactory | @TaskExecutorFactory | | MinionEventObserverFactory | @EventObserverFactory | After annotating the classes, put them under the package of name `org.apache.pinot.*.plugin.minion.tasks.*`, then they will be auto-registered by the controller and minion. ### Example See [SimpleMinionClusterIntegrationTest](https://github.com/apache/pinot/blob/master/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/SimpleMinionClusterIntegrationTest.java) where the `TestTask` is plugged-in. ## Task Manager UI In the Pinot Data Explorer, select **Minion Tasks** from the left navigation to open the **Minion Task Manager** page. This page focuses on minion queue troubleshooting and task drill-downs. Controller-wide scheduler details live on the [Cluster Manager page](/architecture-and-concepts/components/exploring-pinot), where **Cron Scheduler Information** and **Periodic Tasks** are shown separately. The Minion Task Manager landing page shows four summary tiles: * **Task Types** * **Minion Instances** * **Running Tasks** * **Waiting Tasks** Below the summary tiles is the task-queue table. This table shows which task types are active in Helix and lets you drill into each queue. This one shows which types of Minion Task have been used. Essentially which task types have created their task queues in Helix. ![](/files/AiUho1rhzOSgPCG6qVKy) \*\* Clicking into a task type shows the tables using that task type, along with queue-management actions such as stopping or cleaning up the queue. ![](/files/vmVtsAVHpu0o55wWxokb) \*\* Then clicking into any table in this list, one can see how the task is configured for that table. And the task metadata if there is one in ZK. For example, MergeRollupTask tracks a watermark in ZK. If the task is cron scheduled, the current and next schedules are also shown in this page like below. ![](/files/JQfD6Vv6SjbyhpYWNZOB) \*\* ![](/files/Yda5QBAHuNM1XuyhCkJy) \*\* At the bottom of this page is a list of tasks generated for this table for this specific task type. Like here, one MergeRollup task has been generated and completed. The task list also includes a **Status Filter** control so you can focus on a single task state, and a **Sub Tasks (Total/Completed/Running/Waiting/Error/Other)** column that summarizes the subtasks for each task. The **Other** bucket combines `UNKNOWN`, `DROPPED`, `TIMED_OUT`, and `ABORTED` subtasks. Clicking into a task opens task details including start and finish times, runtime configuration, and an **Operations** accordion with a **Delete Task** action for removing the task and its subtasks from the queue. The task detail page also lists the subtasks generated for that task (as context, one minion task can have multiple subtasks to process data in parallel). The subtask table has its own **Status Filter** control, which is useful when a task fanout creates many subtasks across multiple minion workers. In this example, it happened to have one sub-task here, and it shows when it starts and stops and which minion worker it's running. ![](/files/6LqFsTu2fzRz4ZmjVCPh) \*\* Clicking into this subtask shows more details such as the input task config, progress, and error information if the task failed. If the subtask has already been assigned to a minion worker, the page also includes a **Minion Log Files** panel so you can refresh the file list and download logs from that minion directly in the UI. ![](/files/uQqQbDqTc035fzOEDeCe) \*\* ## Task-related metrics There is a controller job that runs every 5 minutes by default, controlled by `controller.minion.task.metrics.emitter.frequencyPeriod`, and emits metrics about Minion tasks scheduled in Pinot. The following metrics are emitted for each task type: * ***NumMinionTasksInProgress***: Number of running tasks * ***NumMinionSubtasksRunning***: Number of running sub-tasks * ***NumMinionSubtasksWaiting***: Number of waiting sub-tasks (unassigned to a minion as yet) * ***NumMinionSubtasksError***: Number of error sub-tasks (completed with an error/exception) * ***PercentMinionSubtasksInQueue***: Percent of sub-tasks in waiting or running states * ***PercentMinionSubtasksInError***: Percent of sub-tasks in error * ***MaxSubtaskWaitTimeMs***: Per-table, per-task-type controller gauge for the longest current wait time across subtasks in `WAITING`. Pinot emits `0` when no subtasks are waiting, so alerts can self-resolve after the queue drains. * ***MaxSubtaskRunningTimeMs***: Per-table, per-task-type controller gauge for the longest current runtime across subtasks in `RUNNING`. Pinot emits `0` when no subtasks are running. The controller also emits metrics about how tasks are cron scheduled: * **cronSchedulerJobScheduled:** Number of current cron schedules registered to be triggered regularly according their cron expressions. It's a Gauge. * **cronSchedulerJobTrigger:** Number of cron scheduled triggered, as a Meter. * **cronSchedulerJobSkipped:** Number of late cron scheduled skipped, as a Meter. * **cronSchedulerJobExecutionTimeMs:** Time used to complete task generation, as a Timer. For each task, the minion will emit these metrics: * ***TASK\_QUEUEING***: Task queueing time (task\_dequeue\_time - task\_inqueue\_time), assuming the time drift between helix controller and pinot minion is minor, otherwise the value may be negative * ***TASK\_EXECUTION***: Task execution time, which is the time spent on executing the task * ***NUMBER\_OF\_TASKS***: number of tasks in progress on that minion. Whenever a Minion starts a task, increase the Gauge by 1, whenever a Minion completes (either succeeded or failed) a task, decrease it by 1 * **NUMBER\_TASKS*****\_*****EXECUTED:** Number of tasks executed, as a Meter. * **NUMBER\_TASKS*****\_*****COMPLETED:** Number of tasks completed, as a Meter. * **NUMBER\_TASKS*****\_*****CANCELLED:** Number of tasks cancelled, as a Meter. * **NUMBER\_TASKS*****\_*****FAILED:** Number of tasks failed, as a Meter. Different from fatal failure, the task encountered an error which can not be recovered from this run, but it may still succeed by retrying the task. * **NUMBER\_TASKS*****\_*****FATAL*****\_*****FAILED:** Number of tasks fatal failed, as a Meter. Different from failure, the task encountered an error, which will not be recoverable even with retrying the task. # Table Explore the table component in Apache Pinot, a fundamental building block for organizing and managing data in Pinot clusters, enabling effective data processing and analysis. Pinot stores data in tables. A Pinot table is conceptually identical to a relational database table with rows and columns. Columns have the same name and data type, known as the table's [schema](/architecture-and-concepts/components/table/schema). Pinot schemas are defined in a JSON file. Because that schema definition is in its own file, multiple tables can share a single schema. Each table can have a unique name, indexing strategy, partitioning, data sources, and other metadata. Pinot table types include: * **real-time:** Ingests data from a streaming source like Apache Kafka® * **offline:** Loads data from a batch source * **hybrid:** Loads data from both a batch source and a streaming source Pinot breaks a table into multiple [segments](/architecture-and-concepts/components/table/segment) and stores these segments in a deep-store such as Hadoop Distributed File System (HDFS) as well as Pinot servers. In the Pinot cluster, a table is modeled as a [Helix resource](https://helix.apache.org/Concepts.html) and each segment of a table is modeled as a [Helix Partition](https://helix.apache.org/Concepts.html). {% hint style="info" %} Table naming in Pinot follows typical naming conventions, such as starting names with a letter, not ending with an underscore, and using only alphanumeric characters. {% endhint %} Pinot supports the following types of tables: | Type | Description | | ------------- | ----------------------------------------------------------------------------------------------------------------------------- | | **Offline** | Offline tables ingest pre-built Pinot segments from external data stores and are generally used for batch ingestion. | | **Real-time** | Real-time tables ingest data from streams (such as Kafka) and build segments from the consumed data. | | **Hybrid** | Hybrid Pinot tables have both real-time as well as offline tables under the hood. By default, all tables in Pinot are hybrid. | {% hint style="info" %} The user querying the database does not need to know the type of the table. They only need to specify the table name in the query. e.g. regardless of whether we have an offline table `myTable_OFFLINE`, a real-time table `myTable_REALTIME`, or a hybrid table containing both of these, the query will be: ```sql select count(*) from myTable ``` {% endhint %} [Table configuration](/reference/configuration-reference/table) is used to define the table properties, such as name, type, indexing, routing, and retention. It is written in JSON format and is stored in Zookeeper, along with the table schema. Use the following properties to make your tables faster or leaner: * Segment * Indexing * Tenants ## Segments A table is comprised of small chunks of data known as segments. Learn more about how Pinot creates and manages segments [here](/architecture-and-concepts/components/table/segment). For offline tables, segments are built outside of Pinot and uploaded using a distributed executor such as Spark or Hadoop. For details, see [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion). For real-time tables, segments are built in a specific interval inside Pinot. You can tune the following for the real-time segments. ### Flush The Pinot real-time consumer ingests the data, creates the segment, and then flushes the in-memory segment to disk. Pinot allows you to configure when to flush the segment in the following ways: * **Number of consumed rows**: After consuming the specified number of rows from the stream, Pinot will persist the segment to disk. * **Number of rows per segment**: Pinot learns and then estimates the number of rows that need to be consumed. The learning phase starts by setting the number of rows to 100,000 (this value can be changed) and adjusts it to reach the appropriate segment size. Because Pinot corrects the estimate as it goes along, the segment size might go significantly over the correct size during the learning phase. You should set this value to optimize the performance of queries. * **Max time duration to wait**: Pinot consumers wait for the configured time duration after which segments are persisted to the disk. **Replicas**\ A segment can have multiple replicas to provide higher availability. You can configure the number of replicas for a table segment [using the CLI](/operate-pinot/deployment/cli#change-num-replicas). **Completion Mode**\ By default, if the in-memory segment in the [non-winner server](/architecture-and-concepts/components/cluster/server) is equivalent to the committed segment, then the non-winner server builds and replaces the segment. If the available segment is not equivalent to the committed segment, the server just downloads the committed segment from the controller. However, in certain scenarios, the segment build can get very memory-intensive. In these cases, you might want to enforce the non-committer servers to just download the segment from the controller instead of building it again. You can do this by setting `completionMode: "DOWNLOAD"` in the table configuration. For details, see [Completion Config](https://docs.pinot.apache.org/architecture-and-concepts/components/pages/-LwAZ7-lfgk5mLEIMIyL#controlling-segment-build-vs.-segment-download-on-real-time-servers). **Download Scheme** A Pinot server might fail to download segments from the deep store, such as HDFS, after its completion. However, you can configure servers to download these segments from peer servers instead of the deep store. Currently, only HTTP and HTTPS download schemes are supported. More methods, such as gRPC/Thrift, are planned be added in the future. For more details about peer segment download during real-time ingestion, refer to this design doc on [bypass deep store for segment completion.](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#BypassingdeepstorerequirementforRealtimesegmentcompletion-Configchange) ## Indexing You can create multiple indices on a table to increase the performance of the queries. The following types of indices are supported: * [Forward Index](/build-with-pinot/indexing/forward-index) * Dictionary-encoded forward index with bit compression * Raw value forward index * Sorted forward index with run-length encoding * [Inverted Index](/build-with-pinot/indexing/inverted-index) * Bitmap inverted index * Sorted inverted index * [Star-tree Index](/build-with-pinot/indexing/star-tree-index) * [Range Index](/build-with-pinot/indexing/range-index) * [Text Index](/build-with-pinot/indexing/text-search-support) * [Geospatial](/build-with-pinot/indexing/geospatial-support) For more details on each indexing mechanism and corresponding configurations, see [Indexing](/build-with-pinot/indexing). Set up [Bloomfilters](/build-with-pinot/indexing/bloom-filter) on columns to make queries faster. You can also keep segments in off-heap instead of on-heap memory for faster queries. ### Pre-aggregation Aggregate the real-time stream data as it is consumed to reduce segment sizes. We add the metric column values of all rows that have the same values for all dimension and time columns and create a single row in the segment. This feature is only available on `REALTIME` tables. The only supported aggregation is `SUM`. The columns to pre-aggregate need to satisfy the following requirements: * All metrics should be listed in `noDictionaryColumns`. * No multi-value dimensions * All dimension columns are treated to have a dictionary, even if they appear as `noDictionaryColumns` in the config. The following table config snippet shows an example of enabling pre-aggregation during real-time ingestion: {% code title="pinot-table-realtime.json" %} ```javascript "tableIndexConfig": { "noDictionaryColumns": ["metric1", "metric2"], "aggregateMetrics": true, ... } ``` {% endcode %} ## Tenants Each table is associated with a tenant. A segment resides on the server, which has the same tenant as itself. For details, see [Tenant](/architecture-and-concepts/components/cluster/tenant). Optionally, override if a table should move to a server with different tenant based on segment status. The example below adds a `tagOverrideConfig` under the `tenants` section for real-time tables to override tags for consuming and completed segments. ```javascript "broker": "brokerTenantName", "server": "serverTenantName", "tagOverrideConfig" : { "realtimeConsuming" : "serverTenantName_REALTIME" "realtimeCompleted" : "serverTenantName_OFFLINE" } } ``` In the above example, the consuming segments will still be assigned to `serverTenantName_REALTIME` hosts, but once they are completed, the segments will be moved to `serverTenantName_OFFLINE`. You can specify the full name of *any* tag in this section. For example, you could decide that completed segments for this table should be in Pinot servers tagged as `allTables_COMPLETED`). To learn more about, see the [Moving Completed Segments](/operate-pinot/tuning/realtime#moving-completed-segments-to-different-hosts) section. ## Hybrid table A hybrid table is a table composed of two tables, one offline and one real-time, that share the same name. In a hybrid table, offline segments can be pushed periodically. The retention on the offline table can be set to a high value because segments are coming in on a periodic basis, whereas the retention on the real-time part can be small. Once an offline segment is pushed to cover a recent time period, the brokers automatically switch to using the offline table for segments for that time period and use the real-time table only for data not available in the offline table. To learn how time boundaries work for hybrid tables, see [Broker](/architecture-and-concepts/components/cluster/broker). A typical use case for hybrid tables is pushing deduplicated, cleaned-up data into an offline table every day while consuming real-time data as it arrives. Data can remain in offline tables for as long as a few years, while the real-time data would be cleaned every few days. ## Examples Create a table config for your data, or see [`examples`](https://github.com/apache/pinot/tree/master/pinot-tools/src/main/resources/examples) for all possible batch/streaming tables. **Prerequisites** * [Set up the cluster](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster) * [Create broker and server tenants](/architecture-and-concepts/components/cluster/tenant#create-a-tenant) ## Offline table creation {% tabs %} {% tab title="Docker" %} ```bash docker run \ --network=pinot-demo \ --name pinot-batch-table-creation \ ${PINOT_IMAGE} AddTable \ -schemaFile examples/batch/airlineStats/airlineStats_schema.json \ -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json \ -controllerHost pinot-controller \ -controllerPort 9000 \ -exec ``` **Sample console output** ```bash Executing command: AddTable -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json -schemaFile examples/batch/airlineStats/airlineStats_schema.json -controllerHost pinot-controller -controllerPort 9000 -exec Sending request: http://pinot-controller:9000/schemas to controller: a413b0013806, version: Unknown {"status":"Table airlineStats_OFFLINE succesfully added"} ``` {% endtab %} {% tab title="Using launcher scripts" %} ```bash bin/pinot-admin.sh AddTable \ -schemaFile examples/batch/airlineStats/airlineStats_schema.json \ -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json \ -exec ``` {% endtab %} {% tab title="curl" %} ```bash # add schema curl -F schemaName=@airlineStats_schema.json localhost:9000/schemas # add table curl -i -X POST -H 'Content-Type: application/json' \ -d @airlineStats_offline_table_config.json localhost:9000/tables ``` {% endtab %} {% endtabs %} Check out the table config in the [Rest API](http://localhost:9000/help#!/Table/alterTableStateOrListTableConfig) to make sure it was successfully uploaded. ## Streaming table creation {% tabs %} {% tab title="Docker" %} **Start Kafka** ``` docker run \ --network pinot-demo --name=kafka \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \ -d apache/kafka:4.0.0 ``` **Create a Kafka topic** ``` docker exec \ -t kafka \ /opt/kafka/bin/kafka-topics.sh \ --bootstrap-server kafka:9092 \ --partitions=1 --replication-factor=1 \ --create --topic flights-realtime ``` **Create a streaming table** ``` docker run \ --network=pinot-demo \ --name pinot-streaming-table-creation \ ${PINOT_IMAGE} AddTable \ -schemaFile examples/stream/airlineStats/airlineStats_schema.json \ -tableConfigFile examples/docker/table-configs/airlineStats_realtime_table_config.json \ -controllerHost pinot-controller \ -controllerPort 9000 \ -exec ``` **Sample output** ``` Executing command: AddTable -tableConfigFile examples/docker/table-configs/airlineStats_realtime_table_config.json -schemaFile examples/stream/airlineStats/airlineStats_schema.json -controllerHost pinot-controller -controllerPort 9000 -exec Sending request: http://pinot-controller:9000/schemas to controller: 8fbe601012f3, version: Unknown {"status":"Table airlineStats_REALTIME succesfully added"} ``` {% endtab %} {% tab title="Using launcher scripts" %} **Start Kafka-Zookeeper** ``` bin/pinot-admin.sh StartZookeeper -zkPort 2181 ``` **Start Kafka** ``` bin/pinot-admin.sh StartKafka -zkAddress=localhost:2181/kafka -port 19092 ``` **Create stream table** ``` bin/pinot-admin.sh AddTable \ -schemaFile examples/stream/airlineStats/airlineStats_schema.json \ -tableConfigFile examples/stream/airlineStats/airlineStats_realtime_table_config.json \ -exec ``` {% endtab %} {% endtabs %} Check out the table config in the [Rest API](http://localhost:9000/help#!/Table/alterTableStateOrListTableConfig) to make sure it was successfully uploaded. ## Logical table A logical table provides a unified query interface over multiple physical tables. This is useful for geographic partitioning, table sharding strategies, or creating abstraction layers over complex table hierarchies. For details, see [Logical Table](/architecture-and-concepts/components/table/logical-table). ## Hybrid table creation To create a hybrid table, you have to create the offline and real-time tables individually. You don't need to create a separate hybrid table. ```javascript "OFFLINE": { "tableName": "pinotTable", "tableType": "OFFLINE", "segmentsConfig": { ... }, "tableIndexConfig": { ... }, "tenants": { "broker": "myBrokerTenant", "server": "myServerTenant" }, "metadata": { ... } }, "REALTIME": { "tableName": "pinotTable", "tableType": "REALTIME", "segmentsConfig": { ... }, "tableIndexConfig": { ... "streamConfigs": { ... }, }, "tenants": { "broker": "myBrokerTenant", "server": "myServerTenant" }, "metadata": { ... } } } ``` # Segment Discover the segment component in Apache Pinot for efficient data storage and querying within Pinot clusters, enabling optimized data processing and analysis. Pinot tables are stored in one or more independent shards called segments. A small table may be contained by a single segment, but Pinot lets tables grow to an unlimited number of segments. There are different processes for creating segments (see [ingestion](/build-with-pinot/ingestion)). Segments have time-based partitions of table data, and are stored on Pinot [servers](/architecture-and-concepts/components/cluster/server) that scale horizontally as needed for both storage and computation. Pinot achieves this by breaking the data into smaller chunks known as **segments** (similar to **shards/partitions** in relational databases). Segments can be seen as **time-based partitions**. A **segment is a horizontal shard representing a chunk of table data** with some number of rows. The segment stores data for all columns of the table. Each segment packs the data in a **columnar fashion**, along with the **dictionaries and indices** for the columns. The segment is laid out in a columnar format so that it can be directly mapped into memory for serving queries. Columns can be **single or multi-valued** and the following types are supported: **STRING, BOOLEAN, INT, LONG, FLOAT, DOUBLE, BIG\_DECIMAL, TIMESTAMP, and BYTES**. Columns may be declared to be **metric or dimension (or specifically as a time dimension)** in the schema. Columns can have default null values. For example, the default null value of a integer column can be 0. The default value for bytes columns must be hex-encoded before it's added to the schema. Pinot uses **dictionary encoding** to store values as a dictionary ID. Columns may be configured to be “no-dictionary” column in which case raw values are stored. Dictionary IDs are encoded using minimum number of bits for efficient storage (*e.g.* a column with a cardinality of 3 will use only 2 bits for each dictionary ID). A **forward index** is built for each column and compressed for efficient memory use. In addition, you can optionally configure **inverted indices** for any set of columns. Inverted indices take up more storage, but improve query performance. Specialized indexes like **Star-Tree index** are also supported. For more details, see [Indexing](/build-with-pinot/indexing). ## Creating a segment Once the table is configured, we can load some data. Loading data involves generating pinot segments from raw data and pushing them to the pinot cluster. Data can be loaded in batch mode or streaming mode. For more details, see the[ ingestion overview](/build-with-pinot/ingestion) page. ### Load data in batch #### **Prerequisites** 1. [Set up a cluster](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster) 2. [Create broker and server tenants](/architecture-and-concepts/components/cluster/tenant#create-a-tenant) 3. [Create an offline table](/architecture-and-concepts/components/table#offline-table-creation) Below are instructions to generate and push segments to Pinot via standalone scripts. For a production setup, you should use frameworks such as Hadoop or Spark. For more details on setting up data ingestion jobs, see [Import Data.](/build-with-pinot/ingestion) #### Job Spec YAML To generate a segment, we need to first create a job spec YAML file. This file contains all the information regarding data format, input data location, and pinot cluster coordinates. Note that this assumes that the controller is **RUNNING** to fetch the table config and schema. If not, you will have to configure the spec to point at their location. For full configurations, see [Ingestion Job Spec](/reference/configuration-reference/job-specification). {% code title="job-spec.yml" %} ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'examples/batch/baseballStats/rawdata' includeFileNamePattern: 'glob:**/*.csv' excludeFileNamePattern: 'glob:**/*.tmp' outputDirURI: 'examples/batch/baseballStats/segments' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' configs: tableSpec: tableName: 'baseballStats' schemaURI: 'http://localhost:9000/tables/baseballStats/schema' tableConfigURI: 'http://localhost:9000/tables/baseballStats' segmentNameGeneratorSpec: pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushParallelism: 2 pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` {% endcode %} #### Create and push segment To create and push the segment in one go, use the following: {% tabs %} {% tab title="Docker" %} ``` docker run \ --network=pinot-demo \ --name pinot-data-ingestion-job \ ${PINOT_IMAGE} LaunchDataIngestionJob \ -jobSpecFile examples/docker/ingestion-job-specs/airlineStats.yaml ``` **Sample Console Output** ``` SegmentGenerationJobSpec: !!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec excludeFileNamePattern: null executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner, segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner, segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner} includeFileNamePattern: glob:**/*.avro inputDirURI: examples/batch/airlineStats/rawdata jobType: SegmentCreationAndTarPush outputDirURI: examples/batch/airlineStats/segments overwriteOutput: true pinotClusterSpecs: - {controllerURI: 'http://pinot-controller:9000'} pinotFSSpecs: - {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file} pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000, segmentUriPrefix: null, segmentUriSuffix: null} recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader, configClassName: null, configs: null, dataFormat: avro} segmentNameGeneratorSpec: null tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema', tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats} Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS Finished building StatsCollector! Collected stats for 403 documents Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389 Using fixed bytes value dictionary for column: Origin, size: 294 Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1 Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303 ...... ...... Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"} Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"} ``` {% endtab %} {% tab title="Using launcher scripts" %} ``` bin/pinot-admin.sh LaunchDataIngestionJob \ -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml ``` {% endtab %} {% endtabs %} Alternately, you can separately create and then push, by changing the jobType to `SegmentCreation` or `SegmenTarPush`. #### Templating Ingestion Job Spec The Ingestion job spec supports templating with Groovy Syntax. This is convenient if you want to generate one ingestion job template file and schedule it on a daily basis with extra parameters updated daily. e.g. you could set `inputDirURI` with parameters to indicate the date, so that the ingestion job only processes the data for a particular date. Below is an example that templates the date for input and output directories. ```yaml inputDirURI: 'examples/batch/airlineStats/rawdata/${year}/${month}/${day}' outputDirURI: 'examples/batch/airlineStats/segments/${year}/${month}/${day}' ``` You can pass in arguments containing values for `${year}, ${month}, ${day}` when kicking off the ingestion job: `-values $param=value1 $param2=value2`... {% tabs %} {% tab title="Docker" %} ``` docker run \ --network=pinot-demo \ --name pinot-data-ingestion-job \ ${PINOT_IMAGE} LaunchDataIngestionJob \ -jobSpecFile examples/docker/ingestion-job-specs/airlineStats.yaml -values year=2014 month=01 day=03 ``` {% endtab %} {% endtabs %} This ingestion job only generates segments for date `2014-01-03` ### Load data in streaming **Prerequisites** 1. [Set up a cluster](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster) 2. [Create broker and server tenants](/architecture-and-concepts/components/cluster/tenant#create-a-tenant) 3. [Create a real-time table and set up a real-time stream](/architecture-and-concepts/components/table#streaming-table-creation) Below is an example of how to publish sample data to your stream. As soon as data is available to the real-time stream, it starts getting consumed by the real-time servers. #### Kafka {% tabs %} {% tab title="Docker" %} Run below command to stream JSON data into Kafka topic: **flights-realtime** ``` docker run \ --network pinot-demo \ --name=loading-airlineStats-data-to-kafka \ ${PINOT_IMAGE} StreamAvroIntoKafka \ -avroFile examples/stream/airlineStats/sample_data/airlineStats_data.avro \ -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092 -zkAddress pinot-zookeeper:2181/kafka ``` {% endtab %} {% tab title="Using launcher scripts" %} Run below command to stream JSON data into Kafka topic: **flights-realtime** ``` bin/pinot-admin.sh StreamAvroIntoKafka \ -avroFile examples/stream/airlineStats/sample_data/airlineStats_data.avro \ -kafkaTopic flights-realtime -kafkaBrokerList localhost:19092 -zkAddress localhost:2181/kafka ``` {% endtab %} {% endtabs %} # Deep Store Leverage Apache Pinot's deep store component for efficient large-scale data storage and management, enabling impactful data processing and analysis. The deep store (or deep storage) is the permanent store for [segment](/architecture-and-concepts/components/table/segment) files. It is used for backup and restore operations. New [server](/architecture-and-concepts/components/cluster/server) nodes in a cluster will pull down a copy of segment files from the deep store. If the local segment files on a server gets damaged in some way (or accidentally deleted), a new copy will be pulled down from the deep store on server restart. The deep store stores a compressed version of the segment files and it typically won't include any indexes. These compressed files can be stored on a local file system or on a variety of other file systems. For more details on supported file systems, see [File Systems](/build-with-pinot/ingestion/formats-filesystems/file-systems). Note: Deep store by itself is not sufficient for restore operations. Pinot stores metadata such as table config, schema, segment metadata in Zookeeper. For restore operations, both Deep Store as well as Zookeeper metadata are required. ## How do segments get into the deep store? There are several different ways that segments are persisted in the deep store. For offline tables, the batch ingestion job writes the segment directly into the deep store, as shown in the diagram below: ![Batch job writing a segment into the deep store](/files/WXtcsPZ1EnG1STriokgI) The ingestion job then sends a notification about the new segment to the controller, which in turn notifies the appropriate server to pull down that segment. For real-time tables, by default, a segment is first built-in memory by the server. It is then uploaded to the lead controller (as part of the Segment Completion Protocol sequence), which writes the segment into the deep store, as shown in the diagram below: ![Server sends segment to Controller, which writes segments into the deep store](/files/u8TWACCzi7vlJr58X6d1) Having all segments go through the controller can become a system bottleneck under heavy load, in which case you can use the peer download policy, as described in [Decoupling Controller from the Data Path](/operate-pinot/deployment/decoupling-controller-from-the-data-path). When using this configuration, the server will directly write a completed segment to the deep store, as shown in the diagram below: ![Server writing a segment into the deep store](/files/hGVREAMTj0V4egRjHTO7) ## Configuring the deep store For hands-on examples of how to configure the deep store, see the following tutorials: * [Use OSS as Deep Storage for Pinot](/tutorials/deep-storage/use-oss-as-deep-storage-for-pinot) * [Use S3 as Deep Storage for Pinot](/tutorials/deep-storage/use-s3-as-deep-store-for-pinot) # Segment Threshold Learn how segment thresholds work in Pinot. The segment threshold determines when a segment is committed in real-time tables. When data is first ingested from a streaming provider like Kafka, Pinot stores the data in a consuming segment. This segment is on the disk of the server(s) processing a particular partition from the streaming provider. However, it's not until a segment is committed that the segment is written to the [deep store](/architecture-and-concepts/components/table/segment/deep-store). The segment threshold decides when that should happen. ## Why is the segment threshold important? The segment threshold is important because it ensures segments are a reasonable size. When queries are processed, smaller segments may increase query latency due to more overhead (number of threads spawned, meta data processing, and so on). Larger segments may cause servers to run out of memory. When a server is restarted, the consuming segment must start consuming from the first row again, causing a lag between Pinot and the streaming provider. *Mark Needham explains the segment threshold* # Segment Retention In this Apache Pinot concepts guide, we'll learn how segment retention works. Segments in Pinot tables have a retention time, after which the segments are deleted. Typically, offline tables retain segments for a longer period of time than real-time tables. The removal of segments is done by the retention manager. By default, the retention manager runs once every 6 hours. The retention manager purges two types of segments: * Expired segments: Segments whose end time has exceeded the retention period. * Replaced segments: Segments that have been replaced as part of the [merge rollup task.](/operate-pinot/segment-management/minion-merge-rollup-task) There are a couple of scenarios where segments in offline tables won't be purged: * If the segment doesn't have an end time. This would happen if the segment doesn't contain a time column. * If the segment's table has a `segmentIngestionType` of `REFRESH`. ## Handling segments with invalid end times By default, when a segment's end time is invalid or missing, the retention manager skips it entirely, and the segment is never deleted regardless of the retention policy. To enable automatic cleanup of such segments, you can enable the creation time fallback by setting `controller.retentionManager.enableCreationTimeFallback` to `true` in the cluster configuration. When enabled, the retention manager will use the segment's creation time (`segmentZKMetadata.getCreationTime()`) as a fallback if the end time is invalid. This configuration is **dynamic** and does not require a controller restart to take effect. It can be updated through the cluster config change listener. If the retention period isn't specified, segments aren't purged from tables. The retention manager initially moves these segments into a *Deleted Segments* area, from where they will eventually be permanently removed. The duration that deleted segments are kept is controlled by the `controller.deleted.segments.retentionInDays` configuration (default: 7 days). When deleting a table via the API, you can override this behavior by passing a `retention` query parameter. For example, `DELETE /tables/{tableName}?retention=0d` deletes all segments immediately without moving them to the deleted-segments area. See the [Controller API Examples](/reference/api-reference/controller-api#delete-tablestablename) for more details. # Schema Explore the Schema component in Apache Pinot, vital for defining the structure and data types of Pinot tables, enabling efficient data processing and analysis. Each table in Pinot is associated with a schema. A schema defines: * Fields in the table with their data types. * Whether the table uses column-based or table-based null handling. For more information, see [Null value support](/build-with-pinot/querying-and-sql/sql-syntax/null-value-support). The schema is stored in Zookeeper along with the table configuration. {% hint style="info" %} Schema naming in Pinot follows typical database table naming conventions, such as starting names with a letter, not ending with an underscore, and using only alphanumeric characters {% endhint %} ### Categories A schema also defines what category a column belongs to. Columns in a Pinot table can be categorized into three categories: | Category | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Dimension** | Dimension columns are typically used in slice and dice operations for answering business queries. Some operations for which dimension columns are used: - `GROUP BY` - group by one or more dimension columns along with aggregations on one or more metric columns - Filter clauses such as `WHERE` | | **Metric** | These columns represent the quantitative data of the table. Such columns are used for aggregation. In data warehouse terminology, these can also be referred to as fact or measure columns. Some operation for which metric columns are used: - Aggregation - `SUM`, `MIN`, `MAX`, `COUNT`, `AVG` etc - Filter clause such as `WHERE` | | **DateTime** | This column represents time columns in the data. There can be multiple time columns in a table, but only one of them can be treated as primary. The primary time column is the one that is present in the [segment config](/reference/configuration-reference/table#segments-config). The primary time column is used by Pinot to maintain the time boundary between offline and real-time data in a hybrid table and for retention management. A primary time column is mandatory if the table's push type is `APPEND` and optional if the push type is `REFRESH` . Common operations that can be done on time column: - `GROUP BY` - Filter clauses such as `WHERE` | Pinot does not enforce strict rules on which of these categories columns belong to, rather the categories can be thought of as hints to Pinot to do internal optimizations. For example, metrics may be stored without a dictionary and can have a different default null value. The categories are also relevant when doing segment merge and rollups. Pinot uses the dimension and time fields to identify records against which to apply merge/rollups. Metrics aggregation is another example where Pinot uses dimensions and time are used as the key, and automatically aggregates values for the metric columns. For configuration details, see [Schema configuration reference](/reference/configuration-reference/schema). ### Date and time fields Since Pinot doesn't have a dedicated `DATETIME` datatype support, you need to input time in either STRING, LONG, or INT format. However, Pinot needs to convert the date into an understandable format such as epoch timestamp to do operations. You can refer to [DateTime field spec configs](/reference/configuration-reference/schema#datetimefieldspecs) for more details on supported formats. ### Creating a schema First, Make sure your [cluster is up](/architecture-and-concepts/components/cluster#set-up-a-pinot-cluster) and running. Let's create a schema and put it in a JSON file. For this example, we have created a schema for flight data. {% hint style="info" %} For more details on constructing a schema file, see the [Schema configuration reference](/reference/configuration-reference/schema). {% endhint %} {% code title="flights-schema.json" %} ```javascript { "schemaName": "flights", "enableColumnBasedNullHandling": true, "dimensionFieldSpecs": [ { "name": "flightNumber", "dataType": "LONG", "notNull": true }, { "name": "tags", "dataType": "STRING", "singleValueField": false, "defaultNullValue": "null" } ], "metricFieldSpecs": [ { "name": "price", "dataType": "DOUBLE", "notNull": true, "defaultNullValue": 0 } ], "dateTimeFieldSpecs": [ { "name": "millisSinceEpoch", "dataType": "LONG", "format": "EPOCH", "granularity": "15:MINUTES" }, { "name": "hoursSinceEpoch", "dataType": "INT", "notNull": true, "format": "EPOCH|HOURS", "granularity": "1:HOURS" }, { "name": "dateString", "dataType": "STRING", "format": "SIMPLE_DATE_FORMAT|yyyy-MM-dd", "granularity": "1:DAYS" } ] } ``` {% endcode %} Then, we can upload the sample schema provided above using either a Bash command or REST API call. {% tabs %} {% tab title="pinot-admin.sh" %} ```bash bin/pinot-admin.sh AddSchema -schemaFile flights-schema.json -exec OR bin/pinot-admin.sh AddTable -schemaFile flights-schema.json -tableFile flights-table.json -exec ``` {% endtab %} {% tab title="curl" %} ``` curl -F schemaName=@transcript-schema.json localhost:9000/schemas ``` {% endtab %} {% endtabs %} Check out the schema in the [Rest API](http://localhost:9000/help#!/Schema/getSchema) to make sure it was successfully uploaded # Time Boundary Learn about time boundaries in hybrid tables. Learn about time boundaries in hybrid tables. Hybrid tables are when we have offline and real-time tables with the same name. When querying these tables, the Pinot broker decides which records to read from the offline table and which to read from the real-time table. It does this using the time boundary. ## How is the time boundary determined? The time boundary is determined by looking at the maximum end time of the offline segments and the segment ingestion frequency specified for the offline table. If it's set to hourly, then: ``` timeBoundary = Maximum end time of offline segments - 1 hour ``` Otherwise: ``` timeBoundary = Maximum end time of offline segments - 1 day ``` It is possible to force the hybrid table to use max(all offline segments' `end time`) by calling the API (V 0.12.0+) ``` curl -X POST \ "http://localhost:9000/tables/{tableName}/timeBoundary" \ -H "accept: application/json" ``` Note that this will not automatically update the time boundary as more segments are added to the offline table, and must be called each time a segment with more recent end time is uploaded to the offline table. You can revert back to using the derived time boundary by calling API: ``` curl -X DELETE \ "http://localhost:9000/tables/{tableName}/timeBoundary" \ -H "accept: application/json" ``` ## Querying When a Pinot broker receives a query for a hybrid table, the broker sends a time boundary annotated version of the query to the offline and real-time tables. For example, if we executed the following query: ```sql SELECT count(*) FROM events ``` The broker would send the following query to the offline table: ```sql SELECT count(*) FROM events_OFFLINE WHERE timeColumn <= $timeBoundary ``` And the following query to the real-time table: ```sql SELECT count(*) FROM events_REALTIME WHERE timeColumn > $timeBoundary ``` The results of the two queries are merged by the broker before being returned to the client. # Logical Table Learn about Logical Tables in Apache Pinot, which provide a unified query interface over multiple physical tables for flexible data organization. A logical table in Pinot provides a unified query interface over multiple physical tables. Instead of querying individual tables separately, users can query a single logical table that transparently routes the query to all underlying physical tables and aggregates the results. ## Overview Logical tables are useful for: * **Geographic/Regional partitioning**: Split data by region (e.g., `ordersUS`, `ordersEU`, `ordersAPAC`) while providing a unified `orders` table for queries * **Table partitioning strategies**: Organize data across multiple physical tables based on business logic * **Time-based table splitting**: Combine historical and recent data from different physical tables {% hint style="info" %} Logical tables require that all underlying physical tables share the same schema structure. A schema with the same name as the logical table must be created before creating the logical table. {% endhint %} ## How It Works When you query a logical table, Pinot: 1. Resolves the logical table name to its list of physical tables 2. Routes the query to all relevant physical tables (both offline and realtime) 3. Aggregates results from all physical tables 4. Returns a unified result set to the client For hybrid logical tables (containing both offline and realtime physical tables), Pinot uses a configurable time boundary strategy to determine which segments to query from each table type, avoiding duplicate data. ## Segment Pruning Optimization Pinot performs automatic cross-table segment pruning when querying logical tables. Instead of pruning segments independently for each physical table, segment pruning operates once across all physical tables collectively. This optimization is particularly beneficial for queries using ORDER BY with LIMIT, where the `SelectionQuerySegmentPruner` can now prune segments across the entire logical table. For example, with a logical table spanning three physical tables (US, EU, APAC), a query like: ```sql SELECT * FROM orders ORDER BY createdTime DESC LIMIT 10 ``` Previously, the pruner would prune segments within each physical table independently, potentially returning more segments than necessary. Now, pruning happens across all physical tables together, allowing the pruner to identify and return only the minimum set of segments needed to satisfy the query requirements. **Key benefits:** * Improved query performance by reducing segments processed * Automatic optimization with no configuration changes required * Particularly effective for ORDER BY + LIMIT queries across logical tables * Single-table behavior remains unchanged ## Logical Table Configuration A logical table configuration defines the mapping between the logical table and its physical tables. ### Configuration Properties | Property | Description | Required | | ------------------------ | --------------------------------------------------------- | ---------------------------------- | | `tableName` | Name of the logical table | Yes | | `brokerTenant` | The broker tenant to use for routing | Yes | | `physicalTableConfigMap` | Map of physical table names to their configurations | Yes | | `refOfflineTableName` | Reference offline table for table config metadata | Required if offline tables exist | | `refRealtimeTableName` | Reference realtime table for table config metadata | Required if realtime tables exist | | `query` | Query configuration (timeout, response size limits, etc.) | No | | `quota` | Quota configuration for rate limiting | No | | `timeBoundaryConfig` | Time boundary configuration for hybrid tables | Required for hybrid logical tables | ### Example Configuration ```json { "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { "ordersUS_OFFLINE": {}, "ordersEU_OFFLINE": {}, "ordersAPAC_OFFLINE": {} }, "refOfflineTableName": "ordersUS_OFFLINE" } ``` ### Hybrid Logical Table Configuration For logical tables that combine both offline and realtime physical tables: ```json { "tableName": "events", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { "eventsHistorical_OFFLINE": {}, "eventsRecent_OFFLINE": {}, "eventsLive_REALTIME": {} }, "refOfflineTableName": "eventsHistorical_OFFLINE", "refRealtimeTableName": "eventsLive_REALTIME", "timeBoundaryConfig": { "boundaryStrategy": "min", "parameters": { "includedTables": ["eventsRecent_OFFLINE"] } } } ``` ## Creating a Logical Table ### Step 1: Create the Schema Create a schema that matches the structure of your physical tables: ```json { "schemaName": "orders", "dimensionFieldSpecs": [ { "name": "orderId", "dataType": "STRING" }, { "name": "customerId", "dataType": "STRING" }, { "name": "region", "dataType": "STRING" }, { "name": "productId", "dataType": "STRING" }, { "name": "status", "dataType": "STRING" } ] } ``` Upload the schema: ```bash curl -F schemaName=@orders_schema.json localhost:9000/schemas ``` ### Step 2: Create the Logical Table ```bash curl -X POST -H 'Content-Type: application/json' \ -d '{ "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { "ordersUS_OFFLINE": {}, "ordersEU_OFFLINE": {}, "ordersAPAC_OFFLINE": {} }, "refOfflineTableName": "ordersUS_OFFLINE" }' \ http://localhost:9000/logicalTables ``` ## Managing Logical Tables ### List Logical Tables ```bash curl http://localhost:9000/logicalTables ``` ### Get Logical Table Configuration ```bash curl http://localhost:9000/logicalTables/{tableName} ``` ### Update Logical Table ```bash curl -X PUT -H 'Content-Type: application/json' \ -d '{ "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { "ordersUS_OFFLINE": {}, "ordersEU_OFFLINE": {}, "ordersAPAC_OFFLINE": {}, "ordersANZ_OFFLINE": {} }, "refOfflineTableName": "ordersUS_OFFLINE" }' \ http://localhost:9000/logicalTables/orders ``` ### Delete Logical Table ```bash curl -X DELETE http://localhost:9000/logicalTables/{tableName} ``` {% hint style="warning" %} Deleting a logical table only removes the logical table configuration. The underlying physical tables and their data are not affected. {% endhint %} ## Querying Logical Tables Query a logical table just like any other Pinot table: ```sql -- Query the logical table SELECT COUNT(*) FROM orders -- Filter by region SELECT orderId, customerId, region, status FROM orders WHERE region = 'us' LIMIT 10 -- Aggregate across all regions SELECT region, COUNT(*) as orderCount FROM orders GROUP BY region ORDER BY region ``` Logical tables work with both the single-stage and multi-stage query engines. ## Time Boundary Configuration For hybrid logical tables that contain both offline and realtime physical tables, you must configure a time boundary strategy to avoid querying duplicate data. ### Available Strategies | Strategy | Description | | -------- | -------------------------------------------------------- | | `min` | Uses the minimum time boundary from the specified tables | ### Configuration Example ```json { "timeBoundaryConfig": { "boundaryStrategy": "min", "parameters": { "includedTables": ["eventsRecent_OFFLINE"] } } } ``` The `includedTables` parameter specifies which physical tables should be considered when computing the time boundary. ## Query Configuration Logical tables support query-level configurations: ```json { "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { ... }, "refOfflineTableName": "ordersUS_OFFLINE", "query": { "timeoutMs": 30000, "disableGroovy": true, "maxServerResponseSizeBytes": 1000000, "maxQueryResponseSizeBytes": 5000000 } } ``` | Property | Description | | ---------------------------- | -------------------------------------- | | `timeoutMs` | Query timeout in milliseconds | | `disableGroovy` | Disable Groovy functions in queries | | `maxServerResponseSizeBytes` | Maximum response size from each server | | `maxQueryResponseSizeBytes` | Maximum total query response size | ## Quota Configuration Apply rate limiting to logical tables: ```json { "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { ... }, "refOfflineTableName": "ordersUS_OFFLINE", "quota": { "maxQueriesPerSecond": 100 } } ``` {% hint style="info" %} Storage quota (`quota.storage`) is not supported for logical tables since they don't store data directly. {% endhint %} ## Managing Logical Tables via the Controller UI The Pinot Controller UI provides browsing and in-place management for logical tables, accessible directly from the main Tables page. ### Accessing Logical Tables 1. Open the Controller UI (default: `http://:9000`). 2. Navigate to **Tables** in the left sidebar. 3. The Tables page displays physical tables and logical tables in separate sections. 4. Click a logical table name to open its detail page, which shows: * Current configuration (JSON) * Physical table mappings ### Supported Operations | Operation | Description | | ---------- | ------------------------------------------------------------------------ | | **List** | View all logical tables with search and filter | | **View** | Inspect the logical table's configuration and physical table assignments | | **Update** | Edit the logical table configuration in-place | | **Delete** | Remove a logical table from the cluster | {% hint style="info" %} Create logical tables with `POST /logicalTables`. Get, update, and delete are available at `/logicalTables/{tableName}` using GET, PUT, and DELETE. {% endhint %} ## Quick Start Example Try the logical table quickstart to see the feature in action: {% tabs %} {% tab title="Docker" %} ```bash docker run \ -p 9000:9000 \ apachepinot/pinot:latest QuickStart \ -type LOGICAL_TABLE ``` {% endtab %} {% tab title="Launcher scripts" %} ```bash ./bin/pinot-admin.sh QuickStart -type LOGICAL_TABLE ``` {% endtab %} {% endtabs %} This quickstart: 1. Creates three physical tables: `ordersUS_OFFLINE`, `ordersEU_OFFLINE`, and `ordersAPAC_OFFLINE` 2. Creates a logical table `orders` that unifies all three 3. Demonstrates queries on both physical and logical tables ## Validation Rules When creating or updating a logical table, Pinot validates: * Table name does not end with `_OFFLINE` or `_REALTIME` * All physical tables exist (unless marked as `multiCluster`) * Physical tables are in the same database as the logical table * Schema with the same name as the logical table exists * Broker tenant exists * Reference table names (`refOfflineTableName`, `refRealtimeTableName`) are set correctly * Time boundary config is provided for hybrid tables ## Limitations * All physical tables must have compatible schemas * Storage quota is not supported * Physical tables in the same logical table should ideally have consistent indexing for optimal query performance ## Pluggable LogicalTableConfig Serialization By default, `LogicalTableConfig` is serialized to and deserialized from ZooKeeper using a built-in JSON format. For advanced use cases requiring a custom storage format, implement `LogicalTableConfigSerDe` and register it via `LogicalTableConfigSerDeProvider`. ### When to Use This * You need a compact binary format for deployments with a very large number of logical tables * Your ZooKeeper schema requires a specific non-default encoding * You are integrating Pinot with an external metadata system with its own serialization requirements ### Implementation **Step 1:** Implement the `LogicalTableConfigSerDe` interface: ```java public class MyCustomSerDe implements LogicalTableConfigSerDe { @Override public byte[] serialize(LogicalTableConfig config) { /* ... */ } @Override public LogicalTableConfig deserialize(byte[] bytes) { /* ... */ } } ``` **Step 2:** Implement `LogicalTableConfigSerDeProvider` to return your custom SerDe. **Step 3:** Register the provider using the Java Service Provider Interface (SPI) by creating the file: ``` META-INF/services/org.apache.pinot.spi.config.table.logical.LogicalTableConfigSerDeProvider ``` containing the fully-qualified class name of your provider implementation. {% hint style="info" %} This is an advanced extension point for specialized deployments. Most users should rely on the default JSON-based serialization. {% endhint %} ## See Also * [Table Configuration](/reference/configuration-reference/table) * [Schema Configuration](/architecture-and-concepts/components/table/schema) # Pinot Data Explorer Pinot Data Explorer is a user-friendly interface in Apache Pinot for interactive data exploration, querying, and visualization. Once you have set up a cluster, you can start exploring the data and the APIs using the Pinot Data Explorer. Navigate to in your browser to open the Data Explorer UI. ## Cluster Manager The first screen that you'll see when you open the Pinot Data Explorer is the Cluster Manager. The Cluster Manager provides a UI to operate and manage your cluster, giving you an overview of tenants, instances, tables, and their current status. ![Pinot Cluster Manager](/files/DARy8CP1nFdEzdiTEHOi) ### Task management The Pinot Data Explorer separates cluster-wide task controls from minion queue troubleshooting: * Select **Minion Tasks** in the left navigation to open the **Minion Task Manager** page. This page summarizes task types, minion instances, running tasks, and waiting tasks, then lets you drill into task queues, tasks, and subtasks. * Stay on **Cluster Manager** for controller-wide scheduling details. The **Cron Scheduler Information** accordion shows the controller scheduler state and metadata, and the **Periodic Tasks** accordion lists registered periodic tasks and lets you run one immediately. The run dialog can target all tables or an individual table and table type. If you want to view the contents of a server, click on its instance name. You'll then see the following: ![Pinot Server](/files/fEDN6vJyNFJJGubGci4q) ### Table management To view a table, click on its name from the tables list. From the table detail screen, you can edit or delete the table, edit or adjust its schema, and perform several other operations. ![baseballStats Table](/files/EK0x233H8wG67ZvsNeeM) For example, if we want to add *yearID* to the list of inverted indexes, click on **Edit Table,** add the extra column, and click **Save:** ![Edit Table](/files/SBNVmPlmMSNnRPQPj8IF) #### Pause and resume consumption For real-time tables, the table detail screen includes a **Pause/Resume Consumption** button ([#15657](https://github.com/apache/pinot/pull/15657)). This lets you pause ingestion on a real-time table directly from the UI without issuing REST API calls, and resume it when ready. This is useful during maintenance windows or when you need to temporarily halt data ingestion. #### Consuming segments info A **Consuming Segments Info** button ([#15623](https://github.com/apache/pinot/pull/15623)) is available on real-time tables, providing a quick view of all currently consuming segments. This shows details such as the partition, current offset, and consumption state, making it easier to monitor real-time ingestion health. #### Reset segment The UI now supports a **Reset Segment** operation ([#16078](https://github.com/apache/pinot/pull/16078)), allowing you to reset a segment directly from the table detail screen. This is helpful when a segment is stuck in an error state and needs to be re-processed. #### Segment state filter A segment state filter ([#16085](https://github.com/apache/pinot/pull/16085)) has been added to the table detail screen. You can filter segments by their state (e.g., ONLINE, CONSUMING, ERROR) to quickly locate segments that need attention, which is especially valuable for tables with a large number of segments. #### Table rebalance The table detail screen also provides access to table rebalance operations. Several UI fixes and improvements ([#15511](https://github.com/apache/pinot/pull/15511)) have been made to improve the reliability and usability of the rebalance workflow, including better parameter validation and progress display. ### Logical table management Starting with Pinot 1.4 and later, the Data Explorer includes a logical table management UI ([#17878](https://github.com/apache/pinot/pull/17878)). Logical tables are collections of physical tables (REALTIME and OFFLINE) that can be queried as a single unified table. The logical tables listing is accessible from the main **Tables** page, alongside physical tables and schemas. From there you can: * **Browse** all logical tables in the cluster with search support. * **View details** of a logical table, including its configuration, the list of physical tables it maps to, and metadata. * **Edit** a logical table's configuration. * **Delete** a logical table with a confirmation dialog. For more information about logical tables, see the [Logical Table Support](/reference/release-notes/releases/1.4.0#logical-table-support-design) section in the 1.4.0 release notes. ### Materialized view discovery and management The Data Explorer also includes a **Data Sources** hub for queryable assets in the cluster. Use **Data Sources** in the left navigation to switch between physical tables and materialized views. The **Materialized Views** page lists each MV together with its base tables, watermark, VALID and STALE partition counts, last refresh time, staleness SLO, and any metadata errors returned by the controller. Click a materialized view name to open its detail page, which shows the stored `definedSQL`, split spec, partition runtime state, raw metadata JSON, and controls to refresh the page data or drop the MV. The UI remains a discovery and monitoring surface. Query execution still happens through the broker: you can query the MV table directly, or let brokers transparently rewrite eligible SSE base-table queries when `pinot.broker.query.enable.materialized.view.rewrite=true`. For the creation model and query contract, see [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views). ## Query Console Navigate to [Query Console](http://localhost:9000/#/query) to see the querying interface. The Query Console lets you run SQL queries against your Pinot cluster and view the results interactively. We can see our `baseballStats` table listed on the left (you will see `meetupRSVP` or `airlineStats` if you used the streaming or the hybrid [quick start](/start-here/quick-start)). Click on the table name to display all the names along with the data types of the columns of the table. You can also execute a sample query `select * from baseballStats limit 10` by typing it in the text box and clicking the **Run Query** button. `Cmd + Enter` can also be used to run the query when focused on the console. ![](/files/-MEkx7TNGEzYa6TCW6TR) Here are some sample queries you can try: ```sql select playerName, max(hits) from baseballStats group by playerName order by max(hits) desc ``` ```sql select sum(hits), sum(homeRuns), sum(numberOfGames) from baseballStats where yearID > 2010 ``` ```sql select * from baseballStats order by league ``` Pinot uses SQL for querying. For the complete syntax reference, see the [SQL Syntax and Operators Reference](/build-with-pinot/querying-and-sql/sql-syntax/sql-reference). For query options, examples, and engine details, see [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot). ### Time-series query execution The Query Console also supports time-series query execution ([#16305](https://github.com/apache/pinot/pull/16305)), introduced as part of the Time Series Engine beta. The current Controller UI adds a dedicated page for running M3QL queries against the controller's Prometheus-compatible `/query_range` endpoint ([#16286](https://github.com/apache/pinot/pull/16286)). It provides time-range controls and shows the raw JSON response directly in the UI. ## REST API The [Pinot Admin UI](http://localhost:9000/help) contains all the APIs that you will need to operate and manage your cluster. It provides a set of APIs for Pinot cluster management including health check, instances management, schema and table management, data segments management. ![](/files/-M1BxN51vPW0p9FFuJCK) Let's check out the tables in this cluster by going to [Table -> List all tables in cluster](http://localhost:9000/help#/Table/listTables), click **Try it out**, and then click **Execute**. We can see the`baseballStats` table listed here. We can also see the exact cURL call made to the controller API. ![List all tables in cluster](/files/hzBWe1nL7J7hGCoNDbh0) You can look at the configuration of this table by going to [Tables -> Get/Enable/Disable/Drop a table](http://localhost:9000/help#!/Table/alterTableStateOrListTableConfig), click **Try it out**, type `baseballStats` in the table name, and then click **Execute**. Let's check out the schemas in the cluster by going to [Schema -> List all schemas in the cluster](http://localhost:9000/help#!/Schema/listSchemaNames), click **Try it out**, and then click **Execute**. We can see a schema called `baseballStats` in this list. ![List all schemas in the cluster](/files/zuBRsaJPUS1dd3tza3Mo) Take a look at the schema by going to [Schema -> Get a schema](http://localhost:9000/help#!/Schema/getSchema), click **Try it out**, type `baseballStats` in the schema name, and then click **Execute**. ![baseballStats Schema](/files/GigsKxoztI7D4jmVa180) Finally, let's check out the data segments in the cluster by going to [Segment -> List all segments](http://localhost:9000/help#!/Segment/getSegments), click **Try it out**, type in `baseballStats` in the table name, and then click **Execute**. There's 1 segment for this table, called `baseballStats_OFFLINE_0`. To learn how to upload your own data and schema, see [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) or [Stream ingestion](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion). # Overview Build applications and data workflows with Apache Pinot using task-oriented guidance. Use this section when you are designing tables, ingesting data, querying Pinot, choosing indexes, or connecting Pinot to applications and tools. The goal here is to help you decide what to do next and then take you to the right detailed docs without forcing you through raw reference first. ## Core build workflows {% content-ref url="/pages/JHGSeFAN7cpJaw9dZyzo" %} [Data modeling](/build-with-pinot/data-modeling) {% endcontent-ref %} {% content-ref url="/pages/YqYKvoLQRBDeRhikxqKe" %} [Ingestion](/build-with-pinot/ingestion) {% endcontent-ref %} {% content-ref url="/pages/ylJExJhigHcd3NiRoRFT" %} [Querying & SQL](/build-with-pinot/querying-and-sql) {% endcontent-ref %} {% content-ref url="/pages/6iTiVOEu4Bqoz6Yeobcj" %} [Indexing](/build-with-pinot/indexing) {% endcontent-ref %} {% content-ref url="/pages/ZeU0T4mM3Dqq7C5CKZvi" %} [Connectors, clients & APIs](/build-with-pinot/connectors-clients-apis) {% endcontent-ref %} ## When to use Reference If you already know the exact property, endpoint, or plugin you need, jump to the [Reference](/reference/reference) section. Build-focused pages in this section explain how pieces fit together. Reference pages stay dense on purpose. ## What this page covered This page introduced the task-oriented `Build with Pinot` structure and pointed to the main workflows for modeling, ingestion, querying, indexing, and integration. ## Next step Start with the workflow that matches your immediate task, such as [Data modeling](/build-with-pinot/data-modeling) or [Ingestion](/build-with-pinot/ingestion). ## Related pages * [Querying & SQL](/build-with-pinot/querying-and-sql) * [Functions](/functions/functions) * [Connectors, clients & APIs](/build-with-pinot/connectors-clients-apis) * [Reference](/reference/reference) # Data modeling Build Pinot tables by getting schema, table shape, logical-table, and schema-evolution decisions right before ingestion starts. Pinot works best when the table shape is clear before data lands. Start here to understand the structure that every ingestion and query decision depends on: schema design, table composition, logical-table layout, and how schemas evolve without breaking existing pipelines. If you need dense JSON config or controller endpoints, jump to the [Reference](/reference/reference) section instead. This section stays narrative and decision-oriented. ## Start Here * [Schema and Table Shape](/build-with-pinot/data-modeling/schema) * [Logical Tables](/build-with-pinot/data-modeling/logical-tables) * [Schema Evolution](/build-with-pinot/data-modeling/schema-evolution) ## Related Existing Docs * [Schema](/architecture-and-concepts/components/table/schema) * [Table](/architecture-and-concepts/components/table) * [Logical Table](/architecture-and-concepts/components/table/logical-table) * [Schema Evolution Tutorial](/tutorials/data-ingestion/schema-evolution) * [Schema Reference](/reference/configuration-reference/schema) * [Table Reference](/reference/configuration-reference/table) ## What this page covered This landing page defines the scope of Pinot data modeling and points to the core pages that matter first. ## Next step Read [Schema and Table Shape](/build-with-pinot/data-modeling/schema) to lock in the table structure before designing ingestion. ## Related pages * [Schema and Table Shape](/build-with-pinot/data-modeling/schema) * [Logical Tables](/build-with-pinot/data-modeling/logical-tables) * [Schema Evolution](/build-with-pinot/data-modeling/schema-evolution) # Schema and Table Shape Understand Pinot schema design, table shape, null handling, and the schema fields that drive query and ingestion behavior. A Pinot schema defines the columns that exist in a table and how Pinot should treat them. The important part is not only the column list, but also the shape of the table: which fields are dimensions, metrics, and time fields, how nulls behave, and whether the table is built for offline, realtime, or hybrid ingestion. Pinot stores schema and table metadata separately, but the two should be designed together. Keep the schema narrow enough to match the data you actually query, and keep the table config dense enough for reference pages rather than this narrative overview. ## What to design The schema answers four practical questions: * What columns exist? * What data type does each column use? * Which columns are dimensions, metrics, or date-time fields? * How should Pinot handle missing values and time semantics? ## Good defaults Use column names that are stable and business-facing. Prefer simple types that match the source data. Add only the fields you need at query time, because schema changes are additive and should be deliberate. For time columns, keep one primary time field in mind for retention and hybrid-table boundary behavior. For null handling, decide early whether the table needs column-based or table-based semantics. For new schemas, model time columns with `dateTimeFieldSpecs` only. Pinot's REST schema submission paths now reject the deprecated `TimeFieldSpec` (`fieldType=TIME`) and require `DateTimeFieldSpec` instead. Legacy schemas that are already stored in the cluster can still load internally for backward compatibility. ## Example schema ```json { "schemaName": "orders", "enableColumnBasedNullHandling": true, "dimensionFieldSpecs": [ { "name": "orderId", "dataType": "STRING" }, { "name": "customerId", "dataType": "STRING" }, { "name": "region", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "amount", "dataType": "DOUBLE", "defaultNullValue": 0 } ], "dateTimeFieldSpecs": [ { "name": "eventTime", "dataType": "LONG", "format": "EPOCH", "granularity": "1:DAYS" } ] } ``` ## When to use the reference pages Use the [schema reference](/reference/configuration-reference/schema) when you need the exact JSON fields, validation rules, or date-time field formats. Use the [table reference](/reference/configuration-reference/table) when you need indexing, retention, or routing configuration. ## What this page covered This page covered the parts of Pinot schema design that shape ingestion and query behavior. ## Next step Read [Logical Tables](/build-with-pinot/data-modeling/logical-tables) if one query name should route to multiple physical tables. ## Related pages * [Data Modeling](/build-with-pinot/data-modeling) * [Logical Tables](/build-with-pinot/data-modeling/logical-tables) * [Schema Evolution](/build-with-pinot/data-modeling/schema-evolution) * [Schema Reference](/reference/configuration-reference/schema) # Logical Tables Use logical tables when one query name should span multiple physical Pinot tables without exposing the partitioning scheme to users. Logical tables are a naming and routing layer on top of physical tables. They let you split data by region, age, or operating mode while keeping one user-facing table name. Use a logical table when the split is an implementation detail, not part of the query contract. Keep the physical tables aligned on schema, and use a reference physical table only as a metadata anchor. ## When they help Logical tables are most useful when you need one of these patterns: Different physical tables per region or business unit. Separate offline and realtime tables that still answer one business question. Time-sliced tables that should be queried together. ## Design rules Keep the underlying schemas aligned. Keep the logical name stable. Prefer this pattern only when the underlying split is operationally meaningful; do not use it to hide a modeling problem that should instead be solved with cleaner ingestion. For hybrid-style layouts, make the time boundary explicit so Pinot does not double count overlapping data. ## Example pattern ```json { "tableName": "orders", "brokerTenant": "DefaultTenant", "physicalTableConfigMap": { "ordersUS_OFFLINE": {}, "ordersEU_OFFLINE": {} }, "refOfflineTableName": "ordersUS_OFFLINE" } ``` ## Learn more The original logical-table walkthrough lives in [Logical Table](/architecture-and-concepts/components/table/logical-table). ## What this page covered This page covered when to use logical tables and how they hide physical table splits from readers. ## Next step Read [Schema Evolution](/build-with-pinot/data-modeling/schema-evolution) if the schema needs to grow after the table is already in production. ## Related pages * [Data Modeling](/build-with-pinot/data-modeling) * [Schema and Table Shape](/build-with-pinot/data-modeling/schema) * [Schema Evolution](/build-with-pinot/data-modeling/schema-evolution) * [Original Logical Table Doc](/architecture-and-concepts/components/table/logical-table) # Schema Evolution Evolve Pinot schemas safely by adding columns, reloading segments, and deciding when a new table is the cleaner path. Pinot schema evolution is intentionally narrow. The safe path is to add columns, reload the affected segments, and backfill only when the table type and data flow support it. If the change is more invasive than that, create a new table instead of forcing the old one to stretch. ## What is safe Additive schema changes are the normal path. New columns can be introduced without rewriting the whole table, as long as the ingestion flow and segment reload behavior are understood. ## What is not safe Renaming a column, dropping a column, or changing a column type is not a small schema tweak. Treat those as table redesign work. ## Typical flow 1. Add the new column to the schema (with an appropriate `defaultNullValue` when older rows should show a default). 2. Update the table config or ingestion config if the new field needs transforms, indexes, or upsert strategy entries. 3. Ensure **new consuming segments** start with the updated schema. For an explicit barrier, call [forceCommit](/reference/api-reference/controller-api#post-tablestablenameforcecommit) and poll its status; for transform changes, prefer the pause procedure below. 4. Reload completed segments after that boundary so the segments committed under the old plan also expose the new column metadata (filled with `defaultNullValue` unless you backfill). 5. Backfill historical data if the use case needs real values instead of defaults (offline / hybrid path). ## Realtime consuming segments: when to pause, reload, or force commit Older docs sometimes said “always pause consumption when adding a column.” That is stronger than necessary. Consuming segments are built with the schema and table config that were current when the mutable segment started. Rows already indexed in that mutable segment are not rewritten in place when you change the schema. The practical goal is: * **Completed (immutable) segments** — reload so the new column appears (typically as `defaultNullValue`). * **In-flight consuming segments** — commit them and start new consumers that load the latest schema/config. * **Transforms / derived columns** — stop the old consumer from continuing with the transform plan it loaded when the mutable segment started. Server reload of a consuming segment **requests a force commit** when `pinot.server.instance.reload.consumingSegment` is `true` (the default). The consumer seals asynchronously and a replacement consumer then starts with the latest schema and table config. Reload job completion is not a hard barrier that new consumers are already ONLINE. For an explicit barrier, call `POST /tables/{tableName}/forceCommit` and poll status; see the [force commit API](/reference/api-reference/controller-api#post-tablestablenameforcecommit). ### Decision table (add a column on an existing table) | Table situation | Risk if you only update schema and wait | Recommended steps | | ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **OFFLINE only** | None for stream consumers | Update schema → [reload segments](/operate-pinot/segment-management/segment-reload) → backfill/repush if you need non-default values. | | **Realtime, plain column** (present in the stream or filled only via `defaultNullValue`; no new transform) | Current consuming segment keeps the old schema until it commits; queries on that mutable segment may omit the column or use a virtual default | 1) Update schema 2) `POST /segments/{table}/reload` (consuming reload requests a force commit when enabled) **or** `POST /tables/{table}/forceCommit` and poll it 3) Reload completed segments when they need physical column metadata or new indexes 4) Optional offline backfill for history | | **Realtime with ingestion transforms** (new column populated by `transformConfigs` / Groovy / built-ins, or transform logic changes) | High: the active consumer keeps using its old transform plan, so the segment can contain incorrect or missing derived values | 1) Prefer **pause and wait for completion** → apply schema + table config → reload completed segments → **resume** (clean boundary), **or** 2) apply schema/config → **forceCommit and poll** → reload the segments committed under the old plan 3) Do not assume rows already indexed by the old consumer are repaired in place | | **Realtime full upsert** | Same consumer-boundary issue as plain RT for an additive column; standard full-upsert metadata remains valid across a normal force commit | Follow the plain-RT steps for an additive schema change. Do **not** treat restart as a migration path for immutable upsert settings such as mode, primary/comparison columns, hash function, or delete/out-of-order fields; create a new table and reingest instead. | | **Realtime partial upsert** | Same column/transform issues as above, plus force commit / reload-consuming is **blocked or skipped** in the default `RESTRICTED` mode because replicas can diverge on winner selection | 1) Update the additive schema/config 2) Before pause, force commit, or reload-consuming, set Helix **cluster config** `pinot.server.consuming.segment.consistency.mode=PROTECTED` (via controller cluster configs — not `pinot-server.conf`; see [Consuming Segment Consistency Mode](/build-with-pinot/ingestion/upsert-dedup/upsert#consuming-segment-consistency-mode)) 3) Use a maintenance window and verify upsert results 4) Allowed `partialUpsertStrategies` / `defaultPartialUpsertStrategy` changes require a controlled server restart and are not retroactive 5) Immutable core upsert settings require a new table and reingestion | | **Hybrid (OFFLINE + REALTIME)** | Offline and realtime can disagree until both sides are updated | Evolve schema once → reload offline segments (and backfill) → treat the realtime side with the matching row in this table | ### Why incorrect values can appear * A **consuming** segment’s mutable index is created with a fixed schema/transform pipeline at start. Adding a column or transform mid-segment does not retroactively recompute earlier rows in that mutable segment. * **Completed** segments only gain the new column through reload (defaults) or rebuild/backfill (real values). * **Upsert / dedup** keep cross-segment state inside the server table data manager. Allowed partial-upsert strategy changes require a controlled server restart and are not retroactive. Core upsert/dedup identity and ordering settings are immutable; use a new table and reingest instead of relying on restart. ### Minimal realtime runbook (additive column) For a plain additive column: ```bash # 1) Upload updated schema curl -F schemaName=@myTable.schema http://localhost:9000/schemas # 2) If transforms/indexes changed, PATCH/PUT the table config as well # 3) For an explicit consumer barrier, force commit and poll curl -X POST "http://localhost:9000/tables/myTable/forceCommit" curl -X GET "http://localhost:9000/tables/forceCommitStatus/" # wait until numberOfSegmentsYetToBeCommitted == 0 # 4) If physical metadata/indexes are required, reload the completed segments. # Use segment-specific reloads to avoid force-committing the replacement consumer. curl -X POST "http://localhost:9000/segments/myTable//reload" ``` For a transform change, use a clean pause boundary: ```bash # 1) Pause; pauseConsumption force-commits the current consumers curl -X POST "http://localhost:9000/tables/myTable/pauseConsumption" curl -X GET "http://localhost:9000/tables/myTable/pauseStatus" # wait until the consuming segments have committed # 2) Upload the schema and table config # 3) Reload completed segments while the table is paused curl -X POST "http://localhost:9000/segments/myTable/reload?type=REALTIME" # 4) Resume with consumers built from the new schema/config curl -X POST "http://localhost:9000/tables/myTable/resumeConsumption" ``` ## Reference material Step-by-step offline quickstart walkthrough: [Schema Evolution tutorial](/tutorials/data-ingestion/schema-evolution). Operational APIs: [Segment reload](/operate-pinot/segment-management/segment-reload), [Force commit](/reference/api-reference/controller-api#post-tablestablenameforcecommit), [Pause stream ingestion](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion#pause-stream-ingestion). ## What this page covered * The additive schema-evolution path and the cases where a new table is safer. * When pause, reload, and forceCommit are required for realtime consumers. * Upsert-specific limits (config changes vs schema adds). ## Next step Read the ingestion pages to see how schema design affects batch and stream pipelines. ## Related pages * [Data Modeling](/build-with-pinot/data-modeling) * [Schema and Table Shape](/build-with-pinot/data-modeling/schema) * [Logical Tables](/build-with-pinot/data-modeling/logical-tables) * [Ingestion transformations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations) * [Upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) * [Original Schema Evolution Tutorial](/tutorials/data-ingestion/schema-evolution) # Ingestion Plan Pinot ingestion around batch, stream, upsert, dedup, formats, filesystems, and transformation choices. Ingestion is where Pinot tables become real. Start here to choose the right path for batch or stream data, then refine the design with upsert, dedup, file format, filesystem, transform, and aggregation decisions. The detailed controller and table-config material belongs in [Reference](/reference/reference). This section stays focused on data flow and operational choices. ## Start Here * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) - for data that arrives in files or lands in a warehouse-style batch flow. * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) - for Kafka-style or other event streams that should be queryable quickly. * [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup) - for tables that need one canonical row per key instead of raw event history. * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) - for source formats, file systems, and deep-storage choices. * [Transformations and Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations) - for ingest-time cleanup and pre-aggregation decisions. ## Related Existing Docs * [Import Data](/build-with-pinot/ingestion) * [Data Ingestion Overview](/build-with-pinot/ingestion) * [Ingestion Transformations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations) * [Ingestion Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-aggregations) * [Supported Data Formats](/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats) * [File Systems](/build-with-pinot/ingestion/formats-filesystems/file-systems) ## What this page covered This landing page defines the ingestion subtree and points to the main decision pages. ## Next step Read [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) if your source data arrives in files or prebuilt segments. ## Related pages * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) * [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup) * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) * [Transformations and Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations) # Batch Ingestion Choose batch ingestion when Pinot should load prebuilt data from files, warehouses, or distributed processing jobs. Batch ingestion builds Pinot segments outside the cluster and pushes them into Pinot after the data is already shaped. Use it when the data changes in larger chunks, when you need deterministic backfills, or when the pipeline already produces files or segment artifacts. The most important design choice is not the framework, but the output contract: what the schema looks like, what the table expects, and where the segments land. ## Common batch paths Spark-based ingestion. Hadoop-style distributed ingestion. Backfill jobs for historical ranges. Pinot ships `LaunchBackfillIngestionJob` for the case where the backfilled input may have fewer files than the original ingestion — see [Backfill Data](/build-with-pinot/ingestion/batch-ingestion/backfill-data). Dimension tables and other specialized offline loads. ## What to decide early Decide on the file format, the deep-storage target, and the segment push workflow before you optimize the job itself. Most batch ingestion problems come from mismatched assumptions at those boundaries. ## Select input files Use `includeFileNamePattern` and `excludeFileNamePattern` in the ingestion job spec to filter input paths. Both accept Java NIO `PathMatcher` patterns with either the `glob:` or `regex:` prefix: ```yaml includeFileNamePattern: 'glob:**/*.avro' excludeFileNamePattern: 'regex:.*[.]tmp' ``` The pattern must match the whole normalized path, not only the file name. See [File name patterns](/reference/configuration-reference/job-specification#file-name-patterns) for Java regex syntax, job-spec template escaping, and URI normalization details. ## Learn more The original step-by-step batch docs live in [Import Data](/build-with-pinot/ingestion) and [Data Ingestion Overview](/build-with-pinot/ingestion). ## What this page covered This page covered when to choose batch ingestion and the main design decisions that shape it. ## Next step Read [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) if the source system is a live event stream. ## Related pages * [Ingestion](/build-with-pinot/ingestion) * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) * [Original Batch Docs](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) # Batch Ingestion Guide Batch ingestion of data into Apache Pinot. With batch ingestion you create a table using data already present in a file system such as S3. This is particularly useful when you want to use Pinot to query across large data with minimal latency or to test out new features using a simple data file. ## Choosing a Batch Ingestion Mode Pinot provides several batch ingestion modes. Use the table below to pick the one that fits your environment and data scale. ### Decision Guide | Mode | Best For | Infrastructure | Data Scale | Status | | ---------------------- | ------------------------------------------------ | ----------------- | -------------- | -------------------------- | | Standalone | Dev/test, small jobs, scripted pipelines | None (single JVM) | Up to a few GB | Recommended for dev | | Spark 3 | Production batch ingestion | Spark 3.x cluster | GB to TB+ | Recommended for production | | Hadoop | Existing MapReduce pipelines | Hadoop cluster | GB to TB+ | Legacy | | Flink | Streaming-first orgs, backfill, upsert bootstrap | Flink cluster | GB to TB+ | Active | | LaunchDataIngestionJob | CLI wrapper for Standalone | None | Up to a few GB | Convenience tool | ### When to Use Each Mode **Standalone** is the simplest option and requires no distributed computing framework. It runs segment generation in a single JVM process, making it ideal for development, testing, and small production jobs where data volumes are modest (up to a few GB). It is also well suited for scripted CI/CD pipelines. **Spark 3** is the recommended choice for production batch ingestion at scale. It distributes segment generation across a Spark 3.x cluster, enabling you to process datasets ranging from gigabytes to terabytes and beyond. If you are setting up a new Spark-based pipeline, use this mode. **Hadoop** uses MapReduce to generate segments on a Hadoop cluster. It is considered legacy and is primarily useful if you have existing MapReduce infrastructure and pipelines that you cannot migrate away from. **Flink** is a good fit for organizations that already run Apache Flink. It supports both batch and streaming modes and is especially useful for backfilling offline tables or bootstrapping upsert tables, since the Flink connector can write partitioned segments that participate correctly in upsert semantics. **LaunchDataIngestionJob** is a CLI convenience wrapper that invokes the Standalone runner under the hood. Use it when you want to trigger ingestion from a shell command or cron job without writing custom code. ### Maven Artifact Coordinates All artifacts use the group ID `org.apache.pinot`. Replace `${pinot.version}` with your Pinot release version. | Mode | Artifact ID | Notes | | ---------- | ---------------------------------- | ---------------------------------------------------- | | Standalone | `pinot-batch-ingestion-standalone` | Included in the Pinot binary distribution | | Spark 3 | `pinot-batch-ingestion-spark-3` | Located in `plugins-external/pinot-batch-ingestion/` | | Hadoop | `pinot-batch-ingestion-hadoop` | Located in `plugins-external/pinot-batch-ingestion/` | | Flink | `pinot-flink-connector` | Located in `pinot-connectors/` | | Common | `pinot-batch-ingestion-common` | Shared library used by all modes | Example Maven dependency for Spark 3: ```xml org.apache.pinot pinot-batch-ingestion-spark-3 ${pinot.version} ``` *** ## Getting Started To ingest data from a filesystem, perform the following steps, which are described in more detail in this page: 1. Create schema configuration 2. Create table configuration 3. Upload schema and table configs 4. Upload data Batch ingestion currently supports the following mechanisms to upload the data: * Standalone * [Hadoop](/build-with-pinot/ingestion/batch-ingestion/hadoop) * [Spark](/build-with-pinot/ingestion/batch-ingestion/spark) * [Flink](/build-with-pinot/ingestion/batch-ingestion/flink) Here's an example using standalone local processing. First, create a table using the following CSV data. ``` studentID,firstName,lastName,gender,subject,score,timestampInEpoch 200,Lucy,Smith,Female,Maths,3.8,1570863600000 200,Lucy,Smith,Female,English,3.5,1571036400000 201,Bob,King,Male,Maths,3.2,1571900400000 202,Nick,Young,Male,Physics,3.6,1572418800000 ``` ## Create schema configuration In our data, the only column on which aggregations can be performed is `score`. Secondly, `timestampInEpoch` is the only timestamp column. So, on our schema, we keep `score` as metric and `timestampInEpoch` as timestamp column. ```json { "schemaName": "transcript", "dimensionFieldSpecs": [ { "name": "studentID", "dataType": "INT" }, { "name": "firstName", "dataType": "STRING" }, { "name": "lastName", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "subject", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [{ "name": "timestampInEpoch", "dataType": "LONG", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` Here, we have also defined two extra fields: format and granularity. The format specifies the formatting of our timestamp column in the data source. Currently, it's in milliseconds, so we've specified `1:MILLISECONDS:EPOCH`. ## **Create table configuration** We define a table `transcript` and map the schema created in the previous step to the table. For batch data, we keep the `tableType` as `OFFLINE`. ```json { "tableName": "transcript", "tableType": "OFFLINE", "segmentsConfig": { "replication": 1, "timeColumnName": "timestampInEpoch", "timeType": "MILLISECONDS", "retentionTimeUnit": "DAYS", "retentionTimeValue": 365 }, "tenants": { "broker":"DefaultTenant", "server":"DefaultTenant" }, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY" }, "continueOnError": true, "rowTimeValueCheck": true, "segmentTimeValueCheck": false }, "metadata": {} } ``` ## Upload schema and table configs Now that we have both the configs, upload them and create a table by running the following command: ```bash bin/pinot-admin.sh AddTable \\ -tableConfigFile /path/to/table-config.json \\ -schemaFile /path/to/table-schema.json -exec ``` Check out the table config and schema in the `\[Rest API]` to make sure it was successfully uploaded. ## Upload data We now have an empty table in Pinot. Next, upload the CSV file to this empty table. A table is composed of multiple segments. The segments can be created in the following three ways: * Minion based ingestion\\ * Upload API\\ * Ingestion jobs ### Minion-based ingestion Refer to [SegmentGenerationAndPushTask](/architecture-and-concepts/components/cluster/minion#segmentgenerationandpushtask) ### Upload API There are 2 controller APIs that can be used for a quick ingestion test using a small file. {% hint style="danger" %} **When these APIs are invoked, the controller has to download the file and build the segment locally.** **Hence, these APIs are NOT meant for production environments and for large input files.** {% endhint %} #### /ingestFromFile This API creates a segment using the given file and pushes it to Pinot. All steps happen on the controller. Example usage: To upload a JSON file `data.json` to a table called `foo_OFFLINE`, use below command **Note that** query params need to be URLEncoded. For example, *{"inputFormat":"json"}* in the command below needs to be converted to *%7B%22inputFormat%22%3A%22json%22%7D.* ```bash curl -X POST -F file=@data.json \ -H "Content-Type: multipart/form-data" \ "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE& batchConfigMapStr={"inputFormat":"json"}" ``` The `batchConfigMapStr` can be used to pass in additional properties needed for decoding the file. For example, in case of csv, you may need to provide the delimiter ```bash curl -X POST -F file=@data.csv \ -H "Content-Type: multipart/form-data" \ "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE& batchConfigMapStr={ "inputFormat":"csv", "recordReader.prop.delimiter":"|" }" ``` #### /ingestFromURI This API creates a segment using file at the given URI and pushes it to Pinot. Properties to access the FS need to be provided in the batchConfigMap. All steps happen on the controller.\ Example usage: {% hint style="warning" %} `/ingestFromURI` only allows remote files by default. Controller-local `file:///...` URIs and `LocalPinotFS`-backed reads are rejected unless you explicitly set `controller.ingestFromURI.allowLocalFileSystem=true` in `controller.conf` and restart the controller. {% endhint %} ``` curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE &batchConfigMapStr={ "inputFormat":"json", "input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region":"us-central", "input.fs.prop.accessKey":"foo", "input.fs.prop.secretKey":"bar" } &sourceURIStr=s3://test.bucket/path/to/json/data/data.json" ``` If you intentionally ingest from a file local to the controller host in a trusted environment, enable the controller config above and use a `file:///...` URI such as: ```bash curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE&batchConfigMapStr={\"inputFormat\":\"json\"}&sourceURIStr=file:///var/tmp/pinot/input.json" ``` ### Ingestion jobs Segments can be created and uploaded using tasks known as `DataIngestionJobs`. A job also needs a config of its own. We call this config the `JobSpec`. For our CSV file and table, the `JobSpec` should look like this: ``` executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner' # Recommended to set jobType to SegmentCreationAndMetadataPush for production environment where Pinot Deep Store is configured jobType: SegmentCreationAndTarPush inputDirURI: '/tmp/pinot-quick-start/rawdata/' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '/tmp/pinot-quick-start/segments/' overwriteOutput: true pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'transcript' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 ``` For more detail, refer to [Ingestion job spec](/reference/configuration-reference/job-specification). Now that we have the job spec for our table `transcript`, we can trigger the job using the following command: ```bash bin/pinot-admin.sh LaunchDataIngestionJob \\ -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yaml ``` Once the job successfully finishes, head over to the `\[query console]` and start playing with the data. ### Segment push job type There are 3 ways to upload a Pinot segment: * Segment tar push * Segment URI push * Segment metadata push #### Segment tar push This is the original and default push mechanism. Tar push requires the segment to be stored locally or can be opened as an InputStream on PinotFS. So we can stream the entire segment tar file to the controller. **The push job will:** 1. Upload the entire segment tar file to the Pinot controller. **Pinot controller will:** 1. Save the segment into the controller segment directory(Local or any PinotFS). 2. Extract segment metadata. 3. Add the segment to the table. #### Segment URI push This push mechanism requires the segment tar file stored on a deep store with a globally accessible segment tar URI. URI push is light-weight on the client-side, and the controller side requires equivalent work as the tar push. **The push job will:** 1. POST this segment tar URI to the Pinot controller. **Pinot controller will:** 1. Download segment from the URI and save it to controller segment directory (local or any PinotFS). 2. Extract segment metadata. 3. Add the segment to the table. #### Segment metadata push This push mechanism also requires the segment tar file stored on a deep store with a globally accessible segment tar URI. Metadata push is light-weight on the controller side, there is no deep store download involves from the controller side. **The push job will:** 1. Download the segment based on URI. 2. Extract metadata. 3. Upload metadata to the Pinot Controller. **Pinot Controller will:** 1. Add the segment to the table based on the metadata. **4. Segment Metadata Push with copyToDeepStore** This extends the original Segment Metadata Push for cases, where the segments are pushed to a location not used as deep store. The ingestion job can still do metadata push but ask Pinot Controller to copy the segments into deep store. Those use cases usually happen when the ingestion jobs don't have direct access to deep store but still want to use metadata push for its efficiency, thus using a staging location to keep the segments temporarily. NOTE: the staging location and deep store have to use same storage scheme, like both on s3. This is because the copy is done via PinotFS.copyDir interface that assumes so; but also because this does copy at storage system side, so segments don't need to go through Pinot Controller at all. To make this work, grant Pinot controllers access to the staging location. For example on AWS, this may require adding an access policy like this example for the controller EC2 instances: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::metadata-push-staging", "arn:aws:s3:::metadata-push-staging/*" ] } ] } ``` Then use metadata push to add one extra config like this one: ``` ... jobType: SegmentCreationAndMetadataPush ... outputDirURI: 's3://metadata-push-staging/stagingDir/' ... pushJobSpec: copyToDeepStoreForMetadataPush: true ... ``` ### Consistent data push and rollback Pinot supports atomic update on segment level, which means that when data consisting of multiple segments are pushed to a table, as segments are replaced one at a time, queries to the broker during this upload phase may produce inconsistent results due to interleaving of old and new data. See [consistent-push-and-rollback.md](/operate-pinot/segment-management/consistent-push-and-rollback) for how to enable this feature. ### Segment fetchers When Pinot segment files are created in external systems (Hadoop/spark/etc), there are several ways to push those data to the Pinot controller and server: 1. Push segment to shared NFS and let pinot pull segment files from the location of that NFS. See [Segment URI Push](#segment-uri-push). 2. Push segment to a Web server and let pinot pull segment files from the Web server with HTTP/HTTPS link. See [Segment URI Push](#segment-uri-push). 3. Push segment to PinotFS(HDFS/S3/GCS/ADLS) and let pinot pull segment files from PinotFS URI. See [Segment URI Push](#segment-uri-push) and [Segment Metadata Push](#segment-metadata-push). 4. Push segment to other systems and implement your own segment fetcher to pull data from those systems. The first three options are supported out of the box within the Pinot package. As long your remote jobs send Pinot controller with the corresponding URI to the files, it will pick up the file and allocate it to proper Pinot servers and brokers. To enable Pinot support for PinotFS, you'll need to provide [PinotFS](/build-with-pinot/ingestion/formats-filesystems/file-systems) configuration and proper Hadoop dependencies. ### Persistence By default, Pinot does not come with a storage layer, so all the data sent, won't be stored in case of a system crash. In order to persistently store the generated segments, you will need to change controller and server configs to add deep storage. Checkout [File systems](/build-with-pinot/ingestion/formats-filesystems/file-systems) for all the info and related configs. ### Tuning #### **Standalone** Since pinot is written in Java, you can set the following basic Java configurations to tune the segment runner job - * Log4j2 file location with `-Dlog4j2.configurationFile` * Plugin directory location with `-Dplugins.dir=/opt/pinot/plugins` * JVM props, like `-Xmx8g -Xms4G` If you are using the docker, you can set the following under `JAVA_OPTS` variable. #### Hadoop You can set `-D mapreduce.map.memory.mb=8192` to set the mapper memory size when submitting the Hadoop job. The Hadoop batch plugin is under `plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/`. Point `plugins.dir` at both `plugins` and `plugins-external` (semicolon-separated) and put the shaded Hadoop plugin jar on `HADOOP_CLASSPATH`. See [Hadoop](/build-with-pinot/ingestion/batch-ingestion/hadoop). #### Spark You can add config `spark.executor.memory` to tune the memory usage for segment creation when submitting the Spark job. The Spark 3 batch plugin is under `plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/`. `pinot-all` does not include it. Point `plugins.dir` at both `plugins` and `plugins-external`, and put the shaded Spark plugin jar on the Spark classpath. See [Spark](/build-with-pinot/ingestion/batch-ingestion/spark). # SQL Insert Into From Files Insert a file into Pinot from Query Console {% hint style="info" %} This feature is supported after the 0.11.0 release. Reference PR: {% endhint %} ### Prerequisite * Ensure you have available Pinot Minion instances deployed within the cluster. * Pinot version is 0.11.0 or above ### How it works 1. Parse the query with the table name and directory URI along with a list of options for the ingestion job. 2. Call controller minion task execution API endpoint to schedule the task on minion 3. Response has the schema of table name and task job id. ### Usage Syntax > INSERT INTO \[database.]table FROM FILE dataDirURI OPTION ( k=v ) \[, OPTION (k=v)]\* ### Example ``` SET taskName = 'myTask-s3'; SET input.fs.className = 'org.apache.pinot.plugin.filesystem.S3PinotFS'; SET input.fs.prop.accessKey = 'my-key'; SET input.fs.prop.secretKey = 'my-secret'; SET input.fs.prop.region = 'us-west-2'; INSERT INTO "baseballStats" FROM FILE 's3://my-bucket/public_data_set/baseballStats/rawdata/' ``` ## Insert Rows into Pinot We are actively developing this feature... The details will be revealed soon. # Upload Pinot Segment Using CLI Upload existing Pinot segments to a controller. This guide explains how to upload already-built Pinot segments to a Pinot controller, which REST endpoint to call, and when to use tar push, URI push, or metadata push. Use this flow when your segment `.tar.gz` files already exist outside Pinot, for example when migrating from an old cluster, backfilling from another system, or re-registering segments that already live in deep storage. Before you upload, do the following: 1. [Create a schema configuration](/start-here/first-table-and-schema#4-save-the-schema) or confirm one exists that matches the segment you want to upload. 2. [Create a table configuration](/reference/configuration-reference/table) or confirm one exists that matches the segment you want to upload. 3. If needed, upload the schema and table configs. ```bash pinot-admin.sh AddTable \\ -tableConfigFile /path/to/table-config.json \\ -schemaFile /path/to/table-schema.json -exec ``` 4. Make sure the controller can read the segment source: * For tar push, the client must be able to stream the segment tar file to the controller. * For URI push and metadata push, the controller must be able to access the URI scheme you use. For PinotFS-backed schemes such as HDFS, S3, GCS, and ADLS, configure the matching [Pinot file system](/build-with-pinot/ingestion/formats-filesystems/file-systems). For custom schemes, implement a [segment fetcher](/develop-and-contribute/extending-pinot/segment-fetchers). ## Controller upload endpoints The controller exposes three upload endpoints: | Endpoint | Use case | Content type | Notes | | ---------------------------- | ---------------------------------------- | ------------------------------------------- | ----------------------------------------------------- | | `POST /v2/segments` | Preferred single-segment upload endpoint | `multipart/form-data` or `application/json` | Recommended for tar push, URI push, and metadata push | | `POST /segments` | Legacy single-segment upload endpoint | `multipart/form-data` or `application/json` | Still supported, but prefer `/v2/segments` | | `POST /segments/batchUpload` | Batch metadata push | `multipart/form-data` | Only supports metadata push for multiple segments | `/v2/segments` is the endpoint to document and use by default. The legacy `/segments` endpoint is still present for backward compatibility. Its JSON-based URI push path keeps the original `DOWNLOAD_URI` instead of moving the segment into a Pinot-chosen final location, so new integrations should use `/v2/segments`. ## Common request options ### Query parameters All three upload modes use the same query parameters: | Query parameter | Required | Default | Description | | ------------------------------ | -------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------- | | `tableName` | Recommended for single upload, required for batch upload | None | Table name to upload into. Pinot can sometimes derive it from the segment metadata, but you should pass it explicitly. | | `tableType` | No | `OFFLINE` | `OFFLINE` or `REALTIME` | | `enableParallelPushProtection` | No | `false` | Reject concurrent uploads for the same segment | | `allowRefresh` | No | `true` | Allow an existing segment to be refreshed instead of failing the upload | Example: ``` POST /v2/segments?tableName=myTable&tableType=OFFLINE&enableParallelPushProtection=false&allowRefresh=true ``` ### Headers | Header | Required | Applies to | Description | | ---------------------------- | ---------------------------------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `UPLOAD_TYPE` | No for tar push, yes for URI and metadata push | All uploads | `SEGMENT` (default), `URI`, or `METADATA` | | `DOWNLOAD_URI` | Yes for URI push and metadata push | URI push, metadata push | Source URI of the segment tar file | | `COPY_SEGMENT_TO_DEEP_STORE` | No | Metadata push | If `true`, controller copies the segment from the source URI into Pinot deep store and rewrites the stored download URI | | `CRYPTER` | No | All uploads | Crypter class name if the uploaded payload is encrypted | ### Offline upsert upload validation For offline upsert tables, Pinot applies an extra upload-time validation when it can resolve a partition column from the table config. Pinot checks `instanceAssignmentConfigMap.OFFLINE` first, then legacy `replicaGroupStrategyConfig.partitionColumn`, and finally a single-column `segmentPartitionConfig`. When one of those configs identifies the partition column, every uploaded segment must expose exactly one partition id for that column in segment metadata. This applies to both single-segment upload endpoints and `POST /segments/batchUpload`. For example, if Pinot resolves `playerId` as the partition column, the segment metadata must include one value such as `column.playerId.partitionValues=2`. Uploads are rejected with `400 BAD_REQUEST` when the partition metadata is missing for that column or lists multiple partition ids such as `2,3`. ## Push modes ### Tar push Tar push is the original and default upload mode. Use it when the client can stream the full segment tar file to the controller. **Request shape** * Endpoint: `POST /v2/segments` * Content type: `multipart/form-data` * Headers: `UPLOAD_TYPE` omitted or set to `SEGMENT` * Body: one multipart file part containing the segment `.tar.gz` **What the controller does** 1. Stores the uploaded segment in the controller's segment directory or deep store. 2. Extracts segment metadata. 3. Adds or refreshes the segment in the target table. Example: ```bash curl -X POST "http://localhost:9000/v2/segments?tableName=myTable&tableType=OFFLINE" \\ -F "file=@/path/to/myTable_2024-01-01_2024-01-02_0.tar.gz" ``` If you prefer the Pinot CLI, `pinot-admin.sh UploadSegment` uses tar push for local segment directories: ```bash pinot-admin.sh UploadSegment \\ -controllerHost localhost \\ -controllerPort 9000 \\ -segmentDir /path/to/local/dir \\ -tableName myTable ``` ### URI push URI push is best when the segment tar file already exists in deep storage or another controller-readable remote system. **Request shape** * Endpoint: `POST /v2/segments` * Content type: `application/json` * Headers: * `UPLOAD_TYPE: URI` * `DOWNLOAD_URI: ` * Body: empty JSON payload is fine; the controller uses the headers **What the controller does** 1. Downloads the segment tar from `DOWNLOAD_URI`. 2. Stores it in the controller's segment directory or deep store. 3. Extracts metadata. 4. Adds or refreshes the segment in the table. Example: ```bash curl -X POST "http://localhost:9000/v2/segments?tableName=myTable&tableType=OFFLINE" \\ -H "Content-Type: application/json" \\ -H "UPLOAD_TYPE: URI" \\ -H "DOWNLOAD_URI: s3://bucket/pinot-segments/myTable_2024-01-01_2024-01-02_0.tar.gz" \\ -d '{}' ``` Use URI push only when the controller can resolve the URI scheme. If the source is on HDFS, S3, GCS, ADLS, or a custom system, configure Pinot with the appropriate [Pinot file system](/build-with-pinot/ingestion/formats-filesystems/file-systems) or [segment fetcher](/develop-and-contribute/extending-pinot/segment-fetchers). ### Metadata push Metadata push is the most controller-efficient option when the segment tar already exists in a reachable storage system. Instead of uploading the full segment tar, the client uploads segment metadata and tells the controller where the tar already lives. **Request shape** * Endpoint: `POST /v2/segments` * Content type: `multipart/form-data` * Headers: * `UPLOAD_TYPE: METADATA` * `DOWNLOAD_URI: ` * Optional: `COPY_SEGMENT_TO_DEEP_STORE: true` * Body: one multipart file part containing the metadata tarball for the segment The metadata tarball contains the segment metadata files, typically `creation.meta` and `metadata.properties`. **What the controller does** 1. Reads the uploaded metadata bundle. 2. Uses `DOWNLOAD_URI` as the segment download location. 3. Adds or refreshes the segment in the table without downloading the full tar just to inspect metadata. If you set `COPY_SEGMENT_TO_DEEP_STORE: true`, the controller copies the segment from `DOWNLOAD_URI` into Pinot deep store and stores the final deep-store URI in segment metadata. This is useful when the ingestion job writes to a staging location instead of the final deep-store path. Example: ```bash curl -X POST "http://localhost:9000/v2/segments?tableName=myTable&tableType=OFFLINE" \\ -H "UPLOAD_TYPE: METADATA" \\ -H "DOWNLOAD_URI: s3://staging-bucket/segments/myTable_2024-01-01_2024-01-02_0.tar.gz" \\ -H "COPY_SEGMENT_TO_DEEP_STORE: true" \\ -F "file=@/path/to/myTable_2024-01-01_2024-01-02_0.metadata.tar.gz" ``` `COPY_SEGMENT_TO_DEEP_STORE` is only useful for metadata push. The staging URI and Pinot deep store should use the same storage scheme because the copy happens through PinotFS. ### Batch metadata push If you need to metadata-push many segments in one call, use `POST /segments/batchUpload`. **Request shape** * Endpoint: `POST /segments/batchUpload` * Content type: `multipart/form-data` * Query parameters: `tableName` and `tableType` are required * Header: `UPLOAD_TYPE: METADATA` * Body: one multipart part containing an uber tarball with: * each segment's `creation.meta` * each segment's `metadata.properties` * an `all_segments_metadata` file mapping segment names to `DOWNLOAD_URI` values This endpoint is only for metadata push. ## Job types and Pinot Admin mapping If you are pushing from a batch ingestion job, the `jobType` maps to controller upload mode like this: | Job type | Push mode | Controller endpoint | | --------------------------------------------------------- | ------------------- | ---------------------------- | | `SegmentTarPush` or `SegmentCreationAndTarPush` | Tar push | `POST /v2/segments` | | `SegmentUriPush` or `SegmentCreationAndUriPush` | URI push | `POST /v2/segments` | | `SegmentMetadataPush` or `SegmentCreationAndMetadataPush` | Metadata push | `POST /v2/segments` | | `SegmentMetadataPush` with `batchSegmentUpload: true` | Batch metadata push | `POST /segments/batchUpload` | For ingestion jobs, define the push behavior in the [ingestion job spec](/reference/configuration-reference/job-specification). Example: ```yaml executionFrameworkSpec: name: standalone segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner segmentMetadataPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner jobType: SegmentCreationAndMetadataPush pinotClusterSpecs: - controllerURI: http://localhost:9000 pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000 copyToDeepStoreForMetadataPush: true ``` Then launch it with: ```bash pinot-admin.sh LaunchDataIngestionJob \\ -jobSpecFile /path/to/job-spec.yaml ``` ## Choosing the right mode | Mode | Use it when | Tradeoff | | ------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | Tar push | The client has the segment tar locally and can upload it directly | Largest payload sent to controller | | URI push | The segment tar already exists at a controller-readable URI | Controller still downloads the full segment tar | | Metadata push | The segment tar already exists remotely and you want the lightest controller-side registration path | Requires a metadata bundle and a valid `DOWNLOAD_URI` | For production clusters with deep store configured, `SegmentCreationAndMetadataPush` is generally the preferred ingestion-job mode. # Spark Batch ingestion of data into Apache Pinot using Apache Spark. Pinot supports Apache Spark 3.x as a processor to create and push segment files to the database. Pinot distribution is bundled with the Spark code to process your files and convert and upload them to Pinot. To set up Spark, do one of the following: * Use the Spark-Pinot Connector. For more information, see the [ReadMe](https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-spark-3-connector/README.md). * Follow the instructions below. You can follow the [local install guide](/start-here/install/local#1-download-or-build-apache-pinot) to build Pinot from source. The resulting JAR file can be found in `pinot/target/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar` If you do build Pinot from Source, you should consider opting into using the `build-shaded-jar` jar profile with `-Pbuild-shaded-jar`. While Pinot does not bundle spark into its jar, it does bundle certain hadoop libraries. Next, you need to change the execution config in the [job spec](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion#create-schema-configuration) to the following: ``` # executionFrameworkSpec: Defines ingestion jobs to be running. executionFrameworkSpec: # name: execution framework name name: 'spark' # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentGenerationJobRunner' # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentTarPushJobRunner' # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentUriPushJobRunner' #segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentMetadataPushJobRunner' # extraConfigs: extra configs for execution framework. extraConfigs: # stagingDir is used in distributed filesystem to host all the segments then move this directory entirely to output directory. stagingDir: your/local/dir/staging ``` Spark job specs can filter input paths with `includeFileNamePattern` and `excludeFileNamePattern`. Both properties accept Java NIO `glob:` and `regex:` patterns; see [File name patterns](/reference/configuration-reference/job-specification#file-name-patterns) for working examples and path-normalization details. ## Required jars and plugins.dir `pinot-all-*-jar-with-dependencies.jar` does **not** bundle Spark or Hadoop batch-ingestion plugins. Those live under `plugins-external/` in the binary distribution (see `pinot-assembly.xml`). You must put them on the Spark classpath **and** point `plugins.dir` at directories that contain them. To run Spark ingestion you need: * `pinot-all` jar — under `lib/` in the package * `pinot-batch-ingestion-spark-3` shaded plugin jar — under `plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/` * Any other plugins your job uses (record readers, file systems) — under `plugins/` (for example `pinot-avro`, `pinot-parquet`, `pinot-s3`) `plugins.dir` accepts a **semicolon-separated** list of directories. Include both `plugins` and `plugins-external` so record readers and the Spark batch runners load correctly: ``` -Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins;${PINOT_DISTRIBUTION_DIR}/plugins-external ``` Put the Spark batch plugin and `pinot-all` on the driver (and executor) classpath with `spark.driver.extraClassPath` / `spark.executor.extraClassPath`: ``` spark.driver.extraClassPath => ${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/pinot-batch-ingestion-spark-3-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar ``` The complete local-mode `spark-submit` command: ``` export PINOT_VERSION=1.4.0 #set to the Pinot version you have installed export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin SPARK_BATCH_PLUGIN=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/pinot-batch-ingestion-spark-3-${PINOT_VERSION}-shaded.jar PINOT_ALL_JAR=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar spark-submit \ --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \ --master local --deploy-mode client \ --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins;${PINOT_DISTRIBUTION_DIR}/plugins-external" \ --conf "spark.driver.extraClassPath=${SPARK_BATCH_PLUGIN}:${PINOT_ALL_JAR}" \ --conf "spark.executor.extraClassPath=${SPARK_BATCH_PLUGIN}:${PINOT_ALL_JAR}" \ local://${PINOT_ALL_JAR} \ -jobSpecFile /path/to/spark_job_spec.yaml ``` Ensure environment variables `PINOT_ROOT_DIR` and `PINOT_VERSION` are set properly. **Note**: You should change the `master` to `yarn` and `deploy-mode` to `cluster` for production environments. {% hint style="info" %} The `spark-core` dependency is not included in Pinot jars since the 0.10.0 release. If you run into runtime issues, make sure your Spark environment provides the dependency, or [build from source](/start-here/getting-started) with the matching Spark profile. {% endhint %} ### Running on YARN The example below uses YARN with `client` deploy mode so the Spark driver can read the unpacked Pinot distribution referenced by `plugins.dir`. Before running it: * Build Pinot from source with option `-DuseProvidedHadoop` * Keep the unpacked Pinot distribution available on the submit host. * Copy the ingestion spec YAML to S3, HDFS, or another location accessible through `--files`. * Add the Spark batch plugin and `pinot-all` through `--jars` so Spark distributes them to executors. * Point the driver and executor classpaths at the distributed jar names. For YARN `cluster` deploy mode, the driver runs remotely. You must separately distribute and unpack the `plugins` and `plugins-external` directories, then set `plugins.dir` to paths that exist in the remote driver container. **Example** ``` spark-submit \ --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \ --master yarn --deploy-mode client \ --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins;${PINOT_DISTRIBUTION_DIR}/plugins-external" \ --conf "spark.driver.extraClassPath=pinot-batch-ingestion-spark-3-${PINOT_VERSION}-shaded.jar:pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \ --conf "spark.executor.extraClassPath=pinot-batch-ingestion-spark-3-${PINOT_VERSION}-shaded.jar:pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \ --jars "${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/pinot-batch-ingestion-spark-3-${PINOT_VERSION}-shaded.jar,${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" \ --files s3://path/to/spark_job_spec.yaml \ ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \ -jobSpecFile spark_job_spec.yaml ``` ### FAQ Q - **I am getting the following exception - `Class has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0`** Since 0.8.0 release, Pinot binaries are compiled with JDK 11. If you are using Spark along with Hadoop 2.7+, you need to use the Java8 version of Pinot. Currently, you need to [build jdk 8 version from source](/start-here/getting-started). Q - **I am not able to find `pinot-batch-ingestion-spark` jar.** Since Pinot 0.10.0, the Spark batch plugin is located under `plugins-external/pinot-batch-ingestion/` in the binary distribution (in older versions it lived under `plugins/`). Current Spark 3 builds ship `pinot-batch-ingestion-spark-3-*-shaded.jar`. Q - **Spark is not able to find the jars** **leading to** **`java.nio.file.NoSuchFileException`** This means the classpath for spark job has not been configured properly. If you are running spark in a distributed environment such as Yarn or k8s, make sure both `spark.driver.extraClassPath` and `spark.executor.extraClassPath` are set. Also, the jars in `driver.extraClassPath` should be added to `--jars` argument in `spark-submit` so that spark can distribute those jars to all the nodes in your cluster. You also need to take provide appropriate scheme with the file path when running the jar. In this doc, we have used `local://` but it can be different depending on your cluster setup. Q - **Spark job failing while pushing the segments.** It can be because of misconfigured `controllerURI` in job spec yaml file. If the controllerURI is correct, make sure it is accessible from all the nodes of your YARN or k8s cluster. Q - **My data gets overwritten during ingestion.** Set [segmentPushType](/reference/configuration-reference/table#segments-config) to `APPEND` in the tableConfig. If already set to `APPEND`, this is likely due to a missing `timeColumnName` in your table config. If you can't provide a time column, use our[ segment name generation configs](/reference/configuration-reference/job-specification#segment-name-generator-spec) in ingestion spec. Generally using `inputFile` segment name generator should fix your issue. Q - **I am getting `java.lang.RuntimeException: java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins/*`** Removing `-Dplugins.dir=...` from `spark.driver.extraJavaOptions` can fix this when the plugin jars are already on the Spark classpath via `extraClassPath` and `--jars`. Prefer pointing `plugins.dir` at real local paths that exist on the driver (`plugins` and `plugins-external` under the distribution). Q - Getting `Class not found:` exception (for example `SparkSegmentGenerationJobRunner`) The Spark/Hadoop batch runners are **not** inside `pinot-all`. Confirm that: 1. `extraClassPath` for both driver and executors includes the shaded jar under `plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-3/` 2. That same jar is listed in `--jars` for cluster mode 3. `plugins.dir` includes `${PINOT_DISTRIBUTION_DIR}/plugins-external` (semicolon-separated with `plugins` if you also need record readers / file-system plugins from `plugins/`) 4. The job spec uses the Spark 3 package: `org.apache.pinot.plugin.ingestion.batch.spark3.*` # Flink Batch ingestion of data into Apache Pinot using Apache Flink. Apache Pinot supports using Apache Flink as a processing framework to generate and upload segments. The Pinot distribution includes a [PinotSink](https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/PinotSink.java) that can be integrated into Flink applications (streaming or batch) to directly write data as segments into Pinot tables. The `PinotSink` supports offline tables, realtime tables, and upsert tables (full upsert only). Data is buffered in memory and flushed as segments when the configured threshold is reached, then uploaded to the Pinot cluster. ## Requirements * **Flink 2.2.0 or later** – Uses the new Flink 2.x `Sink` API. Java 21 support is included. * **Java 11+** – Flink 2.x requires a minimum of Java 11. ## Maven Dependency To use the Pinot Flink Connector in your Flink job, add the following dependency to your `pom.xml`: ```xml org.apache.pinot pinot-flink-connector ${pinot.version} ``` Replace `${pinot.version}` with the Pinot version you're using. For the latest stable version, check the [Apache Pinot releases](https://pinot.apache.org/download/). **Note**: The connector transitively includes dependencies for: * `pinot-controller` - For controller client APIs * `pinot-segment-writer-file-based` - For segment generation * `flink-streaming-java` - Flink 2.x core dependency ## Offline Table Ingestion ### Quick Start Example ```java // Set up Flink environment and data source StreamExecutionEnvironment execEnv = StreamExecutionEnvironment.getExecutionEnvironment(); execEnv.setParallelism(2); // Configure row type RowTypeInfo typeInfo = new RowTypeInfo( new TypeInformation[]{Types.FLOAT, Types.FLOAT, Types.STRING, Types.STRING}, new String[]{"lon", "lat", "address", "name"}); DataStream srcRows = execEnv.fromData(...); // Create a PinotAdminClient to fetch Pinot schema and table config String controllerUrl = "http://localhost:9000"; URI controllerUri = URI.create(controllerUrl); String controllerAddress = controllerUri.getAuthority(); String controllerPath = controllerUri.getPath(); if (controllerPath != null && !controllerPath.isEmpty() && !"/".equals(controllerPath)) { controllerAddress += controllerPath.endsWith("/") ? controllerPath.substring(0, controllerPath.length() - 1) : controllerPath; } Properties properties = new Properties(); properties.setProperty(PinotAdminTransport.ADMIN_TRANSPORT_SCHEME, controllerUri.getScheme()); try (PinotAdminClient client = new PinotAdminClient(controllerAddress, properties)) { // Fetch Pinot schema Schema schema = client.getSchemaClient().getSchemaObject("starbucksStores"); // Fetch Pinot table config TableConfig tableConfig = client.getTableClient().getTableConfigObjectForType("starbucksStores", TableType.OFFLINE); // Create Flink Pinot Sink (Flink 2.x API) srcRows.sinkTo(new PinotSink<>( new FlinkRowGenericRowConverter(typeInfo), tableConfig, schema, controllerUrl)); } execEnv.execute(); ``` Passing `controllerUrl` to `PinotSink` lets the sink inject the `push.controllerUri` and default `outputDirURI` values it needs for segment upload. ### Table Configuration The `PinotSink` uses the TableConfig to determine batch ingestion settings for segment generation and upload. Here's an example table configuration: ```json { "tableName": "starbucksStores_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { // ... }, "tenants": { // ... }, "tableIndexConfig": { // ... }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "HOURLY", "batchConfigMaps": [ { "outputDirURI": "file:///tmp/pinotoutput", "overwriteOutput": "false", "push.controllerUri": "http://localhost:9000" } ] } } } ``` Required configurations: * `outputDirURI` - Directory where segments are written before upload * `push.controllerUri` - Pinot controller URL for segment upload For a complete executable example, refer to [FlinkQuickStart.java](https://github.com/apache/pinot/blob/master/pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/FlinkQuickStart.java). ## Realtime Table Ingestion ### Non-Upsert Realtime Tables For standard realtime tables without upsert, use the same approach as offline tables, but specify `REALTIME` as the table type: ```java // Reuse the PinotAdminClient setup from the offline example above... // Fetch table config for realtime table Schema schema = client.getSchemaClient().getSchemaObject("myTable"); TableConfig tableConfig = client.getTableClient().getTableConfigObjectForType("myTable", TableType.REALTIME); // Same sink configuration srcRows.sinkTo(new PinotSink<>( new FlinkRowGenericRowConverter(typeInfo), tableConfig, schema, controllerUrl)); execEnv.execute(); ``` ### Upsert Tables #### Full Upsert Tables Flink connector supports backfilling full upsert tables where each record contains all columns. The uploaded segments will correctly participate in upsert semantics based on the comparison column value. **Requirements:** 1. **Partitioning**: Data must be partitioned using the same strategy as the upstream stream (e.g., Kafka) 2. **Parallelism**: Flink job parallelism must match the number of upstream stream/table partitions 3. **Comparison Column**: The values of the comparison column must have ordering consistent with the upstream stream. This ensures that Pinot can correctly resolve which record is the latest for a given key. See [Pinot upsert comparison column docs](/build-with-pinot/ingestion/upsert-dedup/upsert#comparison-column) for important considerations. **Example:** ```java // Set up Flink environment StreamExecutionEnvironment execEnv = StreamExecutionEnvironment.getExecutionEnvironment(); execEnv.setParallelism(2); // MUST match number of partitions in stream/table // Configure row type matching your upsert table schema RowTypeInfo typeInfo = new RowTypeInfo( new TypeInformation[]{Types.INT, Types.STRING, Types.STRING, Types.FLOAT, Types.LONG, Types.BOOLEAN}, new String[]{"playerId", "name", "game", "score", "timestampInEpoch", "deleted"}); DataStream srcRows = execEnv.fromData(...); String controllerUrl = "http://localhost:9000"; URI controllerUri = URI.create(controllerUrl); String controllerAddress = controllerUri.getAuthority(); String controllerPath = controllerUri.getPath(); if (controllerPath != null && !controllerPath.isEmpty() && !"/".equals(controllerPath)) { controllerAddress += controllerPath.endsWith("/") ? controllerPath.substring(0, controllerPath.length() - 1) : controllerPath; } Properties properties = new Properties(); properties.setProperty(PinotAdminTransport.ADMIN_TRANSPORT_SCHEME, controllerUri.getScheme()); try (PinotAdminClient client = new PinotAdminClient(controllerAddress, properties)) { Schema schema = client.getSchemaClient().getSchemaObject("myUpsertTable"); TableConfig tableConfig = client.getTableClient().getTableConfigObjectForType("myUpsertTable", TableType.REALTIME); // IMPORTANT: Partition data by primary key using the SAME logic as the stream srcRows.partitionCustom( (Partitioner) (key, partitions) -> key % partitions, r -> (Integer) r.getField("playerId")) // Primary key field .sinkTo(new PinotSink<>( new FlinkRowGenericRowConverter(typeInfo), tableConfig, schema, controllerUrl)); } execEnv.execute(); ``` **How Partitioning Works:** When uploading segments for upsert tables, Pinot uses a special segment naming convention [UploadedRealtimeSegmentName](https://github.com/apache/pinot/blob/7f701245ca71c482ce71456e4e5082bfa82d5e14/pinot-common/src/main/java/org/apache/pinot/common/utils/UploadedRealtimeSegmentName.java) that encodes the partition ID. The format is: ``` {prefix}__{tableName}__{partitionId}__{uploadTimeMs}__{sequenceId} ``` Example: `flink__myTable__0__1724045187__1` Each Flink subtask generates segments for a specific partition based on its subtask index. The segments are then assigned to the same server instances that handle that partition for stream-consumed segments, ensuring correct upsert behavior across all segments. **Configuration Options:** You can customize segment generation using additional constructor parameters: ```java new PinotSink<>( recordConverter, tableConfig, schema, segmentFlushMaxNumRecords, // Default: 500,000, number of rows per segment executorPoolSize, // Default: 5, number of threads to use to upload segment segmentNamePrefix, // Default: "flink" segmentUploadTimeMs // Default: current time, upload time value to encode in segment name ) ``` #### Partial Upsert Tables **WARNING**: Flink-based upload is **not recommended** for partial upsert tables. In partial upsert tables, uploaded segments contain only a subset of columns or an intermdiate row for a primary key. If the uploaded row is not in its final state and subsequent updates arrive via the stream, the partial upsert merger may produce inconsistent results between replicas. This can lead to data inconsistency that is difficult to detect and resolve. For partial upsert tables, prefer stream-based ingestion only or ensure uploaded data represents the final state for each primary key. ## Advanced Configuration ### Segment Flush Control Control when segments are flushed and uploaded: ```java // Same setup as previous examples... long segmentFlushMaxNumRecords = 1000000; // Flush after 1M records int executorPoolSize = 10; // Thread pool size for async uploads srcRows.sinkTo(new PinotSink<>( new FlinkRowGenericRowConverter(typeInfo), tableConfig, schema, segmentFlushMaxNumRecords, executorPoolSize )); ``` ### Segment Naming Customize segment naming and upload time for better organization: ```java // Same setup as previous examples... String segmentNamePrefix = "flink_job_daily"; Long segmentUploadTimeMs = 1724045185000L; // Group segments by upload run time srcRows.sinkTo(new PinotSink<>( new FlinkRowGenericRowConverter(typeInfo), tableConfig, schema, DEFAULT_SEGMENT_FLUSH_MAX_NUM_RECORDS, DEFAULT_EXECUTOR_POOL_SIZE, segmentNamePrefix, segmentUploadTimeMs )); ``` ## Migration from Flink 1.x > **Important:** The connector now requires Flink 2.2.0 or later and Java 11+. The old `PinotSinkFunction` (based on Flink 1.x `SinkFunction` API) is deprecated and does not work with Flink 2.x. ### Old API (Flink 1.x – Deprecated) ```java // This no longer works with Flink 2.x srcRows.addSink(new PinotSinkFunction<>(...)); ``` ### New API (Flink 2.x – Required) ```java // Use this for Flink 2.2.0 and later srcRows.sinkTo(new PinotSink<>(...)); ``` **Key Changes:** * Replace `.addSink()` with `.sinkTo()` * Replace `PinotSinkFunction` with `PinotSink` * Update your Flink version to 2.2.0 or later * Java 21 is now supported ## Additional Resources * [Design Proposal](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=177045634) - Original design motivation * [PR #13107](https://github.com/apache/pinot/pull/13107) - Externally partitioned segments for upsert tables * [PR #13837](https://github.com/apache/pinot/pull/13837) - Flink connector enhancements for upsert backfill * [PR #18250](https://github.com/apache/pinot/pull/18250) - Flink 2.2.0 upgrade * [Table Configuration Reference](/reference/configuration-reference/table) * [Schema Configuration Reference](/reference/configuration-reference/schema) # Hadoop Batch ingestion of data into Apache Pinot using Apache Hadoop. ## Segment Creation and Push Pinot supports [Apache Hadoop](https://hadoop.apache.org) as a processor to create and push segment files to the database. Pinot distribution is bundled with the Hadoop batch-ingestion plugin to process your files and convert and upload them to Pinot. You can follow the [local install guide](/start-here/install/local#1-download-or-build-apache-pinot) to build Pinot from source. The resulting JAR file can be found in `pinot/target/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar` {% hint style="info" %} `pinot-all-*-jar-with-dependencies.jar` does **not** include the Hadoop batch-ingestion plugin. That plugin is packaged under `plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/` in the binary distribution. You must put it on the Hadoop classpath and include `plugins-external` in `plugins.dir`. {% endhint %} Next, you need to change the execution config in the job spec to the following - ``` # executionFrameworkSpec: Defines ingestion jobs to be running. executionFrameworkSpec: # name: execution framework name name: 'hadoop' # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner' # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentTarPushJobRunner' # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentUriPushJobRunner' # segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface. segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentMetadataPushJobRunner' # extraConfigs: extra configs for execution framework. extraConfigs: # stagingDir is used in distributed filesystem to host all the segments then move this directory entirely to output directory. stagingDir: your/local/dir/staging ``` Hadoop job specs can filter input paths with `includeFileNamePattern` and `excludeFileNamePattern`. Both properties accept Java NIO `glob:` and `regex:` patterns; see [File name patterns](/reference/configuration-reference/job-specification#file-name-patterns) for working examples and path-normalization details. You can check out the sample job spec here. Finally execute the hadoop job using the command - ``` export PINOT_VERSION=1.4.0 #set to the Pinot version you have installed export PINOT_DISTRIBUTION_DIR=${PINOT_ROOT_DIR}/build/ HADOOP_BATCH_PLUGIN=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pinot-batch-ingestion-hadoop-${PINOT_VERSION}-shaded.jar PINOT_ALL_JAR=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar # plugins.dir accepts a semicolon-separated list. Include plugins (record readers, FS) and plugins-external (Hadoop batch runners). export HADOOP_CLIENT_OPTS="-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins;${PINOT_DISTRIBUTION_DIR}/plugins-external -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" export HADOOP_CLASSPATH="${HADOOP_BATCH_PLUGIN}:${PINOT_ALL_JAR}:${HADOOP_CLASSPATH}" hadoop jar \ ${PINOT_ALL_JAR} \ org.apache.pinot.tools.admin.PinotAdministrator \ LaunchDataIngestionJob \ -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/hadoopIngestionJobSpec.yaml ``` Ensure environment variables `PINOT_ROOT_DIR` and `PINOT_VERSION` are set properly. {% hint style="warning" %} **ClassNotFoundException for Hadoop runners** usually means `plugins-external` is missing from `plugins.dir` or the shaded `pinot-batch-ingestion-hadoop-*-shaded.jar` is not on `HADOOP_CLASSPATH`. `pinot-all` alone is not enough. {% endhint %} ## Data Preprocessing before Segment Creation We’ve seen some requests that data should be massaged (like partitioning, sorting, resizing) before creating and pushing segments to Pinot. The MapReduce job called `SegmentPreprocessingJob` would be the best fit for this use case, regardless of whether the input data is of AVRO or ORC format. Check the below example to see how to use `SegmentPreprocessingJob`. In Hadoop properties, set the following to enable this job: ``` enable.preprocessing = true preprocess.path.to.output = ``` In table config, specify the operations in `preprocessing.operations` that you'd like to enable in the MR job, and then specify the exact configs regarding those operations: ``` { "OFFLINE": { "metadata": { "customConfigs": { “preprocessing.operations”: “resize, partition, sort”, // To enable the following preprocessing operations "preprocessing.max.num.records.per.file": "100", // To enable resizing "preprocessing.num.reducers": "3" // To enable resizing } }, ... "tableIndexConfig": { "aggregateMetrics": false, "autoGeneratedInvertedIndex": false, "bloomFilterColumns": [], "createInvertedIndexDuringSegmentGeneration": false, "invertedIndexColumns": [], "loadMode": "MMAP", "nullHandlingEnabled": false, "segmentPartitionConfig": { // To enable partitioning "columnPartitionMap": { "item": { "functionName": "murmur", "numPartitions": 4 } } }, "sortedColumn": [ // To enable sorting "actorId" ], "streamConfigs": {} }, "tableName": "tableName_OFFLINE", "tableType": "OFFLINE", "tenants": { ... } } } ``` ### preprocessing.num.reducers Minimum number of reducers. Optional. Fetched when partitioning gets disabled and resizing is enabled. This parameter is to avoid having too many small input files for Pinot, which leads to the case where Pinot server is holding too many small segments, causing too many threads. ### preprocessing.max.num.records.per.file Maximum number of records per reducer. Optional.Unlike, “preprocessing.num.reducers”, this parameter is to avoid having too few large input files for Pinot, which misses the advantage of muti-threading when querying. When not set, each reducer will finally generate one output file. When set (e.g. M), the original output file will be split into multiple files and each new output file contains at most M records. It does not matter whether partitioning is enabled or not. For more details on this MR job, refer to this [document](https://docs.google.com/document/d/1BnjjVj3OLuo-vmOt0WjqEFbUC9AZgCDuDxCtLEFPM34/edit?usp=sharing). # Backfill Data Batch ingestion of backfill data into Apache Pinot. ## Introduction Pinot batch ingestion involves two parts: routine ingestion job(hourly/daily) and backfill. Here are some examples to show how routine batch ingestion works in Pinot offline table: * [Batch Ingestion Overview](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) * [Batch Ingestion in Practice](/tutorials/data-ingestion/batch-data-ingestion-in-practice) **High-level description** 1. Organize raw data into buckets (eg: /var/pinot/airlineStats/rawdata/2014/01/01). Each bucket typically contains several files (eg: /var/pinot/airlineStats/rawdata/2014/01/01/airlineStats\_data\_2014-01-01\_0.avro) 2. Run a Pinot batch ingestion job, which points to a specific date folder like ‘/var/pinot/airlineStats/rawdata/2014/01/01’. The segment generation job will convert each such avro file into a Pinot segment for that day and give it a unique name. 3. Run Pinot segment push job to upload those segments with those uniques names via a Controller API {% hint style="info" %} **IMPORTANT**: The segment name is the unique identifier used to uniquely identify that segment in Pinot. If the controller gets an upload request for a segment with the same name - it will attempt to replace it with the new one. {% endhint %} This newly uploaded data can now be queried in Pinot. However, sometimes users will make changes to the raw data which need to be reflected in Pinot. This process is known as 'Backfill'. ## How to backfill data in Pinot Pinot supports data modification only at the segment level, which means you must update entire segments for doing backfills. The high level idea is to repeat steps 2 (segment generation) and 3 (segment upload) mentioned above: * Backfill jobs must run at the same granularity as the daily job. E.g., if you need to backfill data for 2014/01/01, specify that input folder for your backfill job (e.g.: ‘/var/pinot/airlineStats/rawdata/2014/01/01’) * The backfill job will then generate segments with the same name as the original job (with the new data). * When uploading those segments to Pinot, the controller will replace the old segments with the new ones (segment names act like primary keys within Pinot) one by one. ### Edge case example Backfill jobs expect the same number of (or more) data files on the backfill date. So the segment generation job will create the same number of (or more) segments than the original run. For example, assuming table airlineStats has 2 segments(airlineStats\_2014-01-01\_2014-01-01\_0, airlineStats\_2014-01-01\_2014-01-01\_1) on date 2014/01/01 and the backfill input directory contains only 1 input file. Then the segment generation job will create just one segment: airlineStats\_2014-01-01\_2014-01-01\_0. After the segment push job, only segment airlineStats\_2014-01-01\_2014-01-01\_0 got replaced and stale data in segment airlineStats\_2014-01-01\_2014-01-01\_1 are still there. **If the raw data is modified in such a way that the original time bucket has fewer input files than the first ingestion run, backfill will fail.** ### Complete backfill with LaunchBackfillIngestionJob To safely handle the case described above — where the backfill input may have fewer files than the original ingestion — Pinot ships a dedicated CLI command, `LaunchBackfillIngestionJob`. Instead of relying on segment-name collision to replace stale data, it uses Pinot's segment-lineage machinery to atomically replace the existing segments in a date range with the newly generated ones. See [Consistent Data Push and Rollback](/operate-pinot/segment-management/consistent-push-and-rollback) for the underlying lineage semantics. #### How it works `LaunchBackfillIngestionJob` reuses the same `ingestionJobSpec.yaml` as `LaunchDataIngestionJob` and runs in four steps: | Step | Description | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1. Fetch existing segments | Calls the controller's `selectSegments` API to list every OFFLINE-table segment that overlaps `[startDate, endDate)`. These are the segments that will be replaced. | | 2. Generate new segments | Runs a local `SegmentCreation` pass over the input directory. The segment-name generator is forced to `simple` with a unique `_` postfix, so newly generated segment names never collide with the existing ones (and the lineage replace can therefore reference both sets unambiguously). | | 3. Create a lineage entry | Calls `startReplaceSegments` with `segmentsFrom = ` and `segmentsTo = `. The old segments continue to serve queries until the lineage entry is finalized. | | 4. Push and finalize | Uploads the new segments via `SegmentTarPush`, then calls `endReplaceSegments`, which atomically switches brokers onto the new segment set and retires the old one. If the push fails, the lineage entry is rolled back and any partially uploaded segments are cleaned up. | Because the old segment set is recorded explicitly in step 3 — rather than inferred from name collisions — the failure mode described in [Edge case example](#edge-case-example), where a shrunken backfill input leaves stale segments behind, cannot occur. `LaunchBackfillIngestionJob` currently supports **OFFLINE tables only**. #### Running the command ```bash bin/pinot-admin.sh LaunchBackfillIngestionJob \ -jobSpecFile /path/to/ingestionJobSpec.yaml \ -startDate 2014-01-01 \ -endDate 2014-01-02 ``` | Option | Required | Description | | ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `-jobSpecFile` | yes | Same job-spec YAML used by `LaunchDataIngestionJob`. | | `-startDate` | yes | Backfill start date, **inclusive**. Format `yyyy-MM-dd`, parsed as **UTC**. | | `-endDate` | yes | Backfill end date, **exclusive**. Format `yyyy-MM-dd`, parsed as **UTC**. | | `-partitionColumn` | no | Column on which segments were partitioned via `BoundedColumnValue`. Used together with `-partitionColumnValue` to scope which segments are replaced. | | `-partitionColumnValue` | no | Only segments whose `BoundedColumnValue` partition contains this value are eligible for replacement. | All authentication options inherited from `LaunchDataIngestionJob` (`-authToken`, `-user`, `-password`, `-authProvider`, `-authTokenUrl`) are supported. {% hint style="info" %} Because dates are parsed as UTC day boundaries, callers in non-UTC time zones should account for the offset when building `-startDate` / `-endDate`. A "January 1st" backfill local to `America/Los_Angeles`, for example, spans `2014-01-01T08:00Z` to `2014-01-02T08:00Z`, which does not align with the UTC-day boundaries the command uses. {% endhint %} #### Filtering by partition column For tables whose offline segments are partitioned by `BoundedColumnValue`, the backfill can be scoped to a subset of partitions using `-partitionColumn` and `-partitionColumnValue`: ```bash bin/pinot-admin.sh LaunchBackfillIngestionJob \ -jobSpecFile /path/to/ingestionJobSpec.yaml \ -startDate 2014-01-01 \ -endDate 2014-01-02 \ -partitionColumn region \ -partitionColumnValue us-east-1 ``` The command computes the partition ID for `us-east-1` using the column's recorded partition function, then keeps only segments whose partition metadata contains that ID. Segments with no `BoundedColumnValue` partition metadata, or those covering more than one partition, are skipped. # Dimension Table Batch ingestion of data into Apache Pinot using dimension tables. Dimension tables are a special kind of offline table designed for join-like enrichment of fact data at query time. They are used together with the [lookup UDF](/build-with-pinot/querying-and-sql/sql-syntax/lookup-udf-join) (single-stage engine) or the [lookup join strategy](/build-with-pinot/querying-and-sql/multi-stage-query/join-strategies/lookup-join-strategy) (multi-stage engine) to decorate query results with reference data. ## When to use dimension tables Use a dimension table when you need to enrich a large fact table with attributes from a small, relatively static reference dataset at query time. Common examples include: * Looking up a human-readable team name from a team ID. * Enriching clickstream events with product catalog attributes. * Decorating transaction records with customer or store metadata. If any of the following apply, a regular offline or real-time table is a better fit: * The reference data is large (hundreds of millions of rows or multiple gigabytes). * The data changes frequently and requires real-time ingestion. * You need time-based partitioning, retention policies, or a hybrid table setup. * You need to query the reference data with complex aggregations independently. ## How dimension tables work When a table is marked as a dimension table, Pinot replicates all of its segments to **every server** in the tenant. On each server the data is loaded into an in-memory hash map keyed by the table's primary key, which enables constant-time lookups during query execution. Because the data is fully replicated and held in memory, dimension tables must be **small enough to fit comfortably in each server's heap**. They are not intended for large datasets. ### Memory loading modes Pinot supports two loading modes controlled by the `disablePreload` setting in `dimensionTableConfig`: | Mode | `disablePreload` | Memory usage | Lookup speed | Description | | ------------------------- | ---------------- | ------------ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Fast lookup** (default) | `false` | Higher | Faster | All rows are fully materialized into an in-memory hash map (`Object[] -> Object[]`). Every column value is stored in the map for constant-time retrieval. | | **Memory-optimized** | `true` | Lower | Slightly slower | Only the primary key and a segment/docId reference are stored in the hash map. Column values are read from the segment on each lookup. This trades lookup speed for lower heap usage. | Choose the memory-optimized mode when the dimension table is relatively large and you want to reduce heap pressure, at the cost of slightly slower lookups. ## Size limits and memory considerations * **Cluster-level maximum size**: The controller configuration property `controller.dimTable.maxSize` sets the maximum storage quota allowed for any single dimension table. The default is **200 MB**. Table creation fails if the requested `quota.storage` exceeds this limit. * **Heap impact**: In fast-lookup mode, the entire table is materialized in Java heap on every server. A table that is 100 MB on disk may consume significantly more memory after deserialization. Monitor server heap usage when adding or growing dimension tables. * **Replication overhead**: Because every server in the tenant holds a full copy, adding a dimension table multiplies its memory footprint by the number of servers. {% hint style="warning" %} As a guideline, keep dimension tables under a few hundred thousand rows and well under the `controller.dimTable.maxSize` limit. Tables that approach or exceed available heap will cause out-of-memory errors on servers. {% endhint %} ## Configuration ### Table configuration Mark a table as a dimension table by setting the following properties in the table config: | Property | Required | Description | | -------------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `isDimTable` | Yes | Set to `true` to designate the table as a dimension table. | | `ingestionConfig.batchIngestionConfig.segmentIngestionType` | Yes | Must be set to `REFRESH`. Dimension tables use segment replacement rather than append semantics so that the in-memory hash map is rebuilt with the latest data. | | `segmentAssignmentConfigMap.OFFLINE.segmentAssignmentStrategy` | No | If set, must be `allservers`. Leaving it unset also works because Pinot automatically uses all-servers assignment for dimension tables. | | `quota.storage` | Recommended | Storage quota for the table. Must not exceed the cluster-level `controller.dimTable.maxSize` (default 200 MB). | | `dimensionTableConfig.disablePreload` | No | Set to `true` to use memory-optimized mode (store only primary key and segment reference instead of full rows). Defaults to `false` (fast lookup). | | `dimensionTableConfig.errorOnDuplicatePrimaryKey` | No | Set to `true` to fail segment loading if duplicate primary keys are detected across segments. Defaults to `false` (last-loaded segment wins). | ### Schema configuration Dimension table schemas use `dimensionFieldSpecs` instead of `metricFieldSpecs`. A `primaryKeyColumns` array is **required** -- it defines the key used for lookups. {% hint style="warning" %} Dimension tables always use the `allservers` segment assignment strategy so every segment is replicated to every server in the tenant. If you set the strategy explicitly, do so under `segmentAssignmentConfigMap.OFFLINE.segmentAssignmentStrategy`. Do not configure balanced, replica-group, or round-robin segment assignment for a dimension table. Pinot now rejects those values during table validation. {% endhint %} ### Example table configuration ```json { "OFFLINE": { "tableName": "dimBaseballTeams_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { }, "segmentAssignmentConfigMap": { "OFFLINE": { "segmentAssignmentStrategy": "allservers" } }, "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "REFRESH" } }, "quota": { "storage": "200M" }, "isDimTable": true, "dimensionTableConfig": { "disablePreload": false, "errorOnDuplicatePrimaryKey": false } } } ``` ### Example schema configuration ```json { "schemaName": "dimBaseballTeams", "primaryKeyColumns": ["teamID"], "dimensionFieldSpecs": [ { "dataType": "STRING", "name": "teamID" }, { "dataType": "STRING", "name": "teamName" }, { "dataType": "STRING", "name": "teamAddress" } ] } ``` ## Querying with the LOOKUP function The primary way to use a dimension table is through the `LOOKUP` UDF in the single-stage query engine. This function performs a primary-key lookup against the dimension table and returns a column value. ### Syntax ``` LOOKUP('dimTable', 'dimColToLookUp', 'dimJoinKey1', factJoinKey1 [, 'dimJoinKey2', factJoinKey2 ]*) ``` * `dimTable` -- name of the dimension table (string literal). * `dimColToLookUp` -- column to retrieve from the dimension table (string literal). * `dimJoinKey` / `factJoinKey` -- pairs of join keys: the dimension table column name (string literal) and the corresponding fact table column expression. ### Single-key lookup ```sql SELECT playerName, teamID, LOOKUP('dimBaseballTeams', 'teamName', 'teamID', teamID) AS teamName, LOOKUP('dimBaseballTeams', 'teamAddress', 'teamID', teamID) AS teamAddress FROM baseballStats LIMIT 10 ``` ### Composite-key lookup When the dimension table has a composite primary key, provide multiple key pairs in the same order as `primaryKeyColumns` in the schema: ```sql SELECT customerId, LOOKUP('billing', 'city', 'customerId', customerId, 'creditHistory', creditHistory) AS city FROM transactions LIMIT 10 ``` ### Multi-stage engine In the multi-stage engine (MSE), use a standard `JOIN` with the lookup join strategy hint instead of the `LOOKUP` UDF: ```sql SELECT /*+ lookupJoinStrategy(dim_billing) */ t.customerId, b.city FROM transactions t JOIN billing b ON t.customerId = b.customerId LIMIT 10 ``` For details, see [lookup join strategy](/build-with-pinot/querying-and-sql/multi-stage-query/join-strategies/lookup-join-strategy). ## Refresh and update strategies Because dimension tables use `segmentIngestionType: REFRESH`, uploading a new segment **replaces** the existing segment and triggers a full reload of the in-memory hash map on every server. There is no incremental update mechanism. Typical refresh patterns: * **Scheduled batch job**: Run a periodic ingestion job (e.g., daily or hourly) that rebuilds the segment from the source of truth and uploads it to Pinot. * **On-demand refresh**: Trigger a segment upload through the Pinot REST API whenever the reference data changes. {% hint style="info" %} During a refresh, the old hash map remains active for lookups until the new one is fully loaded. There is no query downtime during a refresh, but there is a brief period where the old data is served. {% endhint %} ### Handling duplicate primary keys When multiple segments contain the same primary key, the default behavior is last-loaded-segment-wins (segments are ordered by creation time). Set `errorOnDuplicatePrimaryKey: true` in `dimensionTableConfig` to fail fast if duplicates are detected. With `REFRESH` ingestion, there is typically only one segment, so duplicates across segments are uncommon. ## Performance best practices * **Keep tables small.** Dimension tables are loaded entirely into memory on every server. Target thousands to low hundreds of thousands of rows. * **Use narrow schemas.** Include only the columns needed for lookups to reduce memory consumption. * **Choose the right loading mode.** Use fast lookup (default) for the best query performance. Switch to memory-optimized mode (`disablePreload: true`) only if heap usage is a concern. * **Set a storage quota.** Always configure `quota.storage` to prevent accidentally uploading oversized data. * **Minimize refresh frequency.** Each refresh triggers a full reload of the hash map. Avoid refreshing more often than necessary. * **Monitor server heap.** After adding a dimension table, check server JVM heap metrics to confirm adequate headroom. ## Limitations * **Offline only.** Dimension tables must be offline tables. They cannot be real-time or hybrid tables. * **Full replication.** All segments are replicated to every server in the tenant, so memory usage scales with the number of servers. * **No incremental updates.** The entire segment must be replaced on each refresh; row-level updates are not supported. * **Primary key required.** The schema must define `primaryKeyColumns`. Lookups without a primary key are not supported. * **Single-stage LOOKUP UDF limitations.** Dimension table column references in the `LOOKUP` function must be string literals, not column identifiers, because they reference a table that is not part of the query's FROM clause. * **No time-based partitioning or retention.** Dimension tables do not support segment retention policies or time-based partitioning. # Stream Ingestion Choose stream ingestion when Pinot should consume events continuously and expose new rows quickly. Stream ingestion keeps Pinot close to the source of truth. Use it when rows should be queryable soon after they are emitted, and when the system needs a steady flow rather than periodic batch loads. ## Core decisions Pick the stream connector and partitioning strategy. Choose how Pinot should flush, commit, and complete segments. Decide whether the table should remain purely realtime or later become hybrid. ## What matters most The stream has to support the consumption mode you choose. The table config has to describe the partitioning, replicas, and segment lifecycle clearly enough that the servers can behave predictably under load. ## Learn more The existing walk-throughs in [Import Data](/build-with-pinot/ingestion) and [Data Ingestion Overview](/build-with-pinot/ingestion) still contain the detailed mechanics. ## What this page covered This page covered the stream-ingestion model and the main lifecycle choices behind it. ## Next step Read [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup) if the stream should collapse duplicate keys or keep only the latest row. ## Related pages * [Ingestion](/build-with-pinot/ingestion) * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) * [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup) * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) * [Original Stream Docs](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion) # Stream Ingestion Guide This guide shows you how to ingest a stream of records into a Pinot table. Apache Pinot lets users consume data from streams and push it directly into the database. This process is called stream ingestion. Stream ingestion makes it possible to query data within seconds of publication. Stream ingestion provides support for checkpoints for preventing data loss. To set up Stream ingestion, perform the following steps, which are described in more detail in this page: 1. Create schema configuration 2. Create table configuration 3. Create ingestion configuration 4. Upload table and schema spec Here's an example where we assume the data to be ingested is in the following format: ```bash {"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"Maths","score":3.8,"timestamp":1571900400000} {"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"History","score":3.5,"timestamp":1571900400000} {"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Maths","score":3.2,"timestamp":1571900400000} {"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Chemistry","score":3.6,"timestamp":1572418800000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Geography","score":3.8,"timestamp":1572505200000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"English","score":3.5,"timestamp":1572505200000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Maths","score":3.2,"timestamp":1572678000000} {"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Physics","score":3.6,"timestamp":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"Maths","score":3.8,"timestamp":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"English","score":3.5,"timestamp":1572678000000} {"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"History","score":3.2,"timestamp":1572854400000} {"studentID":212,"firstName":"Nick","lastName":"Young","gender":"Male","subject":"History","score":3.6,"timestamp":1572854400000} ``` ## Create schema configuration The schema defines the fields along with their data types. The schema also defines whether fields serve as `dimensions`, `metrics`, or `timestamp`. For more details on schema configuration, see [First table and schema](/start-here/first-table-and-schema#4-save-the-schema). For our sample data, the schema configuration looks like this: {% code title="/tmp/pinot-quick-start/transcript-schema.json" %} ```bash { "schemaName": "transcript", "dimensionFieldSpecs": [ { "name": "studentID", "dataType": "INT" }, { "name": "firstName", "dataType": "STRING" }, { "name": "lastName", "dataType": "STRING" }, { "name": "gender", "dataType": "STRING" }, { "name": "subject", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "score", "dataType": "FLOAT" } ], "dateTimeFieldSpecs": [{ "name": "timestamp", "dataType": "LONG", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` {% endcode %} ## Create table configuration with ingestion configuration The next step is to create a table where all the ingested data will flow and can be queried. For details about each table component, see the [table](/reference/configuration-reference/table) reference. The table configuration contains an ingestion configuration (`ingestionConfig`), which specifies how to ingest streaming data into Pinot. For details, see the [ingestion configuration](/reference/configuration-reference/ingestion) reference. ### Example table config with `ingestionConfig` For our sample data and schema, the table config will look like this: ```json { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", }, "metadata": { "customConfigs": {} }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "realtime.segment.flush.threshold.rows": "0", "stream.kafka.decoder.prop.format": "JSON", "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "streamType": "kafka", "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "realtime.segment.flush.threshold.segment.rows": "50000", "stream.kafka.broker.list": "localhost:9876", "realtime.segment.flush.threshold.time": "3600000", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.topic.name": "transcript-topic" } ] }, "transformConfigs": [], "continueOnError": true, "rowTimeValueCheck": true, "segmentTimeValueCheck": false }, "isDimTable": false } } ``` ### Example `ingestionConfig` with multiple stream configs > **⚠️ Important Bug Fix**: A few issues were identified in the previous implementation, which have been fixed via [PR #17953](https://github.com/apache/pinot/pull/17953) and [PR #17217](https://github.com/apache/pinot/pull/17217). These fixes are not present in Pinot 1.4.0. If you plan to use this feature, make sure to cherry-pick these PRs or use Pinot 1.5.0 release. Pinot allows `streamConfigMaps` to contain more than one stream config. For the example below, assume the sample data is duplicated to two Kafka topics, `transcript-topic1` and `transcript-topic2`, and the table should ingest from both topics: ```json { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", }, "metadata": { "customConfigs": {} }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "realtime.segment.flush.threshold.rows": "0", "stream.kafka.decoder.prop.format": "JSON", "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "streamType": "kafka", "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "realtime.segment.flush.threshold.segment.rows": "50000", "stream.kafka.broker.list": "localhost:9876", "realtime.segment.flush.threshold.time": "3600000", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.topic.name": "transcript-topic1" }, { "realtime.segment.flush.threshold.rows": "0", "stream.kafka.decoder.prop.format": "JSON", "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "streamType": "kafka", "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer", "realtime.segment.flush.threshold.segment.rows": "50000", "stream.kafka.broker.list": "localhost:9876", "realtime.segment.flush.threshold.time": "3600000", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.topic.name": "transcript-topic2" } ] }, "transformConfigs": [], "continueOnError": true, "rowTimeValueCheck": true, "segmentTimeValueCheck": false }, "isDimTable": false } } ``` When `streamConfigMaps` contains multiple entries, Pinot validates the combined configuration before allowing the table to be created or updated: * Each entry must still be a valid stream config on its own. * All entries must use the same `streamType`. * Segment flush settings must match across all entries: * `realtime.segment.flush.threshold.rows` * `realtime.segment.flush.threshold.time` * `realtime.segment.flush.threshold.variance.fraction` * `realtime.segment.flush.threshold.segment.size` * `realtime.segment.flush.threshold.segment.rows` * Topic names must be unique across the entries. * Multiple stream configs are not supported together with `pauselessConsumptionEnabled=true`. * Multiple stream configs are not supported for upsert tables. Once validated, Pinot applies the rest of the ingestion config normally: * Transform functions apply to records from every configured stream. * Existing instance-assignment strategies continue to work as usual. * [Partition changes](#handle-partition-changes-in-streams) are handled the same way as for a single stream config. * Underlying ingestion still works in `LOWLEVEL` mode, where: * `transcript-topic1` segments are named like `transcript__0__0__20250101T0000Z` * `transcript-topic2` segments are named like `transcript__10000__0__20250101T0000Z` ## Upload schema and table config Now that we have our table and schema configurations, let's upload them to the Pinot cluster. As soon as the configs are uploaded, Pinot will start ingesting available records from the topic. {% tabs %} {% tab title="Docker" %} ``` docker run \ --network=pinot-demo \ -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \ --name pinot-streaming-table-creation \ apachepinot/pinot:latest AddTable \ -schemaFile /tmp/pinot-quick-start/transcript-schema.json \ -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \ -controllerHost pinot-quickstart \ -controllerPort 9000 \ -exec ``` {% endtab %} {% tab title="Launcher Script" %} ```bash bin/pinot-admin.sh AddTable \ -schemaFile /path/to/transcript-schema.json \ -tableConfigFile /path/to/transcript-table-realtime.json \ -exec ``` {% endtab %} {% endtabs %} ## Tune the stream config ### Skip ahead when a partition falls too far behind If a realtime partition is too far behind, Pinot can skip the backlog at segment-commit time and start the next segment from the latest stream offset instead of the previous segment's `nextOffset`. Use the following stream configs inside `streamConfigMaps`: ```json { "realtime.segment.offsetAutoReset.enable": "true", "realtime.segment.offsetAutoReset.offsetThreshold": "1000000", "realtime.segment.offsetAutoReset.timeThresholdSeconds": "3600" } ``` Pinot checks the thresholds when it seals a consuming segment: * If `latestOffset - nextOffset` exceeds `realtime.segment.offsetAutoReset.offsetThreshold`, Pinot skips to the latest offset. * If the next offset is older than `realtime.segment.offsetAutoReset.timeThresholdSeconds`, Pinot also skips to the latest offset. * Set at least one threshold to a positive value. If both thresholds are unset or non-positive, Pinot keeps the normal `nextOffset`. This is useful when you prefer to drop an ingestion backlog instead of replaying a partition that has fallen far behind. ### Handle decode errors during stream consumption By default, Pinot logs decode errors and silently drops the problematic row so that ingestion continues uninterrupted. However, in some scenarios you may prefer to stop ingestion immediately when a decode error occurs so you can investigate and resolve the issue. You can control this behavior using the `stopOnDecodeError` configuration parameter in your stream config: ```json { "tableName": "transcript", "tableType": "REALTIME", ... "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [{ "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", "stopOnDecodeError": "true", ... }] } }, ... } ``` **Configuration options:** * `"stopOnDecodeError": "true"` - Consumption will immediately stop when a decode error occurs, and the error will be logged with full stack trace. This allows you to investigate the root cause in the server logs before resuming. * `"stopOnDecodeError": "false"` (default) - The first decode error will be logged, and subsequent errors will be silently dropped. The row with the decode error is skipped, and ingestion continues normally. Use `"stopOnDecodeError": "true"` when: * Data quality is critical and you need to catch decode errors immediately * You're developing or testing a new decoder implementation * You want to investigate unexpected data format issues Use `"stopOnDecodeError": "false"` (default) when: * You expect occasional malformed messages in your stream * You want ingestion to be resilient and continue despite individual bad records * Data loss from a few dropped rows is acceptable ### Throttle stream consumption There are some scenarios where the message rate in the input stream can come in bursts which can lead to long GC pauses on the Pinot servers or affect the ingestion rate of other real-time tables on the same server. If this happens to you, throttle the consumption rate during stream ingestion to better manage overall performance. There are two independent throttling mechanisms available: 1. Message-rate–based throttling (table level, records/sec) 2. Byte-rate–based throttling (server level, bytes/sec) Both mechanisms can be enabled simultaneously. #### Message-rate–based throttling (table level) Stream consumption throttling can be tuned using the stream config `partition.consumption.rate.limit` for a direct per-partition cap or `topic.consumption.rate.limit` for a topic-wide cap that Pinot divides by the topic partition count. Here is the sample configuration on how to configure the consumption throttling: ```json { "tableName": "transcript", "tableType": "REALTIME", ... "ingestionConfig": { "streamIngestionConfig":, "streamConfigMaps": { "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", ... "topic.consumption.rate.limit": 1000 } }, ... ``` Some things to keep in mind while tuning this config are: * When you use `topic.consumption.rate.limit`, Pinot divides the topic-wide rate by the number of partitions in the topic and applies the result to each partition consumer. This doesn't take replication factor into account.\ \ **Example**\ topic.consumption.rate.limit - 1000\ num partitions in Kafka topic - 4\ replication factor in table - 3\ \ Pinot will impose a fixed limit of 1000 / 4 = 250 records per second on each partition. \\ * In case of multi-tenant deployment (where you have more than 1 table in the same server instance), you need to make sure that the rate limit on one table doesn't step on/starve the rate limiting of another table. So, when there is more than 1 table on the same server (which is most likely to happen), you may need to re-tune the throttling threshold for all the streaming tables.\\ * The `pinot.server.consumption.rate.limit` setting must be configured in the server's instance configuration, not in the table configuration. This server-wide rows/sec cap is enforced in addition to any table-level `partition.consumption.rate.limit` or `topic.consumption.rate.limit` cap. \ \\ Once throttling is enabled for a table, you can verify by searching for a log that looks similar to: {% code overflow="wrap" %} ```markdown A consumption rate limiter is set up for topic in table with rate limit: (topic rate limit: , partition count: ) ``` {% endcode %} In addition, you can monitor `CONSUMPTION_RATE_LIMIT` (`consumptionRateLimit`) for the effective per-partition configured cap and `CONSUMPTION_QUOTA_UTILIZATION` (`consumptionQuotaUtilization`) for the per-partition utilization. Note that any configuration change for `topic.consumption.rate.limit` in the stream config will **NOT** take effect immediately. The new configuration will be picked up from the next consuming segment. In order to enforce the new configuration, you need to trigger forceCommit APIs. Refer to [Pause Stream Ingestion](#pause-stream-ingestion) for more details. ``` $ curl -X POST {controllerHost}/tables/{tableName}/forceCommit ``` #### Byte-rate–based throttling (server level) In addition to message-rate throttling, Pinot supports byte-based stream consumption throttling at the server level. This throttling mechanism limits the total number of bytes consumed per second by a Pinot server, across all real-time tables and partitions hosted on that server. **When to use byte-based throttling** Byte-based throttling is especially useful when: * Message sizes vary significantly * Ingestion pressure is driven by payload size rather than record count * You want to cap network, direct memory, or disk IO usage at the server level * Multiple real-time tables coexist on the same server **Configuration** Byte-based throttling is configured via cluster config, not via table or stream configs. **Config key** `pinot.server.consumption.rate.limit.bytes` The value is specified in bytes per second. **Updating the configuration** The configuration can be updated dynamically using the Cluster Config API. This limits each Pinot server to consume at most 3,000,000 bytes/sec (\~3 MB/sec) across all real-time tables. **Example using curl** ``` curl -X POST '{controllerHost}/cluster/configs' -H 'Content-Type: application/json' -d '{ "pinot.server.consumption.rate.limit.bytes": "3000000" }' ``` **How byte-based throttling works** * The byte rate limit is enforced per server * The limit applies collectively to all consuming partitions and tables hosted on that server * This throttling is independent of table-level message-rate throttling **Interaction with message-rate throttling** If both throttles are enabled: * Table-level `topic.consumption.rate.limit` controls records/sec per table * Server-level `pinot.server.consumption.rate.limit.bytes` controls bytes/sec per server * Pinot enforces both limits * Consumption is throttled as soon as either limit is reached This allows precise control when both message count and payload size matter. **Dynamic updates and propagation** * Byte-based throttling is updated dynamically via the Cluster Config Change Listener * No server restart is required * Changes take effect automatically as servers receive the updated cluster config **Verifying throttling** Once enabled, Pinot logs messages indicating that a server-level byte consumption limiter has been applied. You can also monitor throttling behavior using these metrics: * `SERVER_CONSUMPTION_RATE_LIMIT` (`serverConsumptionRateLimit`) reports the configured server-wide cap. Pinot sets it to `-1` when server-level rate limiting is disabled. * `SERVER_CONSUMPTION_QUOTA_UTILIZATION` (`serverConsumptionQuotaUtilization`) reports the server-wide utilization percentage. If you previously read the server-wide value from `consumptionQuotaUtilization{table="realtimeRowsConsumed"}`, switch dashboards and alerts to `serverConsumptionQuotaUtilization`. ## Custom ingestion support You can also write an ingestion plugin if the platform you are using is not supported out of the box. For a walkthrough, see [Stream Ingestion Plugin](/develop-and-contribute/plugin-architecture/write-custom-plugins/write-your-stream). ## Pause stream ingestion There are some scenarios in which you may want to pause the real-time ingestion while your table is available for queries. For example, if there is a problem with the stream ingestion and, while you are troubleshooting the issue, you still want the queries to be executed on the already ingested data. For these scenarios, you can first issue a Pause request to a Controller host. After troubleshooting with the stream is done, you can issue another request to Controller to resume the consumption. ```bash $ curl -X POST {controllerHost}/tables/{tableName}/pauseConsumption $ curl -X POST "{controllerHost}/tables/{tableName}/resumeConsumption?comment=maintenance-complete" ``` For tables with many consuming segments, you can batch the pause request so the controller commits fewer segments at a time: ```bash $ curl -X POST "{controllerHost}/tables/{tableName}/pauseConsumption?batchSize=50&batchStatusCheckIntervalSec=5&batchStatusCheckTimeoutSec=180" ``` `batchSize` limits how many consuming segments Pinot commits in one batch. `batchStatusCheckIntervalSec` and `batchStatusCheckTimeoutSec` control how often and how long the controller waits for each batch to finish before moving on or failing the pause request. When a `Pause` request is issued, the controller instructs the real-time servers hosting your table to commit their consuming segments immediately. However, the commit process may take some time to complete. Note that `Pause` and `Resume` requests are async. An `OK` response means that instructions for pausing or resuming has been successfully sent to the real-time server. If you want to know if the consumption has actually stopped or resumed, issue a pause status request. ```bash $ curl -X GET {controllerHost}/tables/{tableName}/pauseStatus ``` The pause status response includes the current `pauseFlag`, the set of `consumingSegments`, the stored `reasonCode`, any persisted `comment`, and the controller-side `timestamp` for the current pause state. When topic-level pause APIs have paused only some stream topics, Pinot also returns `indexOfInactiveTopics` with the zero-based topic indexes that remain inactive. When no explicit comment was stored, Pinot returns a default paused or unpaused message. It's worth noting that consuming segments on real-time servers are stored in volatile memory, and their resources are allocated when the consuming segments are first created. These resources cannot be altered if consumption parameters are changed midway through consumption. It may take hours before these changes take effect. Furthermore, if the parameters are changed in an incompatible way (for example, changing the underlying stream with a completely new set of offsets, or changing the stream endpoint from which to consume messages), it will result in the table getting into an error state. The pause and resume feature is helpful in these instances. When a pause request is issued by the operator, consuming segments are committed without starting new mutable segments. Instead, new mutable segments are started only when the resume request is issued. This mechanism provides the operators as well as developers with more flexibility. It also enables Pinot to be more resilient to the operational and functional constraints imposed by underlying streams. There is another feature called `Force Commit` which utilizes the primitives of the pause and resume feature. When the operator issues a force commit request, the current mutable segments will be committed and new ones started right away. Operators can now use this feature for all compatible table config parameter changes to take effect immediately. ```bash $ curl -X POST {controllerHost}/tables/{tableName}/forceCommit ``` Optional filters and batching (do not combine `partitions` and `segments`): ```bash $ curl -X POST "{controllerHost}/tables/{tableName}/forceCommit?partitions=0,1&batchSize=50&batchStatusCheckIntervalSec=5&batchStatusCheckTimeoutSec=180" $ curl -X POST "{controllerHost}/tables/{tableName}/forceCommit?segments=table__0__12__20250610T2140Z,table__1__12__20250610T2140Z" ``` Real-time tables can also be paused automatically by Pinot itself. In particular, when a table exceeds `quota.storage`, the controller marks it paused with reason code `STORAGE_QUOTA_EXCEEDED` and stops creating new consuming segments during periodic validation. Once the table is back within quota, Pinot clears that pause state and allows segment creation to resume. If you want to resume sooner after fixing the quota issue, you can still call `resumeConsumption` manually. (v 0.12.0+) Once submitted, the forceCommit API returns a jobId that can be used to get the current progress of the forceCommit operation. A sample response and status API call: ```bash $ curl -X POST {controllerHost}/tables/{tableName}/forceCommit { "forceCommitJobId": "6757284f-b75b-45ce-91d8-a277bdbc06ae", "forceCommitStatus": "SUCCESS", "jobMetaZKWriteStatus": "SUCCESS" } $ curl -X GET {controllerHost}/tables/forceCommitStatus/6757284f-b75b-45ce-91d8-a277bdbc06ae { "jobId": "6757284f-b75b-45ce-91d8-a277bdbc06ae", "segmentsForceCommitted": "[\"airlineStats__0__0__20230119T0700Z\",\"airlineStats__1__0__20230119T0700Z\",\"airlineStats__2__0__20230119T0700Z\"]", "submissionTimeMs": "1674111682977", "numberOfSegmentsYetToBeCommitted": 0, "jobType": "FORCE_COMMIT", "segmentsYetToBeCommitted": [], "tableName": "airlineStats_REALTIME" } ``` {% hint style="info" %} The forceCommit request just triggers a regular commit before the consuming segments reach the end criteria, so it follows the same mechanism as regular commit. It is a one-shot request and is not retried automatically upon failure. It is idempotent enough that you may keep issuing it until success if needed. **HTTP 200 is async acceptance**, not proof that commits finished. `forceCommitStatus=SUCCESS` means the controller initiated the operation; wait until `numberOfSegmentsYetToBeCommitted` is `0` on the status API. `jobMetaZKWriteStatus=FAILED` means commit may still run but Pinot could not persist a trackable job id. A ZK status entry records submission time and the consuming segments included. Pending progress is derived by comparing that list to the latest IdealState / metadata. The status entry is not deleted on success or failure and can become stale; Pinot keeps a bounded number of force-commit jobs in ZK (default 100 via `controller.force.commit.maxJobsInZK`). Full parameter tables, mutual exclusion of `partitions` vs `segments`, and failure modes: [Force commit API](/reference/api-reference/controller-api#post-tablestablenameforcecommit). {% endhint %} For incompatible parameter changes, an option is added to the resume request to handle the case of a completely new set of offsets. Operators can now follow a three-step process: First, issue a pause request. Second, change the consumption parameters. Finally, issue the resume request with the appropriate option. These steps will preserve the old data and allow the new data to be consumed immediately. All through the operation, queries will continue to be served. ```bash $ curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption?consumeFrom=smallest $ curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption?consumeFrom=largest ``` ## Handle partition changes in streams If a Pinot table is configured to consume using a [Low Level](#create-table-configuration-with-ingestion-configuration) (partition-based) stream type, then it is possible that the partitions of the table change over time. In Kafka, for example, the number of partitions may increase. In Kinesis, the number of partitions may increase *or* decrease -- some partitions could be merged to create a new one, or existing partitions split to create new ones. Pinot runs a periodic task called `RealtimeSegmentValidationManager` that monitors such changes and starts consumption on new partitions (or stops consumptions from old ones) as necessary. Since this is a controller [periodic task](/architecture-and-concepts/components/cluster/controller#running-the-periodic-task-manually), it may take some time for Pinot to recognize new partitions and start consuming from them. This may delay the data in new partitions appearing in the results that pinot returns. If you want to recognize the new partitions sooner, then [manually trigger](/architecture-and-concepts/components/cluster/controller#running-the-periodic-task-manually) the periodic task so as to recognize such data immediately. For Kafka low-level consumers, the same task can also repair a partition when Pinot is missing the `CONSUMING` segment for a Kafka partition that still exists upstream. Operators can either wait for the next `RealtimeSegmentValidationManager` run or trigger it manually. When Pinot recreates the missing consuming segment, it uses the repair offset selected by validation. If Pinot still has LLC metadata for that partition, it can resume from the stored end offset; otherwise, it typically starts from Kafka's earliest retained offset for that partition, so recovery is limited by Kafka retention. ## Infer ingestion status of real-time tables Often, it is important to understand the rate of ingestion of data into your real-time table. This is commonly done by looking at the consumption lag of the consumer. The lag itself can be observed in many dimensions. Pinot supports observing consumption lag along the offset dimension and time dimension, whenever applicable (as it depends on the specifics of the connector). The ingestion status of a connector can be observed by querying either the `/consumingSegmentsInfo` API or the table's `/debug` API, as shown below: {% code overflow="wrap" lineNumbers="true" %} ```shell # GET /tables/{tableName}/consumingSegmentsInfo curl -X GET "http:///tables/meetupRsvp/consumingSegmentsInfo" -H "accept: application/json" # GET /debug/tables/{tableName} curl -X GET "http://localhost:9000/debug/tables/meetupRsvp?type=REALTIME&verbosity=1" -H "accept: application/json" ``` {% endcode %} A sample response from a Kafka-based real-time table is shown below. The ingestion status is displayed for each of the CONSUMING segments in the table. ```json { "_segmentToConsumingInfoMap": { "meetupRsvp__0__0__20221019T0639Z": [ { "serverName": "Server_192.168.0.103_7000", "consumerState": "CONSUMING", "lastConsumedTimestamp": 1666161593904, "partitionToOffsetMap": { // <<-- Deprecated. See currentOffsetsMap for same info "0": "6" }, "partitionOffsetInfo": { "currentOffsetsMap": { "0": "6" // <-- Current consumer position }, "latestUpstreamOffsetMap": { "0": "6" // <-- Upstream latest position }, "recordsLagMap": { "0": "0" // <-- Lag, in terms of #records behind latest }, "recordsAvailabilityLagMap": { "0": "2" // <-- Lag, in terms of time } } } ], ``` | Term | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | currentOffsetsMap | Current consuming offset position per partition | | latestUpstreamOffsetMap | (Wherever applicable) Latest offset found in the upstream topic partition | | recordsLagMap | (Whenever applicable) Defines how far behind the current record's offset / pointer is from upstream latest record. This is calculated as the difference between the `latestUpstreamOffset` and `currentOffset` for the partition when the lag computation request is made. | | recordsAvailabilityLagMap | (Whenever applicable) Defines how soon after record ingestion was the record consumed by Pinot. This is calculated as the difference between the time the record was consumed and the time at which the record was ingested upstream. | When Pinot cannot compute a lag value, it returns the string sentinel `NOT_CALCULATED` instead of a numeric placeholder. This happens, for example, when the stream connector cannot provide the latest upstream offset for a partition or when the last consumed record does not have a valid upstream ingestion timestamp. Pinot uses the same sentinel in both the `/consumingSegmentsInfo` and `/debug` responses. ## Monitor real-time ingestion Real-time ingestion includes 3 stages of message processing: Decode, Transform, and Index. In each of these stages, a failure can happen which may or may not result in an ingestion failure. The following metrics are available to investigate ingestion issues: 1. Decode stage -> an error here is recorded as `INVALID_REALTIME_ROWS_DROPPED` 2. Transform stage -> possible errors here are: 1. When a message gets dropped due to the [FILTER](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations#filtering) transform, it is recorded as `REALTIME_ROWS_FILTERED` 2. When the transform pipeline sets the `$INCOMPLETE_RECORD_KEY$` key in the message, it is recorded as `INCOMPLETE_REALTIME_ROWS_CONSUMED` , only when `continueOnError` configuration is enabled. If the `continueOnError` is not enabled, the ingestion fails. 3. Index stage -> When there is failure at this stage, the ingestion typically stops and marks the partition as ERROR. There is yet another metric called `ROWS_WITH_ERROR` which is the sum of all error counts in the 3 stages above. Furthermore, the metric `REALTIME_CONSUMPTION_EXCEPTIONS` gets incremented whenever there is a transient/permanent stream exception seen during consumption. These metrics can be used to understand why ingestion failed for a particular table partition before diving into the server logs. # Ingest from Apache Kafka This guide shows you how to ingest a stream of records from an Apache Kafka topic into a Pinot table. Learn how to ingest data from Kafka, a stream processing platform. You should have a local cluster up and running, following the instructions in [Set up a cluster](/operate-pinot/deployment/setup-cluster). {% hint style="info" %} This guide uses the Kafka 3.0 connector (`kafka30`). Pinot also supports a **Kafka 4.0 connector** for KRaft-mode Kafka clusters. See [Kafka Connector Versions](/build-with-pinot/ingestion/stream-ingestion/kafka-connector-versions) for details on choosing the right connector. {% endhint %} ## Install and Launch Kafka Let's start by downloading Kafka to our local machine. {% tabs %} {% tab title="Docker" %} To pull down the latest Docker image, run the following command: ```bash docker pull apache/kafka:4.0.0 ``` {% endtab %} {% tab title="Launcher Scripts" %} Download Kafka from [kafka.apache.org/quickstart#quickstart\_download](https://kafka.apache.org/quickstart#quickstart_download) and then extract it: ```bash tar -xzf kafka_2.13-4.0.0.tgz cd kafka_2.13-4.0.0 ``` {% endtab %} {% endtabs %} Next we'll spin up a Kafka broker. Kafka 4.0 uses KRaft mode by default and does not require ZooKeeper: {% tabs %} {% tab title="Docker" %} ```bash docker run --network pinot-demo --name=kafka \ -e KAFKA_NODE_ID=1 \ -e KAFKA_PROCESS_ROLES=broker,controller \ -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093 \ -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \ -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \ -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \ apache/kafka:4.0.0 ``` Note: The --network pinot-demo flag is optional and assumes that you have a Docker network named pinot-demo that you want to connect the Kafka container to. {% endtab %} {% tab title="Launcher Scripts" %} Kafka 4.0 uses KRaft mode by default. Generate a cluster ID and format the storage directory, then start the broker: **Start Kafka Broker (KRaft mode)** ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties bin/kafka-server-start.sh config/server.properties ``` {% endtab %} {% endtabs %} ## Data Source We're going to generate some JSON messages from the terminal using the following script: ```python import datetime import uuid import random import json while True: ts = int(datetime.datetime.now().timestamp()* 1000) id = str(uuid.uuid4()) count = random.randint(0, 1000) print( json.dumps({"ts": ts, "uuid": id, "count": count}) ) ``` *datagen.py* If you run this script (`python datagen.py`), you'll see the following output: ```json {"ts": 1644586485807, "uuid": "93633f7c01d54453a144", "count": 807} {"ts": 1644586485836, "uuid": "87ebf97feead4e848a2e", "count": 41} {"ts": 1644586485866, "uuid": "960d4ffa201a4425bb18", "count": 146} ``` ## Ingesting Data into Kafka Let's now pipe that stream of messages into Kafka, by running the following command: {% tabs %} {% tab title="Docker" %} ```bash python datagen.py | docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events; ``` {% endtab %} {% tab title="Launcher Scripts" %} ```bash python datagen.py | bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events; ``` {% endtab %} {% endtabs %} We can check how many messages have been ingested by running the following command: {% tabs %} {% tab title="Docker" %} ```bash docker exec -i kafka /opt/kafka/bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic events ``` {% endtab %} {% tab title="Launcher Scripts" %} ```bash bin/kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic events ``` {% endtab %} {% endtabs %} **Output** ``` events:0:11940 ``` And we can print out the messages themselves by running the following command {% tabs %} {% tab title="Docker" %} ```bash docker exec -i kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events ``` {% endtab %} {% tab title="Launcher Scripts" %} ```bash bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events ``` {% endtab %} {% endtabs %} **Output** ```json ... {"ts": 1644586485807, "uuid": "93633f7c01d54453a144", "count": 807} {"ts": 1644586485836, "uuid": "87ebf97feead4e848a2e", "count": 41} {"ts": 1644586485866, "uuid": "960d4ffa201a4425bb18", "count": 146} ... ``` ## Schema A schema defines what fields are present in the table along with their data types in JSON format. Create a file called `/tmp/pinot/schema-stream.json` and add the following content to it. ```json { "schemaName": "events", "dimensionFieldSpecs": [ { "name": "uuid", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "count", "dataType": "INT" } ], "dateTimeFieldSpecs": [{ "name": "ts", "dataType": "TIMESTAMP", "format" : "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" }] } ``` ## Table Config A table is a logical abstraction that represents a collection of related data. It is composed of columns and rows (known as documents in Pinot). The table config defines the table's properties in JSON format. Create a file called `/tmp/pinot/table-config-stream.json` and add the following content to it. ```json { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` ### JSON stream payload formats `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder` still defaults to UTF-8 text JSON when `stream.kafka.decoder.prop.jsonFormat` is unset, so existing Kafka table configs keep their historical behavior. To decode a binary JSON stream, add `stream.kafka.decoder.prop.jsonFormat` to the `streamConfigs` block and pin the payload encoding: ```json "stream.kafka.decoder.prop.jsonFormat": "SQLITE_JSONB" ``` Supported values are `TEXT`, `POSTGRES_JSONB`, `SQLITE_JSONB`, `SMILE`, `CBOR`, and `AUTO`. Use `AUTO` only when a topic can legitimately contain more than one of those encodings. `AUTO` is opt-in, and its CBOR detection only works when each message includes the CBOR self-describe tag. If the producer always emits one known format, pin that format explicitly instead. This setting applies only to stream decoding. Batch ingestion with `JSONRecordReader` still reads text JSON files. ## Create schema and table Create the table and schema by running the appropriate command below: {% tabs %} {% tab title="Docker" %} ```bash docker run --rm -ti --network=pinot-demo -v /tmp/pinot:/tmp/pinot apachepinot/pinot:1.0.0 AddTable -schemaFile /tmp/pinot/schema-stream.json -tableConfigFile /tmp/pinot/table-config-stream.json -controllerHost pinot-controller -controllerPort 9000 -exec ``` {% endtab %} {% tab title="Launcher Scripts" %} ```bash bin/pinot-admin.sh AddTable -schemaFile /tmp/pinot/schema-stream.json -tableConfigFile /tmp/pinot/table-config-stream.json ``` {% endtab %} {% endtabs %} ## Querying Navigate to [localhost:9000/#/query](http://localhost:9000/#/query) and click on the `events` table to run a query that shows the first 10 rows in this table. \_Querying the events table\_ ## Kafka ingestion guidelines ### Kafka connector modules in Pinot Pinot ships two Kafka connector modules: * **`pinot-kafka-3.0`** -- Uses Kafka client library 3.x (currently 3.9.2). This is the default connector included in Pinot distributions. Consumer factory class: `org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory`. * **`pinot-kafka-4.0`** -- Uses Kafka client library 4.x (currently 4.1.1). This connector drops the ZooKeeper-based Scala dependency and uses the pure-Java Kafka client, suitable for KRaft-mode Kafka clusters. Consumer factory class: `org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory`. {% hint style="info" %} The legacy `kafka-0.9` and `kafka-2.x` connector modules have been removed. If you are upgrading from an older Pinot release that used `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory`, update your table configs to use one of the current connector classes listed above. {% endhint %} {% hint style="info" %} Pinot does ***not support*** using high-level Kafka consumers (HLC). Pinot uses low-level (partition-level) consumers to ensure accurate results, reduce operational complexity, scale horizontally, and minimize storage overhead. {% endhint %} #### Migrating from the kafka-2.x connector If your existing table configs reference the removed `kafka-2.x` connector, update the `stream.kafka.consumer.factory.class.name` property: * From: `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` * To (Kafka 3.x): `org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory` * To (Kafka 4.x): `org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory` No other stream config changes are required. The Kafka 3.x connector is compatible with Kafka brokers 2.x and above. The Kafka 4.x connector requires Kafka brokers 4.0 or above. ### Kafka configurations in Pinot #### Use Kafka partition (low) level consumer with SSL Here is an example config which uses SSL based authentication to talk with kafka and schema-registry. Notice there are two sets of SSL options, ones starting with `ssl.` are for kafka consumer and ones with `stream.kafka.decoder.prop.schema.registry.` are for `SchemaRegistryClient` used by `KafkaConfluentSchemaRegistryAvroMessageDecoder`. ``` { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9092", "schema.registry.url": "", "security.protocol": "SSL", "ssl.truststore.location": "", "ssl.keystore.location": "", "ssl.truststore.password": "", "ssl.keystore.password": "", "ssl.key.password": "", "stream.kafka.decoder.prop.schema.registry.rest.url": "", "stream.kafka.decoder.prop.schema.registry.ssl.truststore.location": "", "stream.kafka.decoder.prop.schema.registry.ssl.keystore.location": "", "stream.kafka.decoder.prop.schema.registry.ssl.truststore.password": "", "stream.kafka.decoder.prop.schema.registry.ssl.keystore.password": "", "stream.kafka.decoder.prop.schema.registry.ssl.keystore.type": "", "stream.kafka.decoder.prop.schema.registry.ssl.truststore.type": "", "stream.kafka.decoder.prop.schema.registry.ssl.key.password": "", "stream.kafka.decoder.prop.schema.registry.ssl.protocol": "" } }, "metadata": { "customConfigs": {} } } ``` #### Use Confluent Schema Registry with JSON encoded messages If your Kafka messages are JSON-encoded and registered with Confluent Schema Registry, use the `KafkaConfluentSchemaRegistryJsonMessageDecoder`. This decoder uses the Confluent `KafkaJsonSchemaDeserializer` to decode messages whose JSON schemas are managed by the registry. **When to use this decoder** * Your Kafka producer serializes messages using the Confluent JSON Schema serializer. * Your JSON schemas are registered in Confluent Schema Registry. * You want schema validation and evolution support for JSON messages. If your messages are Avro-encoded and registered with Schema Registry, use `KafkaConfluentSchemaRegistryAvroMessageDecoder` instead (shown in the SSL example above). If your messages are plain JSON without a schema registry, use `JSONMessageDecoder`. **Example table config** ```json { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "created_at", "timeType": "MILLISECONDS", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:9092", "stream.kafka.schema.registry.url": "http://localhost:8081", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://localhost:8081", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` The key configuration properties for this decoder are: * `stream.kafka.decoder.class.name` -- Set to `org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder`. * `stream.kafka.decoder.prop.schema.registry.rest.url` -- The URL of the Confluent Schema Registry. **Authentication** This decoder supports the same authentication options as the Avro schema registry decoder. You can configure SSL or SASL\_SSL authentication for both the Kafka consumer and the Schema Registry client using the `stream.kafka.decoder.prop.schema.registry.*` properties. See the [SSL example](#use-kafka-partition-low-level-consumer-with-ssl) and [SASL\_SSL example](#use-kafka-partition-low-level-consumer-with-sasl_ssl) above for details. For Schema Registry basic authentication, add the following properties: ``` "stream.kafka.decoder.prop.basic.auth.credentials.source": "USER_INFO", "stream.kafka.decoder.prop.schema.registry.basic.auth.user.info": ":" ``` {% hint style="info" %} This decoder was added in Pinot 1.4. Make sure your Pinot deployment is running version 1.4 or later. {% endhint %} #### Consume transactionally-committed messages The Kafka 3.x and 4.x connectors support Kafka transactions. The transaction support is controlled by config `kafka.isolation.level` in Kafka stream config, which can be `read_committed` or `read_uncommitted` (default). Setting it to `read_committed` will ingest transactionally committed messages in Kafka stream only. For example, ``` { "tableName": "transcript", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "timeType": "MILLISECONDS", "schemaName": "transcript", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "transcript-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.isolation.level": "read_committed" } }, "metadata": { "customConfigs": {} } } ``` Note that the default value of this config `read_uncommitted` to read all messages. Also, this config supports low-level consumer only. #### Use Kafka partition (low) level consumer with SASL\_SSL Here is an example config which uses SASL\_SSL based authentication to talk with kafka and schema-registry. Notice there are two sets of SSL options, some for kafka consumer and ones with `stream.kafka.decoder.prop.schema.registry.` are for `SchemaRegistryClient` used by `KafkaConfluentSchemaRegistryAvroMessageDecoder`. ``` "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "mytopic", "stream.kafka.consumer.prop.auto.offset.reset": "largest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.schema.registry.url": "https://xxx", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": "https://xxx", "stream.kafka.decoder.prop.basic.auth.credentials.source": "USER_INFO", "stream.kafka.decoder.prop.schema.registry.basic.auth.user.info": "schema_registry_username:schema_registry_password", "sasl.mechanism": "PLAIN" , "security.protocol": "SASL_SSL" , "sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule required username=\"kafkausername\" password=\"kafkapassword\";", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.autotune.initialRows": "3000000", "realtime.segment.flush.threshold.segment.size": "500M" }, ``` #### Extract record headers as Pinot table columns Pinot's Kafka connector supports automatically extracting record headers and metadata into the Pinot table columns. The following table shows the mapping for record header/metadata to Pinot table column names: | Kafka Record | Pinot Table Column | Description | | ---------------------------------------- | --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Record key: any type | `__key` : String | For simplicity of design, we assume that the record key is always a UTF-8 encoded String | | Record Headers: Map\ | Each header key is listed as a separate column: `__header$HeaderKeyName` : String | For simplicity of design, we directly map the string headers from kafka record to pinot table column | | Record metadata - offset : long | `__metadata$offset` : String | | | Record metadata - partition : int | `__metadata$partition` : String | | | Record metadata - recordTimestamp : long | `__metadata$recordTimestamp` : String | | In order to enable the metadata extraction in a Kafka table, you can set the stream config `metadata.populate` to `true`. In addition to this, if you want to use any of these columns in your table, you have to list them explicitly in your table's schema. For example, if you want to add only the offset and key as dimension columns in your Pinot table, it can listed in the schema as follows: ```json "dimensionFieldSpecs": [ { "name": "__key", "dataType": "STRING" }, { "name": "__metadata$offset", "dataType": "STRING" }, { "name": "__metadata$partition", "dataType": "STRING" }, ... ], ``` Once the schema is updated, these columns are similar to any other pinot column. You can apply ingestion transforms and / or define indexes on them. {% hint style="info" %} Remember to follow the [schema evolution guidelines](/tutorials/data-ingestion/schema-evolution) when updating schema of an existing table! {% endhint %} #### Tell Pinot where to find an Avro schema There is a standalone utility to generate the schema from an Avro file. See [infer the pinot schema from the avro schema and JSON data](/build-with-pinot/ingestion/formats-filesystems/complex-type#infer-the-pinot-schema-from-the-avro-schema-and-json-data) for details. To avoid errors like `The Avro schema must be provided`, designate the location of the schema in your `streamConfigs` section. For example, if your current section contains the following: ```json ... "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "", "stream.kafka.consumer.prop.auto.offset.reset": "largest" ... } ``` Then add this key: `"stream.kafka.decoder.prop.schema"`followed by a value that denotes the location of your schema. #### Subset partition ingestion By default, a Pinot REALTIME table consumes from all partitions of the configured Kafka topic. In some scenarios you may want a table to consume only a subset of the topic's partitions. The `stream.kafka.partition.ids` setting lets you specify exactly which Kafka partitions a table should consume. **When to use subset partition ingestion** * **Split-topic ingestion** -- Multiple Pinot tables share the same Kafka topic, and each table is responsible for a different set of partitions. This is useful when the same topic contains logically distinct data partitioned by key, and you want separate tables (or indexes) for each partition group. * **Multi-table partition assignment** -- You want to distribute the partitions of a high-throughput topic across several Pinot tables for workload isolation, independent scaling, or different retention policies. * **Selective consumption** -- You only need data from specific partitions of a topic (for example, partitions that correspond to a particular region or tenant). **Configuration** Add `stream.kafka.partition.ids` to the `streamConfigMaps` entry in your table config. The value can contain individual zero-based partition IDs, inclusive ranges, or a mix of both in a comma-separated string: ```json "stream.kafka.partition.ids": "0-3,6,8-9" ``` For example, `"0,2,5"` consumes three explicit partitions, `"0-3"` consumes partitions 0 through 3 inclusive, and `"0-3,6,8-9"` mixes both forms in one value. When this setting is present, Pinot will consume only from the resolved set of partitions. When it is absent or blank, Pinot consumes from all partitions of the topic (the default behavior). If the table also uses `segmentPartitionConfig` for partition pruning, keep `numPartitions` aligned with the full Kafka topic partition count, not the size of the consumed subset. Pinot derives realtime segment partition metadata from the full topic partition count even when `stream.kafka.partition.ids` restricts ingestion to a subset. **Example: splitting a topic across two tables** Suppose you have a Kafka topic called `events` with two partitions (0 and 1). You can create two Pinot tables, each consuming from one partition: Table `events_part_0`: ```json { "tableName": "events_part_0", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.partition.ids": "0", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M" } ] } }, "metadata": { "customConfigs": {} } } ``` Table `events_part_1`: ```json { "tableName": "events_part_1", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP" }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.partition.ids": "1", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M" } ] } }, "metadata": { "customConfigs": {} } } ``` **Validation rules and limitations** * Partition IDs must be non-negative integers. Range values are inclusive, and the start of a range must be less than or equal to the end. * Non-integer values (for example, `"abc"`) and malformed ranges (for example, `"5-"`) will cause a validation error. * Duplicate IDs are silently deduplicated after range expansion. For example, `"0,2,0,5"` is treated as `"0,2,5"`. * The partition IDs are sorted internally for stable ordering, regardless of the order specified in the config. * Pinot validates the resolved partition IDs against the Kafka topic metadata before it starts consuming. If a specified partition ID does not exist in the topic, an error is raised. * The resolved set can contain at most 10,000 unique partition IDs. * If you configure `segmentPartitionConfig` on a subset-ingestion realtime table, set `numPartitions` to the full Kafka topic partition count. For example, if a table consumes `"0,3"` from an 8-partition topic, use `8`, not `2`. * When using subset partition ingestion with multiple tables consuming from the same topic, ensure that the partition assignments do not overlap if you want each record to be consumed by exactly one table. Pinot does not enforce non-overlapping partition assignments across tables. * Whitespace around partition IDs, commas, and range bounds is trimmed (for example, `" 0 - 3 , 5 "` is valid). #### Use Protocol Buffers (Protobuf) format Pinot supports decoding Protocol Buffer messages from Kafka using several decoder options depending on your setup. **ProtoBufMessageDecoder (descriptor file based)** Use `ProtoBufMessageDecoder` when you have a pre-compiled `.desc` (descriptor) file for your Protobuf schema. This decoder uses dynamic message parsing and does not require compiled Java classes. Required stream config properties: | Property | Description | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | | `stream.kafka.decoder.prop.descriptorFile` | Path or URI to the `.desc` descriptor file. Supports local file paths, HDFS, and other Pinot-supported file systems. | | `stream.kafka.decoder.prop.protoClassName` | (Optional) Fully qualified Protobuf message name within the descriptor. If omitted, the first message type in the descriptor is used. | Example `streamConfigs`: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-protobuf-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.decoder.prop.descriptorFile": "/path/to/my_message.desc", "stream.kafka.decoder.prop.protoClassName": "mypackage.MyMessage" } ``` **ProtoBufCodeGenMessageDecoder (compiled JAR based)** Use `ProtoBufCodeGenMessageDecoder` when you have a compiled JAR containing your generated Protobuf Java classes. This decoder uses runtime code generation for improved decoding performance. Required stream config properties: | Property | Description | | ------------------------------------------ | ------------------------------------------------------------------- | | `stream.kafka.decoder.prop.jarFile` | Path or URI to the JAR file containing compiled Protobuf classes. | | `stream.kafka.decoder.prop.protoClassName` | Fully qualified Java class name of the Protobuf message (required). | Example `streamConfigs`: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-protobuf-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufCodeGenMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.decoder.prop.jarFile": "/path/to/my-protobuf-classes.jar", "stream.kafka.decoder.prop.protoClassName": "com.example.proto.MyMessage" } ``` **KafkaConfluentSchemaRegistryProtoBufMessageDecoder (Confluent Schema Registry)** Use `KafkaConfluentSchemaRegistryProtoBufMessageDecoder` when your Protobuf schemas are managed by Confluent Schema Registry. This decoder automatically resolves schemas from the registry at runtime. Required stream config properties: | Property | Description | | ---------------------------------------------------- | ------------------------------------- | | `stream.kafka.decoder.prop.schema.registry.rest.url` | URL of the Confluent Schema Registry. | Optional properties: | Property | Description | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | `stream.kafka.decoder.prop.cached.schema.map.capacity` | Maximum number of cached schemas. Default: `1000`. | | `stream.kafka.decoder.prop.schema.registry.*` | SSL and authentication options for connecting to Schema Registry (same pattern as the Avro Confluent decoder). | Example `streamConfigs`: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-protobuf-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081" } ``` #### Use BSON format Pinot supports decoding BSON messages from Kafka using `BSONMessageDecoder`. Use this when each Kafka message contains a single binary-encoded BSON document, such as records produced by a MongoDB change-data-capture pipeline. Example `streamConfigs`: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "mongo-events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.bson.BSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092" } ``` `BSONMessageDecoder` uses the same extractor as batch BSON ingestion, so `ObjectId` values decode to hex strings, `Date` and `BsonTimestamp` values decode to `java.sql.Timestamp`, `Decimal128` values decode to `BigDecimal` with `NaN` and `Infinity` surfaced as `null`, binary values decode to `byte[]`, embedded documents decode to `Map`, and arrays decode to `Object[]`. #### Use Apache Arrow format Pinot supports decoding Apache Arrow IPC streaming format messages from Kafka using `ArrowMessageDecoder`. This is useful when upstream systems produce data serialized in Arrow format. Optional stream config properties: | Property | Description | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `stream.kafka.decoder.prop.arrow.allocator.limit` | Maximum memory (in bytes) for the Arrow allocator. Default: `268435456` (256 MB). | | `stream.kafka.decoder.prop.extractRawTimeValues` | Keep Arrow `Date`, `Time`, and `Timestamp` values as raw integers instead of Pinot's default converted values. Default: `false`. | Example `streamConfigs`: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-arrow-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.arrow.ArrowMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.decoder.prop.arrow.allocator.limit": "536870912", "stream.kafka.decoder.prop.extractRawTimeValues": "true" } ``` {% hint style="info" %} The Arrow decoder expects each Kafka message to contain a complete Arrow IPC stream (schema + record batch). Ensure your producer serializes Arrow data in the IPC streaming format. {% endhint %} When `stream.kafka.decoder.prop.extractRawTimeValues` is `false` (the default), Pinot converts Arrow `Date`, `Time`, and `Timestamp` values during extraction. Set it to `true` to keep raw integers instead: `Date` stays as days since epoch, while `Time` and `Timestamp` stay in the schema's declared Arrow unit. Each Arrow Kafka message can yield zero, one, or many Pinot rows. Empty batches are ignored, single-row batches ingest as one Pinot row, and multi-row batches fan out into multiple Pinot rows. ## Consuming a Subset of Kafka Partitions By default, a Pinot realtime table consumes all partitions of a Kafka topic. You can restrict ingestion to a specific subset of partitions using the `stream.kafka.partition.ids` property. This is useful when: * Splitting a single Kafka topic across multiple Pinot tables for independent scaling * Multi-tenant scenarios where different tables own different partition ranges ### Configuration Add `stream.kafka.partition.ids` to your `streamConfigs` with a comma-separated string of partition IDs, inclusive ranges, or both: ```json "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "myTopic", "stream.kafka.broker.list": "localhost:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.partition.ids": "0-3,6,8-9" } ``` For example, `"0,2,5"` selects individual partitions, `"0-3"` selects an inclusive range, and `"0-3,6,8-9"` mixes both forms in one value. If the table also uses `segmentPartitionConfig`, keep `numPartitions` set to the full Kafka topic partition count rather than the size of the consumed subset. Pinot computes realtime segment partition metadata from the full topic partition count even when ingestion is limited to specific Kafka partitions. ### Notes * Partition IDs must be non-negative integers. Ranges are inclusive and must have `start <= end`. * Pinot validates the resolved partition IDs against the Kafka topic metadata before it starts consuming. * Duplicate IDs in the list are automatically deduplicated. * The resolved set can contain at most 10,000 unique partition IDs. * The total partition count reported to the broker reflects the full Kafka topic size, ensuring correct query routing across tables sharing the same topic. * If you configure `segmentPartitionConfig`, set `numPartitions` to the full Kafka topic partition count. For example, if a table consumes `"1,4"` from a 6-partition topic, use `6`, not `2`. * When splitting a topic between two tables, configure one with even-numbered IDs and another with odd-numbered IDs (for example, `"0,2"` and `"1,3"` for a 4-partition topic). # Ingest from Amazon Kinesis This guide shows you how to ingest a stream of records from an Amazon Kinesis topic into a Pinot table. To ingest events from an Amazon Kinesis stream into Pinot, set the following configs into your table config: ```json { "tableName": "kinesisTable", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kinesis", "stream.kinesis.topic.name": "", "region": "", "accessKey": "", "secretKey": "", "shardIteratorType": "AFTER_SEQUENCE_NUMBER", "stream.kinesis.fetch.timeout.millis": "30000", "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory", "realtime.segment.flush.threshold.rows": "1000000", "realtime.segment.flush.threshold.time": "6h" } }, "metadata": { "customConfigs": {} } } ``` where the Kinesis specific properties are: | Property | Description | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | streamType | This should be set to "kinesis" | | stream.kinesis.topic.name | Kinesis stream name | | region | Kinesis region e.g. us-west-1 | | accessKey | Kinesis access key | | secretKey | Kinesis secret key | | shardIteratorType | AWS shard iterator type Pinot uses when it opens a shard iterator. Pinot passes this value through to the Kinesis client. **Default:** `LATEST` | | maxRecordsToFetch | Specifies the maximum number of records to retrieve in a single `getRecords` API call to Kinesis. This parameter controls the batch size for data retrieval. Can be set between 1 and 10,000 (Kinesis API limit by AWS) Larger values reduce the number of API calls needed but may increase latency and memory usage per batch. Default value is set to max 10000. Only lower this when you have memory constraints | | requests\_per\_second\_limit | Controls the maximum Kinesis read requests per second Pinot will attempt per shard. Pinot applies this budget to both `getRecords` and `getShardIterator` reads, accepts fractional values such as `0.25`, and defaults to `1.0`. Start by dividing Kinesis's 5 `getRecords` requests-per-second shard limit across Pinot replicas, Pinot tables, and any non-Pinot consumers that share the shard. | When Kinesis still returns `ProvisionedThroughputExceededException`, Pinot backs off and retries until the fetch timeout instead of immediately returning an empty batch. This makes fractional `requests_per_second_limit` values practical when several consumers need to share the same shard budget. Kinesis supports authentication using the [DefaultCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). The credential provider looks for the credentials in the following order: * Environment Variables - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` (only recognized by Java SDK) * Java System Properties - `aws.accessKeyId` and `aws.secretKey` * Web Identity Token credentials from the environment or container * Credential profiles file at the default location `(~/.aws/credentials)` shared by all AWS SDKs and the AWS CLI * Credentials delivered through the Amazon EC2 container service if `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` environment variable is set and security manager has permission to access the variable, * Instance profile credentials delivered through the Amazon EC2 metadata service {% hint style="info" %} You must provide all `read` `access level` permissions for Pinot to work with an AWS Kinesis data stream. See the [AWS documentation](/build-with-pinot/ingestion/stream-ingestion/amazon-kinesis) for details. {% endhint %} Although you can also specify the `accessKey` and `secretKey` in the properties above, we don't recommend this insecure method. We recommend using it only for non-production proof-of-concept (POC) setups. You can also specify other AWS fields such as AWS\_SESSION\_TOKEN as environment variables and config and it will work. ### Resharding In Kinesis, whenever you reshard a stream, it is done via split or merge operations on shards. If you split a shard, the shard closes and creates 2 new children shards. So if you started with shard0, and then split it, it would result in shard1 and shard2. Similarly, if you merge 2 shards, both those will close and create a child shard. So in the same example, if you merge shards 1 and 2, you'll end up with shard3 as the active shard, while shard0, shard1, shard2 will remain closed forever. Please check out this recipe for more details: In Pinot, resharding of any stream is detected by periodic task RealtimeValidationManager: [docs](/reference/configuration-reference/controller#realtimesegmentvalidationmanager). This runs hourly. If you rehsard, your new shards will not get detected unless: 1. We finish ingesting from parent shards completely 2. And after 1, the RealtimeValidationManager runs You will see a period where the ideal state will show all segments ONLINE, as parents have naturally completed ingesting, and we're waiting for RealtimeValidationManager to kickstart the ingestion from children. If you need the ingestion to happen sooner, you can manually invoke the RealtimeValidationManager: [docs](/architecture-and-concepts/components/cluster/controller#running-the-periodic-task-manually) ### Limitations 1. `ShardID` is of the format "**shardId-000000000001**". We use the numeric part as `partitionId`. Our `partitionId` variable is integer. If shardIds grow beyond `Integer.MAX\_VALUE`, we will overflow into the partitionId space. 2. Segment size based thresholds for segment completion will not work. It assumes that partition "0" always exists. However, once the shard 0 is split/merged, we will no longer have partition 0. # Ingest from Apache Pulsar This guide shows you how to ingest a stream of records from an Apache Pulsar topic into a Pinot table. Pinot supports consuming data from [Apache Pulsar](https://pulsar.apache.org) via the `pinot-pulsar` plugin. You need to enable this plugin so that Pulsar specific libraries are present in the classpath. Enable the Pulsar plugin with the following config at the time of Pinot setup: `-Dplugins.include=pinot-pulsar` {% hint style="info" %} The `pinot-pulsar` plugin is included in the official binary distribution since Pinot 0.11.0. If you are running an older version, you can download the plugin from [the Apache Pinot external repository](https://repo.startreedata.io/artifactory/external-snapshots/org/apache/pinot/pinot-pulsar/) and add it to the `plugins` directory. {% endhint %} ## Set up Pulsar table Here is a sample Pulsar stream config. You can use the `streamConfigs` section from this sample and make changes for your corresponding table. ```json { "tableName": "pulsarTable", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "timestamp", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "pulsar", "stream.pulsar.topic.name": "", "stream.pulsar.bootstrap.servers": "pulsar://localhost:6650,pulsar://localhost:6651", "stream.pulsar.consumer.prop.auto.offset.reset" : "smallest", "stream.pulsar.fetch.timeout.millis": "30000", "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory", "realtime.segment.flush.threshold.rows": "1000000", "realtime.segment.flush.threshold.time": "6h" } }, "metadata": { "customConfigs": {} } } ``` ## Pulsar configuration options You can change the following Pulsar specifc configurations for your tables | Property | Description | | --------------------------------- | ---------------------------------------------- | | `streamType` | This should be set to "pulsar" | | `stream.pulsar.topic.name` | Your pulsar topic name | | `stream.pulsar.bootstrap.servers` | Comma-separated broker list for Apache Pulsar | | `stream.pulsar.metadata.populate` | set to `true` to populate metadata | | `stream.pulsar.metadata.fields` | set to comma separated list of metadata fields | ### Authentication The Pinot-Pulsar connector supports authentication using security tokens. To generate a token, follow the instructions in [Pulsar documentation](https://pulsar.apache.org/docs/en/security-jwt). Once generated, add the following property to `streamConfigs` to add an authentication token for each request: ``` "stream.pulsar.authenticationToken":"your-auth-token" ``` ### OAuth2 Authentication The Pinot-Pulsar connector supports authentication using OAuth2, for example, if connecting to a StreamNative Pulsar cluster. For more information, see how to [Configure OAuth2 authentication in Pulsar clients](https://pulsar.apache.org/docs/en/security-oauth2/#configure-oauth2-authentication-in-pulsar-clients). Once configured, you can add the following properties to `streamConfigs`: ``` "stream.pulsar.issuerUrl": "https://auth.streamnative.cloud" "stream.pulsar.credsFilePath": "file:///path/to/private_creds_file "stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster" ``` ### TLS support The Pinot-pulsar connector also supports TLS for encrypted connections. You can follow [the official pulsar documentation](https://pulsar.apache.org/docs/en/security-tls-transport/) to enable TLS on your pulsar cluster. Once done, you can enable TLS in pulsar connector by providing the trust certificate file location generated in the previous step. ``` "stream.pulsar.tlsTrustCertsFilePath": "/path/to/ca.cert.pem" ``` Also, make sure to change the brokers url from `pulsar://localhost:6650` to `pulsar+ssl://localhost:6650` so that secure connections are used. For other table and stream configurations, you can headover to [Table configuration Reference](/reference/configuration-reference/table) ### Supported Pulsar versions Pinot currently relies on Pulsar client version 4.0.x. Make sure the Pulsar broker is compatible with this client version. #### Extract record headers as Pinot table columns Pinot's Pulsar connector supports automatically extracting record headers and metadata into the Pinot table columns. Pulsar supports a large amount of per-record metadata. Reference the [official Pulsar documentation](https://pulsar.apache.org/docs/en/concepts-messaging/#message-properties) for the meaning of the metadata fields. The following table shows the mapping for record header/metadata to Pinot table column names: | Pulsar Message | Pinot table Column | Comments | Available By Default | | --------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | -------------------- | | key : String | `__key` : String | | Yes | | properties : Map\ | Each header key is listed as a separate column: `__header$HeaderKeyName` : String | | Yes | | publishTime : Long | `__metadata$publishTime` : String | publish time as determined by the producer | Yes | | brokerPublishTime: Optional | `__metadata$brokerPublishTime` : String | publish time as determined by the broker | Yes | | eventTime : Long | `__metadata$eventTime` : String | | Yes | | messageId : MessageId -> String | `__metadata$messageId` : String | String representation of the MessagId field. The format is ledgerId:entryId:partitionIndex | | | messageId : MessageId -> bytes | `__metadata$messageBytes` : String | Base64 encoded version of the bytes returned from calling MessageId.toByteArray() | | | producerName : String | `__metadata$producerName` : String | | | | schemaVersion : byte\[] | `__metadata$schemaVersion` : String | Base64 encoded value | | | sequenceId : Long | `__metadata$sequenceId` : String | | | | orderingKey : byte\[] | `__metadata$orderingKey` : String | Base64 encoded value | | | size : Integer | `__metadata$size` : String | | | | topicName : String | `__metadata$topicName` : String | | | | index : String | `__metadata$index` : String | | | | redeliveryCount : Integer | `__metadata$redeliveryCount` : String | | | In order to enable the metadata extraction in a Pulsar table, set the stream config `metadata.populate` to `true`. The fields `eventTime`, `publishTime`, `brokerPublishTime`, and `key` are populated by default. If you would like to extract additional fields from the Pulsar Message, populate the `metadataFields` config with a comma separated list of fields to populate. The fields are referenced by the field name in the Pulsar Message. For example, setting: ```json "streamConfigs": { ... "stream.pulsar.metadata.populate": "true", "stream.pulsar.metadata.fields": "messageId,messageIdBytes,eventTime,topicName", ... } ``` Will make the `__metadata$messageId`, `__metadata$messageBytes`, `__metadata$eventTime`, and `__metadata$topicName`, fields available for mapping to columns in the Pinot schema. In addition to this, if you want to use any of these columns in your table, you have to list them explicitly in your table's schema. For example, if you want to add only the offset and key as dimension columns in your Pinot table, it can listed in the schema as follows: ```json "dimensionFieldSpecs": [ { "name": "__key", "dataType": "STRING" }, { "name": "__metadata$messageId", "dataType": "STRING" }, ... ], ``` Once the schema is updated, these columns are similar to any other pinot column. You can apply ingestion transforms and / or define indexes on them. {% hint style="info" %} Remember to follow the [schema evolution guidelines](/tutorials/data-ingestion/schema-evolution) when updating schema of an existing table! {% endhint %} # Configure Indexes Learn how to apply indexes to a Pinot table. This guide assumes that you have followed the [Ingest data from Apache Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) guide. Pinot supports a series of different indexes that can be used to optimize query performance. In this guide, we'll learn how to add indexes to the `events` table that we set up in the [Ingest data from Apache Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) guide. ### Why do we need indexes? If no indexes are applied to the columns in a Pinot segment, the query engine needs to scan through every document, checking whether that document meets the filter criteria provided in a query. This can be a slow process if there are a lot of documents to scan. When indexes are applied, the query engine can more quickly work out which documents satisfy the filter criteria, reducing the time it takes to execute the query. ### What indexes does Pinot support? By default, Pinot creates a forward index for every column. The forward index generally stores documents in insertion order. However, before flushing the segment, Pinot does a single pass over every column to see whether the data is sorted. If data is sorted, Pinot creates a sorted (forward) index for that column instead of the forward index. For real-time tables you can also explicitly tell Pinot that one of the columns should be sorted. For more details, see the \[Sorted Index Documentation]\(../../../../build-with-pinot/indexing/forward-index.md#real-time-tables). For filtering documents within a segment, Pinot supports the following indexing techniques: * Inverted index: Used for exact lookups. * Range index - Used for range queries. * Text index - Used for phrase, term, boolean, prefix, or regex queries. * Geospatial index - Based on H3, a hexagon-based hierarchical gridding. Used for finding points that exist within a certain distance from another point. * JSON index - Used for querying columns in JSON documents. * Star-Tree index - Pre-aggregates results across multiple columns. ## View events table Let's see how we can apply these indexing techniques to our data. To recap, the `events` table has the following fields: | Date Time Fields | Dimensions Fields | Metric Fields | | ---------------- | ----------------- | ------------- | | `ts` | `uuid` | `count` | We might want to write queries that filter on the `ts` and `uuid` columns, so these are the columns on which we would want to configure indexes. Since the data we're ingesting into the Kafka topic is all implicitly ordered by timestamp, this means that the `ts` column already has a sorted index. This means that any queries that filter on this column are already optimised. So that leaves us with the `uuid` column. ## Add an inverted index We're going to add an inverted index to the `uuid` column so that queries that filter on that column will return quicker. We need to add the following line: ```json "invertedIndexColumns": ["uuid"] ``` To the `tableIndexConfig` section. Copy the following to the clipboard: **/tmp/pinot/table-config-stream.json** ```json { "tableName": "events", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts", "schemaName": "events", "replicasPerPartition": "1" }, "tenants": {}, "tableIndexConfig": { "invertedIndexColumns": ["uuid"], "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "events", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "kafka:9092", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "50M", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "metadata": { "customConfigs": {} } } ``` Navigate to [localhost:9000/#/tenants/table/events\_REALTIME](http://localhost:9000/#/tenants/table/events_REALTIME), click on **Edit Table**, paste the next table config, and then click **Save**. Once you've done that, you'll need to click **Reload All Segments** and then **Yes** to apply the indexing change to all segments. ## Check the index has been applied We can check that the index has been applied to all our segments by querying Pinot's REST API. You can find Swagger documentation at [localhost:9000/help](http://localhost:9000/help). The following query will return the indexes defined on the `uuid` column: ```bash curl -X GET "http://localhost:9000/segments/events/metadata?columns=uuid" \ -H "accept: application/json" 2>/dev/null | jq '.[] | [.segmentName, .indexes]' ``` **Output** We're using the [jq command line JSON processor](https://stedolan.github.io/jq/) to extract the fields that we're interested in. ```json [ "events__0__1__20220214T1106Z", { "uuid": { "bloom-filter": "NO", "dictionary": "YES", "forward-index": "YES", "inverted-index": "YES", "null-value-vector-reader": "NO", "range-index": "NO", "json-index": "NO" } } ] [ "events__0__0__20220214T1053Z", { "uuid": { "bloom-filter": "NO", "dictionary": "YES", "forward-index": "YES", "inverted-index": "YES", "null-value-vector-reader": "NO", "range-index": "NO", "json-index": "NO" } } ] ``` We can see from looking at the `inverted-index` property that the index has been applied. ## Querying You can now run some queries that filter on the `uuid` column, as shown below: ```sql SELECT * FROM events WHERE uuid = 'f4a4f' LIMIT 10 ``` You'll need to change the actual `uuid` value to a value that exists in your database, because the UUIDs are generated randomly by our script. # Stream Ingestion with CLP Support for encoding fields with CLP during ingestion. {% hint style="warning" %} This is an experimental feature. Configuration options and usage may change frequently until it is stabilized. {% endhint %} When performing stream ingestion of JSON records using [Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka), users can encode specific fields with [CLP](https://github.com/y-scope/clp) by using a CLP-specific StreamMessageDecoder. CLP is a compressor designed to encode unstructured log messages in a way that makes them more compressible while retaining the ability to search them. It does this by decomposing the message into three fields: * the message's static text, called a log type; * repetitive variable values, called dictionary variables; and * non-repetitive variable values (called encoded variables since we encode them specially if possible). Searches are similarly decomposed into queries on the individual fields. {% hint style="info" %} Although CLP is designed for log messages, other unstructured text like file paths may also benefit from its encoding. {% endhint %} For example, consider this JSON record: ```json { "timestamp": 1672531200000, "message": "INFO Task task_12 assigned to container: [ContainerID:container_15], operation took 0.335 seconds. 8 tasks remaining.", "logPath": "/mnt/data/application_123/container_15/stdout" } ``` If the user specifies the fields `message` and `logPath` should be encoded with CLP, then the StreamMessageDecoder will output: ```json { "timestamp": 1672531200000, "message_logtype": "INFO Task \\x12 assigned to container: [ContainerID:\\x12], operation took \\x13 seconds. \\x11 tasks remaining.", "message_dictionaryVars": [ "task_12", "container_15" ], "message_encodedVars": [ 1801439850948198735, 8 ], "logPath_logtype": "/mnt/data/\\x12/\\x12/stdout", "logPath_dictionaryVars": [ "application_123", "container_15" ], "logPath_encodedVars": [] } ``` *In the fields with the `_logtype` suffix, \x11 is a placeholder for an integer variable, \x12 is a placeholder for a dictionary variable, and \x13 is a placeholder for a float variable. In `message_encoedVars`, the float variable `0.335` is encoded as an integer using CLP's custom encoding.* All remaining fields are processed in the same way as they are in `org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor`. Specifically, fields in the table's schema are extracted from each record and any remaining fields are dropped. ## Configuration ### Table Index Assuming the user wants to encode `message` and `logPath` as in the example, they should change/add the following settings to their `tableIndexConfig` (we omit irrelevant settings for brevity): ```json { "tableIndexConfig": { "streamConfigs": { "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.clplog.CLPLogMessageDecoder", "stream.kafka.decoder.prop.fieldsForClpEncoding": "message,logPath", "stream.kafka.decoder.prop.removeProcessedFields": "true" }, "varLengthDictionaryColumns": [ "message_logtype", "message_dictionaryVars", "logPath_logtype", "logPath_dictionaryVars" ] } } ``` * `stream.kafka.decoder.prop.fieldsForClpEncoding` is a comma-separated list of names for fields that should be encoded with CLP. * `stream.kafka.decoder.prop.removeProcessedFields` is optional. When set to `true`, Pinot removes the original input field after writing the derived CLP columns (`_logtype`, `_dictionaryVars`, `_encodedVars`). The default is `false`, which keeps the original field alongside the derived columns. * We use [variable-length dictionaries](/reference/configuration-reference/table#table-index-config) for the logtype and dictionary variables since their length can vary significantly. ### Schema For the table's schema, users should configure the CLP-encoded fields as follows (we omit irrelevant settings for brevity): ```json { "dimensionFieldSpecs": [ { "name": "message_logtype", "dataType": "STRING", "maxLength": 2147483647 }, { "name": "message_encodedVars", "dataType": "LONG", "singleValueField": false }, { "name": "message_dictionaryVars", "dataType": "STRING", "maxLength": 2147483647, "singleValueField": false }, { "name": "logpath_logtype", "dataType": "STRING", "maxLength": 2147483647 }, { "name": "logpath_encodedVars", "dataType": "LONG", "singleValueField": false }, { "name": "logpath_dictionaryVars", "dataType": "STRING", "maxLength": 2147483647, "singleValueField": false } ] } ``` * We use the maximum possible length for the logtype and dictionary variable columns. * The dictionary and encoded variable columns are multi-valued columns. ## Searching and decoding CLP-encoded fields To decode CLP-encoded fields, use [CLPDECODE](/functions/clp/clpdecode). To search CLP-encoded fields, you can combine `CLPDECODE` with `LIKE`. Note, this may decrease performance when querying a large number of rows. We are working to integrate efficient searches on CLP-encoded columns as another UDF. The development of this feature is being tracked in this [design doc](https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU/edit). ## CLP Forward Index V2 Starting in Pinot 1.3.0, the CLP forward index was upgraded to V2 (`CLPMutableForwardIndexV2`), which is now the default for CLP-encoded columns during real-time ingestion. Key improvements include: ### Dynamic encoding with cardinality monitoring V2 monitors dictionary cardinality during ingestion and dynamically switches encoding modes: * **CLP dictionary encoding**: Used when log type and dictionary variable cardinality remains below a configurable threshold relative to the document count. * **Raw string fallback**: When cardinality exceeds the threshold (docs/cardinality ratio drops below 10), V2 automatically falls back to a raw string forward index to avoid the memory and I/O overhead of maintaining a large dictionary. ### Improved compression V2 uses fixed-byte encoding with Zstandard chunk compression instead of V1's uncompressed fixed-bit encoding. This significantly improves compression ratios for most real-world log data. ### Compression codec options You can select the compression codec for CLP-encoded columns using the `compressionCodec` in `fieldConfig`: | Codec | Description | | ------------ | ------------------------------------------ | | `CLPV2` | CLP V2 with default ZStandard compression | | `CLPV2_ZSTD` | CLP V2 with explicit ZStandard compression | | `CLPV2_LZ4` | CLP V2 with LZ4 compression | | `CLP` | Legacy V1 (uncompressed, pass-through) | Example field config: ```json { "fieldConfigList": [ { "name": "message", "encodingType": "RAW", "compressionCodec": "CLPV2_ZSTD" } ] } ``` ### Immutable CLP Forward Index When mutable (real-time) segments are converted to immutable segments, V2 directly copies the mutable dictionary and index data without re-encoding, eliminating the serialization/deserialization overhead present in V1. The resulting immutable forward index is memory-mapped for efficient random access during queries. # Confluent Schema Registry Decoders Decode Avro, JSON, and Protobuf messages from Kafka using Confluent Schema Registry. Pinot supports decoding Kafka messages serialized with [Confluent Schema Registry](https://docs.confluent.io/platform/current/schema-registry/index.html) for Avro, JSON Schema, and Protocol Buffers formats. These decoders automatically fetch and cache schemas from the registry, ensuring data is deserialized according to the registered schema. ## Available Decoders | Format | Decoder Class | Plugin | | -------------------- | --------------------------------------------------------------------------------------------------- | -------------------------- | | **Avro** | `org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder` | `pinot-confluent-avro` | | **JSON Schema** | `org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder` | `pinot-confluent-json` | | **Protocol Buffers** | `org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder` | `pinot-confluent-protobuf` | ## Common Configuration All Confluent Schema Registry decoders share the same configuration properties: | Property | Required | Default | Description | | ---------------------------- | -------- | ------- | ------------------------------------------- | | `schema.registry.rest.url` | Yes | — | Confluent Schema Registry REST endpoint URL | | `cached.schema.map.capacity` | No | 1000 | Maximum number of schemas to cache locally | ### SSL/TLS Configuration To connect to a Schema Registry endpoint over SSL/TLS, add properties with the `schema.registry.` prefix: | Property | Description | | ----------------------------------------- | ----------------------- | | `schema.registry.ssl.truststore.location` | Path to truststore file | | `schema.registry.ssl.truststore.password` | Truststore password | | `schema.registry.ssl.keystore.location` | Path to keystore file | | `schema.registry.ssl.keystore.password` | Keystore password | | `schema.registry.ssl.key.password` | Private key password | ## Confluent Avro Decoder Decodes Avro-serialized Kafka messages with schema managed by Confluent Schema Registry. ```json { "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-avro-topic", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081" } } ``` ## Confluent JSON Schema Decoder Decodes JSON messages serialized with Confluent's JSON Schema serializer. Messages include a schema ID header that the decoder uses to fetch the JSON Schema from the registry for validation. ```json { "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-json-topic", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.confluent.KafkaConfluentSchemaRegistryJsonMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081" } } ``` {% hint style="info" %} The JSON Schema decoder validates incoming messages against the schema registered in Schema Registry. Messages that don't match the magic byte format (non-Confluent messages) are silently dropped. {% endhint %} ## Confluent Protobuf Decoder Decodes Protocol Buffer messages serialized with Confluent's Protobuf serializer. The decoder fetches the `.proto` schema definition from the registry and deserializes the binary payload. ```json { "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "my-protobuf-topic", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.KafkaConfluentSchemaRegistryProtoBufMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": "http://schema-registry:8081" } } ``` ## SSL/TLS Example To connect to a secured Schema Registry: ```json { "streamConfigs": { "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "stream.kafka.decoder.prop.schema.registry.rest.url": "https://schema-registry:8082", "stream.kafka.decoder.prop.schema.registry.ssl.truststore.location": "/path/to/truststore.jks", "stream.kafka.decoder.prop.schema.registry.ssl.truststore.password": "changeit", "stream.kafka.decoder.prop.schema.registry.ssl.keystore.location": "/path/to/keystore.jks", "stream.kafka.decoder.prop.schema.registry.ssl.keystore.password": "changeit" } } ``` ## How Schema Resolution Works 1. Each Confluent-serialized message starts with a magic byte (`0x00`) followed by a 4-byte schema ID 2. The decoder extracts the schema ID from the message header 3. The schema is fetched from Schema Registry and cached locally (up to `cached.schema.map.capacity`) 4. The message payload is deserialized using the resolved schema 5. Fields are extracted into Pinot's `GenericRow` format for ingestion Messages without the Confluent magic byte prefix are silently dropped and logged as errors. ## See Also * [Ingest from Apache Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) — General Kafka ingestion guide * [Stream Ingestion Connectors](/reference/plugin-reference/stream-ingestion-connectors) — Full connector configuration reference * [Supported Data Formats](/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats) — All supported input formats # Kafka Connector Versions Choose the right Apache Kafka connector version for your Pinot deployment. Apache Pinot provides multiple Kafka connector versions to match different Kafka broker deployments. Choose the connector that matches your Kafka cluster version. ## Available Connectors | Connector Plugin | Kafka Client Version | Notes | | ----------------- | -------------------- | --------------------------------------------------------------------------------------------------------------- | | `pinot-kafka-3.0` | 3.9.x | Recommended for Kafka 3.x clusters. Built with Scala 2.13 (default); use `-Pscala-2.12` profile for Scala 2.12. | | `pinot-kafka-4.0` | 4.1.x | Recommended for Kafka 4.x clusters (KRaft mode). Pure Java — no Scala dependency. | {% hint style="warning" %} The `pinot-kafka-2.0` (kafka20) plugin has been removed. If your table config references `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory`, you must migrate to either `kafka30` or `kafka40`. {% endhint %} ## Kafka 4.0 Connector The Kafka 4.0 connector (`pinot-kafka-4.0`) supports Apache Kafka 4.x brokers running in **KRaft mode** (ZooKeeper-free). It uses pure Java Kafka clients with no Scala dependency, resulting in a smaller deployment footprint. ### When to use Kafka 4.0 * Your Kafka cluster runs Kafka 4.0+ with KRaft mode * You want to eliminate the Scala transitive dependency * You are deploying new Pinot clusters against modern Kafka infrastructure ### Configuration The Kafka 4.0 connector uses the same configuration properties as the Kafka 3.0 connector. The only difference is the `stream.kafka.consumer.factory.class.name`: ```json { "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "your-topic", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "100M" } } ``` ### Migration from Kafka 2.0 or 3.0 To migrate from an older Kafka connector to Kafka 3.0 or 4.0, update the consumer factory class name in your table configuration: | From | To | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory` | `org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory` (Kafka 3.x) or `org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory` (Kafka 4.x) | | `org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory` | `org.apache.pinot.plugin.stream.kafka40.KafkaConsumerFactory` | 2. Ensure the `pinot-kafka-4.0` plugin JAR is available in your Pinot plugin directory. 3. All other `stream.kafka.*` configuration properties remain the same. {% hint style="info" %} The Kafka 4.0 connector is fully compatible with all existing Kafka consumer configuration properties including SSL/TLS, SASL authentication, isolation levels, and Schema Registry integration. See the [main Kafka ingestion guide](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) for detailed configuration examples. {% endhint %} ## Kafka 3.0 Connector The Kafka 3.0 connector (`pinot-kafka-3.0`) supports Apache Kafka 3.x brokers. This is the most widely deployed connector version. ### Scala Version The Kafka 3.0 connector is built with **Scala 2.13** by default. If you need Scala 2.12 compatibility, build Pinot with the `-Pscala-2.12` Maven profile: ``` mvn clean install -Pscala-2.12 -DskipTests ``` ### Configuration ```json { "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "your-topic", "stream.kafka.broker.list": "kafka:9092", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder" } } ``` ## Common Configuration Properties All Kafka connector versions share the same configuration properties. See [Ingest streaming data from Apache Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) for the complete configuration reference, including: * SSL/TLS setup * SASL authentication * Schema Registry integration (Avro, JSON Schema, Protobuf) * Consumer tuning properties * Isolation levels (`read_committed` / `read_uncommitted`) ## Passing Native Kafka Consumer Properties You can pass any native Kafka consumer configuration property using the `stream.kafka.consumer.prop.` prefix: ```json { "streamConfigs": { "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.prop.max.poll.records": "500", "stream.kafka.consumer.prop.fetch.min.bytes": "100000", "stream.kafka.consumer.prop.session.timeout.ms": "30000" } } ``` # Upsert and Dedup Use upsert or dedup when ingesting rows should collapse to one current record per key instead of preserving every event. Upsert and dedup are for tables that ingest repeated keys. Use them when the current value matters more than the raw event history, or when duplicate events should not fan out into duplicate query results. ## Choose the right behavior Use upsert when newer rows should replace older rows for the same primary key. Use dedup when repeated records should be filtered out and only the first or unique representation should remain. ## Operational notes These patterns need a careful schema, a stable primary key, and ingestion flow that understands the table-level metadata Pinot uses to keep the result consistent. The strongest detail still lives in the original docs under [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup). ## What this page covered This page covered the difference between upsert and dedup and when each is the better fit. ## Next step Read [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) to decide how Pinot should read source data and store generated segments. ## Related pages * [Ingestion](/build-with-pinot/ingestion) * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) * [Original Upsert and Dedup Docs](/build-with-pinot/ingestion/upsert-dedup) # Offline Table Upsert Use upsert semantics on batch-ingested offline tables. Pinot supports upsert on `OFFLINE` tables in builds that include [PR #17789](https://github.com/apache/pinot/pull/17789). Use it for batch corrections, replays, and late-arriving records. For a full overview of upsert features (comparison columns, delete columns, TTL, metadata management), see the main [Upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) page. This page covers the OFFLINE-specific configuration and differences. ## How offline upsert works Pinot keeps one row per primary key. For duplicate keys, Pinot keeps the row with the greatest comparison value. If you do not set `comparisonColumns`, Pinot uses the table time column. An offline upsert table must define either `upsertConfig.comparisonColumns` or `segmentsConfig.timeColumnName`. Pinot rejects the table configuration when neither is set; segment creation time is not used as an implicit comparison value. Offline upsert replaces full rows. It does not merge partial rows. ## Configure offline upsert {% stepper %} {% step %} ### Define a primary key Add `primaryKeyColumns` to the schema. ```json { "schemaName": "orders", "primaryKeyColumns": ["order_id"] } ``` {% endstep %} {% step %} ### Enable upsert on the offline table Set `tableType` to `OFFLINE`. Set `upsertConfig.mode` to `FULL`. ```json { "tableName": "orders_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "timeColumnName": "event_time", "retentionTimeUnit": "DAYS", "retentionTimeValue": "30", "replication": "3" }, "upsertConfig": { "mode": "FULL", "comparisonColumns": ["event_time"] } } ``` {% endstep %} {% step %} ### Ingest or replace segments Generate and upload offline segments as usual. Pinot applies upsert semantics when it loads those segments. Use append-style uploads for incremental corrections. Use refresh-style uploads when replacing an existing batch. {% endstep %} {% endstepper %} ## When to use it Use offline upsert when updates arrive in files. Use it for daily corrections. Use it for backfills. Use it for replaying snapshots into offline segments. ## Differences from real-time upsert Offline upsert does not consume a stream. It does not require low-level consumers. It does not depend on stream partitioning. It fits batch ingestion and segment replacement workflows. For stream-based updates, use [Stream ingestion with Upsert](/build-with-pinot/ingestion/upsert-dedup/upsert). ## Operational notes Changing the primary key needs a full rebuild. Changing comparison columns also needs a full rebuild. Reload alone is not enough for these changes. If you use a hybrid table, avoid overlapping offline and realtime time ranges. ## Related topics * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion/batch-ingestion) * [Backfill Data](/build-with-pinot/ingestion/batch-ingestion/backfill-data) * [Create and update a table configuration](/operate-pinot/deployment/setup-table) * [Stream ingestion with Upsert](/build-with-pinot/ingestion/upsert-dedup/upsert) # Stream Ingestion with Upsert Upsert support in Apache Pinot. Pinot provides native upsert support during ingestion. There are scenarios where records need modifications, such as correcting a ride fare or updating a delivery status. Partial upserts are convenient as you only need to specify the columns where values change, and you ignore the rest. ## Table type support Upsert is supported across REALTIME, OFFLINE, and HYBRID table types. The available modes depend on the table type: | Table type | FULL upsert | PARTIAL upsert | Notes | | ---------- | ----------- | -------------- | ---------------------------------------------------------- | | REALTIME | Yes | Yes | Stream-based ingestion with full upsert feature set | | OFFLINE | Yes | No | Batch ingestion; replaces full rows only | | HYBRID | Yes | No | Avoid overlapping time ranges between offline and realtime | For OFFLINE table upsert configuration details, see [Offline Table Upsert](/build-with-pinot/ingestion/upsert-dedup/offline-table-upsert). ## Overview of upserts in Pinot See an overview of how upserts work in Pinot. {% embed url="" %} Apache Pinot 1.0 Upserts overview {% endembed %} ## Enable upserts in Pinot To enable upserts on a Pinot table, do the following: 1. [Define the primary key in the schema](#define-the-primary-key-in-the-schema) 2. [Enable upserts in the table configurations](#enable-upsert-in-the-table-configurations) ### Define the primary key in the schema To update a record, you need a primary key to uniquely identify the record. To define a primary key, add the field `primaryKeyColumns` to the schema definition. For example, the schema definition of `UpsertMeetupRSVP` in the quick start example has this definition. {% code title="upsert\_meetupRsvp\_schema.json" %} ```javascript { "primaryKeyColumns": ["event_id"] } ``` {% endcode %} Note this field expects a list of columns, as the primary key can be a composite. When two records of the same primary key are ingested, *the record with the greater comparison value (timeColumn by default) is used*. When records have the same primary key and event time, then the order is not determined. In most cases, the later ingested record will be used, but this may not be true in cases where the table has a column to sort by. {% hint style="warning" %} **Partition the input stream by the primary key** \ An important requirement for the Pinot upsert table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the [`send`](https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html) API. If the original stream is not partitioned, then a streaming processing job (such as with Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion. Additionally if using `segmentPartitionConfig`to leverage Broker segment pruning then it's important to ensure that the partition function used matches both on the Kafka producer side as well as Pinot. In Kafka default for Java client is 32-bit **murmur2** hash and for all other languages such as Python its **CRC32** (Cyclic Redundancy Check 32-bit). {% endhint %} ### Enable upsert in the table configurations To enable upsert, make the following configurations in the table configurations. ### Upsert modes **Full upsert** The upsert mode defaults to `FULL` . FULL upsert means that a new record will replace the older record completely if they have same primary key. Example config: ```json { "upsertConfig": { "mode": "FULL" } } ``` **Partial upserts** Partial upsert lets you choose to update only specific columns and ignore the rest. To enable the partial upsert, set the `mode` to `PARTIAL` and specify `partialUpsertStrategies` for partial upsert columns. Since `release-0.10.0`, `OVERWRITE` is used as the default strategy for columns without a specified strategy. `defaultPartialUpsertStrategy` is also introduced to change the default strategy for all columns. {% hint style="info" %} Note that **null handling** must be enabled for partial upsert to work. {% endhint %} For example: {% code title="release-0.8.0" %} ```json { "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies":{ "rsvp_count": "INCREMENT", "group_name": "IGNORE", "venue_name": "OVERWRITE" } }, "tableIndexConfig": { "nullHandlingEnabled": true } } ``` {% endcode %} {% code title="release-0.10.0" %} ```javascript { "upsertConfig": { "mode": "PARTIAL", "defaultPartialUpsertStrategy": "OVERWRITE", "partialUpsertStrategies":{ "rsvp_count": "INCREMENT", "group_name": "IGNORE" } }, "tableIndexConfig": { "nullHandlingEnabled": true } } ``` {% endcode %} ### Custom row merger for partial upsert If column-level `partialUpsertStrategies` are not expressive enough, you can provide a custom row merger class with `upsertConfig.partialUpsertMergerClass`. ```json { "upsertConfig": { "mode": "PARTIAL", "partialUpsertMergerClass": "org.apache.pinot.segment.local.upsert.merger.PartialUpsertMyCustomMerger" }, "tableIndexConfig": { "nullHandlingEnabled": true } } ``` When `partialUpsertMergerClass` is set, Pinot instantiates that `PartialUpsertMerger` implementation instead of the built-in columnar partial-upsert merger. The custom merger class must be available on the server classpath and expose a constructor with `(List primaryKeyColumns, List comparisonColumns, UpsertConfig upsertConfig)`. `partialUpsertMergerClass` is mutually exclusive with `partialUpsertStrategies`. Pinot rejects table configs that try to set both at the same time. Pinot supports the following partial upsert strategies: | Strategy | Description | | ---------------- | ------------------------------------------------------------------------------------- | | OVERWRITE | Overwrite the column of the last record | | INCREMENT | Add the new value to the existing values | | APPEND | Add the new item to the Pinot unordered set | | UNION | Add the new item to the Pinot unordered set if not exists | | IGNORE | Ignore the new value, keep the existing value (v0.10.0+) | | MAX | Keep the maximum value betwen the existing value and new value (v0.12.0+) | | MIN | Keep the minimum value betwen the existing value and new value (v0.12.0+) | | FORCE\_OVERWRITE | Always replace the existing value with the incoming value, including `null` (v1.4.0+) | {% hint style="info" %} For partial upsert strategies other than `FORCE_OVERWRITE`, if the value is `null` in either the existing record or the incoming record, Pinot ignores the upsert strategy and keeps the non-null value: (`null`, *newValue*) -> *newValue* (*oldValue*, `null`) -> *oldValue* (`null`, `null`) -> `null` {% endhint %} Use `FORCE_OVERWRITE` when an incoming `null` should clear the previously stored value: (*oldValue*, `null`) -> `null` ### Post-Partial-Upsert Transforms (Derived Columns) When using partial upserts, you may have derived columns that need to be recomputed after the row is merged from the incoming record and the existing record. The `postPartialUpsertTransformConfigs` feature allows you to apply transformation functions to compute derived columns from the fully merged row. **Use Case** Consider an e-commerce table tracking orders: * `order_id`: Primary key * `score`: Points earned from the order * `bonus`: Bonus points awarded * `total`: Derived column that should equal `score + bonus` With partial upserts, incoming records may only contain updated values for `score` or `bonus`. The ingestion-time transforms only see the incoming record, so they cannot correctly compute `total` from a partially merged row. The `postPartialUpsertTransformConfigs` feature allows you to recompute `total` from the complete merged row after the partial upsert merge happens. **Configuration** To enable post-partial-upsert transforms, add the `postPartialUpsertTransformConfigs` configuration to your table's `upsertConfig`: {% code title="Table Config Example" %} ```json { "upsertConfig": { "mode": "PARTIAL", "defaultPartialUpsertStrategy": "OVERWRITE", "partialUpsertStrategies": { "score": "OVERWRITE", "bonus": "OVERWRITE" }, "postPartialUpsertTransformConfigs": [ { "columnName": "total", "transformFunction": "plus(score,bonus)" } ] }, "tableIndexConfig": { "nullHandlingEnabled": true } } ``` {% endcode %} `postPartialUpsertTransformConfigs` uses the same `TransformConfig` shape as ingestion-time transforms: each entry provides a destination `columnName` and a `transformFunction`. Pinot validates these configs before accepting the table: * They are only supported for `PARTIAL` upsert tables. * The destination column must exist in the schema. * The destination column cannot be a primary key, comparison column, `deleteRecordColumn`, or `outOfOrderRecordColumn`. * Each destination column can appear at most once. * The transform function cannot reference its own destination column. **Evaluation Semantics** * Post-partial-upsert transforms are evaluated **after** the partial upsert merge completes * They operate on the **complete merged row**, not just the incoming record * Both incoming and existing column values are available for the transform expression * The transforms use the same [function syntax as ingestion-time transforms](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations) * Transform results are stored in the derived columns as part of the final record **Interaction with Ingestion Transforms** Ingestion-time transforms and post-partial-upsert transforms serve different purposes: | Aspect | Ingestion Transforms | Post-Partial-Upsert Transforms | | ---------------- | --------------------------------------- | --------------------------------------------------------------------------- | | Execution timing | Before ingestion into Pinot | After partial upsert merge, during ingestion | | Input record | Incoming source record | Merged row (incoming + existing) | | Use case | Normalize/clean raw input data | Recompute derived columns from merged state | | Applies to | All table types (upsert and non-upsert) | Partial upsert tables only | | Example | Convert timestamp format | `total = plus(score, bonus)` where `score` and `bonus` come from merged row | Both can be used together: 1. Ingestion transforms normalize the incoming record 2. The normalized incoming record participates in partial upsert merge 3. Post-partial-upsert transforms recompute derived columns from the complete merged row **Example Workflow** Given a partial upsert table with this configuration: ```json { "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "score": "OVERWRITE", "bonus": "OVERWRITE" }, "postPartialUpsertTransformConfigs": [ { "columnName": "total", "transformFunction": "plus(score,bonus)" } ] }, "tableIndexConfig": { "nullHandlingEnabled": true } } ``` Processing these records: 1. **Initial record** (order\_id=123): * Incoming: `{order_id: 123, score: 100, bonus: 10}` * Merge: (first record, no existing row) * Post-transform: `total = plus(100, 10) = 110` * Final: `{order_id: 123, score: 100, bonus: 10, total: 110}` 2. **Update record** (order\_id=123): * Incoming: `{order_id: 123, score: 150}` (only score updated) * Merge: `{order_id: 123, score: 150, bonus: 10}` (bonus preserved from existing row) * Post-transform: `total = plus(150, 10) = 160` * Final: `{order_id: 123, score: 150, bonus: 10, total: 160}` 3. **Another update** (order\_id=123): * Incoming: `{order_id: 123, bonus: 25}` (only bonus updated) * Merge: `{order_id: 123, score: 150, bonus: 25}` (score preserved from existing row) * Post-transform: `total = plus(150, 25) = 175` * Final: `{order_id: 123, score: 150, bonus: 25, total: 175}` {% hint style="info" %} The derived columns computed by post-partial-upsert transforms can be queried like any other column. If you need to use these derived columns in further upsert strategies or transforms, ensure they are defined in your schema. {% endhint %} **None upserts** If set mode to `NONE`, the upsert is disabled. ### Comparison column By default, Pinot uses the value in the time column (`timeColumn` in tableConfig) to determine the latest record. That means, for two records with the same primary key, the record with the larger value of the time column is picked as the latest update. However, there are cases when users need to use another column to determine the order. In such case, you can use option `comparisonColumn` to override the column used for comparison. For example, ```json { "upsertConfig": { "mode": "FULL", "comparisonColumn": "anotherTimeColumn" } } ``` For partial upsert table, the out-of-order events won't be consumed and indexed. For example, for two records with the same primary key, if the record with the smaller value of the comparison column came later than the other record, it will be skipped. {% hint style="info" %} NOTE: Please use `comparisonColumns` for single comparison column instead of `comparisonColumn` as it is currently deprecated. You may see unrecognizedProperties when using the old config, but it's converted to comparisonColumns automatically when adding the table. {% endhint %} #### Multiple comparison columns In some cases, especially where partial upsert might be employed, there may be multiple producers of data each writing to a mutually exclusive set of columns, sharing only the primary key. In such a case, it may be helpful to use one comparison column per producer group so that each group can manage its own specific versioning semantics without the need to coordinate versioning across other producer groups. ```json { "upsertConfig": { "mode": "PARTIAL", "defaultPartialUpsertStrategy": "OVERWRITE", "partialUpsertStrategies":{}, "comparisonColumns": ["secondsSinceEpoch", "otherComparisonColumn"] } } ``` Documents written to Pinot are expected to have exactly 1 non-null value out of the set of comparisonColumns; if more than 1 of the columns contains a value, the document will be rejected. When new documents are written, whichever comparison column is non-null will be compared against only that same comparison column seen in prior documents with the same primary key. Consider the following examples, where the documents are assumed to arrive in the order specified in the array. ```json [ { "event_id": "aa", "orderReceived": 1, "description" : "first", "secondsSinceEpoch": 1567205394 }, { "event_id": "aa", "orderReceived": 2, "description" : "update", "secondsSinceEpoch": 1567205397 }, { "event_id": "aa", "orderReceived": 3, "description" : "update", "secondsSinceEpoch": 1567205396 }, { "event_id": "aa", "orderReceived": 4, "description" : "first arrival, other column", "otherComparisonColumn": 1567205395 }, { "event_id": "aa", "orderReceived": 5, "description" : "late arrival, other column", "otherComparisonColumn": 1567205392 }, { "event_id": "aa", "orderReceived": 6, "description" : "update, other column", "otherComparisonColumn": 1567205398 } ] ``` The following would occur: 1. `orderReceived: 1` * Result: persisted * Reason: first doc seen for primary key "aa" 2. `orderReceived: 2` * Result: persisted (replacing `orderReceived: 1`) * Reason: comparison column (`secondsSinceEpoch`) larger than that previously seen 3. `orderReceived: 3` * Result: rejected * Reason: comparison column (`secondsSinceEpoch`) smaller than that previously seen 4. `orderReceived: 4` * Result: persisted (replacing `orderReceived: 2`) * Reason: comparison column (`otherComparisonColumn`) larger than previously seen (never seen previously), despite the value being smaller than that seen for `secondsSinceEpoch` 5. `orderReceived: 5` * Result: rejected * Reason: comparison column (`otherComparisonColumn`) smaller than that previously seen 6. `orderReceived: 6` * Result: persist (replacing `orderReceived: 4`) * Reason: comparison column (`otherComparisonColumn`) larger than that previously seen ### Metadata time-to-live (TTL) In Pinot, the metadata map is stored in heap memory. To decrease in-memory data and improve performance, minimize the time primary key entries are stored in the metadata map (metadata time-to-live (TTL)). Limiting the TTL is especially useful for primary keys with high cardinality and frequent updates. Since the metadata TTL is applied on the first comparison column, the time unit of upsert TTL is the same as the first comparison column. #### Configure how long primary keys are stored in metadata To configure how long primary keys are stored in metadata, specify the length of time in `metadataTTL.` For example: ``` { "upsertConfig": { "mode": "FULL", "snapshot": "ENABLE", "preload": "ENABLE", "metadataTTL": 86400 } } ``` In this example, Pinot will retain primary keys in metadata for 1 day. Note that enabling upsert snapshot is required for metadata TTL for in-memory validDocsIDs recovery. ### Delete column Upsert Pinot table can support soft-deletes of primary keys. This requires the incoming record to contain a dedicated boolean single-field column that serves as a delete marker for a primary key. Once the real-time engine encounters a record with delete column set to `true` , the primary key will no longer be part of the queryable set of documents. This means the primary key will not be visible in the queries, unless explicitly requested via query option `skipUpsert=true`. ```json { "upsertConfig": { ... "deleteRecordColumn": } } ``` Note that the `delete` column has to be a single-value boolean column. ```json // In the Schema { ... { "name": "", "dataType": "BOOLEAN" }, ... } ``` {% hint style="warning" %} For an existing upsert table, treat `deleteRecordColumn` as immutable. Pinot now rejects adding, removing, or changing it through the controller update APIs unless you force the update, because servers cache the delete-column choice when they initialize upsert state. {% endhint %} A deleted primary key can be revived by ingesting a record with the same primary, but with higher comparison column value(s). Note that when reviving a primary key in a partial upsert table, the revived record will be treated as the source of truth for all columns. This means any previous updates to the columns will be ignored and overwritten with the new record's values. ### Deleted Keys time-to-live (TTL) The above config `deleteRecordColumn` only soft-deletes the primary key. To decrease in-memory data and improve performance, minimize the time deleted-primary-key entries are stored in the metadata map (deletedKeys time-to-live (TTL)). Limiting the TTL is especially useful for deleted-primary-keys where there are no future updates foreseen. #### Configure how long deleted-primary-keys are stored in metadata To configure how long primary keys are stored in metadata, specify the length of time in `deletedKeysTTL` For example: ``` "upsertConfig": { "mode": "FULL", "deleteRecordColumn": , "deletedKeysTTL": 86400 } } ``` In this example, Pinot will retain the deleted-primary-keys in metadata for 1 day. {% hint style="info" %} Note that the value of this field `deletedKeysTTL` should be the same as the unit of comparison column. If your comparison column is having values which corresponds to seconds, this config should also have values in seconds (see above example). `metadataTTL` and `deletedKeysTTL` do not work with multiple comparison columns and comparison/time column must be of `NUMERIC` type. {% endhint %} ### Data consistency with deletes and compaction together When using `deletedKeysTTL` together with `UpsertCompactionTask`, there can be a scenario where a segment containing deleted-record (where `deleteRecordColumn` = true was set for the primary key) gets compacted first and a previous old record is not yet compacted. During server restart, now the old record is added to the metadata manager map and is treated as non-deleted. To prevent data inconsistencies in this scenario, we have added a new config `enableDeletedKeysCompactionConsistency` which when set to true, will ensure that the deleted records are not compacted until all the previous records from all other segments are compacted for the deleted primary-key. ```json { "upsertConfig": { "mode": "FULL", "deleteRecordColumn": , "deletedKeysTTL": 86400, "enableDeletedKeysCompactionConsistency": true } } ``` ### Data consistency when queries and upserts happen concurrently Upserts in Pinot enable real-time updates and ensure that queries always retrieve the latest version of a record, making them a powerful feature for managing mutable data efficiently. However, in applications with extremely high QPS and high ingestion rates, queries and upserts happening concurrently can sometimes lead to inconsistencies in query results. For example, consider a table with 1 million primary keys. A distinct count query should always return 1 million, regardless of how new records are ingested and older records are invalidated. However, at high ingestion and query rates, the query may occasionally return a count slightly above or below 1 million. This happens because queries determine valid records by acquiring *validDocIds* bitmaps from multiple segments, which indicate which documents are currently valid. Since acquiring these bitmaps is not atomic with respect to ongoing upserts, a query may capture an inconsistent view of the data, leading to overcounting or undercounting of valid records. This is a classic concurrency issue where reads and writes happen simultaneously, leading to temporary inconsistencies. Typically, such issues are resolved using locks or snapshots to maintain a stable view of the data during query execution. To address this, two new consistency modes - **SYNC** and **SNAPSHOT** - have been introduced for upsert enabled tables to ensure consistent query results even when queries and upserts occur concurrently and at very high throughput. By default, the consistency mode is **NONE**, meaning the system operates as before. The **SYNC** mode ensures consistency by blocking upserts while queries execute, guaranteeing that queries always see a stable upserted data view. However, this can introduce write latency. Alternatively, the **SNAPSHOT** mode creates a consistent snapshot of *validDocIds* bitmaps for queries to use. This allows upserts to continue without blocking queries, making it more suitable for workloads with both high query and write rates.\ These new consistency modes provide flexibility, allowing applications to balance consistency guarantees against performance trade-offs based on their specific requirements. ``` { "upsertConfig": { "consistencyMode": "SYNC", // or "SNAPSHOT", "NONE" ... } } ``` For **SNAPSHOT** mode, one can configure how often the upsert view should be refreshed via a upsertConfig called `upsertViewRefreshIntervalMs`, which is 3000ms by default. Both the write and query threads can refresh the upsert view when it gets stale according to this config. Changing this config requires server restarts. Pinot also tracks newly added segments on the server for a bounded time via `newSegmentTrackingTimeMs` (default `10000`). During that window, Pinot can include those newly added segments as optional segments while broker routing catches up, which helps queries see a more complete upserted view immediately after segment addition. Setting `newSegmentTrackingTimeMs` to `0` disables this tracking. When `consistencyMode` is `SYNC` or `SNAPSHOT`, `newSegmentTrackingTimeMs` must stay positive. One can further adjust the view's freshness during query time without restarting servers via a query option called `upsertViewFreshnessMs` . By default, this query option matches with that upsertConfig `upsertViewRefreshIntervalMs` , but if a query sets it to a smaller value, the upsert view may get refreshed sooner for the query; and if set to 0, the query simply forces to refresh upsert view every time. For debugging purposes, there's a query option called `skipUpsertView`. If set to `true`, it bypasses the consistent upsert view maintained by SYNC or SNAPSHOT modes. This effectively executes the query as if it were in NONE mode. ### Use strictReplicaGroup for routing The upsert Pinot table can use only the low-level consumer for the input streams. As a result, it uses the [partitioned replica-group assignment](/operate-pinot/segment-management/segment-assignment#partitioned-replica-group-segment-assignment) implicitly for the segments. Moreover, upsert poses the additional requirement that **all segments of the same partition must be served from the same server** to ensure the data consistency across the segments. Accordingly, it requires to use `strictReplicaGroup` as the routing strategy. To use that, configure `instanceSelectorType` in `Routing` as the following: ```json { "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` {% hint style="warning" %} Using implicit partitioned replica-group assignment from low-level consumer won't persist the instance assignment (mapping from partition to servers) to the ZooKeeper, and new added servers will be automatically included without explicit reassigning instances (usually through rebalance). This can cause new segments of the same partition assigned to a different server and break the requirement of upsert. To prevent this, we recommend using explicit [partitioned replica-group instance assignment](/operate-pinot/segment-management/instance-assignment#partitioned-replica-group-instance-assignment) to ensure the instance assignment is persisted. Note that `numInstancesPerPartition` should always be `1` in `replicaGroupPartitionConfig`. {% endhint %} {% hint style="warning" %} Do not enable [Adaptive Server Selection](/operate-pinot/tuning/query-routing-using-adaptive-server-selection) on upsert tables. The adaptive selector does not respect `strictReplicaGroup` boundaries and may route a query to servers across multiple replica groups, breaking the upsert consistency guarantee that all segments of the same partition are served from the same server. See [#12507](https://github.com/apache/pinot/issues/12507) for details. {% endhint %} ### Enable validDocIds snapshots for upsert metadata recovery Upsert snapshot support is also added in `release-0.12.0`. To enable the snapshot, set `snapshot` to `ENABLE`. For example: ```json { "upsertConfig": { "mode": "FULL", "snapshot": "ENABLE" } } ``` Upsert maintains metadata in memory containing which docIds are valid in a particular segment (ValidDocIndexes). This metadata gets lost during server restarts and needs to be recreated again.\ \ ValidDocIndexes can not be recovered easily after out-of-TTL primary keys get removed. Enabling snapshots addresses this problem by adding functions to store and recover validDocIds snapshot for Immutable Segments The snapshots are taken on every segment commit to ensure that they are consistent with the persisted data in case of abrupt shutdown.\ \ We recommend that you enable this feature so as to speed up server boot times during restarts. {% hint style="info" %} For upsert tables that use `metadataTTL` or `deletedKeysTTL`, [segment reload](/operate-pinot/segment-management/segment-reload) rebuilds upsert metadata from the persisted `validDocIds` snapshot instead of rescanning every row in the immutable segment. This prevents reload from resurrecting keys that TTL expiry or delete handling had already removed from the upsert metadata. If reload would have to download a different copy of the segment, Pinot fails the reload when the segment CRC changes because the local docId-based snapshot might no longer match the downloaded rows. This applies to `forceDownload=true` reloads and to normal reloads that trigger a CRC-based re-download. {% endhint %} {% hint style="info" %} The lifecycle for validDocIds snapshots are shows as follows, 1. If snapshot is enabled, snapshots for existing segments are taken or refreshed when the next consuming segment gets started. 2. The snapshot files are kept on disk until the segments get removed, e.g. due to data retention or manual deletion. 3. If snapshot is disabled, the existing snapshot for a segment is cleaned up when the segment gets loaded by the server, e.g. when the server restarts. {% endhint %} ### Enable preload for faster server restarts Upsert preload feature can make it faster to restore the upsert states when server restarts. To enable the preload feature, set `preload` to `ENABLE`. Snapshot must also be enabled. For example: ```json { "upsertConfig": { "mode": "FULL", "snapshot": "ENABLE", "preload": "ENABLE" } } ``` \ Under the hood, it uses the validDocIds snapshots to identify the valid docs and restore their upsert metadata quickly instead of performing a whole upsert comparison flow. The flow is triggered before the server is marked as ready, after which the server starts to load the remaining segments without snapshots (hence the name preload). The feature also requires you to specify `pinot.server.instance.max.segment.preload.threads: N` in the server config where N should be replaced with the number of threads that should be used for preload. It's 0 by default to disable the preloading feature. {% hint style="warning" %} A bug was introduced in v1.2.0 that when snapshot and preload recovery are enabled but `max.segment.preload.threads` is left as `0`, the preloading mechanism is still enabled but segments fail to load because there are no threads for preloading. This was fixed in newer versions, but for v1.2.0, remember to set `max.segment.preload.threads` to a positive value as well. Server restart is needed for the config change to take effect. {% endhint %} #### Enable commit time compaction for storage optimization {% hint style="warning" %} If you are enabling commit time compaction for an existing table, it is recommended to first pause the ingestion for that table, enable this feature by updating the table-config, and then resume ingestion. {% endhint %} Many Upsert use-cases have a lot of Update events within the segment commit window. For instance, if we had an Upsert table for order status of Uber Eats orders, we would expect a lot of update events for the same order within a 1 hour window. For such use-cases, the committed segments end up with a lot of dead tuples, and you have to wait for the Segment Compaction tasks to prune them, which can take hours. Commit time compaction is a performance optimization feature for upsert tables that removes invalid and obsolete records during the segment commit process itself. This not only reduces the storage bloat of the table immediately, but it can also bring down the segment commit time. To enable commit time compaction, set the `enableCommitTimeCompaction` to `true` in the upsert configuration. For example: ```json { "upsertConfig": { "mode": "FULL", "enableCommitTimeCompaction": true } } ``` **How it works** During segment commit, commit time compaction: * Filters out invalid document IDs. Retains valid records and soft-deleted records. * Generates accurate column statistics for compacted segments * Maintains correct document order while removing obsolete data * Reduces segment size immediately without requiring minion tasks **Configuration requirements** * The feature is enabled per table by setting `enableCommitTimeCompaction=true` in the upsert configuration * Changes take effect after one segment commit cycle (the current consuming segment will be committed without compaction) * Compatible with all types of upsert tables ### Handle out-of-order events There are 2 configs added related to handling out-of-order events. #### dropOutOfOrderRecord To enable dropping of out-of-order record, set the `dropOutOfOrderRecord` to `true`. For example: ```json { "upsertConfig": { ..., "dropOutOfOrderRecord": true } } ``` This feature doesn't persist any out-of-order event to the consuming segment. If not specified, the default value is `false`. * When `false`, the out-of-order record gets persisted to the consuming segment, but the MetadataManager mapping is not updated thus this record is not referenced in query or in any future updates. You can still see the records when using `skipUpsert` query option. * When `true`, the out-of-order record doesn't get persisted at all and the MetadataManager mapping is not updated so this record is not referenced in query or in any future updates. You **cannot** see the records when using `skipUpsert` query option. #### outOfOrderRecordColumn This is to identify out-of-order events programmatically. To enable this config, add a boolean field in your table schema, say `isOutOfOrder` and enable via this config. For example: ```json { "upsertConfig": { ..., "outOfOrderRecordColumn": "isOutOfOrder" } } ``` This feature persists a `true` / `false` value to the `isOutOfOrder` field based on the orderness of the event. You can filter out out-of-order events while using `skipUpsert` to avoid any confusion. For example: ```json select key, val from tbl1 where isOutOfOrder = false option(skipUpsert=false) ``` {% hint style="info" %} Note that `dropOutOfOrderRecord` and `outOfOrderRecordColumn` are only supported when no consistencyMode is set (i.e., `consistencyMode = NONE`). This is because, when a consistencyMode is enabled, rows are added before the valid documents are updated. As a result, out-of-order records cannot be dropped or marked in upsert tables, defeating the purpose of these options. {% endhint %} ### Use custom metadata manager Pinot supports custom PartitionUpsertMetadataManager that handle records and segments updates. ```json { "upsertConfig": { "metadataManagerClass": org.apache.pinot.segment.local.upsert.CustomPartitionUpsertMetadataManager } } ``` #### Adding custom upsert managers You can add custom PartitionUpsertMetadataManager as follows: * Create a new java project. Make sure you keep the package name as `org.apache.pinot.segment.local.upsert.xxx` * In your java project include the dependency {% tabs %} {% tab title="Maven" %} ``` org.apache.pinot pinot-segment-local 1.0.0 ``` {% endtab %} {% tab title="Gradle" %} ``` include 'org.apache.pinot:pinot-common:1.0.0' ``` {% endtab %} {% endtabs %} * Add your custom partition manager that implements PartitionUpsertMetadataManager interface ``` //Example custom partition manager class CustomPartitionUpsertMetadataManager implements PartitionUpsertMetadataManager {} ``` * Add your custom TableUpsertMetadataManager that implements BaseTableUpsertMetadataManager interface ``` //Example custom table upsert metadata manager public class CustomTableUpsertMetadataManager extends BaseTableUpsertMetadataManager {} ``` * Place the compiled JAR in the `/plugins` directory in pinot. You will need to restart all Pinot instances if they are already running. * Now, you can use the custom upsert manager in table configs as follows: ``` { "upsertConfig": { "metadataManagerClass": org.apache.pinot.segment.local.upsert.CustomPartitionUpsertMetadataManager } } ``` :warning: The upsert manager class name is case-insensitive as well. ### Immutable upsert configuration fields {% hint style="danger" %} **Certain upsert and schema configuration fields cannot be modified after table creation.** Changing these fields on an existing upsert table can lead to data inconsistencies or data loss, particularly when servers restart and commit segments. Pinot validates and invalidates documents based on these configurations, so altering them after data has been ingested will cause the existing validDocId snapshots to become inconsistent with the new configuration. The following fields are immutable after table creation: **Schema fields:** * `primaryKeyColumns` **upsertConfig fields:** * `mode` (FULL, PARTIAL, NONE) * `hashFunction` * `comparisonColumns` * `timeColumnName` (when used as the default comparison column) * `deleteRecordColumn` * `dropOutOfOrderRecord` * `outOfOrderRecordColumn` Attempting to update these fields will return an error: ``` Failed to update table '': Cannot modify [] as it may lead to data inconsistencies. Please create a new table instead. ``` **Recommended workaround:** Create a new table with the desired configuration and reingest all data. **Alternative (use with caution):** If you must modify these fields without recreating the table, you can use the `force=true` query parameter on the table config update API. Before doing so, disable SNAPSHOT mode in upsertConfig, pause consumption, and restart all servers. Note that this approach only guarantees consistency for newly ingested keys; existing data may remain inconsistent. {% endhint %} {% hint style="warning" %} For `PARTIAL` upsert tables, Pinot now allows `partialUpsertStrategies` and `defaultPartialUpsertStrategy` to be updated on an existing table through the controller update APIs. These updates are not retroactive: * Existing merged values stay as they are already stored. * The new strategy only applies after each consuming server restarts and rebuilds its partial-upsert handler. * During a rolling restart, replicas can temporarily consume with different strategy versions and diverge on newly merged rows. If you observe row-value drift after the rollout, reset the affected consuming segments so Pinot can rebuild them from the common segment data. See [Segment Lifecycle and Repair](/operate-pinot/segment-management/segment-lifecycle-and-repair). {% endhint %} ### Upsert table limitations There are some limitations for the upsert Pinot tables. * Partial upsert is supported for REALTIME tables only. OFFLINE tables support FULL upsert only. See [Offline Table Upsert](/build-with-pinot/ingestion/upsert-dedup/offline-table-upsert) for details. * The star-tree index cannot be used for indexing, as the star-tree index performs pre-aggregation during the ingestion. * Unlike append-only tables, out-of-order events (with comparison value in incoming record less than the latest available value) won't be consumed and indexed by Pinot partial upsert table, these late events will be skipped. * We cannot change the number of partitions in the source topic after the upsert/dedup table is created (start with a relatively high number of partitions as mentioned in best practices). #### Handling Inconsistencies When a consuming segment commits, the server replaces the mutable segment with a new immutable segment. During this transition, there is a chance that the in-memory upsert metadata (primary key → latest record location) can diverge across replicas. This divergence is generally safe for FULL Upsert tables because replicas eventually converge, but it is unsafe for: * Partial upsert tables: Merge correctness depends on the accurate “latest” record location; wrong pointers can introduce incorrect values for new entries. * Full upsert tables with dropOutOfOrderRecord=true or outOfOrderRecordColumn: Out-of-order detection relies on the current location; wrong metadata can cause incorrect acceptance or rejection. To mitigate that, we added a **Helix cluster config** (not a per-process `pinot-server.conf` key): `pinot.server.consuming.segment.consistency.mode`, with three modes: #### RESTRICTED (default) Blocks force-commit and reload for Partial Upsert and DropOutOfOrder tables. Consuming segment can only commit naturally. Guarantees consistency #### PROTECTED Allows force-commit/reload with post-replacement reconciliation using a temporary map which track the previous immutable segment location of the key. Reconciliation: * Keys still pointing to replaced segment → revert to prior immutable location. * Keys without prior location → removed. * Un reconcilable keys → logged, and metrics emitted for the user to take action. Make sure ParallelSegmentConsumptionPolicy is always ∈ {`DISALLOW_ALWAYS`, `ALLOW_DURING_BUILD_ONLY`}. #### UNSAFE Allows force-commit/reload with no reconciliation. Chances of inconsistencies during commit. This mode is unsafe and not recommended in production settings. #### Monitoring * `pinot.server.tableName.realtimeUpsertInconsistentRows`: Number of primary keys left unreplaced after a segment replacement detects inconsistent metadata across replicas for a full-upsert table. This includes tables that use `dropOutOfOrderRecord=true` or `outOfOrderRecordColumn`. * `pinot.server.tableName.partialUpsertKeysNotReplaced`: Number of primary keys left unreplaced after a segment replacement detects inconsistent metadata across replicas for a partial-upsert table. ### Best practices Unlike other real-time tables, Upsert table takes up more memory resources as it needs to bookkeep the record locations in memory. As a result, it's important to plan the capacity beforehand, and monitor the resource usage. Here are some recommended practices of using Upsert table. #### Create the topic/stream with more partitions. The number of partitions in input streams determines the partition numbers of the Pinot table. The more partitions you have in input topic/stream, more Pinot servers you can distribute the Pinot table to and therefore more you can scale the table horizontally. **Do note that** you can't increase the partitions in future for upsert enabled tables so you need to start with good enough partitions (atleast 2-3X the number of pinot servers) #### Memory usage Upsert table maintains an in-memory map from the primary key to the record location. **So it's recommended to use a simple primary key type and avoid composite primary keys to save the memory cost. Beware when using `JSON` column as primary key, same key-values in different order would be considered as different primary keys**. In addition, consider the `hashFunction` config in the Upsert config, which can be `UUID`, `MD5` or `MURMUR3`. If your primary key column is a valid UUID and you are running out of memory due to a high number of primary keys, the `UUID` hash function can lower memory requirements by up to 35% without bringing in any hash collision risks.\ If the primary key is not a valid UUID, this hash function stores the primary key as is and skips the UUID based compression. `MD5` and `MURMUR3` can also help lower memory requirements. They work for all types of primary key values, but bring in a small risk of hash collision. The generated hash from `MD5` and `MURMUR3` is a 128-bit hash, so this is beneficial when your primary key values are larger than 128-bits. #### Monitoring Set up a dashboard over the metric `pinot.server.upsertPrimaryKeysCount.tableName` to watch the number of primary keys in a table partition. It's useful for tracking its growth which is proportional to the memory usage growth. \*\*\*\* The total memory usage by upsert is roughly `(primaryKeysCount * (sizeOfKeyInBytes + 24))` #### Capacity planning It's useful to plan the capacity beforehand to ensure you will not run into resource constraints later. A simple way is to measure the rate of the primary keys in the input stream per partition and extrapolate the data to a specific time period (based on table retention) to approximate the memory usage. A heap dump is also useful to check the memory usage so far on an upsert table instance. ### Example Putting these together, you can find the table configurations of the quick start examples as the following: ```json { "tableName": "upsertMeetupRsvp", "tableType": "REALTIME", "tenants": {}, "segmentsConfig": { "timeColumnName": "mtime", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1", "replication": "1" }, "tableIndexConfig": { "segmentPartitionConfig": { "columnPartitionMap": { "event_id": { "functionName": "Hashcode", "numPartitions": 2 } } } }, "instanceAssignmentConfigMap": { "CONSUMING": { "tagPoolConfig": { "tag": "DefaultTenant_REALTIME" }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numReplicaGroups": 1, "partitionColumn": "event_id", "numPartitions": 2, "numInstancesPerPartition": 1 } } }, "routing": { "segmentPrunerTypes": [ "partition" ], "instanceSelectorType": "strictReplicaGroup" }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "upsertMeetupRSVPEvents", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:19092" } ] } }, "upsertConfig": { "mode": "FULL", "snapshot": "ENABLE", "preload": "ENABLE" }, "fieldConfigList": [ { "name": "location", "encodingType": "RAW", "indexType": "H3", "properties": { "resolutions": "5" } } ], "metadata": { "customConfigs": {} } } ``` ```json { "tableName": "upsertPartialMeetupRsvp", "tableType": "REALTIME", "tenants": {}, "segmentsConfig": { "timeColumnName": "mtime", "retentionTimeUnit": "DAYS", "retentionTimeValue": "1", "replication": "1" }, "tableIndexConfig": { "segmentPartitionConfig": { "columnPartitionMap": { "event_id": { "functionName": "Hashcode", "numPartitions": 2 } } }, "nullHandlingEnabled": true }, "instanceAssignmentConfigMap": { "CONSUMING": { "tagPoolConfig": { "tag": "DefaultTenant_REALTIME" }, "replicaGroupPartitionConfig": { "replicaGroupBased": true, "numReplicaGroups": 1, "partitionColumn": "event_id", "numPartitions": 2, "numInstancesPerPartition": 1 } } }, "routing": { "segmentPrunerTypes": [ "partition" ], "instanceSelectorType": "strictReplicaGroup" }, "ingestionConfig": { "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "upsertPartialMeetupRSVPEvents", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory", "stream.kafka.broker.list": "localhost:19092" } ] } }, "upsertConfig": { "mode": "PARTIAL", "partialUpsertStrategies": { "rsvp_count": "INCREMENT", "group_name": "UNION", "venue_name": "APPEND" } }, "fieldConfigList": [ { "name": "location", "encodingType": "RAW", "indexType": "H3", "properties": { "resolutions": "5" } } ], "metadata": { "customConfigs": {} } } ``` {% hint style="info" %} Pinot server maintains a primary key to record location map across all the segments served in an upsert-enabled table. As a result, when updating the config for an existing upsert table (e.g. change the columns in the primary key, change the comparison column), servers need to be restarted in order to apply the changes and rebuild the map. {% endhint %} ### Realtime ingestion OOM protection Pinot can apply server-side realtime ingestion backpressure when JVM heap usage is high. The default server mode `pinot.server.instance.ingestion.oom.protection.mode=DISABLE` leaves the feature off; set it to `UPSERT_DEDUP_ONLY` to protect realtime upsert and dedup tables by default, or use `ingestionConfig.streamIngestionConfig.oomProtection` to opt an individual table in or out. For the full configuration keys, defaults, runtime behavior, and metric, see [OOM Protection Using Automatic Query Killing](/operate-pinot/tuning/oom-protection-using-automatic-query-killing#realtime-ingestion-oom-protection-on-servers). ### Parallel consumption during commit, download, and replacement For partial upsert tables, Pinot can pause the next consuming segment while the previous segment is still being finalized so that replicas do not advance with different merged-row state during commit handling. * By default, partial upsert tables do **not** keep consuming in parallel during commit handling. * When pauseless consumption is enabled, Pinot can still continue during the build phase while stopping during download and replacement, depending on `parallelSegmentConsumptionPolicy`. For backward compatibility, partial upsert tables still accept the deprecated table-level flag `upsertConfig.allowPartialUpsertConsumptionDuringCommit`. Setting it to `true` restores the older behavior and allows the replica to keep consuming throughout commit handling, including segment download and replacement: ```json { "upsertConfig": { "mode": "PARTIAL", "allowPartialUpsertConsumptionDuringCommit": true } } ``` If the table-level flag is left at its default `false`, the server-level fallback `pinot.server.instance.upsert.default.allow.partial.upsert.consumption.during.commit` can enable the same legacy behavior for partial upsert tables on that server. New deployments should prefer `parallelSegmentConsumptionPolicy` in `streamIngestionConfig` when they need explicit control over parallel consumption. ## Advanced Server Configuration ### Consuming Segment Consistency Mode For partial upsert tables or tables with `dropOutOfOrderRecord=true` or `outOfOrderRecordColumn` configured, configure how controllers and servers handle segment reloads and force commits via the **Helix cluster config** key `pinot.server.consuming.segment.consistency.mode` (set with the controller cluster-config API / UI, for example `POST /cluster/configs`). Putting this key only in `pinot-server.conf` does **not** take effect — both controller and server load it from cluster config through `ConsumingSegmentConsistencyModeListener`. | Mode | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RESTRICTED` | *(Default)* Skips consuming-segment reload and rejects explicit force commit for partial-upsert tables and upsert tables with `dropOutOfOrderRecord` or `outOfOrderRecordColumn` configured. | | `PROTECTED` | Enables reloads/force commits with upsert metadata reversion during segment replacements. Requires `ParallelSegmentConsumptionPolicy` set to `DISALLOW_ALWAYS` or `ALLOW_DURING_BUILD_ONLY`. | | `UNSAFE` | Allows reloads without metadata reversion. Use only if inconsistency is acceptable or handled externally. | > **Note:** This cluster config is distinct from the table-level `upsertConfig.consistencyMode` setting (SYNC / SNAPSHOT / NONE), which controls query-vs-upsert concurrency on a single server. ## Migrating from deprecated config fields As of Pinot 1.4.0, the following upsert config fields have been renamed: | Deprecated field | New field | Values | | ---------------- | ---------- | --------------------------------- | | `enableSnapshot` | `snapshot` | `ENABLE`, `DISABLE`, or `DEFAULT` | | `enablePreload` | `preload` | `ENABLE`, `DISABLE`, or `DEFAULT` | The new fields use the `Enablement` enum (`ENABLE`, `DISABLE`, `DEFAULT`) instead of boolean values. `DEFAULT` defers to the server-level configuration, which allows table-level overrides when the feature is enabled at the instance level. The deprecated boolean fields still work but will be removed in a future release. Update your table configs to use the new field names. ## Quick Start To illustrate how the full upsert works, the Pinot binary comes with a quick start example. Use the following command to creates a real-time upsert table `meetupRSVP`. ```bash # stop previous quick start cluster, if any bin/quick-start-upsert-streaming.sh ``` You can also run partial upsert demo with the following command ```bash # stop previous quick start cluster, if any bin/quick-start-partial-upsert-streaming.sh ``` As soon as data flows into the stream, the Pinot table will consume it and it will be ready for querying. Head over to the Query Console to check out the real-time data. ![Query the upsert table](/files/-McFcHBaCtmhW380MW0q) For partial upsert you can see only the value from configured column changed based on specified partial upsert strategy. ![Query the partial upsert table](/files/-MeWXEvLeel7PIhymKT8) An example for partial upsert is shown below, each of the event\_id kept being unique during ingestion, meanwhile the value of rsvp\_count incremented. ![Explain partial upsert table](/files/-MeWXEvMVdE6vNB1pwyj) To see the difference from the non-upsert table, you can use a query option `skipUpsert` to skip the upsert effect in the query result. ### FAQ **Can I add a schema column to an existing upsert table without recreating it?** Yes for **additive** columns. Update the schema, then reload and/or [forceCommit](/reference/api-reference/controller-api#post-tablestablenameforcecommit) so new consuming segments pick up the column. Partial-upsert tables and upsert tables with out-of-order handling configured restrict force commit unless the Helix **cluster config** `pinot.server.consuming.segment.consistency.mode` allows it (see [Consuming Segment Consistency Mode](#consuming-segment-consistency-mode) — not `pinot-server.conf`). Changing upsert **configuration** is different: allowed partial-upsert strategy changes require a controlled restart and are not retroactive, while immutable settings such as primary/comparison columns and mode require a new table and reingestion. See the [schema evolution decision table](/build-with-pinot/data-modeling/schema-evolution#decision-table-add-a-column-on-an-existing-table). **Can I change configs like primary key columns and comparison columns in existing upsert table?** Not recommended. Existing segments contain validDocId snapshots computed using the old configuration. Changing the configuration can lead to data inconsistencies as existing snapshots wouldn't be cleaned up, especially if a server restarts with validDocId snapshots while replica server do not. **Avoid changing:** primary key columns, comparison columns, upsert mode, hashFunction, and `deleteRecordColumn`. Pinot now enforces this guard on the controller update APIs. By default, `PUT /tables/{tableName}` and `PUT /tableConfigs/{tableName}` reject backward-incompatible upsert or dedup config changes with `400 Bad Request`. For upsert tables, this includes comparison columns, hash function, mode, `deleteRecordColumn`, out-of-order settings, and the table time column when Pinot is using it as the default comparison column. For dedup tables, this includes the dedup hash function, dedup time column, and the table time column when Pinot is using it as the default dedup time column. `partialUpsertStrategies` and `defaultPartialUpsertStrategy` are the exception for `PARTIAL` upsert tables. Pinot accepts those updates without `force=true`, but the new strategy only affects merges that happen after each server restarts and reloads the table config. Existing merged values are not rewritten automatically, and replicas can temporarily diverge during a rolling restart. If that happens, reset the affected consuming segments after the rollout. You can still bypass the guard with `force=true` on `PUT /tables/{tableName}` or `forceTableSchemaUpdate=true` on `PUT /tableConfigs/{tableName}`, but Pinot recommends using that only for controlled recovery or migration workflows. If changes are unavoidable: **Best option:** Create a new table and reingest all data. **Alternative:** Disable SNAPSHOT, pause consumption and restart all the servers. This will work for new incoming keys only; consistency across existing data is not guaranteed. # Segment Compaction on Upserts Use segment compaction on upsert-enabled real-time tables. ## Overview of segment compaction Compacting a segment replaces the completed segment with a compacted segment that only contains the latest version of records. For more information about how to use upserts on a real-time table in Pinot, see [Stream Ingestion with Upsert](/build-with-pinot/ingestion/upsert-dedup/upsert). The Pinot upsert feature stores all versions of the record ingested into immutable segments on disk. Even though the previous versions are not queried, they continue to add to the storage overhead. To remove older records (no longer used in query results) and reclaim storage space, we need to compact Pinot segments periodically. Segment compaction is done via a new minion task. To schedule Pinot tasks periodically, see the [Minion documentation](/architecture-and-concepts/components/cluster/minion). ## Compact segments on upserts in a real-time table To compact segments on upserts, complete the following steps: 1. Ensure task scheduling is enabled and a minion is available. 2. Add the following to your table configuration. These configurations (except `schedule)`determine which segments to compact. ``` "task": { "taskTypeConfigsMap": { "UpsertCompactionTask": { "schedule": "0 */5 * ? * *", "bufferTimePeriod": "7d", "invalidRecordsThresholdPercent": "30", "invalidRecordsThresholdCount": "100000", "tableMaxNumTasks": "100", "validDocIdsType": "SNAPSHOT" } } } ``` * `bufferTimePeriod:` To compact segments once they are complete, set to `“0d”`. To delay compaction (as the configuration above shows by 7 days (`"7d"`)), specify the number of days to delay compaction after a segment completes. * `invalidRecordsThresholdPercent` (Optional) Limits the older records allowed in the completed segment as a percentage of the total number of records in the segment. In the example above, the completed segment may be selected for compaction when 30% of the records in the segment are old. * `invalidRecordsThresholdCount` (Optional) Limits the older records allowed in the completed segment by record count. In the example above, if the segment contains more than 100K records, it may be selected for compaction. * `tableMaxNumTasks` (Optional) Limits the number of tasks allowed to be scheduled. * `validDocIdsType` (Optional) Specifies the source of validDocIds to fetch when running data compaction. Valid values are `SNAPSHOT`, `SNAPSHOT_WITH_DELETE`, `IN_MEMORY`, and `IN_MEMORY_WITH_DELETE`. `SNAPSHOT` remains the default even when `upsertConfig.deleteRecordColumn` is configured, and Pinot honors the configured value as-is. * `SNAPSHOT`: Default validDocIds type. Loads the `validDocIds` snapshot from the Pinot segment. `upsertConfig.snapshot` must not be `DISABLE` for this type. * `SNAPSHOT_WITH_DELETE`: Loads the delete-aware `queryableDocIds` snapshot from the Pinot segment. `upsertConfig.snapshot` must not be `DISABLE`, and `upsertConfig.deleteRecordColumn` must be set. * `IN_MEMORY`: Loads the `validDocIds` bitmap from the real-time server's memory. * `IN_MEMORY_WITH_DELETE`: Loads the delete-aware `queryableDocIds` bitmap from the real-time server's memory. `upsertConfig.deleteRecordColumn` must be set for this type. {% hint style="warning" %} When using the two in-memory types, if the server gets restarted, the upsert view gets back consistent once server re-ingests the data it has ingested before starting. The in-memory bitmaps are updated when server ingests data into consuming segment, even before the consuming segment gets committed. So if server gets restarted whlie still consuming data, the upsert view gets back consistent once it catches up the previously ingested data. Instead, the bitmap snapshots are only taken after committing the segment, thus can be more consistent on server restarts, but is eventually consistent as well if server gets restarted while ingesting data. {% endhint %} {% hint style="info" %} Because segment compaction is an expensive operation, we **do not recommend** setting `invalidRecordsThresholdPercent and invalidRecordsThresholdCount` too low (close to 1). By default, all configurations above are `0`, so no thresholds are applied. {% endhint %} ## Example The following example includes a dataset with 24M records and 240K unique keys that have each been duplicated 100 times. After ingesting the data, there are 6 segments (5 completed segments and 1 consuming segment) with a total estimated size of 22.8MB. ![](/files/I1QlUMU81gzERZUpCBDp) *Example dataset* Submitting the query `“set skipUpsert=true; select count(*) from transcript_upsert”` before compaction produces 24,000,000 results: ![](/files/eYwPUmxp0L3bkmAjkKiG) *Results before segment compaction* After the compaction tasks are complete, the [Minion Task Manager UI](/architecture-and-concepts/components/cluster/minion#task-manager-ui) reports the following. ![](/files/ua1o9CnIKG1gn6iZNits) *Minion compaction task completed* Segment compactions generates a task for each segment to compact. Five tasks were generated in this case because 90% of the records (3.6–4.5M records) are considered ready for compaction in the completed segments, exceeding the configured thresholds. {% hint style="info" %} If a completed segment only contains old records, Pinot immediately deletes the segment (rather than creating a task to compact it). {% endhint %} Submitting the query again shows the count matches the set of 240K unique keys. ![](/files/Vzjg9UkVMwve0aqBtn1d) *Results after segment compaction* Once segment compaction has completed, the total number of segments remain the same and the total estimated size drops to 2.77MB. {% hint style="info" %} To further improve query latency, merge small segments into larger one. {% endhint %} # Stream Ingestion with Dedup Deduplication support in Apache Pinot. Pinot provides native support for deduplication (dedup) during the real-time ingestion (v0.11.0+). ## Prerequisites for enabling dedup To enable dedup on a Pinot table, make the following table configuration and schema changes: ### Define the primary key in the schema To be able to dedup records, a primary key is needed to uniquely identify a given record. To define a primary key, add the field `primaryKeyColumns` to the schema definition. {% code title="schemaWithPK.json" %} ```javascript { "primaryKeyColumns": ["id"] } ``` {% endcode %} Note this field expects a list of columns, as the primary key can be composite. While ingesting a record, if its primary key is found to be already present, the record will be dropped. ### Partition the input stream by the primary key An important requirement for the Pinot dedup table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the [`send`](https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html) API. If the original stream is not partitioned, then a streaming processing job (e.g. Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion. ### Use strictReplicaGroup for routing The dedup Pinot table can use only the low-level consumer for the input streams. As a result, it uses the [partitioned replica-group assignment](/operate-pinot/segment-management/segment-assignment#partitioned-replica-group-segment-assignment) for the segments. Moreover, dedup poses the additional requirement that all segments of the same partition must be served from the same server to ensure the data consistency across the segments. Accordingly, it requires `strictReplicaGroup` as the routing strategy. To use that, configure `instanceSelectorType` in `Routing` as the following: {% code title="routing" %} ```json { "routing": { "instanceSelectorType": "strictReplicaGroup" } } ``` {% endcode %} {% hint style="warning" %} instance assignment is persisted. Note that `numInstancesPerPartition` should always be `1` in `replicaGroupPartitionConfig`. {% endhint %} ### Other limitations * The incoming stream must be partitioned by the primary key such that, all records with a given primaryKey must be consumed by the same Pinot server instance. ## Enable dedup in the table configurations To enable dedup for a REALTIME table, add the following to the table config. {% code title="tableConfigWithDedup.json" %} ```json { ... "dedupConfig": { "dedupEnabled": true, "hashFunction": "NONE" }, ... } ``` {% endcode %} Supported values for `hashFunction` are `NONE`, `MD5` and `MURMUR3`, with the default being `NONE`. ## Metadata TTL Server stores the existing primary keys in dedup metadata map kept on JVM heap. As the dedup metadata grows, the heap memory pressure increases, which may affect the performance of ingestion and queries. One can set a positive metadata TTL to enable the TTL mechanism to keep the metadata size bounded. By default, the table's time colum is used as the dedup time column. The time unit of TTL is the same as the dedup time column. The TTL should be set long enough so that new records can be deduplicated before the primary keys gets removed. Time column must be `NUMERIC` data type when `metadataTTl` is enabled. ```json { ... "dedupConfig": { "dedupEnabled": true, "hashFunction": "NONE", "dedupTimeColumn": "mtime", "metadataTTL": 30000 }, ... } ``` ## Enable preload for faster server restarts When ingesting new records, the server has to read the metadata map to check for duplicates. But when server restarts, the documents in existing segments are all unique as ensured by the dedup logic during real-time ingestion. So we can do write-only to bootstrap the metadata map faster. Set `preload` to `ENABLE` to turn this on. ```json { ... "dedupConfig": { "dedupEnabled": true, "hashFunction": "NONE", "dedupTimeColumn": "mtime", "metadataTTL": 30000, "preload": "ENABLE" }, ... } ``` The feature also requires you to specify `pinot.server.instance.max.segment.preload.threads: N` in the server config where N should be replaced with the number of threads that should be used for preload. It's 0 by default to disable the preloading feature. This preloading thread pool is shared with [upsert table's preloading](/build-with-pinot/ingestion/upsert-dedup/upsert#enable-preload-for-faster-server-restarts). ## Ignore non-default tier segments during dedup metadata construction When real-time segments are moved to a non-default tier, they are often already outside the metadata TTL window. In that case, you can skip them when Pinot rebuilds dedup metadata on the server. Set `dedupConfig.ignoreNonDefaultTiers` to one of the following values: * `ENABLE`: Always skip immutable segments on non-default tiers when constructing dedup metadata. * `DISABLE`: Always include immutable segments on non-default tiers. * `DEFAULT`: Use the server-level default from `pinot.server.instance.dedup.default.ignore.non.default.tiers`. ```json { "dedupConfig": { "dedupEnabled": true, "ignoreNonDefaultTiers": "ENABLE" } } ``` The server-level default is `false`, so tables with `ignoreNonDefaultTiers: "DEFAULT"` continue to include non-default-tier segments unless you override the server setting. ## Parallel consumption during commit, download, and replacement Pinot protects dedup correctness by limiting how much a slow replica can keep consuming while the previous segment is still being finalized. * For non-pauseless dedup tables, Pinot disallows parallel consumption during commit by default. * For pauseless dedup tables, Pinot still allows consumption during the build phase, but it stops the slow replica during segment download and replacement so the dedup metadata stays consistent with the lead replica. For backward compatibility, dedup tables still accept the deprecated table-level flag `dedupConfig.allowDedupConsumptionDuringCommit`. Setting it to `true` restores the older behavior and allows the replica to continue consuming throughout commit handling, including download and replacement: ```json { "dedupConfig": { "dedupEnabled": true, "allowDedupConsumptionDuringCommit": true } } ``` If the table-level flag is left at its default `false`, the server-level fallback `pinot.server.instance.dedup.default.allow.dedup.consumption.during.commit` can enable the same legacy behavior for dedup tables on that server. New deployments should prefer `parallelSegmentConsumptionPolicy` in `streamIngestionConfig` when they need explicit control over parallel consumption. ## Immutable dedup configuration fields {% hint style="danger" %} **Certain dedup and schema configuration fields cannot be modified after table creation.** Changing these fields on an existing dedup table can lead to data inconsistencies or data loss between replicas. Pinot uses these configurations to determine which records to keep or discard, so altering them after data has been ingested will cause existing metadata to become inconsistent with the new configuration. The following fields are immutable after table creation: **Schema fields:** * `primaryKeyColumns` **dedupConfig fields:** * `hashFunction` * `dedupTimeColumn` * `timeColumnName` (when used as the default dedup time column) Attempting to update these fields will return an error: ``` Failed to update table '': Cannot modify [] as it may lead to data inconsistencies. Please create a new table instead. ``` **Recommended workaround:** Create a new table with the desired configuration and reingest all data. **Alternative (use with caution):** If you must modify these fields without recreating the table, you can use the `force=true` query parameter on the table config update API. Before doing so, pause consumption and restart all servers. Note that this approach only guarantees consistency for newly ingested keys; existing data may remain inconsistent. {% endhint %} ## Best practices Unlike other real-time tables, Dedup table takes up more memory resources as it needs to bookkeep the primary key and its corresponding segment reference, in memory. As a result, it's important to plan the capacity beforehand, and monitor the resource usage. Here are some recommended practices of using Dedup table. * Create the Kafka topic with more partitions. The number of Kafka partitions determines the partition numbers of the Pinot table. The more partitions you have in the Kafka topic, more Pinot servers you can distribute the Pinot table to and therefore more you can scale the table horizontally. **But note that**, [like upsert tables](https://docs.pinot.apache.org/build-with-pinot/ingestion/upsert-dedup/pages/-MKlOW6W3IGSChedI1sT#create-the-topicstream-with-more-partitions.), you can't increase the partitions in future for dedup enabled tables so you need to start with good enough partitions (atleast 2-3X the number of pinot servers). * For Dedup tables, updating primary key columns or the dedupTimeColumn is not recommended, as it may lead to data loss and inconsistencies between replicas. If a change is unavoidable, ensure that consumption is paused and all servers are restarted for the change to take effect. Even then, consistency is not guaranteed. * Dedup table maintains an in-memory map from the primary key to the segment reference. So it's recommended to use a simple primary key type and avoid composite primary keys to save the memory cost. In addition, consider the `hashFunction` config in the Dedup config, which can be `MD5` or `MURMUR3`, to store the 128-bit hashcode of the primary key instead. This is useful when your primary key takes more space. But keep in mind, this hash may introduce collisions, though the chance is very low. * **Monitoring**: Set up a dashboard over the metric `pinot.server.dedupPrimaryKeysCount.tableName` to watch the number of primary keys in a table partition. It's useful for tracking its growth which is proportional to the memory usage growth. * **Capacity planning:** It's useful to plan the capacity beforehand to ensure you will not run into resource constraints later. A simple way is to measure the amount of the primary keys in the Kafka throughput per partition and time the primary key space cost to approximate the memory usage. A heap dump is also useful to check the memory usage so far on an dedup table instance. # Formats and Filesystems Match Pinot ingestion to the right input formats and deep-storage filesystems without overcomplicating the table design. Pinot supports several source formats and deep-storage choices. Pick these early, because they affect how segments are produced, moved, and recovered. ## Source formats Use the original format docs when you need the exact supported file types or loader behavior. The main landing page is [Supported Data Formats](/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats). ## Filesystems and deep storage Choose the deep-storage backend that matches your operational environment. The detailed filesystem docs still live under [File Systems](/build-with-pinot/ingestion/formats-filesystems/file-systems). ## Keep it simple Do not mix format decisions with schema design. The schema says what the data means; the filesystem says where segments survive after Pinot produces them. ## What this page covered This page covered how source formats and deep storage fit into the ingestion design. ## Next step Read [Transformations and Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations) if data needs cleanup or pre-aggregation before query time. ## Related pages * [Ingestion](/build-with-pinot/ingestion) * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) * [Upsert and Dedup](/build-with-pinot/ingestion/upsert-dedup) * [Supported Data Formats](/build-with-pinot/ingestion/formats-filesystems/pinot-input-formats) * [File Systems](/build-with-pinot/ingestion/formats-filesystems/file-systems) # Supported Data Formats This section contains a collection of guides that will show you how to import data from a Pinot-supported input format. Pinot offers support for various popular input formats during ingestion. By changing the input format, you can reduce the time spent doing serialization-deserialization and speed up the ingestion. ## Configuring input formats To change the input format, adjust the `recordReaderSpec` config in the ingestion job specification. ``` recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' configs: key1 : 'value1' key2 : 'value2' ``` The configuration consists of the following keys: * **`dataFormat`**: Name of the data format to consume. * **`className`**: Name of the class that implements the `RecordReader` interface. This class is used for parsing the data. * **`configClassName`**: Name of the class that implements the `RecordReaderConfig` interface. This class is used the parse the values mentioned in `configs` * **`configs`**: Key-value pair for format-specific configurations. This field is optional. ## Supported input formats Pinot supports multiple input formats out of the box. Specify the corresponding readers and the associated custom configurations to switch between formats. ### CSV ``` dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' configs: fileFormat: 'default' #should be one of default, rfc4180, excel, tdf, mysql header: 'columnName separated by delimiter' delimiter: ',' multiValueDelimiter: '-' ``` CSV Record Reader supports the following configs: * **`fileFormat`**: `default`, `rfc4180`, `excel`, `tdf`, `mysql` * **`header`**: Header of the file. The `columnNames` should be separated by the delimiter mentioned in the configuration. * **`delimiter`**: The character seperating the columns. * **`multiValueDelimiter`**: The character separating multiple values in a single column. This can be used to split a column into a list. * **`skipHeader`**: Skip header record in the file. Boolean. * **`ignoreEmptyLines`**: Ignore empty lines (instead of filling them with default values). Boolean. * **`ignoreSurroundingSpaces`**: ignore spaces around column names and values. Boolean * **`quoteCharacter`**: Single character used for quotes in CSV files. * **`recordSeparator`**: Character used to separate records in the input file. Default is or `\r` depending on the platform. * **`nullStringValue`**: String value that represents null in CSV files. Default is empty string. * **`stopOnError`**: Stop processing the file when Pinot encounters a malformed CSV record. Boolean. Default is `false`. By default, Pinot attempts to recover from malformed data rows and continue reading the rest of the file. Set `stopOnError: true` if you want batch ingestion to stop at the first malformed record instead. Pinot still validates the CSV header during initialization, so an invalid header or first record fails fast. {% hint style="info" %} Your CSV file may have raw text fields that cannot be reliably delimited using any character. In this case, explicitly set the **multiValueDelimeter** field to empty in the ingestion config.\ \ `multiValueDelimiter: ''` {% endhint %} ### Avro Use `extractRawTimeValues` when you need raw Avro temporal logical-type values instead of Pinot's default converted values. ``` dataFormat: 'avro' className: 'org.apache.pinot.plugin.inputformat.avro.AvroRecordReader' configs: extractRawTimeValues: true ``` The Avro record reader converts the data in file to a `GenericRecord`. A Java class or `.avro` file is not required. By default, `extractRawTimeValues` is `false`, so Pinot converts Avro temporal logical types during extraction. Set `extractRawTimeValues` to `true` to keep the raw Avro integer values for `date`, `time-millis`, `time-micros`, `timestamp-millis`, `timestamp-micros`, and `timestamp-nanos`. `decimal` and `uuid` always convert. We use the following conversion table to translate between Avro and Pinot data types. The conversions are done using the offical Avro methods present in `org.apache.avro.Conversions`. | Avro Data Type | Pinot Data Type | Comment | | ----------------- | --------------- | ----------------------------------------------------------------------------- | | INT | INT | | | LONG | LONG | | | FLOAT | FLOAT | | | DOUBLE | DOUBLE | | | BOOLEAN | BOOLEAN | | | STRING | STRING | | | ENUM | STRING | | | BYTES | BYTES | | | FIXED | BYTES | | | MAP | JSON | | | ARRAY | JSON | | | RECORD | JSON | | | UNION | JSON | | | DECIMAL | BYTES | | | UUID | UUID | Supports `string` with `logicalType: "uuid"` and 16-byte `fixed` UUID values. | | DATE | STRING | `yyyy-MM-dd` format | | TIME\_MILLIS | STRING | `HH:mm:ss.SSS` format | | TIME\_MICROS | STRING | `HH:mm:ss.SSSSSS` format | | TIMESTAMP\_MILLIS | TIMESTAMP | | | TIMESTAMP\_MICROS | TIMESTAMP | | ### JSON ``` dataFormat: 'json' className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader' ``` For batch ingestion, `dataFormat: 'json'` uses `JSONRecordReader` and reads newline-delimited UTF-8 text JSON files only. For stream ingestion, use `org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder`. When `stream..decoder.prop.jsonFormat` is unset, the decoder keeps the same UTF-8 text JSON behavior. You can also pin the stream payload format to `TEXT`, `POSTGRES_JSONB`, `SQLITE_JSONB`, `SMILE`, or `CBOR`, or set `AUTO` to opt into per-message detection for mixed streams. `AUTO` is opt-in and only detects CBOR when the payload includes the self-describe tag. If you already know the wire format, pin it explicitly instead of using `AUTO`. For a Kafka example and the config key details, see [Ingest streaming data from Apache Kafka](/build-with-pinot/ingestion/stream-ingestion/import-from-apache-kafka) and [Ingestion Configuration](/reference/configuration-reference/ingestion). ### BSON ``` dataFormat: 'bson' className: 'org.apache.pinot.plugin.inputformat.bson.BSONRecordReader' ``` Use the BSON record reader for `mongodump`-style files that store BSON documents back to back with the standard BSON length prefix. Pinot also detects and reads gzip-compressed BSON files automatically. For batch ingestion, `dataFormat: bson` resolves to `BSONRecordReader` without any extra `configClassName`. For stream ingestion, configure your stream decoder as `org.apache.pinot.plugin.inputformat.bson.BSONMessageDecoder`; each stream message must contain exactly one BSON document. `BSONRecordExtractor` converts decoded BSON values into Pinot-compatible Java values before schema coercion: | BSON type | Pinot-side Java value | Notes | | ----------------- | --------------------- | --------------------------------------------------------------------------- | | `ObjectId` | `String` | 24-character hex string | | `DateTime` | `java.sql.Timestamp` | Millisecond precision | | `BsonTimestamp` | `java.sql.Timestamp` | Second precision; the intra-second ordinal is dropped | | `Decimal128` | `BigDecimal` | `NaN` and `Infinity` become `null`; negative zero becomes `BigDecimal.ZERO` | | `Binary` | `byte[]` | Includes UUID binary subtypes | | Embedded document | `Map` | Converted recursively | | Array | `Object[]` | Converted recursively | ### Thrift ``` dataFormat: 'thrift' className: 'org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReader' configs: thriftClass: 'ParserClassName' ``` {% hint style="info" %} Thrift requires the generated class using `.thrift` file to parse the data. The `.class` file should be available in the Pinot's `classpath`. You can put the files in the `lib/` folder of Pinot distribution directory. {% endhint %} ### Parquet ``` dataFormat: 'parquet' className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader' ``` Since 0.11.0 release, the Parquet record reader determines whether to use `ParquetAvroRecordReader` or `ParquetNativeRecordReader` to read records. The reader looks for the `parquet.avro.schema` or `avro.schema` key in the parquet file footer, and if present, uses the Avro reader. You can change the record reader manually in case of a misconfiguration. ``` dataFormat: 'parquet' className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader' ``` {% hint style="warning" %} For the support of DECIMAL and other parquet native data types, always use `ParquetNativeRecordReader`. {% endhint %} To keep Parquet temporal values in their raw integer form instead of Pinot's default converted values, set `extractRawTimeValues` on `ParquetRecordReader`. ``` dataFormat: 'parquet' className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader' configs: extractRawTimeValues: true ``` When `extractRawTimeValues` is `false` (the default), Pinot converts Parquet `DATE`, `TIME_*`, and `TIMESTAMP_*` values during extraction. Set it to `true` to keep their raw integer values instead. When you use `ParquetRecordReader`, Pinot forwards this config to whichever underlying reader it selects (`ParquetAvroRecordReader` or `ParquetNativeRecordReader`). `DECIMAL` and `UUID` always convert. `ParquetNativeRecordReader` preserves primitive values in their native Pinot-compatible form during extraction. For example, a Parquet `BOOLEAN` stays a Pinot `BOOLEAN` instead of being stringified. | Parquet Data Type | Pinot Data Type | Comment | | -------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------- | | BOOLEAN | BOOLEAN | Preserved as a native boolean value. | | INT96 | LONG | Parquet`INT96` type converts **nanoseconds** to Pinot `INT64` type of **milliseconds** | | INT64 | LONG | | | INT32 | INT | | | FLOAT | FLOAT | | | DOUBLE | DOUBLE | | | BINARY | BYTES | | | FIXED-LEN-BYTE-ARRAY | BYTES | | | DECIMAL | DOUBLE | | | ENUM | STRING | | | UTF8 | STRING | | | REPEATED | MULTIVALUE/MAP (represented as MV | if parquet original type is LIST, then it is converted to MULTIVALUE column otherwise a MAP column. | For `ParquetAvroRecordReader` , you can refer to the [Avro section above](#avro) for the type conversions. #### LIST and MAP wrapper extraction Parquet LIST and MAP wrapper structs are now properly unwrapped when ingesting via `ParquetNativeRecordReader` and `ParquetAvroRecordReader`. Previously, schema-identified LIST and MAP columns had their wrapper elements exposed in the data: * An `array` field previously came back as `[{"element": "abc"}, {"element": "xyz"}]` — now it correctly comes back as `["abc", "xyz"]`. * A `map` field previously came back as `{"key_value":[{"key":"k","value":"v"}]}` — now it correctly comes back as `{"k":"v"}`. Real struct fields named `element` are preserved; only schema-identified LIST and MAP wrappers are normalized. The readers support both the standard 3-level Parquet LIST encoding and legacy 2-level encodings (repeated primitive, repeated multi-field group, or repeated single-field group not named `element`). **Backward incompatibility:** If you have ingestion pipelines or transform expressions that worked around the previous broken shape (for example, selecting `data.element` instead of `data` for an array column), you will need to update those queries and transforms. **MAP ordering:** Parquet itself does not preserve source MAP entry order. Pinot canonicalizes ingested MAP and JSON output by sorting map keys when it serializes the value, so query results are deterministic but do not preserve the original insertion order. If the original pair order matters, model the field as `LIST>` instead. ### ORC ``` dataFormat: 'orc' className: 'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader' ``` ORC record reader supports the following data types - | ORC Data Type | Java Data Type | | ------------- | -------------------- | | BOOLEAN | String | | SHORT | Integer | | INT | Integer | | LONG | Integer | | FLOAT | Float | | DOUBLE | Double | | STRING | String | | VARCHAR | String | | CHAR | String | | LIST | Object\[] | | MAP | Map\ | | DATE | Long | | TIMESTAMP | Long | | BINARY | byte\[] | | BYTE | Integer | {% hint style="info" %} In LIST and MAP types, the object should only belong to one of the data types supported by Pinot. {% endhint %} ### Protocol Buffers ``` dataFormat: 'proto' className: 'org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReader' configs: descriptorFile: 'file:///path/to/sample.desc' ``` The reader requires a descriptor file to deserialize the data present in the files. You can generate the descriptor file (`.desc`) from the `.proto` file using the command - ``` protoc --include_imports --descriptor_set_out=/absolute/path/to/output.desc /absolute/path/to/input.proto ``` ### Apache Arrow The Arrow input format plugin supports reading data in [Apache Arrow IPC format](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format). This is useful for ingesting data from systems that produce Arrow-formatted output. {% hint style="success" %} The `pinot-arrow` plugin is included in the standard Pinot binary distribution (tarball and Docker image). No additional installation steps are required to use Apache Arrow format for data ingestion. {% endhint %} #### Batch ingestion For batch ingestion from Arrow IPC files: ``` dataFormat: 'arrow' className: 'org.apache.pinot.plugin.inputformat.arrow.ArrowRecordReader' ``` The `ArrowRecordReader` reads Arrow IPC files for batch ingestion. Note that Arrow IPC files require seekable channels, so **gzip compression is not supported**. To preserve raw Arrow temporal values instead of Pinot's default converted values, set `extractRawTimeValues` on `ArrowRecordReader`: ```yaml dataFormat: 'arrow' className: 'org.apache.pinot.plugin.inputformat.arrow.ArrowRecordReader' configs: extractRawTimeValues: true ``` When `extractRawTimeValues` is `false` (the default), Pinot converts Arrow `Date`, `Time`, and `Timestamp` values during extraction. Set it to `true` to keep raw integers instead: `Date` stays as days since epoch, while `Time` and `Timestamp` stay in the schema's declared Arrow unit. #### Direct segment generation Standard batch ingestion job specs continue to use `ArrowRecordReader`. If you build segments directly in application code, Pinot also exposes an optional Arrow column-major path through `SegmentIndexCreationDriverImpl.init(config, columnReaderFactory)`. Use `ArrowFileColumnReaderFactory` for Arrow IPC files on disk: ```java SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl(); try (ArrowFileColumnReaderFactory factory = new ArrowFileColumnReaderFactory(arrowFile)) { driver.init(config, factory); driver.build(); } ``` The file-backed factory reads one Arrow record batch at a time during segment build. For direct integrations, it also exposes these config keys: * **`arrowAllocatorLimit`**: Maximum Arrow off-heap allocator size in bytes for the file-backed column-major path. The default is `268435456` (256 MB). * **`extractRawTimeValues`**: Keeps Arrow `Date`, `Time`, and `Timestamp` values as raw integers instead of Pinot's default converted values. The default is `false`. If you already manage an `ArrowReader` and allocator in process, use `ArrowColumnReaderFactory` instead. That path supports the same `extractRawTimeValues` behavior, but allocator sizing remains caller-managed. #### Stream ingestion For stream ingestion, the Arrow decoder converts Arrow columnar batches to Pinot rows: ``` stream.kafka.decoder.class.name=org.apache.pinot.plugin.inputformat.arrow.ArrowMessageDecoder ``` **Configuration properties:** | Property | Default | Description | | ----------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------- | | `arrow.allocator.limit` | 268435456 (256 MB) | Memory limit for Arrow's off-heap allocator in bytes | | `extractRawTimeValues` | `false` | Keep Arrow `Date`, `Time`, and `Timestamp` values as raw integers instead of Pinot's default converted values | Arrow type conversions are handled automatically: UTF-8 text becomes `String`, `Date` becomes `LocalDate`, `Time` becomes `LocalTime`, `Timestamp` becomes `Timestamp`, Arrow Maps become flattened `Map`, and Arrow Lists become `Object[]`. Dictionary-encoded columns are decoded against their logical type before extraction. Each Arrow Kafka message should contain a complete IPC stream. Empty batches are skipped, single-row batches ingest as one Pinot row, and multi-row batches fan out into multiple Pinot rows. # File Systems This section contains a collection of short guides to show you how to import data from a Pinot-supported file system. FileSystem is an abstraction provided by Pinot to access data stored in distributed file systems (DFS). Pinot uses distributed file systems for the following purposes: * **Batch ingestion job**: To read the input data (CSV, Avro, Thrift, etc.) and to write generated segments to DFS. * **Controller**: When a segment is uploaded to the controller, the controller saves it in the configured DFS. * **Server**:- When a server(s) is notified of a new segment, the server copies the segment from remote DFS to their local node using the DFS abstraction. ## Supported file systems Pinot lets you choose a distributed file system provider. The following file systems are supported by Pinot: * [Amazon S3](/build-with-pinot/ingestion/formats-filesystems/file-systems/amazon-s3) * [Google Cloud Storage](/build-with-pinot/ingestion/formats-filesystems/file-systems/import-from-gcp) * [HDFS](/build-with-pinot/ingestion/formats-filesystems/file-systems/import-from-hdfs) * [Azure Data Lake Storage](/build-with-pinot/ingestion/formats-filesystems/file-systems/import-from-adls-azure) ## Enabling a file system To use a distributed file system, you need to enable plugins. To do that, specify the plugin directory and include the required plugins: ``` -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-plugin-to-include-1,pinot-plugin-to-include-2 ``` You can change the file system in the `controller` and `server` configuration. In the following configuration example, the URI is `s3://bucket/path/to/file` and `scheme` refers to the file system URI prefix `s3`. ``` #CONTROLLER pinot.controller.storage.factory.class.[scheme]=className of the pinot file system pinot.controller.segment.fetcher.protocols=file,http,[scheme] pinot.controller.segment.fetcher.[scheme].class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` ``` #SERVER pinot.server.storage.factory.class.[scheme]=className of the Pinot file system pinot.server.segment.fetcher.protocols=file,http,[scheme] pinot.server.segment.fetcher.[scheme].class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` You can also change the file system during ingestion. In the ingestion job spec, specify the file system with the following configuration: ``` pinotFSSpecs - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS ``` # Amazon S3 This guide shows you how to import data from files stored in Amazon S3. Enable the [Amazon S3](https://aws.amazon.com/s3/) file system backend by including the `pinot-s3` plugin. In the controller or server configuration, add the config: ``` -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-s3 ``` ## S3A URI scheme support Starting in Pinot 1.3.0, the `pinot-s3` plugin supports both the `s3://` and `s3a://` URI schemes. Both schemes use the same underlying AWS SDK v2 client and identical configuration — the only difference is the URI prefix. This allows Pinot to integrate with Hadoop-based ecosystems and tools that standardize on the `s3a://` scheme. To use the `s3a://` scheme, specify it in your deep store paths and file system configuration: ``` controller.data.dir=s3a://path/to/data/directory/ pinot.controller.storage.factory.class.s3a=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3a.region=us-east-1 pinot.controller.segment.fetcher.protocols=file,http,s3a pinot.controller.segment.fetcher.s3a.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` All configuration properties documented below work identically for both the `s3` and `s3a` schemes. {% hint style="info" %} By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify `-Dplugins.include`, you need to put all the plugins you want to use, e.g. `pinot-json`, `pinot-avro` , `pinot-kafka-3.0...` {% endhint %} You can configure the S3 file system using the following options: | Configuration | Description | | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | region | The AWS Data center region in which the bucket is located | | accessKey | (Optional) AWS access key required for authentication. This should only be used for testing purposes as we don't store these keys in secret. | | secretKey | (Optional) AWS secret key required for authentication. This should only be used for testing purposes as we don't store these keys in secret. | | endpoint | (Optional) Override endpoint for s3 client. | | disableAcl | If this is set to`false`, bucket owner is granted full access to the objects created by pinot. Default value is `true`. | | serverSideEncryption | (Optional) The server-side encryption algorithm used when storing this object in Amazon S3 (Now supports `aws:kms`), set to null to disable SSE. | | ssekmsKeyId | (Optional, but **required** when `serverSideEncryption=aws:kms`) Specifies the AWS KMS key ID to use for object encryption. All GET and PUT requests for an object protected by AWS KMS will fail if not made via SSL or using SigV4. | | ssekmsEncryptionContext | (Optional) Specifies the AWS KMS Encryption Context to use for object encryption. The value of this header is a base64-encoded UTF-8 string holding JSON with the encryption context key-value pairs. | | requestChecksumCalculation | (Optional) Controls the AWS SDK checksum mode for S3 requests. Default: `WHEN_REQUIRED`. Options: `WHEN_SUPPORTED`, `WHEN_REQUIRED`. | | responseChecksumValidation | (Optional) Controls the AWS SDK checksum mode for S3 responses. Default: `WHEN_REQUIRED`. Options: `WHEN_SUPPORTED`, `WHEN_REQUIRED`. | | useLegacyMd5Plugin | (Optional) When set to `true`, uses the LegacyMd5Plugin to restore pre-2.30.0 MD5 checksum behavior. Default: `false`. | | enableCrossRegionAccess | (Optional) If you want to copy objects b/w two buckets that lie in different regions. Defaults to `true` if not configured. | Each of these properties should be prefixed by `pinot.[node].storage.factory.s3.` where `node` is either `controller` or `server` depending on the config e.g. ``` pinot.controller.storage.factory.s3.region=ap-southeast-1 ``` S3 Filesystem supports authentication using the [DefaultCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). The credential provider looks for the credentials in the following order - * Environment Variables - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` (only recognized by Java SDK) * Java System Properties - `aws.accessKeyId` and `aws.secretKey` * Web Identity Token credentials from the environment or container * Credential profiles file at the default location `(~/.aws/credentials)` shared by all AWS SDKs and the AWS CLI * Credentials delivered through the Amazon EC2 container service if `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` environment variable is set and security manager has permission to access the variable, * Instance profile credentials delivered through the Amazon EC2 metadata service You can also specify the accessKey and secretKey using the properties. However, this method is not secure and should be used only for POC setups. ## Checksum validation {% hint style="info" %} Checksum configuration is available starting in Pinot 1.4. {% endhint %} Starting with AWS SDK 2.30.0, the S3 client enables request and response checksum validation by default. Pinot exposes configuration properties to control this behavior. ### Request and response checksums By default, Pinot sets both `requestChecksumCalculation` and `responseChecksumValidation` to `WHEN_REQUIRED`. This keeps Pinot on the AWS SDK's required-only checksum path unless the S3 API explicitly requires checksum calculation or validation. If you want the S3 client to calculate or validate checksums whenever the API supports it, set both properties to `WHEN_SUPPORTED`: ``` pinot.controller.storage.factory.s3.requestChecksumCalculation=WHEN_SUPPORTED pinot.controller.storage.factory.s3.responseChecksumValidation=WHEN_SUPPORTED ``` | Value | Behavior | | --------------- | --------------------------------------------------------------------------------- | | WHEN\_REQUIRED | Only use checksum calculation or validation when the S3 API requires it (default) | | WHEN\_SUPPORTED | Use checksum calculation or validation whenever the S3 API supports it | ### LegacyMd5Plugin for S3-compatible stores Some S3-compatible object stores (e.g. MinIO, Ceph, or older AWS configurations) require the legacy `Content-MD5` header on requests. After the AWS SDK 2.30.0 upgrade, these stores may return errors like: ``` Missing required content hash for this request: Content-MD5 or x-amz-content-sha256 ``` To restore the pre-2.30.0 MD5 checksum behavior, enable the `useLegacyMd5Plugin` option: ``` pinot.controller.storage.factory.s3.useLegacyMd5Plugin=true ``` This adds the LegacyMd5Plugin to the S3 client, which sends the `Content-MD5` header that these stores expect. {% hint style="warning" %} Only enable `useLegacyMd5Plugin` if your S3-compatible store requires the legacy MD5 header. For standard AWS S3, the default checksum behavior is recommended. {% endhint %} ## Examples ### Job spec ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 's3://pinot-bucket/pinot-ingestion/batch-input/' outputDirURI: 's3://pinot-bucket/pinot-ingestion/batch-output/' overwriteOutput: true pinotFSSpecs: - scheme: s3 className: org.apache.pinot.plugin.filesystem.S3PinotFS configs: region: 'ap-southeast-1' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'students' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` ### Controller config ``` controller.data.dir=s3://path/to/data/directory/ controller.local.temp.dir=/path/to/local/temp/directory controller.enable.split.commit=true pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.region=ap-southeast-1 pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` ### Server config ``` pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=ap-southeast-1 pinot.server.storage.factory.s3.httpclient.maxConnections=50 pinot.server.storage.factory.s3.httpclient.socketTimeout=30s pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` ### Minion config ``` pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.minion.storage.factory.s3.region=ap-southeast-1 pinot.minion.segment.fetcher.protocols=file,http,s3 pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` # Azure Data Lake Storage This guide shows you how to import data from files stored in Azure Data Lake Storage Gen2 (ADLS Gen2) Enable the Azure Data Lake Storage using the `pinot-adls` plugin. In the controller or server, add the config: ``` -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-adls ``` {% hint style="info" %} By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify `-Dplugins.include`, you need to put all the plugins you want to use, e.g. `pinot-json`, `pinot-avro` , `pinot-kafka-3.0...` {% endhint %} Azure Blob Storage provides the following options: * `accountName`: Name of the Azure account under which the storage is created. * `accessKey`: Access key required for the authentication. * `fileSystemName`: Name of the file system to use, for example, the container name (similar to the bucket name in S3). * `enableChecksum`: Enable MD5 checksum for verification. Default is `false`. Each of these properties should be prefixed by `pinot.[node].storage.factory.class.adl2.` where `node` is either `controller` or `server` depending on the config, like this: ``` pinot.controller.storage.factory.class.adl2.accountName=test-user ``` ### Examples #### Job spec ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'adl2://path/to/input/directory/' outputDirURI: 'adl2://path/to/output/directory/' overwriteOutput: true pinotFSSpecs: - scheme: adl2 className: org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS configs: accountName: 'my-account' accessKey: 'foo-bar-1234' fileSystemName: 'fs-name' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'students' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` #### Controller config ``` controller.data.dir=adl2://path/to/data/directory/ controller.local.temp.dir=/path/to/local/temp/directory controller.enable.split.commit=true pinot.controller.storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS pinot.controller.storage.factory.adl2.accountName=my-account pinot.controller.storage.factory.adl2.accessKey=foo-bar-1234 pinot.controller.storage.factory.adl2.fileSystemName=fs-name pinot.controller.segment.fetcher.protocols=file,http,adl2 pinot.controller.segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` #### Server config ``` pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS pinot.server.storage.factory.adl2.accountName=my-account pinot.server.storage.factory.adl2.accessKey=foo-bar-1234 pinot.server.storage.factory.adl2.fileSystemName=fs-name pinot.server.segment.fetcher.protocols=file,http,adl2 pinot.server.segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` #### Minion config ``` storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS storage.factory.adl2.accountName=my-account storage.factory.adl2.fileSystemName=fs-name storage.factory.adl2.accessKey=foo-bar-1234 segment.fetcher.protocols=file,http,adl2 segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` # HDFS This guide shows you how to configure HDFS for use with Pinot, including data import and deep storage. Enable the [Hadoop distributed file system (HDFS)](https://hadoop.apache.org/) using the `pinot-hdfs` plugin. In the controller or server, add the config: ``` -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-hdfs ``` {% hint style="info" %} By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify `-Dplugins.include`, you need to put all the plugins you want to use, e.g. `pinot-json`, `pinot-avro` , `pinot-kafka-3.0...` {% endhint %} HDFS implementation provides the following options: * `hadoop.conf.path`: Absolute path of the directory containing Hadoop XML configuration files, such as **hdfs-site.xml, core-site.xml** . * `hadoop.write.checksum`: Create checksum while pushing an object. Default is `false` * `hadoop.kerberos.principle` * `hadoop.kerberos.keytab` * `hadoop.allow.insecure`: Set to `true` to access an HDFS cluster without Kerberos by installing the configured remote user as the JVM-wide Hadoop login user. Default is `false`. * `hadoop.user.name`: Hadoop username to use when `hadoop.allow.insecure` is `true`. Each of these properties should be prefixed by `pinot.[node].storage.factory.class.hdfs.` where `node` is either `controller` or `server` depending on the config The `kerberos` configs should be used only if your Hadoop installation is secured with Kerberos. Refer to the [Hadoop in secure mode documentation](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html) for information on how to secure Hadoop using Kerberos. For an HDFS cluster without Kerberos, configure both insecure-access properties on the storage factory. This mode changes the global Hadoop login user for the Pinot JVM, so use it only when that process should act as one HDFS identity. ```properties pinot.controller.storage.factory.hdfs.hadoop.allow.insecure=true pinot.controller.storage.factory.hdfs.hadoop.user.name=pinot ``` You must provide proper Hadoop dependencies jars from your Hadoop installation to your Pinot startup scripts. ``` export HADOOP_HOME=/local/hadoop/ export HADOOP_VERSION=2.7.1 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar" ``` ## Push HDFS segment to Pinot Controller To push HDFS segment files to Pinot controller, send the HDFS path of your newly created segment files to the Pinot Controller. The controller will download the files. This curl example requests tells the controller to download segment files to the proper table: ``` curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file. ``` ## Examples ### Job spec Standalone Job: ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'hdfs:///path/to/input/directory/' outputDirURI: 'hdfs:///path/to/output/directory/' includeFileNamePath: 'glob:**/*.csv' overwriteOutput: true pinotFSSpecs: - scheme: hdfs className: org.apache.pinot.plugin.filesystem.HadoopPinotFS configs: hadoop.conf.path: 'path/to/conf/directory/' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'students' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` Hadoop Job: ```yaml executionFrameworkSpec: name: 'hadoop' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentUriPushJobRunner' extraConfigs: stagingDir: 'hdfs:///path/to/staging/directory/' jobType: SegmentCreationAndTarPush inputDirURI: 'hdfs:///path/to/input/directory/' outputDirURI: 'hdfs:///path/to/output/directory/' includeFileNamePath: 'glob:**/*.csv' overwriteOutput: true pinotFSSpecs: - scheme: hdfs className: org.apache.pinot.plugin.filesystem.HadoopPinotFS configs: hadoop.conf.path: '/etc/hadoop/conf/' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'students' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` ### Controller config ``` controller.data.dir=hdfs://path/to/data/directory/ controller.local.temp.dir=/path/to/local/temp/directory controller.enable.split.commit=true pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.controller.storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory/ pinot.controller.segment.fetcher.protocols=file,http,hdfs pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle= pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab= ``` ### Server config ``` pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.server.storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory/ pinot.server.segment.fetcher.protocols=file,http,hdfs pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle= pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab= ``` ### Minion config ``` storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory segment.fetcher.protocols=file,http,hdfs segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher segment.fetcher.hdfs.hadoop.kerberos.principle= segment.fetcher.hdfs.hadoop.kerberos.keytab= ``` ## HDFS as deep storage To use HDFS as deep storage, configure each Pinot component with the HDFS plugin and the appropriate storage factory and segment fetcher properties. The sections below provide complete configuration and startup examples for each component. ### Server setup #### Configuration ``` pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.server.storage.factory.hdfs.hadoop.conf.path=/path/to/hadoop/conf/directory/ # For server, instructing the HadoopPinotFS plugin to use the specified keytab and principal when accessing HDFS paths pinot.server.storage.factory.hdfs.hadoop.kerberos.principle= pinot.server.storage.factory.hdfs.hadoop.kerberos.keytab= pinot.server.segment.fetcher.protocols=file,http,hdfs pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle= pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab= pinot.set.instance.id.to.hostname=true pinot.server.instance.dataDir=/path/in/local/filesystem/for/pinot/data/server/index pinot.server.instance.segmentTarDir=/path/in/local/filesystem/for/pinot/data/server/segment pinot.server.grpc.enable=true pinot.server.grpc.port=8090 ``` #### Executable ``` export HADOOP_HOME=/path/to/hadoop/home export HADOOP_VERSION=2.7.1 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export GC_LOG_LOCATION=/path/to/gc/log/file export PINOT_VERSION=0.10.0 export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/ export SERVER_CONF_DIR=/path/to/pinot/conf/dir/ export ZOOKEEPER_ADDRESS=localhost:2181 export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-server.log" ${PINOT_DISTRIBUTION_DIR}/bin/start-server.sh -zkAddress ${ZOOKEEPER_ADDRESS} -configFileName ${SERVER_CONF_DIR}/server.conf ``` ### Controller setup #### Configuration ``` controller.data.dir=hdfs://path/in/hdfs/for/controller/segment controller.local.temp.dir=/tmp/pinot/ controller.zk.str= controller.enable.split.commit=true controller.access.protocols.http.port=9000 controller.helix.cluster.name=PinotCluster pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.controller.storage.factory.hdfs.hadoop.conf.path=/path/to/hadoop/conf/directory/ # For controller, instructing the HadoopPinotFS plugin to use the specified keytab and principal when accessing the HDFS path defined in controller.data.dir pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle= pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab= pinot.controller.segment.fetcher.protocols=file,http,hdfs pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle= pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab= controller.vip.port=9000 controller.port=9000 pinot.set.instance.id.to.hostname=true pinot.server.grpc.enable=true ``` #### Executable ``` export HADOOP_HOME=/path/to/hadoop/home export HADOOP_VERSION=2.7.1 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export GC_LOG_LOCATION=/path/to/gc/log/file export PINOT_VERSION=0.10.0 export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/ export SERVER_CONF_DIR=/path/to/pinot/conf/dir/ export ZOOKEEPER_ADDRESS=localhost:2181 export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms8G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-controller.log" ${PINOT_DISTRIBUTION_DIR}/bin/start-controller.sh -configFileName ${SERVER_CONF_DIR}/controller.conf ``` ### Broker setup #### Configuration ``` pinot.set.instance.id.to.hostname=true pinot.server.grpc.enable=true ``` #### Executable ``` export HADOOP_HOME=/path/to/hadoop/home export HADOOP_VERSION=2.7.1 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export GC_LOG_LOCATION=/path/to/gc/log/file export PINOT_VERSION=0.10.0 export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/ export SERVER_CONF_DIR=/path/to/pinot/conf/dir/ export ZOOKEEPER_ADDRESS=localhost:2181 export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-broker.log" ${PINOT_DISTRIBUTION_DIR}/bin/start-broker.sh -zkAddress ${ZOOKEEPER_ADDRESS} -configFileName ${SERVER_CONF_DIR}/broker.conf ``` ## Kerberos authentication When using HDFS with Kerberos security enabled, Pinot provides two ways to authenticate: ### 1. Automatic authentication (recommended) By configuring the `storage.factory` Kerberos properties shown above, Pinot will automatically handle Kerberos authentication using the specified keytab and principal. This eliminates the need for manual `kinit` commands and ensures continuous authentication even after ticket expiration. #### Why these properties are required The `storage.factory` Kerberos properties serve a critical purpose in Pinot's HDFS integration: **For Controller:** * The controller uses `controller.data.dir` to store segment metadata and other data in HDFS * When `controller.data.dir` points to an HDFS path (e.g., `hdfs://namenode:8020/pinot/data`), the HadoopPinotFS plugin needs Kerberos credentials to access it * Without `storage.factory` Kerberos properties, the controller would fail to read/write to HDFS, causing segment upload and metadata operations to fail * These properties enable the HadoopPinotFS plugin to programmatically authenticate using the keytab file **For Server:** * The server uses HadoopPinotFS for various HDFS operations including segment downloads and deep storage access * When servers need to access segments stored in HDFS deep storage, they require valid Kerberos credentials * The `storage.factory` properties provide persistent authentication that survives across server restarts and ticket expirations #### Understanding the two sets of Kerberos properties You may notice two sets of Kerberos properties in the configuration: 1. **`storage.factory` properties (recommended):** * `pinot.controller.storage.factory.hdfs.hadoop.kerberos.principal` * `pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab` * `pinot.server.storage.factory.hdfs.hadoop.kerberos.principal` * `pinot.server.storage.factory.hdfs.hadoop.kerberos.keytab` **Purpose:** These properties configure Kerberos authentication for the HadoopPinotFS storage factory, which handles controller and server deep storage operations and general HDFS filesystem operations through the storage factory. **Why needed:** The storage factory is initialized at startup and used throughout the component's lifecycle for HDFS access. Without these properties, any HDFS operation through the storage factory would fail with authentication errors. 2. **`segment.fetcher` properties (legacy, for backward compatibility):** * `pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle` (note: typo "principle" instead of "principal" maintained for compatibility) * `pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab` * `pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle` * `pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab` **Purpose:** These configure Kerberos for the segment fetcher component specifically. **Why both are needed:** While there is some functional overlap, having both ensures complete coverage of all HDFS access patterns, backward compatibility with existing deployments, and independent operation of the segment fetcher. #### Benefits of automatic authentication * Eliminates the need to run `kinit` commands manually, reducing operational overhead and human error * Kerberos tickets typically expire after 24 hours (configurable); with keytab-based authentication, Pinot automatically renews tickets internally, preventing service disruptions * Keytab files provide secure, long-term credentials without storing passwords in scripts or configuration ### 2. Manual authentication (legacy) Alternatively, you can manually authenticate using `kinit` before starting Pinot components: ```bash kinit -kt ``` **Limitations of manual authentication:** * **Ticket expiration:** Kerberos tickets typically expire after 24 hours, requiring re-authentication * **Service interruption:** If tickets expire while Pinot is running, HDFS operations will fail until re-authentication * **Operational burden:** Requires monitoring and manual intervention, especially problematic for 24/7 production systems * **Automation challenges:** Difficult to integrate into automated deployment pipelines {% hint style="warning" %} Manual authentication is not recommended for production environments. Always use the `storage.factory` Kerberos properties for production deployments. {% endhint %} ## Troubleshooting ### HDFS FileSystem issues If you receive an error that says `No FileSystem for scheme"hdfs"`, the problem is likely to be a class loading issue. To fix, try adding the following property to `core-site.xml`: `fs.hdfs.impl org.apache.hadoop.hdfs.DistributedFileSystem` And then export `/opt/pinot/lib/hadoop-common-.jar` in the classpath. ### Kerberos authentication issues #### Error: "Failed to authenticate with Kerberos" **Possible causes:** 1. **Incorrect keytab path:** Ensure the keytab file path is absolute and accessible by the Pinot process 2. **Wrong principal name:** Verify the principal name matches the one in the keytab file 3. **Keytab file permissions:** The keytab file must be readable by the user running Pinot (typically `chmod 400` or `chmod 600`) **Solution:** ```bash # Verify keytab contains the correct principal klist -kt /path/to/your.keytab # Test authentication manually kinit -kt /path/to/your.keytab your-principal@YOUR.REALM # Check if authentication succeeded klist ``` #### Error: "GSSException: No valid credentials provided" **Cause:** This typically occurs when the `storage.factory` Kerberos properties are not set, the keytab file path is incorrect or the file doesn't exist, or the Kerberos configuration (`krb5.conf`) is not properly configured. **Solution:** 1. Verify all `storage.factory` Kerberos properties are correctly set in the configuration 2. Ensure the keytab file exists and has correct permissions 3. Check that `/etc/krb5.conf` (or `$JAVA_HOME/jre/lib/security/krb5.conf`) is properly configured with your Kerberos realm settings #### Error: "Unable to obtain Kerberos password" or "Clock skew too great" **Cause:** Time synchronization issue between Pinot server and Kerberos KDC. **Solution:** ```bash # Check time synchronization date # Ensure NTP is running and synchronized sudo systemctl status ntpd # Or for chrony sudo systemctl status chronyd ``` Kerberos requires clock synchronization within 5 minutes (default) between client and KDC. #### Error: "HDFS operation fails after running for several hours" **Cause:** This typically indicates that manual `kinit` was used instead of `storage.factory` properties, and Kerberos tickets have expired (default 24 hours). **Solution:** 1. Configure `storage.factory` Kerberos properties to enable automatic ticket renewal 2. Remove any manual `kinit` from startup scripts 3. Restart Pinot components to apply the configuration #### Verifying Kerberos configuration To verify your Kerberos setup is working correctly: ```bash # 1. Test keytab authentication kinit -kt /path/to/your.keytab your-principal@YOUR.REALM # 2. Verify you can list HDFS directories hdfs dfs -ls / # 3. Check Pinot logs for authentication messages tail -f /path/to/pinot/logs/pinot-controller.log | grep -i kerberos tail -f /path/to/pinot/logs/pinot-server.log | grep -i kerberos # 4. Look for successful authentication messages like: # "Login successful for user using keytab file " ``` #### Best practices 1. **Use absolute paths** for keytab files in configuration 2. **Secure keytab files** with appropriate permissions (400 or 600) 3. **Use service principals** (e.g., `pinot/hostname@REALM`) rather than user principals for production 4. **Monitor Kerberos ticket expiration** in logs to ensure automatic renewal is working 5. **Keep keytab files backed up** in secure locations 6. **Test configuration** in a non-production environment first # Google Cloud Storage This guide shows you how to import data from GCP (Google Cloud Platform). Enable the [Google Cloud Storage](https://cloud.google.com/products/storage/) using the `pinot-gcs` plugin. In the controller or server, add the config: ``` -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcs ``` {% hint style="info" %} By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify `-Dplugins.include`, you need to put all the plugins you want to use, e.g. `pinot-json`, `pinot-avro` , `pinot-kafka-3.0...` {% endhint %} GCP file systems provides the following options: * `projectId` - The name of the Google Cloud Platform project under which you have created your storage bucket. * `gcpKey` - Location of the json file containing GCP keys. You can refer [Creating and managing service account keys](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) to download the keys. Each of these properties should be prefixed by `pinot.[node].storage.factory.class.gs.` where `node` is either `controller` or `server` depending on the configuration, like this: ``` pinot.controller.storage.factory.class.gs.projectId=test-project ``` ### Examples #### Job spec ```yaml executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: 'gs://my-bucket/path/to/input/directory/' outputDirURI: 'gs://my-bucket/path/to/output/directory/' overwriteOutput: true pinotFSSpecs: - scheme: gs className: org.apache.pinot.plugin.filesystem.GcsPinotFS configs: projectId: 'my-project' gcpKey: 'path-to-gcp json key file' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: 'students' pinotClusterSpecs: - controllerURI: 'http://localhost:9000' ``` #### Controller config ``` controller.data.dir=gs://path/to/data/directory/ controller.local.temp.dir=/path/to/local/temp/directory controller.enable.split.commit=true pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS pinot.controller.storage.factory.gs.projectId=my-project pinot.controller.storage.factory.gs.gcpKey=path/to/gcp/key.json pinot.controller.segment.fetcher.protocols=file,http,gs pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` #### Server config ``` pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS pinot.server.storage.factory.gs.projectId=my-project pinot.server.storage.factory.gs.gcpKey=path/to/gcp/key.json pinot.server.segment.fetcher.protocols=file,http,gs pinot.server.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` #### Minion config ``` pinot.minion.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS pinot.minion.storage.factory.gs.projectId=my-project pinot.minion.storage.factory.gs.gcpKey=path/to/gcp/key.json pinot.minion.segment.fetcher.protocols=file,http,gs pinot.minion.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ``` # Complex Type (Array, Map) Handling Complex type handling in Apache Pinot. Commonly, ingested data has a complex structure. For example, Avro schemas have [records](https://avro.apache.org/docs/current/specification/_print/#schema-record) and [arrays](https://avro.apache.org/docs/current/specification/_print/#arrays) while JSON supports [objects](https://json-schema.org/understanding-json-schema/reference/object.html) and [arrays](https://json-schema.org/understanding-json-schema/reference/array.html). Apache Pinot's data model supports primitive data types (including int, long, float, double, BigDecimal, string, bytes), and limited multi-value types, such as an array of primitive types. Simple data types allow Pinot to build fast indexing structures for good query performance, but does require some handling of the complex structures. There are three options for complex type handling: * Convert the complex-type data into a JSON string and then build a JSON index. * Use `OPEN_STRUCT` when an object column should stay in one field, but frequently queried keys need columnar storage and secondary indexes. * Use the built-in complex-type handling rules in the ingestion configuration. On this page, we'll show how to handle these complex-type structures with each of these three approaches. We will process some example data, consisting of the field `group` from the [Meetup events Quickstart example](https://github.com/apache/pinot/tree/master/pinot-tools/src/main/resources/examples/stream/meetupRsvp). This object has two child fields and the child `group` is a nested array with elements of object type. ![Example JSON data](/files/-MaW63rfirmhLIFZbZJj) ## JSON indexing Apache Pinot provides a powerful [JSON index](/build-with-pinot/indexing/json-index) to accelerate the value lookup and filtering for the column. To convert an object `group` with complex type to JSON, add the following to your table configuration. {% code title="json\_meetupRsvp\_realtime\_table\_config.json" %} ```javascript { "ingestionConfig":{ "transformConfigs": [ { "columnName": "group_json", "transformFunction": "jsonFormat(\"group\")" } ], }, ... "tableIndexConfig": { "loadMode": "MMAP", "noDictionaryColumns": [ "group_json" ], "jsonIndexColumns": [ "group_json" ] }, } ``` {% endcode %} The config `transformConfigs` transforms the object `group` to a JSON string `group_json`, which then creates the JSON indexing with configuration `jsonIndexColumns`. To read the full spec, see [meetupRsvpJson\_realtime\_table\_config.json](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/examples/stream/meetupRsvpJson/meetupRsvpJson_realtime_table_config.json). Also, note that `group` is a reserved keyword in SQL and therefore needs to be quoted in `transformFunction`. {% hint style="info" %} The `columnName` can't use the same name as any of the fields in the source JSON data, for example, if our source data contains the field `group` and we want to transform the data in that field before persisting it, the destination column name would need to be something different, like `group_json`. {% endhint %} {% hint style="info" %} Note that you do *not* need to worry about the `maxLength` of the field `group_json` on the schema, because `"JSON"` data type does not have a `maxLength` and will not be truncated. This is true even though `"JSON"` is stored as a string internally. {% endhint %} The schema will look like this: {% code title="json\_meetupRsvp\_realtime\_table\_schema.json" %} ```javascript { { "name": "group_json", "dataType": "JSON", } ... } ``` {% endcode %} For the full specification, see [json\_meetupRsvp\_schema.json](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/examples/stream/meetupRsvpJson/meetupRsvpJson_schema.json). With this, you can start to query the nested fields under `group`. For more details about the supported JSON function, see [guide](/build-with-pinot/indexing/json-index)). ## OPEN\_STRUCT storage and per-key indexes Use `OPEN_STRUCT` when your source field is an object or map whose key set evolves over time, but you still want Pinot to store the most important keys as standard columns. Pinot stores an `OPEN_STRUCT` column in two tiers: * Dense keys become materialized child columns named `$`. * Remaining keys are packed into one sparse JSON column named `$__sparse__`. Pinot decides which keys are dense in this order: * Keys listed in `denseKeys` are always materialized. * Other keys are materialized when their fill rate is at least `denseKeyMinFillRate` (default `0.5`). * If more keys qualify than `maxDenseKeys` allows, Pinot keeps the highest-fill-rate keys as dense and writes the rest to the sparse JSON column. Dense keys reuse Pinot's standard column infrastructure, so each materialized key gets a forward index and can also use vetted per-key settings for dictionary, inverted, range, and bloom-filter behavior through `valueFieldConfigs`. If you do not configure a dense key explicitly, Pinot defaults to dictionary encoding plus an inverted index for that key. ### Define the schema Declare the object column as `OPEN_STRUCT`. `childFieldSpecs` is optional, but it is useful when some keys should always keep a specific type: ```json { "complexFieldSpecs": [ { "name": "attributes", "dataType": "OPEN_STRUCT", "fieldType": "COMPLEX", "childFieldSpecs": { "customerId": { "name": "customerId", "dataType": "STRING", "fieldType": "DIMENSION" }, "country": { "name": "country", "dataType": "STRING", "fieldType": "DIMENSION" } } } ] } ``` ### Configure dense keys and per-key indexes Add an `open_struct` entry to the field's `indexes` object in `fieldConfigList`: ```json { "fieldConfigList": [ { "name": "attributes", "indexes": { "open_struct": { "denseKeys": ["customerId", "country"], "denseKeyMinFillRate": 0.5, "maxDenseKeys": 32, "sparseJsonIndex": true, "valueFieldConfigs": [ { "name": "customerId", "indexes": { "inverted": {} } }, { "name": "country", "indexes": { "bloom": {} } } ] } } } ] } ``` ### Query OPEN\_STRUCT keys Access a key with the item operator. The same syntax works in projections, filters, and aggregations: ```sql SELECT attributes['customerId'], MIN(attributes['customerId']), MAX(attributes['customerId']), DISTINCTCOUNT(attributes['customerId']) FROM events WHERE attributes['country'] IN ('US', 'CA') GROUP BY attributes['customerId'] ``` For a materialized key, Pinot reads the generated child column and can use its dictionary, inverted, range, or other configured index. Per-key index filtering supports equality and inequality, `IN`, `NOT IN`, ranges, `IS NULL`, and `IS NOT NULL`. `EXPLAIN PLAN` reports `delegateTo:per_key_index` when the filter uses this path. Keys stored in the shared sparse column are also available through the item operator. Pinot exposes each sparse key as a virtual typed data source, so projections, filters, grouping, and aggregations use the same SQL syntax as dense keys. Sparse keys use scan-based execution by default. Set `sparseJsonIndex` to `true` to build a JSON index over the sparse column; Pinot can use it for compatible string-key equality and `IN` predicates, while other predicates continue to scan the virtual data source. A key that is absent from a document returns its type's default value and is marked null when null handling is enabled. A key that is absent from the segment returns `NULL`; `IS NULL` matches all documents and other predicates match none. Notes: * `OPEN_STRUCT` is a field-level index for single-value `OPEN_STRUCT` columns. * Pinot can still ingest keys that are not listed in `childFieldSpecs`; it infers a stored type from observed values when possible. * When any schema field uses `OPEN_STRUCT`, `$` becomes a reserved character in schema column names because Pinot uses it in generated child-column names. * Use the [schema reference](/reference/configuration-reference/schema) for the exact schema JSON and the [table reference](/reference/configuration-reference/table) for the full `open_struct` config surface. ## Flatten and unnest with ingestion configurations Though JSON indexing is a handy way to process the complex types, there are some limitations: * It’s not performant to group by or order by a JSON field, because `JSON_EXTRACT_SCALAR` is needed to extract the values in the GROUP BY and ORDER BY clauses, which invokes the function evaluation. * It does not work with Pinot's [multi-value column functions](/functions/aggregation#multi-value-column-functions) such as `DISTINCTCOUNTMV`. Alternatively, from Pinot 0.8, you can use the complex-type handling in ingestion configurations to flatten and unnest the complex structure and convert them into primitive types. Then you can reduce the complex-type data into a flattened Pinot table, and query it via SQL. With the built-in processing rules, you do not need to write ETL jobs in another compute framework such as Flink or Spark. To process this complex type, you can add the configuration `complexTypeConfig` to the `ingestionConfig`. For example: {% code title="complexTypeHandling\_meetupRsvp\_realtime\_table\_config.json" %} ```javascript { "ingestionConfig": { "complexTypeConfig": { "delimiter": ".", "fieldsToUnnest": ["group.group_topics"], "collectionNotUnnestedToJson": "NON_PRIMITIVE" } } } ``` {% endcode %} With the `complexTypeConfig` , all the map objects will be flattened to direct fields automatically. And with `unnestFields` , a record with the nested collection will unnest into multiple records. For instance, the example at the beginning will transform into two rows with this configuration example. ![Flattened/unnested data](/files/-MaW63rl5HQKHhbIhk3M) Note that: * The nested field `group_id` under `group` is flattened to `group.group_id`. The default value of the delimiter is `.` You can choose another delimiter by specifying the configuration `delimiter` under `complexTypeConfig`. This flattening rule also applies to maps in the collections to be unnested. * The nested array `group_topics` under `group` is unnested into the top-level, and converts the output to a collection of two rows. Note the handling of the nested field within `group_topics`, and the eventual top-level field of `group.group_topics.urlkey`. All the collections to unnest shall be included in the configuration `fieldsToUnnest`. * Collections not specified in `fieldsToUnnest`will be serialized into JSON string, except for the array of primitive values, which will be ingested as a multi-value column by default. The behavior is defined by the `collectionNotUnnestedToJson` config, which takes the following values: * `NON_PRIMITIVE`- Converts the array to a multi-value column. *(default)* * `ALL`- Converts the array of primitive values to JSON string. * `NONE`- Does not do any conversion. You can find the full specifications of the table config [here](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/examples/stream/meetupRsvpComplexType/meetupRsvpComplexType_realtime_table_config.json) and the table schema [here](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/resources/examples/stream/meetupRsvpComplexType/meetupRsvpComplexType_schema.json). You can then query the table with primitive values using the following SQL query: ```sql SELECT "group.group_topics.urlkey", "group.group_topics.topic_name", "group.group_id" FROM meetupRsvp LIMIT 10 ``` {% hint style="info" %} `.` is a reserved character in SQL, so you need to quote the flattened columns in the query. {% endhint %} ### Infer the Pinot schema from the Avro schema and JSON data When there are complex structures, it can be challenging and tedious to figure out the Pinot schema manually. To help with schema inference, Pinot provides utility tools to take the Avro schema or JSON data as input and output the inferred Pinot schema. To infer the Pinot schema from Avro schema, you can use a command like this: ```bash bin/pinot-admin.sh AvroSchemaToPinotSchema \ -timeColumnName fields.hoursSinceEpoch \ -avroSchemaFile /tmp/test.avsc \ -pinotSchemaName myTable \ -outputDir /tmp/test \ -fieldsToUnnest entries ``` Note you can input configurations like `fieldsToUnnest` similar to the ones in `complexTypeConfig`. And this will simulate the complex-type handling rules on the Avro schema and output the Pinot schema in the file specified in `outputDir`. Similarly, you can use the command like the following to infer the Pinot schema from a file of JSON objects. ```bash bin/pinot-admin.sh JsonToPinotSchema \ -timeColumnName hoursSinceEpoch \ -jsonFile /tmp/test.json \ -pinotSchemaName myTable \ -outputDir /tmp/test \ -fieldsToUnnest payload.commits ``` You can check out an example of this run in this [PR](https://github.com/apache/pinot/pull/6930). # Complex Type Examples (Unnest) Additional examples that demonstrate handling of complex types. Additional examples that demonstrate handling of complex types. ## Unnest Root Level Collection In this example, we would look at un-nesting json records that are batched together as part of a single key at the root level. We will make use of the [ComplexType](/build-with-pinot/ingestion/formats-filesystems/complex-type) configs to persist the individual student records as separate rows in Pinot. ### Sample JSON record ```json { "students": [ { "firstName": "Jane", "id": "100", "scores": { "physics": 91, "chemistry": 93, "maths": 99 } }, { "firstName": "John", "id": "101", "scores": { "physics": 97, "chemistry": 98, "maths": 99 } }, { "firstName": "Jen", "id": "102", "scores": { "physics": 96, "chemistry": 95, "maths": 100 } } ] } ``` ### Pinot Schema The Pinot schema for this example would look as follows. ```json { "schemaName": "students001", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "students.firstName", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.id", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.scores", "dataType": "JSON", "notNull": false, "fieldType": "DIMENSION" } ], "dateTimeFieldSpecs": [ { "name": "ts", "fieldType": "DATE_TIME", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ], "metricFieldSpecs": [] } ``` ### Pinot Table Configuration The Pinot table configuration for this schema would look as follows. ```json { "ingestionConfig": { "complexTypeConfig": { "fieldsToUnnest": [ "students" ] } } } ``` ### Data in Pinot Post ingestion, the student records would appear as separate records in Pinot. Note that the nested field `scores` is captured as a JSON field. ![Unnested Student Records](/files/5vJKVDdqTUoJzLwHlgpC) ## Unnest sibling collections In this example, we would look at un-nesting the sibling collections "student" and "teacher". ### Sample JSON Record ```json { "student": [ { "name": "John" }, { "name": "Jane" } ], "teacher": [ { "physics": "Kim" }, { "chemistry": "Lu" }, { "maths": "Walsh" } ] } ``` ### Pinot Schema ```json { "schemaName": "students002", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "student.name", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false }, { "name": "teacher.physics", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false }, { "name": "teacher.chemistry", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false }, { "name": "teacher.maths", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false } ] } ``` ### Pinot Table configuration ```json "complexTypeConfig": { "fieldsToUnnest": [ "student", "teacher" ] } ``` ### Data in Pinot ![Unnested student records](/files/TjUVl9vhP5byvcrD4HZs) ## Unnest nested collection In this example, we would look at un-nesting the nested collection "students.grades". ### Sample JSON Record ```json { "students": [ { "name": "Jane", "grades": [ { "physics": "A+" }, { "maths": "A-" } ] }, { "name": "John", "grades": [ { "physics": "B+" }, { "maths": "B-" } ] } ] } ``` ### Pinot Schema ```json { "schemaName": "students003", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "students.name", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false }, { "name": "students.grades.physics", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false }, { "name": "students.grades.maths", "dataType": "STRING", "fieldType": "DIMENSION", "notNull": false } ] } ``` ### Pinot Table configuration ```json "complexTypeConfig": { "fieldsToUnnest": [ "students", "students.grades" ] } ``` ### Data in Pinot ![Unnest Nested Collection](/files/5i5ToRnmqm2fpXVXL9G3) ## Unnest Multi Level Array In this example, we would look at un-nesting the array "finalExam" which is located within the array "students". ### Sample JSON Record ```json { "students": [ { "name": "John", "grades": { "finalExam": [ { "physics": "A+" }, { "maths": "A-" } ] } }, { "name": "Jane", "grades": { "finalExam": [ { "physics": "B+" }, { "maths": "B-" } ] } } ] } ``` ### Pinot Schema ```json { "schemaName": "students004", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "students.name", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.grades.finalExam.physics", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.grades.finalExam.maths", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" } ] } ``` ### Pinot Table configuration ```json "complexTypeConfig": { "fieldsToUnnest": [ "students", "students.grades.finalExam" ] } ``` ### Data in Pinot ![Unnested Multi Level Array](/files/ARQnKKAKfT2DZujVl6ev) ## Convert inner collections In this example, the inner collection "grades" is converted into a multi value string column. ### Sample JSON Record ```json { "students": [ { "name": "John", "grades": [ { "physics": "A+" }, { "maths": "A" } ] }, { "name": "Jane", "grades": [ { "physics": "B+" }, { "maths": "B-" } ] } ] } ``` ### Pinot Schema ```json { "schemaName": "students005", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "students.name", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.grades", "dataType": "STRING", "notNull": false, "isSingleValue": false, "fieldType": "DIMENSION" } ] } ``` ### Pinot Table configuration ```json "complexTypeConfig": { "fieldsToUnnest": [ "students" ] } ``` ### Data in Pinot ![Converted Inner Collection](/files/BJsIDOh7MnvRe981v1WL) ## Primitive Array Converted to JSON String In this example, the array of primitives "extra\_curricular" is converted to a Json string. ### Sample JSON Record ```json { "students": [ { "name": "John", "extra_curricular": [ "piano", "soccer" ] }, { "name": "Jane", "extra_curricular": [ "violin", "music" ] } ] } ``` ### Pinot Schema ```json { "schemaName": "students006", "enableColumnBasedNullHandling": false, "dimensionFieldSpecs": [ { "name": "students.name", "dataType": "STRING", "notNull": false, "fieldType": "DIMENSION" }, { "name": "students.extra_curricular", "dataType": "JSON", "notNull": false, "fieldType": "DIMENSION" } ] } ``` ### Pinot Table configuration ```json "complexTypeConfig": { "fieldsToUnnest": [ "students" ], "collectionNotUnnestedToJson": "ALL" } ``` ### Data in Pinot ![Primitives Converted to JSON](/files/Y9VhOO0ZTuZY9YAMOVUl) ## Unnest JsonArrayString collections In this example, the data is `STRING` type and the content is string encoded `JSON ARRAY` . In this case, the Unnest won't happen automatically on a `STRING` field. Users need to first convert the `STRING` field to `ARRAY` or `MAP` field then perform the unnest. Here are the steps: 1. use `enrichmentConfigs` to create the intermediate column `recordArray` with the function: `jsonStringToListOrMap(data_for_unnesting)` ``` "enrichmentConfigs": [ { "enricherType": "generateColumn", "properties": {"fieldToFunctionMap":{"recordArray":"jsonStringToListOrMap(data_for_unnesting)"}}, "preComplexTypeTransform": true } ], ``` 2. configure `complexTypeConfig` to unnest the intermediate field `recordArray` to generate the field `recordArray||name` ``` "complexTypeConfig": { "fieldsToUnnest": [ "recordArray" ], "delimiter": "||" }, ``` ### Sample Record ``` { "key": "value", "data_for_unnesting": [ { "name": "record1" }, { "name": "record2" }, { "name": "record3" } ], "event_time": "2025-04-24T20:45:56.721936" } ``` ### Pinot Schema {% hint style="warning" %} Note the field to ingest is `recordArray||name` not `data_for_unnesting||name` {% endhint %} ``` { "schemaName": "testUnnest", "enableColumnBasedNullHandling": true, "dimensionFieldSpecs": [ { "name": "key", "dataType": "STRING", "fieldType": "DIMENSION" }, { "name": "recordArray||name", "dataType": "STRING", "fieldType": "DIMENSION" } ], "dateTimeFieldSpecs": [ { "name": "event_time", "dataType": "LONG", "fieldType": "DATE_TIME", "format": "EPOCH|MILLISECONDS|1", "granularity": "MILLISECONDS|1" } ] } ``` ### Pinot Table Configuration ``` { "tableName": "testUnnest_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "deletedSegmentsRetentionPeriod": "0d", "segmentPushType": "APPEND", "timeColumnName": "event_time", "retentionTimeUnit": "DAYS", "retentionTimeValue": "180", "minimizeDataMovement": false, "replication": "1" }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant" }, "tableIndexConfig": { "aggregateMetrics": false, "optimizeDictionary": false, "autoGeneratedInvertedIndex": false, "enableDefaultStarTree": false, "nullHandlingEnabled": true, "skipSegmentPreprocess": false, "optimizeDictionaryType": false, "enableDynamicStarTreeCreation": false, "columnMajorSegmentBuilderEnabled": true, "createInvertedIndexDuringSegmentGeneration": true, "optimizeDictionaryForMetrics": false, "noDictionarySizeRatioThreshold": 0, "loadMode": "MMAP", "rangeIndexVersion": 2, "invertedIndexColumns": [ "key" ], "varLengthDictionaryColumns": [ "key" ] }, "metadata": {}, "ingestionConfig": { "transformConfigs": [], "enrichmentConfigs": [ { "enricherType": "generateColumn", "properties": {"fieldToFunctionMap":{"recordArray":"jsonStringToListOrMap(data_for_unnesting)"}}, "preComplexTypeTransform": true } ], "continueOnError": true, "rowTimeValueCheck": true, "complexTypeConfig": { "fieldsToUnnest": [ "recordArray" ], "delimiter": "||" }, "retryOnSegmentBuildPrecheckFailure": false, "segmentTimeValueCheck": false }, "isDimTable": false } ``` ### Data in Pinot ![](/files/6upHQY27WdvXorStt0IA) \*\* # Ingest Records with Dynamic Schemas Storing records with dynamic schemas in a table with a fixed schema. Some domains (e.g., logging) generate records where each record can have a different set of keys, whereas Pinot tables have a relatively static schema. For records with varying keys, it's impractical to store each field in its own table column. However, most (if not all) fields may be important, so fields should not be dropped unnecessarily. Additionally, searching patterns on such table could also be complex and change frequently. Exact match, range query, prefix/suffix match, wildcard search and aggregation functions could be used on any old or newly created keys or values. ## SchemaConformingTransformer The [SchemaConformingTransformer](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/SchemaConformingTransformer.java) is a [RecordTransformer](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/RecordTransformer.java) that can transform records with dynamic schemas such that they can be ingested in a table with a static schema. The transformer takes record fields that don't exist in the schema and stores them in a type of catchall field. Moreover, it builds a `__mergedTextIndex` field and takes advantage of Lucene to fulfill text search. For example, consider this record: ```json { "arrayField":[0, 1, 2, 3], "stringField":"a", "intField_noIndex":9, "string_noIndex":"z", "message": "a", "mapField":{ "arrayField":[0, 1, 2, 3], "stringField":"a", "intField_noIndex":9, "string_noIndex":"z" }, "mapField_noIndex":{ "arrayField":[0, 1, 2, 3], "stringField":"a", }, "nestedFields":{ "arrayField":[0, 1, 2, 3], "stringField":"a", "intField_noIndex":9, "string_noIndex":"z", "mapField":{ "arrayField":[0, 1, 2, 3], "stringField":"a", "intField_noIndex":9, "string_noIndex":"z" } } } ``` Let's say the table's schema contains the following fields: * arrayField * mapField * nestedFields * nestedFields.stringField * json\_data * json\_data\_no\_idx * \_\_mergedTextIndex Without this transformer, `stringField` field and fields ends with `_noIdx` would be dropped. `mapField` and `nestedFields` fields' storage needs to rely on the global setup in complexTransformers without granular customizations. However, with this transformer, the record would be transformed into the following: ```json { "arrayField":[0, 1, 2, 3], "nestedFields.stringField":"a", "json_data":{ "stringField":"a", "mapField":{ "arrayField":[0, 1, 2, 3], "stringField":"a", "stringField":"aA_123" }, "nestedFields":{ "arrayField":[0, 1, 2, 3], "mapField":{ "arrayField":[0, 1, 2, 3], "stringField":"a" } } }, "json_data_no_idx":{ "intField_noIndex":9, "string_noIndex":"z", "mapField":{ "intField_noIndex":9, "string_noIndex":"z" }, "mapField_noIndex":{ "arrayField":[0, 1, 2, 3], "stringField":"a", }, "nestedFields":{ "intField_noIndex":9, "string_noIndex":"z", "mapField":{ "intField_noIndex":9, "string_noIndex":"z" } } }, "__mergedTextIndex": [ // To be explained in following sections ] } ``` Notice that there are 3 reserved (and configurable) fields `json_data`, `json_data_no_idx` and `__mergedTextIndex`. And the transformer does the following: * Flattens nested fields all the way to the leaf node and: * Conducts special treatments if necessary according to the config * If the key path matches the schema, put the data into the dedicated field * Otherwise, put them into `json_data` or `json_data_no_idx` depending on its key suffix * For keys in dedicated columns or json\_data, puts them into `__mergedTextIndex` in the form of "Begin Anchor + value + Separator + key + End Anchor" to power the text matches. * Additional functionalities by configurations * Drop fields `fieldPathsToDrop` * Preserve the subtree without flattening `fieldPathsToPreserveInput` and `fieldPathsToPreserveInputWithIndex` * Skip storaging the fields but still indexing it (`message` in the example) `fieldPathsToSkipStorage` * Skip indexing the fields `unindexableFieldSuffix` * Optimize case insensitive search `optimizeCaseInsensitiveSearch` * Map input key path to a schema name with customizations `columnNameToJsonKeyPathMap` * Support anonymous dot, {'a.b': 'c'} vs {'a': {'b': 'c}} `useAnonymousDotInFieldNames` * Truncate value by length `mergedTextIndexDocumentMaxLength` * Double ingestion to support schema evolution `fieldsToDoubleIngest` ## Table Configurations ### SchemaConformingTransformer Configuration To use the transformer, add the `schemaConformingTransformerConfig` option in the `ingestionConfig` section of your table configuration, as shown in the following example. For example: ```json "schemaConformingTransformerConfig": { "enableIndexableExtras": true, "indexableExtrasField": "json_data", "enableUnindexableExtras": true, "unindexableExtrasField": "json_data_no_idx", "unindexableFieldSuffix": "_noindex", "fieldPathsToDrop": [], "fieldPathsToSkipStorage": [ "message" ], "columnNameToJsonKeyPathMap": {}, "mergedTextIndexField": "__mergedTextIndex", "useAnonymousDotInFieldNames": true, "optimizeCaseInsensitiveSearch": false, "reverseTextIndexKeyValueOrder": true, "mergedTextIndexDocumentMaxLength": 32766, "mergedTextIndexBinaryDocumentDetectionMinLength": 512, "mergedTextIndexPathToExclude": [ "_timestampMillisNegative", "__mergedTextIndex", "_timestampMillis" ], "fieldsToDoubleIngest": [], "jsonKeyValueSeparator": "\u001e", "mergedTextIndexBeginOfDocAnchor": "\u0002", "mergedTextIndexEndOfDocAnchor": "\u0003", "fieldPathsToPreserveInput": [], "fieldPathsToPreserveInputWithIndex": [] } ``` Available configuration options are listed in [SchemaConformingTransformerConfig](https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/ingestion/SchemaConformingTransformerConfig.java). ### Configuration of reserved fields Other index config of 3 reserved columns could be set like: ```json "fieldConfigList": [ { "name": "json_data", "encodingType": "RAW", "indexTypes": [], "compressionCodec": "LZ4", "indexes": null, "properties": { "rawIndexWriterVersion": "4" }, "tierOverwrites": null }, { "name": "json_data_no_idx", "encodingType": "RAW", "indexTypes": [], "compressionCodec": "ZSTANDARD", "indexes": null, "properties": { "rawIndexWriterVersion": "4" }, "tierOverwrites": null }, { "name": "__mergedTextIndex", "encodingType": "RAW", "indexType": "TEXT", "indexTypes": [ "TEXT" ], "compressionCodec": "LZ4", "indexes": null, "properties": { "enableQueryCacheForTextIndex": "false", "luceneAnalyzerClass": , "luceneAnalyzerClassArgTypes": <>, "luceneAnalyzerClassArgs": <>, "luceneMaxBufferSizeMB": "50", "luceneQueryParserClass": , "luceneUseCompoundFile": "true", "noRawDataForTextIndex": "true", "rawIndexWriterVersion": "4" }, "tierOverwrites": null } ] "jsonIndexConfigs": { "json_data": { "disabled": false, "maxLevels": 3, "excludeArray": true, "disableCrossArrayUnnest": true, "maxValueLength": 1000, "skipInvalidJson": true } } ``` Specifically, customizable json index could be set according to [json index indexPaths](/build-with-pinot/indexing/json-index#enable-and-configure-a-json-index). ## Power the text search ### Schema Design With the help of `SchemaConformingTransformer`, all data could be kept even without specifying special dedicated columns in table schema. However, to optimize the storage and various query patterns, dedicated columns should be created based on the usage: * Fields with frequent exact match query, e.g. region, log\_level, runtime\_env * Fields with range query, e.g. timestamp * High frequency fields from messages * Reduce json index size * Optimize group by queries ### Text Search After putting each key/value pairs into the `__mergedTextIndex` field, there will neeed to be `luceneAnalyzerClass` to tokenize the document and `luceneQueryParserClass` to query by tokens. Some example common searching patterns and their queries are: * Exact key/value match TEXT\_MATCH(\_\_mergedTextIndex, '"valuer:key"') * Wildcard value search in a key TEXT\_MATCH(\_\_mergedTextIndex, '/.\* value .\*:key/') * Key exists check TEXT\_MATCH(\_\_mergedTextIndex, '/.\*:key/') * Global value exact match TEXT\_MATCH(\_\_mergedTextIndex, '/"value"/') * Global value wildcard match TEXT\_MATCH(\_\_mergedTextIndex, '/.\* value .\*/') The `luceneAnalyzerClass` and `luceneQueryParserClass` usually need to have similar delimiter set. It also needs to consider the values below. ``` "jsonKeyValueSeparator": "\u001e", "mergedTextIndexBeginOfDocAnchor": "\u0002", "mergedTextIndexEndOfDocAnchor": "\u0003", ``` With given example, each key/value pair would be stored as "\u0002value\u001ekey\u0003". The prefix and suffix match on key or value need to be adjusted accordingly in the `luceneQueryParserClass`. # Transformations and Aggregations Use ingest-time transformations and aggregations when Pinot should normalize or reduce data before it reaches query time. Ingestion transformations clean up source records before they become Pinot rows. Ingestion aggregations reduce repeated values into fewer rows when a realtime table can safely store the summarized shape instead of the raw event stream. ## Transformations Use transformations to rename, reshape, extract, filter, or derive fields while ingesting. Keep the logic close to the table so the pipeline stays understandable. The detailed examples still live in [Ingestion Transformations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations), including [geospatial transform functions](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations#geospatial-functions) such as `ST_Point` and `geoToH3`. ## Aggregations Use ingestion aggregation when the use case only needs summarized realtime data. This can reduce storage and improve query performance, but it changes the data you keep, so use it only when raw rows are not needed later. The detailed examples still live in [Ingestion Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-aggregations). ## What this page covered This page covered when to transform or aggregate data during ingestion instead of waiting for query time. ## Next step Read the reference pages if you need the exact ingestion-config fields or table-config JSON. ## Related pages * [Ingestion](/build-with-pinot/ingestion) * [Batch Ingestion](/build-with-pinot/ingestion/batch-ingestion) * [Stream Ingestion](/build-with-pinot/ingestion/stream-ingestion) * [Formats and Filesystems](/build-with-pinot/ingestion/formats-filesystems) * [Original Ingestion Transformations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations) * [Original Ingestion Aggregations](/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-aggregations) # Ingestion Transformations Raw source data often needs to undergo some transformations before it is pushed to Pinot. Transformations include extracting records from nested objects, applying simple transform functions on certain columns, filtering out unwanted columns, as well as more advanced operations like joining between datasets. A preprocessing job is usually needed to perform these operations. In streaming data sources, you might write a Samza job and create an intermediate topic to store the transformed data. For simple transformations, this can result in inconsistencies in the batch/stream data source and increase maintenance and operator overhead. To make things easier, Pinot supports transformations that can be applied via the [table config](/reference/configuration-reference/table). {% hint style="warning" %} If you **add or change a transformed column** while realtime consumers are running, the current consuming segment can keep using the old transform plan until it commits. Use the [add a new column during ingestion](#add-a-new-column-during-ingestion) steps (or the [schema evolution decision table](/build-with-pinot/data-modeling/schema-evolution#decision-table-add-a-column-on-an-existing-table)) — pause is recommended for transform changes, but plain default-only columns often only need reload / forceCommit. {% endhint %} ## Transformation functions Pinot supports the following functions: * Groovy functions * Built-in functions {% hint style="warning" %} A transformation function cannot mix Groovy and built-in functions; only use one type of function at a time. {% endhint %} ### Groovy functions Groovy functions can be defined using the syntax: ```javascript Groovy({groovy script}, argument1, argument2...argumentN) ``` Any valid Groovy expression can be used. :warning: **Enabling Groovy** Allowing executable Groovy in ingestion transformation can be a security vulnerability. To enable Groovy for ingestion, set the following controller configuration: `controller.disable.ingestion.groovy=false` If not set, Groovy for ingestion transformation is disabled by default. #### Groovy static analysis Pinot can also apply static analysis to Groovy scripts before compiling them. This is configured with cluster-level Groovy analyzer configs: * `pinot.groovy.all.static.analyzer`: default analyzer for both query-time and ingestion-time Groovy * `pinot.groovy.ingestion.static.analyzer`: ingestion-specific override used when Pinot validates Groovy in table configs * `pinot.groovy.query.static.analyzer`: query-specific override used when Pinot validates Groovy in broker queries If the ingestion-specific or query-specific config is not set, Pinot falls back to `pinot.groovy.all.static.analyzer`. If none of these cluster configs are set, Groovy static analysis is disabled. Each static analyzer config is a JSON object with these fields: * `allowedReceivers`: fully qualified receiver classes Groovy method calls can target * `allowedImports`: fully qualified imports allowed in the script * `allowedStaticImports`: fully qualified static imports allowed in the script * `disallowedMethodNames`: method names Pinot rejects even if the receiver is otherwise allowed * `methodDefinitionAllowed`: whether Groovy method definitions are allowed inside the script Static analysis does not enable Groovy by itself. You still need `controller.disable.ingestion.groovy=false` to use Groovy in ingestion transforms. Use the controller API to inspect and update these configs: * `GET /cluster/configs/groovy/staticAnalyzerConfig/default` returns Pinot's built-in sample config * `GET /cluster/configs/groovy/staticAnalyzerConfig` returns the cluster's current Groovy static analyzer config overrides * `POST /cluster/configs/groovy/staticAnalyzerConfig` updates one or more of the Groovy analyzer configs The full request and response examples for these endpoints are in the [controller API reference](/reference/api-reference/controller-api#groovy-static-analysis-configs). ### Built-in Pinot functions Pinot supports registered scalar functions for ingestion transformations. Functions are annotated with `@ScalarFunction`; for example, [toEpochSeconds](https://github.com/apache/pinot/blob/02cb2d4970c71a2ea5b4c140a860fbf220e11bd3/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/DateTimeFunctions.java#L78). #### Immutable function requirement New or changed persisted ingestion transforms can use only scalar functions whose result depends solely on their explicit arguments (`IMMUTABLE` functions). Pinot applies this rule to: * `ingestionConfig.transformConfigs` in table configs * schema-level `transformFunction` definitions * `upsertConfig.postPartialUpsertTransformConfigs` for partial-upsert tables The check includes nested scalar function calls. For example, `now()`, `ago()`, `agoMV()`, and unseeded `rand()` are not valid in a new or changed persisted transform because they can produce different values for the same input. `rand(seed)` is valid because its result is derived from its explicit seed. Pinot keeps exact existing transform definitions running for compatibility. If you add or modify a persisted transform, replace any non-immutable function with an input-derived alternative before submitting the schema or table config. This validation does not change the functions' query-time behavior. Groovy expressions continue through Pinot's existing Groovy validation rather than scalar-function volatility metadata. Below are some commonly used built-in Pinot functions for ingestion transformations. #### DateTime functions These functions enable time transformations. **toEpochXXX** Converts from epoch milliseconds to a higher granularity. | Function name | Description | | -------------- | ------------------------------------------------------------------------ | | toEpochSeconds | Converts epoch millis to epoch seconds. Usage:`"toEpochSeconds(millis)"` | | toEpochMinutes | Converts epoch millis to epoch minutes Usage: `"toEpochMinutes(millis)"` | | toEpochHours | Converts epoch millis to epoch hours Usage: `"toEpochHours(millis)"` | | toEpochDays | Converts epoch millis to epoch days Usage: `"toEpochDays(millis)"` | **toEpochXXXRounded** Converts from epoch milliseconds to another granularity, rounding to the nearest rounding bucket. For example, `1588469352000` (2020-05-01 42:29:12) is `26474489` minutesSinceEpoch. `` `toEpochMinutesRounded(1588469352000) = 26474480 `` (2020-05-01 42:20:00) | Function Name | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------- | | toEpochSecondsRounded | Converts epoch millis to epoch seconds, rounding to nearest rounding bucket`"toEpochSecondsRounded(millis, 30)"` | | toEpochMinutesRounded | Converts epoch millis to epoch seconds, rounding to nearest rounding bucket`"toEpochMinutesRounded(millis, 10)"` | | toEpochHoursRounded | Converts epoch millis to epoch seconds, rounding to nearest rounding bucket`"toEpochHoursRounded(millis, 6)"` | | toEpochDaysRounded | Converts epoch millis to epoch seconds, rounding to nearest rounding bucket`"toEpochDaysRounded(millis, 7)"` | **fromEpochXXX** Converts from an epoch granularity to milliseconds. | Function Name | Description | | ---------------- | ----------------------------------------------------------------------------------- | | fromEpochSeconds | Converts from epoch seconds to milliseconds `"fromEpochSeconds(secondsSinceEpoch)"` | | fromEpochMinutes | Converts from epoch minutes to milliseconds `"fromEpochMinutes(minutesSinceEpoch)"` | | fromEpochHours | Converts from epoch hours to milliseconds `"fromEpochHours(hoursSinceEpoch)"` | | fromEpochDays | Converts from epoch days to milliseconds `"fromEpochDays(daysSinceEpoch)"` | **Simple date format** Converts simple date format strings to milliseconds and vice versa, per the provided pattern string. | Function name | Description | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | [ToDateTime](/functions/datetime/todatetime) | Converts from milliseconds to a formatted date time string, as per the provided pattern `"toDateTime(millis, 'yyyy-MM-dd')"` | | [FromDateTime](/functions/datetime/fromdatetime) | Converts a formatted date time string to milliseconds, as per the provided pattern `"fromDateTime(dateTimeStr, 'EEE MMM dd HH:mm:ss ZZZ yyyy')"` | {% hint style="info" %} **Note** Letters that are not part of Simple Date Time legend () need to be escaped. For example: `"transformFunction": "fromDateTime(dateTimeStr, 'yyyy-MM-dd''T''HH:mm:ss')"` {% endhint %} #### JSON functions | Function name | Description | | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | json\_format | Converts a JSON/AVRO complex object to a string. This json map can then be queried using [jsonExtractScalar](/functions/transformations) function. `"json_format(jsonMapField)"` | #### Geospatial functions Geospatial scalar functions are available as ingestion transforms. Use them to materialize points, parse WKT/WKB/GeoJSON, convert geometry/geography, or compute H3 grid ids at ingest time. Geometry and geography values are stored as `BYTES` in the schema. | Function name | Returns | Description | | ------------------------------------------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------- | | [ST\_Point](/functions/geospatial/stpoint) / `stPoint` | `BYTES` (Point) | Builds a point from `x`/`y` coordinates; optional third arg marks geography. | | [ST\_Polygon](/functions/geospatial/stpolygon) | `BYTES` (Polygon) | Parses polygon WKT into a planar polygon geometry. | | [ST\_GeomFromText](/functions/geospatial/stgeomfromtext) / `stGeomFromText` | `BYTES` (Geometry) | Parses a WKT string into a geometry. | | [ST\_GeogFromText](/functions/geospatial/stgeogfromtext) / `stGeogFromText` | `BYTES` (Geography) | Parses a WKT string into a geography. | | [ST\_GeomFromWKB](/functions/geospatial/stgeomfromwkb) / `stGeomFromWKB` | `BYTES` (Geometry) | Parses WKB bytes into a geometry. | | [ST\_GeogFromWKB](/functions/geospatial/stgeogfromwkb) / `stGeogFromWKB` | `BYTES` (Geography) | Parses WKB bytes into a geography. | | [ST\_GeomFromGeoJSON](/functions/geospatial/st_geomfromgeojson) / `stGeomFromGeoJson` | `BYTES` (Geometry) | Parses a GeoJSON string into a geometry. | | [ST\_GeogFromGeoJSON](/functions/geospatial/st_geogfromgeojson) / `stGeogFromGeoJson` | `BYTES` (Geography) | Parses a GeoJSON string into a geography. | | [ST\_AsText](/functions/geospatial/stastext) / `stAsText` | `STRING` | Serializes geometry/geography to WKT. | | [ST\_AsBinary](/functions/geospatial/stasbinary) / `stAsBinary` | `BYTES` | Serializes geometry/geography to WKB. | | [ST\_AsGeoJSON](/functions/geospatial/st_asgeojson) / `stAsGeoJson` | `STRING` | Serializes geometry/geography to GeoJSON. | | [ST\_GeometryType](/functions/geospatial/stgeometrytype) / `stGeometryType` | `STRING` | Returns the geometry type name (for example, `Point`, `Polygon`). | | `ST_Area` / `stArea` | `DOUBLE` | Computes planar area for geometry or spherical area in square meters for geography. | | [ST\_Distance](/functions/geospatial/stdistance) / `stDistance` | `DOUBLE` | Distance between two geometries (cartesian) or geographies (meters). | | [ST\_Contains](/functions/geospatial/stcontains) / `stContains` | `INT` (0/1) | Whether the first geometry/geography contains the second. | | `ST_Equals` / `stEquals` | `INT` (0/1) | Whether two geometries are equal. | | `ST_Within` / `stWithin` | `INT` (0/1) | Whether the first geometry is completely inside the second. | | [toSphericalGeography](/functions/geospatial/tosphericalgeography) | `BYTES` (Geography) | Converts a geometry object to spherical geography. | | [toGeometry](/functions/geospatial/togeometry) | `BYTES` (Geometry) | Converts a spherical geography object to geometry. | | `geoToH3` | `LONG` | H3 cell id from `(longitude, latitude, resolution)` or from a point `BYTES` plus resolution. | | [gridDistance](/functions/geospatial/griddistance) | `LONG` | H3 grid distance between two H3 indexes. | | [gridDisk](/functions/geospatial/griddisk) | `LONG[]` | H3 indexes within `k` grid steps of an origin index. | {% hint style="info" %} Function names are case-insensitive in transforms. The full geospatial function reference (including query-time usage and aggregates such as [ST\_Union](/functions/geospatial/stunion)) is in [Geospatial functions](/functions/geospatial). {% endhint %} **Example: materialize a point and an H3 grid id from longitude/latitude columns** ```javascript "ingestionConfig": { "transformConfigs": [ { "columnName": "location", "transformFunction": "ST_Point(lon, lat, 1)" }, { "columnName": "h3Index", "transformFunction": "geoToH3(lon, lat, 7)" } ] } ``` Add `location` as a `BYTES` dimension and `h3Index` as a `LONG` dimension in the schema. The third argument to `ST_Point` marks the value as geography (`1`/`true`) rather than planar geometry. **Example: parse WKT and derive H3 from an existing point column** ```javascript "ingestionConfig": { "transformConfigs": [ { "columnName": "geom", "transformFunction": "ST_GeomFromText(wkt)" }, { "columnName": "h3FromPoint", "transformFunction": "geoToH3(location, 9)" } ] } ``` ## Types of transformation ### Filtering Records can be filtered as they are ingested. A filter function can be specified in the filterConfigs in the ingestionConfigs of the table config. ```javascript "tableConfig": { "tableName": ..., "tableType": ..., "ingestionConfig": { "filterConfig": { "filterFunction": "" } } } ``` If the expression evaluates to true, the record will be filtered out. The expressions can use any of the transform functions described in the previous section. Consider a table that has a column `timestamp`. If you want to filter out records that are older than timestamp 1589007600000, you could apply the following function: ```javascript "ingestionConfig": { "filterConfig": { "filterFunction": "Groovy({timestamp < 1589007600000}, timestamp)" } } ``` Consider a table that has a string column `campaign` and a multi-value column double column `prices`. If you want to filter out records where campaign = 'X' or 'Y' and sum of all elements in prices is less than 100, you could apply the following function: ```javascript "ingestionConfig": { "filterConfig": { "filterFunction": "Groovy({(campaign == \"X\" || campaign == \"Y\") && prices.sum() < 100}, prices, campaign)" } } ``` Filter config also supports SQL-like expression of built-in [scalar functions](/functions/udf#scalar-functions) for filtering records (starting v 0.11.0+). Example: ```javascript "ingestionConfig": { "filterConfig": { "filterFunction": "strcmp(campaign, 'X') = 0 OR strcmp(campaign, 'Y') = 0 OR timestamp < 1589007600000" } } ``` ### Column transformation Transform functions can be defined on columns in the ingestion config of the table config. {% hint style="info" %} Pinot evaluates ingestion transforms against normalized extractor values, not the raw source-format encoding. For typed input formats such as Avro, Parquet, ORC, Thrift, and Protocol Buffers, booleans stay `Boolean`, `Byte` and `Short` values widen to `Integer`, logical dates and times surface as `java.time.LocalDate`, `java.time.LocalTime`, or `java.sql.Timestamp`, multi-value fields surface as `Object[]`, and maps or nested records surface as `Map`. Pinot coerces those intermediate values to the column's declared schema type after the transform step. If you have an older transform that depended on format-specific quirks such as stringified booleans or raw epoch date and time values, update the transform to handle the normalized value or cast it explicitly. For `JSONPATHSTRING`, `JSONPATHSTRINGFAST`, and `JSONPATHSTRINGFIRSTMATCH`, this means typed leaves on those already-materialized trees now render `UUID`, `LocalDate`, and `LocalTime` values as unquoted canonical or ISO-8601 strings instead of JSON-quoted strings. The change is specific to ingestion-time typed objects and does not change behavior for normal JSON-string input. {% endhint %} ```javascript { "tableConfig": { "tableName": ..., "tableType": ..., "ingestionConfig": { "transformConfigs": [{ "columnName": "fieldName", "transformFunction": "" }] }, ... } ``` For example, imagine that our source data contains the `prices` and `timestamp` fields. We want to extract the maximum price and store that in the `maxPrices` field and convert the timestamp into the number of hours since the epoch and store it in the `hoursSinceEpoch` field. You can do this by applying the following transformation: {% code title="pinot-table-offline.json" %} ```javascript { "tableName": "myTable", ... "ingestionConfig": { "transformConfigs": [{ "columnName": "maxPrice", "transformFunction": "Groovy({prices.max()}, prices)" // groovy function }, { "columnName": "hoursSinceEpoch", "transformFunction": "toEpochHours(timestamp)" // built-in function }] } } ``` {% endcode %} Below are some examples of commonly used functions. #### String concatenation Concat `firstName` and `lastName` to get `fullName` ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "fullName", "transformFunction": "Groovy({firstName+' '+lastName}, firstName, lastName)" }] } ``` #### Find an element in an array Find max value in array `bids` ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "maxBid", "transformFunction": "Groovy({bids.max{ it.toBigDecimal() }}, bids)" }] } ``` #### Time transformation Convert `timestamp` from `MILLISECONDS` to `HOURS` ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "hoursSinceEpoch", "transformFunction": "Groovy({timestamp/(1000*60*60)}, timestamp)" }] } ``` #### Column name change Change name of the column from `user_id` to `userId` ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "userId", "transformFunction": "user_id" }] } ``` #### Rename fields from a Kafka JSON message Kafka JSON payloads often use keys that aren’t great Pinot column names. Common examples are keys containing `-`, such as `event-id`. Map the source key to a schema-friendly column using `transformConfigs`. Reference the source key with a quoted identifier. ```javascript "ingestionConfig": { "transformConfigs": [ { "columnName": "event_id", "transformFunction": "\"event-id\"" }, { "columnName": "event_timestamp", "transformFunction": "\"event-timestamp\"" }, { "columnName": "user_id", "transformFunction": "\"user-id\"" } ] } ``` {% hint style="info" %} Add the destination columns (for example, `event_id`) to your Pinot schema. {% endhint %} #### Extract value from a column containing space Pinot doesn't support columns that have spaces, so if a source data column has a space, we'll need to store that value in a column with a supported name. To extract the value from `first Name` into the column `firstName`, run the following: ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "firstName", "transformFunction": "\"first Name \"" }] } ``` #### Ternary operation If `eventType` is `IMPRESSION` set `impression` to `1`. Similar for `CLICK`. ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "impressions", "transformFunction": "Groovy({eventType == 'IMPRESSION' ? 1: 0}, eventType)" }, { "columnName": "clicks", "transformFunction": "Groovy({eventType == 'CLICK' ? 1: 0}, eventType)" }] } ``` #### AVRO Map Store an AVRO Map in Pinot as two multi-value columns. Sort the keys, to maintain the mapping.\ 1\) The keys of the map as `map_keys`\ 2\) The values of the map as `map_values` ```javascript "ingestionConfig": { "transformConfigs": [{ "columnName": "map2_keys", "transformFunction": "Groovy({map2.sort()*.key}, map2)" }, { "columnName": "map2_values", "transformFunction": "Groovy({map2.sort()*.value}, map2)" }] } ``` #### Chaining transformations Transformations can be chained. This means that you can use a field created by a transformation in another transformation function. For example, we might have the following JSON document in the `data` field of our source data: ```json { "userId": "12345678__foo__othertext" } ``` We can apply one transformation to extract the `userId` and then another one to pull out the numerical part of the identifier: ```javascript "ingestionConfig": { "transformConfigs": [ { "columnName": "userOid", "transformFunction": "jsonPathString(data, '$.userId')" }, { "columnName": "userId", "transformFunction": "Groovy({Long.valueOf(userOid.substring(0, 8))}, userOid)" } ] } ``` Pinot validates these chains by dependency, not by list order. A transform can consume a column produced by another transform even if the consumer appears earlier in `transformConfigs`, as long as the dependency graph is valid. {% hint style="info" %} Intermediate transform outputs do not need schema columns when they are only consumed by later transforms. Final outputs that Pinot stores or aggregates still need schema columns. For example, the following pattern is valid even when `message_obj` is not in the schema: ```json "ingestionConfig": { "transformConfigs": [ { "columnName": "message_obj", "transformFunction": "JSONEXTRACTOBJECT(message)" }, { "columnName": "level", "transformFunction": "JSONPATHSTRING(message_obj, '$.level', null)" }, { "columnName": "msg_time", "transformFunction": "JSONPATHSTRING(message_obj, '$.time', null)" } ] } ``` In this example, `message_obj` is an intermediate column, so Pinot materializes it during transformation and drops it before indexing. The leaf columns such as `level` and `msg_time` are the stored outputs, so those columns still belong in the schema. {% endhint %} ### Flattening There are 2 kinds of flattening: #### One record into many This is not natively supported as of yet. You can **write a custom Decoder/RecordReader if you want to use this**. Once the Decoder generates the multiple GenericRows from the provided input record, a List\ should be set into the destination GenericRow, with the key `$MULTIPLE_RECORDS_KEY$`. The segment generation drivers will treat this as a special case and handle the multiple records case. #### Extract attributes from complex objects The multi-stage query engine (MSE) supports flattening array columns at query time using `CROSS JOIN UNNEST(...)`. This is the recommended approach for expanding array fields without pre-flattening data at ingestion. ```sql SET useMultistageEngine=true; SELECT t.id, elem FROM myTable AS t CROSS JOIN UNNEST(t.arrayColumn) AS u(elem) ``` For full syntax including `WITH ORDINALITY`, filtering, and aggregation after unnesting, see the [Unnest operator](/build-with-pinot/querying-and-sql/multi-stage-query/operator-types/unnest) page. ## Add a new column during ingestion This section applies when the new or changed column is populated through **ingestion transforms** (or you change transform logic) on a **realtime** table. The active consuming segment builds its transform pipeline when the mutable segment starts. Mid-segment schema/config updates do not rewrite rows already indexed in that consumer, so values in the current consuming segment can be wrong or missing until you start a new consumer. You do **not** always need a full pause for every schema add. Use the [schema evolution decision table](/build-with-pinot/data-modeling/schema-evolution#decision-table-add-a-column-on-an-existing-table): | Change | Typical action | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Plain column / `defaultNullValue` only | Update schema, then reload (consuming reload requests a force commit when allowed) or use [forceCommit](/reference/api-reference/controller-api#post-tablestablenameforcecommit) and poll it | | New or changed **transform** | Prefer pause → apply schema + table config → reload completed segments → resume; or apply schema/config → forceCommit and poll → reload the segments committed under the old plan | | Partial-upsert strategy for the new column | Update schema and strategy config together, then use a controlled server restart; immutable core upsert settings require a new table and reingestion. See [schema evolution](/build-with-pinot/data-modeling/schema-evolution). | ### Transform-safe procedure (pause boundary) To ensure accurate transformed values with a clean consumer boundary: 1. Pause consumption (and wait for pause status success): ```bash curl -X POST {controllerHost}/tables/{tableName}/pauseConsumption curl -X GET {controllerHost}/tables/{tableName}/pauseStatus ``` 2. Apply new table or schema configurations. 3. [Reload segments](/operate-pinot/segment-management/segment-reload) using the [Pinot Controller API](/operate-pinot/segment-management/segment-reload#use-the-pinot-controller-api-to-reload-segments) or [Pinot Admin Console](/operate-pinot/segment-management/segment-reload#use-the-pinot-admin-console-to-reload-segments). 4. Resume consumption: ```bash curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption ``` ### Alternative: force commit without a long pause After applying the new schema and table config, if you can tolerate committing current consumers immediately: ```bash curl -X POST {controllerHost}/tables/{tableName}/forceCommit curl -X GET {controllerHost}/tables/forceCommitStatus/{jobId} ``` Wait until `numberOfSegmentsYetToBeCommitted` is `0`, then reload the segments committed under the old transform plan so Pinot recomputes the new column. Details: [Force commit API](/reference/api-reference/controller-api#post-tablestablenameforcecommit). # Ingestion Aggregations Many data analytics use-cases only need aggregated data. For example, data used in charts can be aggregated down to one row per time bucket per dimension combination. Doing this results in much less storage and better query performance. Configuring this for a table is done via the Aggregation Config in the [table config](/reference/configuration-reference/table). {% hint style="warning" %} Note that Ingestion aggregation only works with realtime Pinot tables. Furthermore, this is done at a segment level. Cross-segment aggregation still requires query-time processing {% endhint %} ## Aggregation Config The aggregation config controls the aggregations that happen during real-time data ingestion. Offline aggregations must be handled separately. Below is a description of the config, which is defined in the ingestion config of the table config. ```json { "tableConfig": { "tableName": "...", "ingestionConfig": { "aggregationConfigs": [{ "columnName": "aggregatedFieldName", "aggregationFunction": "()" }] } } } ``` ## Requirements The following are required for ingestion aggregation to work: * Ingestion aggregation config is effective only for real-time tables. (There is no ingestion time aggregation support for offline tables. We need use [Merge/Rollup Task](/operate-pinot/segment-management/minion-merge-rollup-task) or pre-process aggregations in the offline data flow using batch processing engines like Spark/MapReduce). * [Stream ingestion](/build-with-pinot/ingestion/stream-ingestion/stream-ingestion) type must be lowLevel. * All metrics must have aggregation configs. * All metric columns must be single-value and configured as `noDictionaryColumns`. * All dimension and time columns used as aggregation keys must be single-value. They can still be configured as dictionary or no-dictionary columns in the table config. * Metrics aggregation cannot be enabled together with upsert or dedup, and it is not supported when the schema contains `COMPLEX` columns. * `aggregatedFieldName` must be in the Pinot schema and `originalFieldName` must not exist in Pinot schema {% hint style="info" %} When metrics aggregation is enabled, Pinot groups rows in the consuming segment by the dictionary ids of the dimension and time key columns. If one of those key columns is configured as no-dictionary in the table config, Pinot still creates a transient dictionary for that column in the consuming segment so ingestion aggregation can run. The committed segment is rebuilt from the table config, so committed segments still honor the configured no-dictionary setting. {% endhint %} ## Example Scenario Here is an example of sales data, where only the daily sales aggregates per product are needed. You can also find it when running RealtimeQuickStart, there is a table called `dailySales` ![](/files/VeK39E02SUdPaiWorHyl) \*\* ### Example Input Data ```json {"customerID":205,"product_name": "car","price":1500.00,"timestamp":1571900400000} {"customerID":206,"product_name": "truck","price":2200.00,"timestamp":1571900400000} {"customerID":207,"product_name": "car","price":1300.00,"timestamp":1571900400000} {"customerID":208,"product_name": "truck","price":700.00,"timestamp":1572418800000} {"customerID":209,"product_name": "car","price":1100.00,"timestamp":1572505200000} {"customerID":210,"product_name": "car","price":2100.00,"timestamp":1572505200000} {"customerID":211,"product_name": "truck","price":800.00,"timestamp":1572678000000} {"customerID":212,"product_name": "car","price":800.00,"timestamp":1572678000000} {"customerID":213,"product_name": "car","price":1900.00,"timestamp":1572678000000} {"customerID":214,"product_name": "car","price":1000.00,"timestamp":1572678000000} ``` ### Schema Note that the schema only reflects the final table structure. ```json { "schemaName": "dailySales", "dimensionFieldSpecs": [ { "name": "product_name", "dataType": "STRING" } ], "metricFieldSpecs": [ { "name": "sales_count", "dataType": "LONG" }, { "name": "total_sales", "dataType": "DOUBLE" } ], "dateTimeFieldSpecs": [ { "name": "daysSinceEpoch", "dataType": "LONG", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:MILLISECONDS" } ] } ``` ### Table Config From the below aggregation config example, note that `price` exists in the input data while `total_sales` exists in the Pinot Schema. ```json { "tableName": "daily_sales", "ingestionConfig": { "transformConfigs": [ { "columnName": "daysSinceEpoch", "transformFunction": "toEpochDays(\"timestamp\")" } ], "aggregationConfigs": [ { "columnName": "total_sales", "aggregationFunction": "SUM(price)" }, { "columnName": "sales_count", "aggregationFunction": "COUNT(*)" } ] } "tableIndexConfig": { "noDictionaryColumns": [ "sales_count", "total_sales" ] } } ``` ### Example Final Table ![](/files/TbYk4FEDzvDJgbKsLQOU) \*\* | product\_name | sales\_count | total\_sales | daysSinceEpoch | | ------------- | ------------ | ------------ | -------------- | | car | 2 | 2800.00 | 18193 | | truck | 1 | 2200.00 | 18193 | | truck | 1 | 700.00 | 18199 | | car | 2 | 3200.00 | 18200 | | truck | 1 | 800.00 | 18202 | | car | 3 | 3700.00 | 18202 | ## Allowed Aggregation Functions | function name | notes | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | MAX | | | MIN | | | SUM | | | COUNT | Specify as `COUNT(*)` | | DISTINCTCOUNTHLL | Specify as `DISTINCTCOUNTHLL(field, log2m)`, default is 12. See [function reference](/functions/aggregation/distinctcounthll) for how to define `log2m`. Cannot be changed later, a new field must be used. The schema for the output field should be `BYTES` type. | | DISTINCTCOUNTHLLPLUS | Specify as `DISTINCTCOUNTHLLPLUS(field, s, p)`. See [function reference](/functions/sketch/distinctcounthllplus) for how to define `s` and `p`, they cannot be changed later. The schema for the output field should be `BYTES` type. | | SUMPRECISION | Specify as `SUMPRECISION(field, precision)`, precision must be defined. Used to compute the maximum possible size of the field. Cannot be changed later, a new field must be used. The schema for the output field should be `BIG_DECIMAL` type. | ## Frequently Asked Questions ### Why not use a Startree? Startrees can only be added to real-time segments after the segments has sealed, and creating startrees is CPU-intensive. Ingestion Aggregation works for consuming segments and uses no additional CPU. Startrees take additional memory to store, while ingestion aggregation stores less data than the original dataset. ### When to not use ingestion aggregation? If the original rows in non-aggregated form are needed, then ingestion-aggregation cannot be used. ### I already use the `aggregateMetrics` setting? The `aggregateMetrics` works the same as Ingestion Aggregation, but only allows for the SUM function. The current changes are backward compatible, so no need to change your table config unless you need a different aggregation function. ### Does this config work for offline data? Ingestion Aggregation only works for real-time ingestion. For offline data, the offline process needs to generate the aggregates separately. ### Why do all metrics need to be aggregated? If a metric isn't aggregated then it will result in more than one row per unique set of dimensions. ### Can aggregation key columns be no-dictionary? Yes. Single-value dimension and time columns can be configured as no-dictionary columns and still participate in ingestion aggregation. Pinot creates transient dictionaries for those key columns only in the consuming segment so it can build the aggregation key. Metric columns are different: they must remain single-value no-dictionary columns because the aggregated values are updated in place. ### Why no data show up when I enabled AggregationConfigs? 1. Check if ingestion is normal without AggregationConfigs, this is to isolate the problem 2. Check Pinot Server log for any warning or error log, especially related to class `MutableSegmentImpl`and method `aggregateMetrics`. 3. For JSON data, please ensure you don't double quote numbers, as they are parsed as string internally and won't be able to do the value based aggregation, e.g. sum. Using the above example, data ingestion not working with row: `{"customerID":205,"product_name": "car","price":"1500.00","timestamp":1571900400000}` , the major issue here is that price number is double quoted so it won't show up. Below is a sample stacktrace: ``` 2024/11/04 00:24:27.760 ERROR [RealtimeSegmentDataManager_dailySales__0__0__20241104T0824Z] [dailySales__0__0__20241104T0824Z] Caught exception while indexing the record at offset: 9 , row: { "fieldToValueMap" : { "price" : "1000.00", "daysSinceEpoch" : 18202, "sales_count" : 0, "total_sales" : 0.0, "product_name" : "car", "timestamp" : 1572678000000 }, "nullValueFields" : [ "sales_count", "total_sales" ] } java.lang.ClassCastException: class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap') at org.apache.pinot.segment.local.aggregator.SumValueAggregator.applyRawValue(SumValueAggregator.java:25) ~[classes/:?] at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.aggregateMetrics(MutableSegmentImpl.java:855) ~[classes/:?] at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:577) ~[classes/:?] at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.processStreamEvents(RealtimeSegmentDataManager.java:641) ~[classes/:?] at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.consumeLoop(RealtimeSegmentDataManager.java:477) ~[classes/:?] at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:734) ~[classes/:?] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] ``` # Querying & SQL Learn how to query Apache Pinot, choose the right query engine, and find SQL and function guidance quickly. Use this section to decide how to query Pinot, how much SQL support you need, which query engine to use, and where to look for execution controls such as quotas, cancellation, and cursors. Narrative guidance lives here. Dense syntax and endpoint detail is linked where needed. ## Start here {% content-ref url="/pages/-LtSZUOogdMvPabsf93k" %} [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) {% endcontent-ref %} {% content-ref url="/pages/l6j6tUAH628nqyQD4rSh" %} [SQL syntax](/build-with-pinot/querying-and-sql/sql-syntax) {% endcontent-ref %} {% content-ref url="/pages/sNyqXVWInQCk7v2xHswX" %} [SQL DDL](/build-with-pinot/querying-and-sql/sql-syntax/sql-ddl) {% endcontent-ref %} {% content-ref url="/pages/Az3jdJGQ6HGwR4rG8mN7" %} [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views) {% endcontent-ref %} {% content-ref url="/pages/kaS1dx1rjV1tKqPDNIQj" %} [Overview](/functions/functions) {% endcontent-ref %} {% content-ref url="/pages/qPhpMkSMXnNy4Sy2K07g" %} [Query Engines (SSE vs MSE)](/build-with-pinot/querying-and-sql/sse-vs-mse) {% endcontent-ref %} {% content-ref url="/pages/Zx6h36wDZpJKu2AKV1MB" %} [Query options, quotas, cancellation & cursors](/build-with-pinot/querying-and-sql/query-execution-controls) {% endcontent-ref %} ## Deep dives For explain plans, joins, optimizer behavior, and operator details, continue into the multi-stage query docs and engine-specific material linked from [SSE vs MSE](/build-with-pinot/querying-and-sql/sse-vs-mse) and [SQL syntax](/build-with-pinot/querying-and-sql/sql-syntax). ## What this page covered This page mapped the main query workflows in Pinot: learning the query path, understanding SQL behavior, finding functions, choosing between SSE and MSE, using controller-managed SQL DDL for tables and materialized views, and tuning execution controls. ## Next step Read [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) if you want the end-to-end query flow, or [SSE vs MSE](/build-with-pinot/querying-and-sql/sse-vs-mse) if you are deciding which engine to use. ## Related pages * [Build with Pinot](/build-with-pinot/build-with-pinot) * [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views) * [Functions](/functions/functions) * [Reference](/reference/reference) # Querying Pinot A practical entry point for querying Pinot. Pinot queries run through the broker and are written in SQL. This page is the wayfinding layer for people who want to query data, understand which engine to use, and know where to look when a query needs tuning. ## How to start 1. Write the query in Pinot SQL. 2. Decide whether the single-stage engine is enough or whether you need multi-stage features such as joins and subqueries. 3. Use query options to control runtime behavior. 4. Inspect the plan or result shape when you need to debug performance. ```sql SET useMultistageEngine = true; SELECT city, COUNT(*) FROM stores GROUP BY city LIMIT 10; ``` ## What matters most Pinot SQL uses the Apache Calcite parser with the `MYSQL_ANSI` dialect. In practice, that means you should pay attention to identifier quoting, literal quoting, and engine-specific capabilities. If you are debugging a slow or surprising query, the most useful follow-up pages are: * [SQL syntax](/build-with-pinot/querying-and-sql/sql-syntax) * [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) * [Grouping algorithm](/build-with-pinot/querying-and-sql/sql-syntax/grouping-algorithm) * [Query quotas](/build-with-pinot/querying-and-sql/query-execution-controls/query-quotas) * [Query cancellation](/build-with-pinot/querying-and-sql/query-execution-controls/query-cancellation) * [Cursor pagination](/build-with-pinot/querying-and-sql/query-execution-controls/query-using-cursors) * [Correlation IDs](/build-with-pinot/querying-and-sql/query-execution-controls/query-correlation-id) * [Explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan) * [Multi-stage explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan-multi-stage) * [SSE vs MSE](/build-with-pinot/querying-and-sql/sse-vs-mse) ## Group-by quirks (default LIMIT, trimming, ORDER BY) These behaviors catch many users by surprise. Details and tuning knobs live in [Grouping algorithm](/build-with-pinot/querying-and-sql/sql-syntax/grouping-algorithm) and [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options). ### Default LIMIT is 10 On the **single-stage engine (SSE)**, if a query omits `LIMIT`, the broker applies a default of **10** rows (`pinot.broker.default.query.limit`). This applies to selection queries and `GROUP BY` queries, so an SSE group-by without an explicit `LIMIT` returns at most 10 groups. ```sql -- SSE: returns at most 10 groups even if many cities exist SELECT city, COUNT(*) AS cnt FROM stores GROUP BY city; ``` The **multi-stage engine (MSE)** does not apply this broker default. Still set an explicit `LIMIT` whenever you care about result size or want intentional truncation. ### Tail trimming on GROUP BY For SSE group-by with `ORDER BY`, Pinot may **trim tail groups** while aggregating so servers stay within memory limits. Where trimming is enabled, the candidate size is based on `max(minTrimSize, 5 * LIMIT)`; segment trimming is disabled by default, while server and broker reduction have their own defaults. Pinot ranks candidates using the query's `ORDER BY`. See [Grouping algorithm](/build-with-pinot/querying-and-sql/sql-syntax/grouping-algorithm) for the stage-specific settings. Implications: * Results can be approximate when cardinality is high relative to `LIMIT` and trim thresholds. * `numGroupsLimitReached=true` means a group operator reached its hard group cap; increasing a trim size cannot recover groups already dropped at that cap. * Raise the relevant trim sizes when you need a larger candidate set, or raise `LIMIT` when you also need more rows returned. Both choices increase memory use. ### GROUP BY with ORDER BY vs without ORDER BY | Pattern | Execution behavior | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GROUP BY ... ORDER BY ... LIMIT N` | Each trim stage ranks candidate groups by the `ORDER BY` expressions and drops lower-ranked candidates. This is the intended top-N shape, although distributed trimming can still make results approximate when candidate sets are too small. | | `GROUP BY ... LIMIT N` (**no** `ORDER BY`) | There is no ranking. By default, SSE result tables stop admitting unseen group keys after reaching their result size, so processing order can affect which N keys survive. | ```sql -- Ordered top-N cities by count (trim keeps high counts when ORDER BY matches the ranking) SELECT city, COUNT(*) AS cnt FROM stores GROUP BY city ORDER BY cnt DESC LIMIT 20; -- Unordered: at most 20 groups, but not a defined "top" set SELECT city, COUNT(*) AS cnt FROM stores GROUP BY city LIMIT 20; ``` Do **not** assume a stable or ranked group set without `ORDER BY`. For a deterministic subset, set `accurateGroupByWithoutOrderBy=true`; SSE then keeps the lexicographically smallest group keys during server and broker reduction. This does not rank by an aggregate and cannot recover keys dropped by `numGroupsLimit`. See [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options). ### HAVING and post-aggregation On SSE, `HAVING` filters the merged group candidates **after** aggregation and any earlier group trimming. If `HAVING` prefers groups that trimming already dropped (for example `HAVING SUM(x) < 100` with `ORDER BY SUM(x) DESC`), matching groups may be missing—increase trim sizes or adjust ordering. ```sql SELECT city, COUNT(*) AS cnt, SUM(revenue) AS total FROM stores GROUP BY city HAVING COUNT(*) > 100 ORDER BY total DESC LIMIT 50; ``` **Post-aggregation** expressions combine aggregated values (and group keys) in `SELECT`, `HAVING`, or `ORDER BY` after the aggregates are computed: ```sql SELECT city, SUM(revenue) AS total, COUNT(*) AS cnt, SUM(revenue) / COUNT(*) AS avg_revenue FROM stores GROUP BY city HAVING SUM(revenue) / COUNT(*) > 10 ORDER BY avg_revenue DESC LIMIT 50; ``` ### Query options (names and syntax) Use `SET` statements or `OPTION (...)` to pass per-query options. Recognized keys resolve **case-insensitively** to the canonical camelCase names listed in [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) (for example `timeoutMs`, `numGroupsLimit`, `minSegmentGroupTrimSize`, `useMultistageEngine`). Prefer canonical spelling: unknown names can be accepted but ignored. ```sql SET timeoutMs = 5000; SET minSegmentGroupTrimSize = 5000; SELECT city, COUNT(*) AS cnt FROM stores GROUP BY city ORDER BY cnt DESC LIMIT 100; ``` ## When to use which engine Single-stage execution is the default path for straightforward filtering, aggregation, and top-K style queries. Use multi-stage execution when you need features that are not available in single-stage mode, such as: * joins * subqueries * window functions * more complex distributed query shapes As a rule of thumb: use SSE for simple filtering, aggregation, and top-K queries; use MSE when your query shape requires joins, subqueries, window functions, or other advanced relational operators. For a detailed comparison, see [SSE vs MSE](/build-with-pinot/querying-and-sql/sse-vs-mse). ## Next step Read [SQL syntax](/build-with-pinot/querying-and-sql/sql-syntax) for the query language itself, then move to [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) or [Explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan) when you need control or diagnostics. ## Related pages * [Querying & SQL controls](/build-with-pinot/querying-and-sql/query-execution-controls) * [SQL syntax](/build-with-pinot/querying-and-sql/sql-syntax) * [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) * [Grouping algorithm](/build-with-pinot/querying-and-sql/sql-syntax/grouping-algorithm) * [Explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan) * [Multi-stage explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan-multi-stage) * [SSE vs MSE](/build-with-pinot/querying-and-sql/sse-vs-mse) * [SQL syntax and operators reference](/build-with-pinot/querying-and-sql/sql-syntax/sql-reference) ## Identifier vs Literal In Pinot SQL: * **Double quotes(")** are used to force string identifiers, e.g. column names * **Single quotes(')** are used to enclose string literals. If the string literal also contains a single quote, escape this with a single quote e.g `'''Pinot'''` to match the string literal `'Pinot'` Misusing those might cause unexpected query results, like the following examples: * `WHERE a='b'` means the predicate on the column `a` equals to a string literal value `'b'` * `WHERE a="b"` means the predicate on the column `a` equals to the value of the column `b` If your column names use reserved keywords (e.g. `timestamp` or `date`) or special characters, you will need to use double quotes when referring to them in queries. Note: Define decimal literals within quotes to preserve precision. ## Example Queries ### Selection ``` //default to limit 10 SELECT * FROM myTable SELECT * FROM myTable LIMIT 100 ``` ``` SELECT "date", "timestamp" FROM myTable ``` ### Aggregation ```sql SELECT COUNT(*), MAX(foo), SUM(bar) FROM myTable ``` ### Grouping on Aggregation ```sql SELECT MIN(foo), MAX(foo), SUM(foo), AVG(foo), bar, baz FROM myTable GROUP BY bar, baz LIMIT 50 ``` {% hint style="info" %} On the single-stage engine, omitting `LIMIT` defaults to **10 groups**. Without `ORDER BY`, which group keys survive a small result size can depend on processing order. See [Group-by quirks](#group-by-quirks-default-limit-trimming-order-by). {% endhint %} ### Ordering on Aggregation ```sql SELECT MIN(foo), MAX(foo), SUM(foo), AVG(foo), bar, baz FROM myTable GROUP BY bar, baz ORDER BY bar, MAX(foo) DESC LIMIT 50 ``` ### Filtering groups with HAVING ```sql SELECT bar, SUM(foo) AS total FROM myTable GROUP BY bar HAVING SUM(foo) > 1000 ORDER BY total DESC LIMIT 50 ``` ### Post-aggregation expressions ```sql SELECT bar, SUM(foo) AS total, COUNT(*) AS cnt, SUM(foo) / COUNT(*) AS avg_foo FROM myTable GROUP BY bar ORDER BY avg_foo DESC LIMIT 50 ``` ### Filtering ```sql SELECT COUNT(*) FROM myTable WHERE foo = 'foo' AND bar BETWEEN 1 AND 20 OR (baz < 42 AND quux IN ('hello', 'goodbye') AND quuux NOT IN (42, 69)) ``` For performant filtering of IDs in a list, see [Filtering with IdSet](/build-with-pinot/querying-and-sql/sql-syntax/filtering-with-idset). ### Filtering with NULL predicate ```sql SELECT COUNT(*) FROM myTable WHERE foo IS NOT NULL AND foo = 'foo' AND bar BETWEEN 1 AND 20 OR (baz < 42 AND quux IN ('hello', 'goodbye') AND quuux NOT IN (42, 69)) ``` ### Selection (Projection) ```sql SELECT * FROM myTable WHERE quux < 5 LIMIT 50 ``` ### Ordering on Selection ```sql SELECT foo, bar FROM myTable WHERE baz > 20 ORDER BY bar DESC LIMIT 100 ``` ### Pagination on Selection Note that results might not be consistent if the `ORDER BY` column has the same value in multiple rows. ```sql SELECT foo, bar FROM myTable WHERE baz > 20 ORDER BY bar DESC LIMIT 50, 100 ``` ### Wild-card match (in WHERE clause only) The example below counts rows where the column `airlineName` starts with `U`: ```sql SELECT COUNT(*) FROM myTable WHERE REGEXP_LIKE(airlineName, '^U.*') GROUP BY airlineName LIMIT 10 ``` **Note:** REGEXP\_LIKE also supports case insensitive search using the `i` flag as the third parameter. ### Case-When Statement Pinot supports the `CASE-WHEN-ELSE` statement, as shown in the following two examples: ```sql SELECT CASE WHEN price > 30 THEN 3 WHEN price > 20 THEN 2 WHEN price > 10 THEN 1 ELSE 0 END AS price_category FROM myTable ``` ```sql SELECT SUM( CASE WHEN price > 30 THEN 30 WHEN price > 20 THEN 20 WHEN price > 10 THEN 10 ELSE 0 END) AS total_cost FROM myTable ``` ### UDF Pinot doesn't currently support injecting functions. Functions have to be implemented within Pinot, as shown below: ```sql SELECT COUNT(*) FROM myTable GROUP BY DATETIMECONVERT(timeColumnName, '1:MILLISECONDS:EPOCH', '1:HOURS:EPOCH', '1:HOURS') ``` For more examples, see [Transform Function in Aggregation Grouping](/functions/transformations). ### BYTES column Pinot supports queries on BYTES column using hex strings. The query response also uses hex strings to represent bytes values. The query below fetches all the rows for a given UID: ```sql SELECT * FROM myTable WHERE UID = 'c8b3bce0b378fc5ce8067fc271a34892' ``` # SQL syntax A narrative guide to Pinot SQL syntax and the main constructs you use most often. Pinot uses the Apache Calcite SQL parser with the `MYSQL_ANSI` dialect. This page is the practical overview: it explains the syntax patterns most people use every day and points to the deeper reference when you need the full operator list. ## Core rules * Use single quotes for string literals. * Use double quotes for identifiers when a column name is reserved or contains special characters. * `SET` statements apply query options before the query runs. * `EXPLAIN PLAN FOR` shows how Pinot will execute a query without returning data. ```sql SET useMultistageEngine = true; SELECT "date", city, COUNT(*) FROM orders WHERE status = 'shipped' GROUP BY "date", city ORDER BY "date" DESC LIMIT 20; ``` ## Common query shapes Pinot supports the usual `SELECT`, `WHERE`, `GROUP BY`, `ORDER BY`, and `LIMIT` patterns. Typical query shapes include: * filtering a table and returning a small result set * grouping and aggregating by one or more dimensions * using `ORDER BY` to rank rows before a `LIMIT` * using `CASE WHEN` and scalar functions in select lists ## Engine-aware syntax Some SQL features depend on the engine: * single-stage execution is best for simple analytic queries * multi-stage execution is required for joins, subqueries, and several advanced distributed patterns * `EXPLAIN PLAN FOR` is the best way to see how Pinot interprets a statement If you are working on a query and do not know whether a feature is supported, check the engine-specific guidance before you assume the syntax is invalid. ## DDL runs on the controller Pinot also supports controller-managed SQL DDL, but it is exposed through the controller rather than the broker query path. Use `POST /sql/ddl` for table statements such as `CREATE TABLE`, `DROP TABLE`, `SHOW TABLES`, and `SHOW CREATE TABLE`, and for materialized-view statements such as `CREATE MATERIALIZED VIEW`, `SHOW MATERIALIZED VIEWS`, `SHOW CREATE MATERIALIZED VIEW`, and `DROP MATERIALIZED VIEW`. This distinction matters because the SSE and MSE engines still execute query statements, not controller metadata changes. If you want the syntax and examples for controller-managed DDL, use [SQL DDL](/build-with-pinot/querying-and-sql/sql-syntax/sql-ddl) and [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views). ## Where the details live This page intentionally stays light. For the full statement-by-statement reference, use the detailed [SQL syntax and operators reference](/build-with-pinot/querying-and-sql/sql-syntax/sql-reference). For controller-managed DDL, use [SQL DDL](/build-with-pinot/querying-and-sql/sql-syntax/sql-ddl). For MV-specific workflow guidance, use [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views). For query controls and diagnostics, use the pages under `query-execution-controls/`. ## What this page covered This page covered the main Pinot SQL rules, the most common statement patterns, and the difference between narrative guidance and the full SQL reference. ## Next step Read [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) for the broader query workflow, or jump to [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) if you want to control runtime behavior. ## Related pages * [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) * [SQL DDL](/build-with-pinot/querying-and-sql/sql-syntax/sql-ddl) * [Materialized Views](/build-with-pinot/querying-and-sql/materialized-views) * [Query options](/build-with-pinot/querying-and-sql/query-execution-controls/query-options) * [Explain plan](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan) # SQL DDL Use controller-managed SQL DDL to create, inspect, list, and drop Pinot tables and materialized views. Pinot supports a controller-managed SQL DDL surface for table and materialized-view metadata operations. Use it when you want a SQL alternative to the JSON-based `/schemas` and `/tables` workflows. The controller accepts one DDL statement per request: * `CREATE TABLE` * `DROP TABLE` * `SHOW TABLES` * `SHOW CREATE TABLE` * `CREATE MATERIALIZED VIEW` * `SHOW MATERIALIZED VIEWS` * `SHOW CREATE MATERIALIZED VIEW` * `DROP MATERIALIZED VIEW` {% hint style="info" %} Run these statements through the controller endpoint `POST /sql/ddl`, not through the broker query API. SSE and MSE still execute query statements; the controller owns table and materialized-view DDL. {% endhint %} ## How it works `POST /sql/ddl` compiles SQL into the same Pinot `Schema` and `TableConfig` model used by the existing controller APIs. That means: * DDL-created tables go through the same controller validation path as `POST /tables`. * DDL-created materialized views persist as regular `OFFLINE` tables plus the same MV task config the JSON APIs use. * `SHOW CREATE TABLE` renders a canonical SQL form of the stored schema and table config that you can review, version, or replay. * `SHOW CREATE MATERIALIZED VIEW` renders canonical MV DDL for the stored definition, including the `AS ` clause and MV-specific properties. * `dryRun=true` lets you compile and validate without persisting any metadata. * Existing broker query APIs still handle `SELECT` statements. The controller endpoint handles metadata changes. ## Supported statement shapes | Statement | Notes | | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | \`CREATE TABLE \[IF NOT EXISTS] \[db.]table (...) \[PRIMARY KEY (...)] TABLE\_TYPE = OFFLINE | REALTIME \[PROPERTIES (...)]\` | | `CREATE TABLE [IF NOT EXISTS] [db.]table WITH (key = value, ...)` | Extension-only form for options-defined tables. A Pinot distribution can install a handler that derives the schema and table config entirely from the `WITH` options. Apache Pinot OSS does not install such a handler, so the default behavior is to reject this form and tell you to use the column-list `TABLE_TYPE = ...` form instead. | | \`DROP TABLE \[IF EXISTS] \[db.]table \[TYPE OFFLINE | REALTIME]\` | | `SHOW TABLES [FROM db]` | Lists tables in the selected database. | | \`SHOW CREATE TABLE \[db.]table \[TYPE OFFLINE | REALTIME]\` | | `CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db.]name [(...)] [REFRESH [INTERVAL] EVERY ...] PROPERTIES (...) AS ` * `SHOW MATERIALIZED VIEWS [FROM db]` * `SHOW CREATE MATERIALIZED VIEW [db.]name` * `DROP MATERIALIZED VIEW [IF EXISTS] [db.]name` If you omit the column list, Pinot infers the MV schema from the `SELECT` projection. If you provide a column list, declare every projected column and alias every computed expression or aggregation so it matches the destination column name. Schema inference is part of the built-in single-source handler. A custom handler can require an explicit column list instead. The DDL needs at least these properties: * `timeColumnName`: the MV column Pinot uses to track watermark progress. * `bucketTimePeriod`: the materialization window size, such as `1h` or `1d`. * `stalenessThresholdMs`: optional freshness SLO for broker rewrite. `0` disables the SLO check. `REFRESH EVERY` is optional. When you provide it, Pinot stores a per-MV schedule using minute, hour, or day units. When you omit it, the MV runs on the cluster-wide `MaterializedViewTask` schedule. Example DDL: ```sql CREATE MATERIALIZED VIEW salesByHourMv REFRESH EVERY 1 HOUR PROPERTIES ( 'timeColumnName' = 'bucket_start_ts', 'bucketTimePeriod' = '1h', 'stalenessThresholdMs' = '900000', 'replication' = '1' ) AS SELECT DATETRUNC('HOUR', event_ts) AS bucket_start_ts, region, SUM(revenue) AS sum_revenue, COUNT(*) AS row_count FROM sales GROUP BY DATETRUNC('HOUR', event_ts), region; ``` Submit the statement through the controller: ```bash curl -X POST "http://localhost:9000/sql/ddl" \ -H "accept: application/json" \ -H "Content-Type: application/json" \ -d @- <<'EOF' {"sql":"CREATE MATERIALIZED VIEW salesByHourMv REFRESH EVERY 1 HOUR PROPERTIES ('timeColumnName' = 'bucket_start_ts', 'bucketTimePeriod' = '1h', 'stalenessThresholdMs' = '900000', 'replication' = '1') AS SELECT DATETRUNC('HOUR', event_ts) AS bucket_start_ts, region, SUM(revenue) AS sum_revenue, COUNT(*) AS row_count FROM sales GROUP BY DATETRUNC('HOUR', event_ts), region"} EOF ``` Pinot generates MV segments through `MaterializedViewTask`. The controller task manager can schedule those tasks automatically, or you can trigger them manually: ``` POST /tasks/schedule?taskType=MaterializedViewTask&tableName=_OFFLINE ``` ## Understand recovery and tuning When a retention delete or an empty refresh clears a covered MV bucket, Pinot keeps tracking that bucket as an empty covered partition instead of dropping it outright. If source data is later backfilled into the same time window, the controller consistency manager re-marks that bucket `STALE` and the next overwrite cycle rebuilds it. By default, the consistency manager runs this empty-bucket recovery sweep every `300000` ms (5 minutes). You can change that interval live, without restarting controllers, through the cluster config key `pinot.materialized.view.consistency.empty.sweep.interval.ms`. Set it with `pinot-admin.sh ClusterConfig` or the controller `/cluster/configs` endpoint. Non-positive values fall back to the 5-minute default. ## When to keep using JSON APIs The existing `POST /schemas`, `POST /tables`, and `PUT /tables/{tableName}` APIs still work for materialized views. Keep using them when your automation already depends on raw Pinot metadata payloads, or when you need a hand-written `MaterializedViewTask` cron that cannot be expressed as `REFRESH EVERY MINUTES|HOURS|DAYS` or `'m|h|d'`. If you are building a downstream extension that needs a different MV task type or query-engine contract, see [Plugins](/develop-and-contribute/plugin-architecture). Custom `MaterializedViewDdlHandler` implementations own that alternate task wiring and any engine-specific validation. ## Enable transparent rewrite for SSE queries To let brokers rewrite eligible base-table queries to a materialized view, set this broker config: ```properties pinot.broker.query.enable.materialized.view.rewrite=true ``` With that switch enabled, Pinot still falls back to the base table unless the MV has usable coverage. In practice, the MV must have a non-zero watermark, and if you set `stalenessThresholdMs`, the MV must still be within that freshness bound. {% hint style="warning" %} Today Pinot splits MV rewrites by watermark only. If a bucket below the watermark is already marked `STALE` or is still tracked as an empty covered bucket waiting for re-materialization, Pinot can still route that time range to the MV until the next overwrite cycle completes. The exposure window is bounded by the consistency-manager debounce plus one scheduling cycle, and delete/backfill races can add up to one empty-bucket recovery sweep interval. {% endhint %} Pinot currently rewrites eligible SSE query shapes that are subsumed by the MV, including exact matches, projection-subset scan queries, and supported aggregation rollups. Scalar expressions used as grouping keys, such as `DATETRUNC`, `UPPER`, `LOWER`, and `SUBSTR`, can match the corresponding projected expression in the MV. Aggregate expressions still require a supported aggregation rollup rule. When a rewrite happens, the broker response includes `materializedViewQueried` with the MV table name that served the query. If you need one eligible query to stay on the base table while leaving broker-side rewrite enabled for everything else, set `enableMaterializedViewRewrite=false` on that query: ```sql SET enableMaterializedViewRewrite = false; SELECT region, SUM(revenue) AS total_revenue, COUNT(*) AS total_rows FROM sales GROUP BY region ORDER BY total_revenue DESC LIMIT 20; ``` This option defaults to `true`, so queries only opt out when they set it to `false`. It disables MV rewrite for that query only and forces the normal base-table path. Pinot also uses the same option internally for the `MaterializedViewTask` materialization query so the minion always reads from the base table instead of rewriting back onto an MV. For example, once the MV is built and the broker switch is on, this base-table query can be served by the MV without changing the SQL: ```sql SELECT region, SUM(revenue) AS total_revenue, COUNT(*) AS total_rows FROM sales GROUP BY region ORDER BY total_revenue DESC LIMIT 20; ``` The response includes the MV table name when rewrite succeeds: ```json { "materializedViewQueried": "salesByHourMv_OFFLINE" } ``` Scalar grouping expressions can also participate in the rewrite. For example, an MV defined with `DATETRUNC('DAY', event_ts)` as a projected grouping key can serve this base-table query: ```sql SELECT DATETRUNC('DAY', event_ts) AS event_day, SUM(revenue) AS total_revenue FROM sales GROUP BY DATETRUNC('DAY', event_ts); ``` The scalar expression must match a projected MV expression. Wrapping an aggregate in another function, such as `ROUND(SUM(revenue))`, is an aggregate expression and rewrites only when Pinot has a compatible aggregation equivalence rule. ## Query the MV table directly You can still query the MV table name directly and re-aggregate its stored values as needed: ```sql SELECT region, SUM(sum_revenue) AS total_revenue, SUM(row_count) AS total_rows FROM salesByHourMv WHERE bucket_start_ts BETWEEN 1746057600000 AND 1746144000000 GROUP BY region ORDER BY total_revenue DESC LIMIT 20; ``` If your MV stores raw sketch columns, query them with the matching merge function on the MV table: * `DISTINCTCOUNTHLL(raw_hll_col)` * `DISTINCTCOUNTHLLPLUS(raw_hllplus_col)` * `DISTINCTCOUNTTHETASKETCH(raw_theta_col)` ## Try the bundled quickstart Pinot ships a full local example that loads the base table, creates the MV table, runs the minion task, and compares base-table answers with MV-table re-aggregation: ```bash bin/pinot-admin.sh QuickStart -type MATERIALIZED_VIEW ``` The quickstart creates `airlineStatsMv`, triggers `MaterializedViewTask`, and gives you a local setup for validating direct MV queries. After you enable the broker rewrite switch, it is also a convenient way to test transparent rewrite behavior. ## Inspect and manage materialized views In the Data Explorer, use **Data Sources** to discover both physical tables and materialized views: * **Data Sources** shows cards for **Tables** and **Materialized Views**. * **Materialized Views** lists each MV with its base tables, watermark, VALID and STALE partition counts, last refresh time, staleness SLO, and any metadata errors. * Clicking an MV opens a detail page with the stored `definedSQL`, split spec, partition state, raw runtime metadata, and controls to refresh the page data or drop the MV. The same controller DDL surface also lets you inspect and remove MVs from SQL: ```sql SHOW MATERIALIZED VIEWS; SHOW CREATE MATERIALIZED VIEW salesByHourMv; DROP MATERIALIZED VIEW IF EXISTS salesByHourMv; ``` `SHOW MATERIALIZED VIEWS` returns raw MV names without the `_OFFLINE` suffix. `SHOW CREATE MATERIALIZED VIEW` emits canonical DDL for the stored MV definition, including an explicit column list even when the original `CREATE MATERIALIZED VIEW` used inferred columns. The controller also exposes dedicated MV endpoints: * `GET /materializedViews` * `GET /materializedViews/{materializedViewTableName}` * `DELETE /materializedViews/{materializedViewTableName}` ## What this page covered This page covered the current materialized-view feature surface in Pinot: how to create and manage an MV through controller SQL DDL, which source tables and aggregations are supported, how broker-side SSE rewrite works, how to query the MV table directly, and where to inspect the MV in the UI and controller API. ## Next step Read [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) for the broader query path, or [Pinot Data Explorer](/architecture-and-concepts/components/exploring-pinot) for the UI walkthrough. ## Related pages * [Querying & SQL](/build-with-pinot/querying-and-sql) * [Querying Pinot](/build-with-pinot/querying-and-sql/querying-pinot) * [Pinot Data Explorer](/architecture-and-concepts/components/exploring-pinot) * [Functions](/functions/functions) # Multi-Stage Query Deep dive into the multi-stage engine (MSE) internals, execution model, and troubleshooting. For an overview of when to use the multi-stage engine (MSE) versus the single-stage engine (SSE), see [Query Engines (SSE vs MSE)](/build-with-pinot/querying-and-sql/sse-vs-mse). This section provides a deep dive into MSE internals. Most of the concepts explained here are related to the engine's execution model and are not required for writing queries. However, understanding them can help you take advantage of MSE's capabilities and troubleshoot issues. # Understanding Stages Learn more about multi-stage stages and how to extract stages from query plans. ### Deep dive into stages As explained in the [Multi-stage query engine](/build-with-pinot/querying-and-sql/sse-vs-mse) reference documentation, the multi-stage query engine breaks down a query into multiple stages. Each stage corresponds to a subset of the query plan and is executed independently. Stages are connected in a tree-like structure where the output of one stage is the input to another stage. The stage that is at the root of the tree sends the final results to the client. The stages that are at the leaves of the tree read from the tables. The intermediate stages process the data and send it to the next stage. When the broker receives a query, it generates a query plan. This is a tree-like structure where each node is an operator. The plan is then optimized, moving and changing nodes to generate a plan that is semantically equivalent (it returns the same rows) but more efficient. During this phase the broker colors the nodes of the plan, assigning them to a stage. The broker also assigns a parallelism to each stage and defines which servers are going to execute each stage. For example, if a stage has a parallelism of 10, then at most 10 servers will execute that stage in parallel. One single server can execute multiple stages in parallel and it can even execute multiple instances of the same stage in parallel. Stages are identified by their stage ID, which is a unique identifier for each stage. In the current implementation the stage ID is a number and the root stage has a stage ID of 0, although this may change in the future. The current implementation has some properties that are worth mentioning: * The leaf stages execute a slightly modified version of the single-stage query engine. Therefore these stages cannot execute joins or aggregations, which are always executed in the intermediate stages. * Intermediate stages execute operations using a new query execution engine that has been created for the multi-stage query engine. This is why some of the functions that are supported in the single-stage query engine are not supported in the multi-stage query engine and vice versa. * An intermediate stage can only have one join, one window function or one set operation. If a query has more than one of these operations, the broker will create multiple stages, each with one of these operations. ### Extracting Stages from Query Plans As explained in [Explain Plan (Multi-Stage)](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan-multi-stage), you can use the `EXPLAIN PLAN` syntax to obtain the logical plan of a query. This logical plan can be used to extract the stages of the query. For example, if the query is: ```sql explain plan for select customer.c_address, orders.o_shippriority from customer join orders on customer.c_custkey = orders.o_custkey limit 10 ``` A possible output of the `EXPLAIN PLAN` command is: ``` LogicalSort(offset=[0], fetch=[10]) PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false]) LogicalSort(fetch=[10]) LogicalProject(c_address=[$0], o_shippriority=[$3]) LogicalJoin(condition=[=($1, $2)], joinType=[inner]) PinotLogicalExchange(distribution=[hash[1]]) LogicalProject(c_address=[$4], c_custkey=[$6]) LogicalTableScan(table=[[default, customer]]) PinotLogicalExchange(distribution=[hash[0]]) LogicalProject(o_custkey=[$5], o_shippriority=[$10]) LogicalTableScan(table=[[default, orders]]) ``` As it happens with all queries, the logical plan forms a tree-like structure. In this default explain format, the tree-like structure is represented with indentation. The root of the tree is the first line, which is the last operator to be executed and marks the root stage. The boundary between stages are the PinotLogicalExchange operators. In the example above, there are four stages: * The root stage starts with the `LogicalSort` operator in the root of operators and ends with the `PinotLogicalSortExchange` operator. This is the last stage to be executed and the only one that is executed in the broker, which will directly send the result to the client once it is computed. * The next stage starts with this `PinotLogicalSortExchange` operator and includes the `LogicalSort` operator, the `LogicalProject` operator, the `LogicalJoin` operator and the two `PinotLogicalExchange` operators. This stage clearly is not a root stage and it is not reading data from the segments, so it is not a leaf stage. Therefore it has to be an intermediate stage. * The join has two children, which are the `PinotLogicalExchange` operators. In this specific case, both sides are very similar. They start with a `PinotLogicalExchange` operator and end with a `LogicalTableScan` operator. All stages that end with a `LogicalTableScan` operator are leaf stages. Now that we have identified the stages, we can understand what each stage is doing by [interpreting multi-stage explain plans](/build-with-pinot/querying-and-sql/multi-stage-query/explain-plan-1#interpreting-multi-stage-explain-plans). # Multistage Lite Mode Introduces the Multistage Engine Lite Mode {% hint style="info" %} MSE Lite Mode was introduced in Pinot 1.4 and is stable as of Pinot 1.5.0. {% endhint %} ![](/files/q0GvNDR0AKVN3x8cunjY) \*\* Multistage Engine (MSE) Lite Mode is an optional, guardrail-oriented execution mode for self-service and high-QPS tenants. Without additional bounds, queries can scan a large number of records or run expensive operations, which can impact the reliability of a shared tenant and create friction in onboarding new use-cases. Lite Mode addresses this by capping the rows returned from each leaf stage and applying tighter resource bounds automatically. It is based on the observation that most of the users need access to advanced SQL features like Window Functions, Subqueries, etc., and aren't interested in scanning a lot of data or running fully Distributed Joins. ### Overview MSE Lite Mode has the following key characteristics: * Users can still use all MSE query features like Window Functions, Subqueries, Joins, etc. * But, the maximum number of rows returned by a Leaf Stage will be set to a user configurable value. The default value is `100,000`. * Query execution follows a scatter-gather paradigm, similar to the Single-stage Engine. This is different from regular MSE that uses shuffles across Pinot Servers. * Leaf stage(s) are run in the Servers, and all other operators are run using a single thread in the Broker. Leaf Stage in a Multistage Engine query usually refers to Table Scan, an optional Project, an optional Filter and an optional Aggregate Plan Node. At present, all joins in MSE Lite Mode are run in the Broker. This may change with the next release, since Colocated Joins can theoretically be run in the Servers. ### Example To illustrate how MSE Lite Mode applies automatic resource bounds, consider the query below based on the `colocated_join` Quickstart. If this query were allowed in production with the regular MSE, it would scan all the rows of the `userFactEvents` table. With Lite Mode, the full scan will be prevented because Lite Mode will automatically add a Sort to the leaf stage with a configurable limit (aka "fetch") value. ```sql SET useMultistageEngine = true; SET usePhysicalOptimizer = true; SET useLiteMode = true; EXPLAIN PLAN FOR WITH ordered_events AS ( SELECT cityName, tripAmount, ROW_NUMBER() OVER ( ORDER BY ts DESC ) as row_num FROM userFactEvents ), filtered_events AS ( SELECT * FROM ordered_events WHERE row_num < 1000 ) SELECT cityName, SUM(tripAmount) as cityTotal FROM filtered_events GROUP BY cityName ``` The query plan for this query would be as follows. The window function, the filter in the filtered-events table, and the aggregation would be run in the Pinot Broker using a single thread. We assume that the Pinot Broker is configured with the lite mode limit value of 100k records: ![](/files/eWAw4x8qIGWTHSSzHVgP) \*\* ```iecst PhysicalAggregate(group=[{0}], agg#0=[$SUM0($1)], aggType=[DIRECT]) PhysicalFilter(condition=[<($3, 1000)]) PhysicalWindow(window#0=[window(order by [2 DESC] rows between UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]) PhysicalExchange(exchangeStrategy=[SINGLETON_EXCHANGE], collation=[[2 DESC]]) PhysicalSort(fetch=[100000], collation=[[2 DESC]]) <== added by Lite Mode PhysicalProject(cityName=[$3], tripAmount=[$7], ts=[$9]) PhysicalTableScan(table=[[default, userFactEvents]]) ``` ### Enabling Lite Mode To use Lite Mode, you can use the following query options. ```sql SET useMultistageEngine=true; SET usePhysicalOptimizer=true; -- enables the new Physical MSE Query Optimizer SET useLiteMode=true; -- enables Lite Mode ``` ### Running Non-Leaf Stages in Pinot Servers By default Lite Mode will run the non-leaf stage in the Broker. If you want to run the non-leaf stages in Pinot Servers, you can set the following query option to false. In this case, a random server will be picked for the non-leaf stage. ```sql SET runInBroker=false; ``` ### Configuration You can set the following configs in your Pinot Broker. | Configuration Key | Default | Description | | ---------------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pinot.broker.multistage.lite.mode.leaf.stage.limit | 100000 | The maximum number of records that a given Leaf Stage instance on a server is allowed to return. This controls the default value of the query option `liteModeLeafStageLimit`. Recommended value is 100k records or lower. | | pinot.broker.multistage.lite.mode.leaf.stage.fanOutAdjustedLimit | -1 | If set to a positive value, Pinot divides this value by the number of workers assigned to the leaf stage and uses the quotient as the per-worker hard limit. This controls the default value of the query option `liteModeLeafStageFanOutAdjustedLimit`. | | pinot.broker.multistage.use.lite.mode | false | Default value of the query option `useLiteMode`. | | pinot.broker.multistage.run.in.broker | true | Whether to run the non-leaf stages in the broker by default. This controls the default value of the query option `runInBroker`. | ### Runtime warnings for implicit leaf-stage limits Lite Mode can enforce a leaf-stage hard limit even when your SQL text does not include a `LIMIT`. This happens when Pinot injects an implicit leaf-stage limit during planning, such as by adding a leaf-stage `PhysicalSort(fetch=...)` or by applying the same hard limit to a leaf-stage aggregate. When that implicit limit is active, the multi-stage broker response includes the following fields: | Field | Meaning | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mseLiteLeafStageLimitReached` | `true` when at least one leaf-stage worker hit the implicit limit at execution time. When this is `true`, the result is incomplete and Pinot also sets `partialResult=true`. | | `mseLiteLeafStageEffectiveLimit` | The effective per-worker limit used for the implicit leaf-stage cap. This field is present only when Pinot injected the implicit limit. | | `mseLiteFanOutAdjustedLimitApplied` | `true` when the effective limit came from `liteModeLeafStageFanOutAdjustedLimit` after Pinot divided it across leaf-stage workers. `false` means Pinot used the normal `liteModeLeafStageLimit` value. This field is present only when Pinot injected the implicit limit. | If Pinot injected the implicit limit but no worker actually hit it, `mseLiteLeafStageLimitReached` remains `false` and Lite Mode does not mark the response as partial for this reason. Example response when the implicit limit is binding: ```json { "numRowsResultSet": 8, "partialResult": true, "mseLiteLeafStageLimitReached": true, "mseLiteLeafStageEffectiveLimit": 2, "mseLiteFanOutAdjustedLimitApplied": false } ``` ### FAQ #### Q1: What is the Lite Mode intended for? Lite Mode was contributed by Uber and is inspired from [their Presto over Pinot architecture](https://www.uber.com/blog/serving-millions-of-apache-pinot-queries-with-neutrino/). Lite Mode is an optional execution mode with tighter scan and resource bounds, designed for use-cases where users need advanced SQL features (window functions, subqueries, etc.) but do not need fully distributed execution of joins or CTEs. One can think of this as an advanced version of the Single-Stage Engine. #### Q2: Why use a single thread in the broker for the non-leaf stages? Using a single thread, or more importantly a single Operator Chain, means that the entire stage can be run without any Exchange. It also keeps the design simple and makes it easy to reason about performance and debugging. #### Q3: Can Lite Mode be used in tandem with server/segment pruning for high QPS use-cases? Yes, if you setup `segmentPrunerTypes` as [described here](/operate-pinot/tuning/routing) in your Table Config, then segments and servers will be pruned. You can use this to scale out Read QPS. # Physical Optimizer Describes the new Multistage Engine Physical Query Optimizer {% hint style="info" %} The Physical Optimizer is an optional query optimizer for the multi-stage engine, introduced in Pinot 1.4 and stable as of Pinot 1.5.0. {% endhint %} We have added a new query optimizer in the Multistage Engine that computes and tracks precise Data Distribution across the entire plan before running some critical optimizations like Sort Pushdown, Aggregate Split/Pushdown, etc. One of the biggest features of this Optimizer is that it can eliminate Shuffles or simplify Exchanges, when applicable, for arbitrarily complex queries, without requiring any Query Hints. To enable this Optimizer for your MSE query, you can use the following Query Options: ```sql SET useMultistageEngine=true; SET usePhysicalOptimizer=true; ``` ## Key Features The examples below are based on the `COLOCATED_JOIN` Quickstart. ### Automatic Colocated Joins and Shuffle Simplification Consider the query below which consists of 3 Joins. With the new query optimizer, the entire query can run without any cross-server data exchange, since the data is partitioned by userUUID into a compatible number of partitions (see the "Setting Up Table Data Distribution" section below). ```sql SET useMultistageEngine = true; SET usePhysicalOptimizer = true; WITH filtered_users AS ( SELECT userUUID FROM userAttributes WHERE userUUID NOT IN ( SELECT userUUID FROM userGroups WHERE groupUUID = 'group-1' ) AND userUUID IN ( SELECT userUUID FROM userGroups WHERE groupUUID = 'group-2' ) ) SELECT userUUID, SUM(tripAmount) FROM userFactEvents WHERE userUUID IN ( SELECT userUUID FROM filtered_users ) GROUP BY userUUID ``` The query plan for this query is shown below. You can see that the entire query leverages `IDENTITY_EXCHANGE`, which is a 1:1 Exchange as defined in Exchange Types below. ```iecst PhysicalExchange(exchangeStrategy=[SINGLETON_EXCHANGE]) PhysicalAggregate(group=[{1}], agg#0=[$SUM0($0)], aggType=[DIRECT]) PhysicalJoin(condition=[=($1, $2)], joinType=[semi]) PhysicalExchange(exchangeStrategy=[IDENTITY_EXCHANGE]) PhysicalProject(tripAmount=[$7], userUUID=[$10]) PhysicalTableScan(table=[[default, userFactEvents]]) PhysicalJoin(condition=[=($0, $1)], joinType=[semi]) PhysicalProject(userUUID=[$0]) PhysicalFilter(condition=[IS NOT TRUE($3)]) PhysicalJoin(condition=[=($1, $2)], joinType=[left]) PhysicalExchange(exchangeStrategy=[IDENTITY_EXCHANGE]) PhysicalProject(userUUID=[$6], userUUID0=[$6]) PhysicalTableScan(table=[[default, userAttributes]]) PhysicalExchange(exchangeStrategy=[IDENTITY_EXCHANGE]) PhysicalAggregate(group=[{0}], agg#0=[MIN($1)], aggType=[DIRECT]) PhysicalProject(userUUID=[$4], $f1=[true]) PhysicalFilter(condition=[=($3, _UTF-8'group-1')]) PhysicalTableScan(table=[[default, userGroups]]) PhysicalExchange(exchangeStrategy=[IDENTITY_EXCHANGE]) PhysicalProject(userUUID=[$4]) PhysicalFilter(condition=[=($3, _UTF-8'group-2')]) PhysicalTableScan(table=[[default, userGroups]]) ``` ### Shuffle Simplification with Different Servers / Partition Count The new optimizer can simplify shuffles even if: * The Servers used by either side of a Join are different * The Partition Count for the join inputs are different In the example below, we have a Join performed across two tables: orange (left) and green (right). The orange table has 4 partitions and the green table has 2 partitions. The servers selected for the Orange and Green tables are \[S0, S1] and \[S0, S2] respectively. The Join is performed in the servers \[S0, S1], because Physical Optimizer by default uses the same Workers as the leftmost input operator. If the hash-function used for partitioning the two tables is the same, we can leverage an Identity Exchange and skip re-partitioning the data on either side of the join. This is because S0 will consist of records from partitions $$P\_0$$ and $$P\_2$$ of the Orange table, which together contain all records that would make up partition $$P\_0$$ modulo 2. i.e. $$ {(P\_0 \cup P\_2)}*{mod 4} = (P\_0)*{mod 2} $$ Note that Identity Exchange does not imply that the servers in the sender and receiver will be the same. It only implies that there will be a 1:1 mapping from senders to receivers. In the example below, the data transfer from S2 to S1 will be over the network. ![](/files/gEUTkhi0W6JLgoNye4FY) \*\* ### Automatically Skip Aggregate Exchange To evaluate something like `GROUP BY userUUID` accurately you would need to distribute records based on the `userUUID` column. The old query optimizer would add a Partitioning Exchange under each Aggregate, unless one used the query hint `is_partitioned_by_group_by_keys`. The Physical Optimizer can detect when data is already partitioned by the required column, and will automatically skip adding an Exchange. This has two advantages: * We avoid unnecessary Data Exchanges * We avoid splitting the Aggregate, since by default when an Aggregate exists on top of an Exchange, a copy of the Aggregate is added under the Exchange (unless `is_skip_leaf_stage_group_by` query hint is set) This optimization can be seen in action in the query example shared above. Since data is already partitioned by `userUUID`, all aggregations are run in `DIRECT` mode, i.e. without splitting the aggregate into multiple aggregates. ### Segment / Server Pruning Similar to the Single Stage Engine, if you have enabled `segmentPrunerTypes` in your table's Routing config, the Physical Optimizer will prune segments and servers using time, partition or other pruner types for the Leaf Stage. e.g. the following query will only select segments which satisfy the following constraint: ``` segmentPartition = Murmur("user-1") % numPartitions ``` ```sql SET useMultistageEngine = true; SET usePhysicalOptimizer = true; WITH user_events AS ( SELECT productCode, tripAmount FROM userFactEvents WHERE userUUID = 'user-1' ORDER BY ts DESC LIMIT 100 ) SELECT productCode, SUM(tripAmount) FROM user_events GROUP BY productCode ``` If partitioning is done in a way that segments corresponding to a given partition are present on only 1 server, then the entire query above will run within a single server, simulating shard-local execution from other systems. ### Solve Constant Queries in Pinot Broker Apache Calcite is capable of detecting Filter Expressions that will always evaluate to False. In such cases, the query plan may not have any Table Scans at all. Physical Optimizer solves such queries within the Broker itself, without involving any servers. ```sql SET useMultistageEngine = true; SET usePhysicalOptimizer = true; SELECT COUNT(*) FROM userFactEvents WHERE userUUID = 'user-1' AND userUUID = 'user-2' ``` ## Worker Assignment At present, Worker Assignment follows these simple rules: * Leaf Stage will have workers assigned based on Table Scan and Filters, using the Routing configs set in the Table Config. * Other Stages will use the same workers as the left-most input stage. * Some Plan Nodes, such as `Sort(fetch=..)`, may require data to be collected in a single Worker. In such a case, that stage will be run on a single Worker, which will be randomly selected from one of the input workers. ## Limitations Some of the features of the existing MSE query optimizer are not yet available in the Physical Optimizer. We aim to add support for most these in Pinot 1.5: * Spools. * Dynamic filters for semi-join # Explain Plan Query execution within Pinot is modeled as a sequence of operators that are executed in a pipelined manner to produce the final result. The `EXPLAIN PLAN FOR` syntax can be used to obtain the execution plan of a query, which can be useful to further optimize them. {% hint style="warning" %} The explain plan output format is still under development and may change in future releases. This under-development label applies to the explain plan output format specifically, not to the core multi-stage engine, which is generally available. Pinot explain plans are human-readable and are intended to be used for debugging and optimization purposes. This is especially important when using the explain plan in automated scripts or tools. The explain plan, even the ones returned as tables or JSON, are not guaranteed to be stable across releases. {% endhint %} Pinot supports different type of explain plans depending on the query engine and the granularity or details we want to obtain. ```mermaid graph LR EXPLAIN STAGE{"single or \n multi-stage?"} SSE_Q_VERBOSE{verbose?} SSE_SIMPLE[brief SSE] SSE_EXTENDED[verbose SSE] MSE_WORKERS[workers MSE] MSE_LOGICAL[logical MSE] MSE_Q_VERBOSE{verbose?} MSE_IMPL_SIMPLE["brief segment MSE"] MSE_IMPL_VERBOSE["verbose segment MSE"] EXPLAIN --> STAGE STAGE -- single --> SSE_Q_VERBOSE STAGE -- multi --> MSE_Q_DISTRIBUTION{workers?} SSE_Q_VERBOSE -- no --> SSE_SIMPLE SSE_Q_VERBOSE -- yes --> SSE_EXTENDED MSE_Q_DISTRIBUTION -- yes --> MSE_WORKERS MSE_Q_DISTRIBUTION -- no --> MSE_Q_LOGICAL{logical?} MSE_Q_LOGICAL -- yes --> MSE_LOGICAL MSE_Q_LOGICAL -- no --> MSE_Q_VERBOSE MSE_Q_VERBOSE -- yes --> MSE_IMPL_VERBOSE MSE_Q_VERBOSE -- no --> MSE_IMPL_SIMPLE ``` ## Different plans for different segments Segments are the basic unit of data storage and processing in Pinot. When a query is executed, it is executed on each segment and the results are merged together. Not all segments have the data distribution, indexes, etc. Therefore the query engine may decide to execute the query differently on different segments. This includes: * Segments that were not refreshed since indexes were added or removed on the table config. * Realtime segments that are being ingested, where some indexes (like range indexes) cannot be used. * Data distribution, specially min and max values for columns, which can affect the query plan. Given a Pinot query can touch thousands of segments, Pinot tries to minimize the number of [different queries](#user-content-fn-1)[^1] shown when explaining a query. By default, Pinot tries to analyze the plan for each segment and returns a simplified plan. How this simplification is done depends on the query engine, you can read more about that below. There is a verbose mode that can be used to show the plan for each segment. This mode is activated by setting the `explainPlanVerbose` query option to true, prefixing `SET explainPlanVerbose=true;` to the explain plan sentence. ## Explain on multi-stage query engine Following the more complex nature of the multi-stage query engine, its explain plan can be customized to get a plan specialized[^2] on different aspects of the query execution. There are 3 different types of explain plans for the multi-stage query engine: | Mode | Syntax by default | Syntax if segment plan is enabled | Description | | ------------ | --------------------------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | Segment plan | `SET explainAskingServers=true;` `EXPLAIN PLAN FOR` | `EXPLAIN PLAN FOR` | Includes the segment specific information (like indexes). | | Logical plan | `EXPLAIN PLAN FOR` or `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR` | `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR` | Simplest multi-stage plan. No index or data shuffle information. | | Workers plan | `EXPLAIN IMPLEMENTATION PLAN FOR` | `EXPLAIN IMPLEMENTATION PLAN FOR` | Used to understand data shuffle between servers. Note: The name of this mode is open to discussion and may change in the future. | {% hint style="info" %} The syntax used to select each explain plan mode is confusing and it may be changed in the future. {% endhint %} ### Segment plan The plan with segments is a detailed representation of the query execution plan that includes the segment specific information, like data distribution, indexes, etc. This mode was introduced in Pinot 1.3.0 and it is planned to be the default in future releases. Meanwhile it can be used by setting the `explainAskingServers` query option to true, prefixing `SET explainAskingServers=true;` to the explain plan sentence. Alternatively this mode can be activated by default by changing the broker configuration `pinot.query.multistage.explain.include.segment.plan` to true. Independently of how it is activated, once this mode is enabled, `EXPLAIN PLAN FOR` syntax will include segment information. #### Verbose and brief mode As explained in `Different plans for different segments`, by default Pinot tries to minimize the number of [different query](#user-content-fn-3)[^3] shown when explaining a query. In multi-stage, the brief mode groups distinct segment plans into `Alternative(segments=[n])` branches, where `n` is the number of segments using that plan. Equivalent plans are aggregated into the same branch, so if the same plan is executed on 100 segments, the brief mode will show it only once and additive stats such as the number of docs will be summed. If some segments use a different plan, brief mode keeps one `Alternative` branch per distinct plan instead of falling back to one child per segment. When every segment shares the same plan, Pinot removes the redundant single `Alternative` wrapper after merging, so the common case still renders as a single child under the combine node. In the verbose mode, one plan is shown per segment, including the segment name and all the segment specific information. This may be useful to know which segments are not using indexes, or which segments are using a different data distribution. #### Example ```sql -- SET explainAskingServer= true is required if -- pinot.query.multistage.explain.include.segment.plan is false, -- optional otherise SET explainAskingServers=true; EXPLAIN PLAN FOR SELECT DISTINCT deviceOS, groupUUID FROM userAttributes AS a JOIN userGroups AS g ON a.userUUID = g.userUUID WHERE g.groupUUID = 'group-1' LIMIT 100 ``` Returns ``` Execution Plan LogicalSort(offset=[0], fetch=[100]) PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false]) LogicalSort(fetch=[100]) PinotLogicalAggregate(group=[{0, 1}]) PinotLogicalExchange(distribution=[hash[0, 1]]) PinotLogicalAggregate(group=[{0, 2}]) LogicalJoin(condition=[=($1, $3)], joinType=[inner]) PinotLogicalExchange(distribution=[hash[1]]) LeafStageCombineOperator(table=[userAttributes]) StreamingInstanceResponse StreamingCombineSelect SelectStreaming(table=[userAttributes], totalDocs=[10000]) Project(columns=[[deviceOS, userUUID]]) DocIdSet(maxDocs=[40000]) FilterMatchEntireSegment(numDocs=[10000]) PinotLogicalExchange(distribution=[hash[1]]) LeafStageCombineOperator(table=[userGroups]) StreamingInstanceResponse StreamingCombineSelect SelectStreaming(table=[userGroups], totalDocs=[2478]) Project(columns=[[groupUUID, userUUID]]) DocIdSet(maxDocs=[50000]) FilterInvertedIndex(predicate=[groupUUID = 'group-1'], indexLookUp=[inverted_index], operator=[EQ]) SelectStreaming(segment=[userGroups_OFFLINE_4], table=[userGroups], totalDocs=[4]) Project(columns=[[groupUUID, userUUID]]) DocIdSet(maxDocs=[10000]) FilterEmpty SelectStreaming(segment=[userGroups_OFFLINE_6], table=[userGroups], totalDocs=[4]) Project(columns=[[groupUUID, userUUID]]) DocIdSet(maxDocs=[10000]) FilterMatchEntireSegment(numDocs=[4]) ``` ### Logical Plan The logical plan is a high-level representation of the query execution plan. This plan is calculated on the broker without asking the servers for their segment specific plans. This means that the logical plan does not include the segment specific information, like data distribution, indexes, etc. In Pinot 1.3.0, the logical plan is enabled by default and can be obtained by using `EXPLAIN PLAN FOR` syntax. Optionally, the segment plan can be enabled by default, in which case the logical plan can be obtained by using `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR` syntax. {% hint style="info" %} The recommended way to ask for logical plan is to use `EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR` given this syntax is available in all versions of Pinot, independently of the configuration. {% endhint %} #### Example: ```sql -- WITHOUT IMPLENTATION qualifier can be used to ensure logical plan is used -- It can be used in any version of Pinot even when segment plan is enabled by default EXPLAIN PLAN WITHOUT IMPLEMENTATION FOR SELECT DISTINCT deviceOS, groupUUID FROM userAttributes AS a JOIN userGroups AS g ON a.userUUID = g.userUUID WHERE g.groupUUID = 'group-1' LIMIT 100 ``` Returns: ``` Execution Plan LogicalSort(offset=[0], fetch=[100]) PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false]) LogicalSort(fetch=[100]) PinotLogicalAggregate(group=[{0, 1}]) PinotLogicalExchange(distribution=[hash[0, 1]]) PinotLogicalAggregate(group=[{0, 2}]) LogicalJoin(condition=[=($1, $3)], joinType=[inner]) PinotLogicalExchange(distribution=[hash[1]]) LogicalProject(deviceOS=[$4], userUUID=[$6]) LogicalTableScan(table=[[default, userAttributes]]) PinotLogicalExchange(distribution=[hash[1]]) LogicalProject(groupUUID=[$3], userUUID=[$4]) LogicalFilter(condition=[=($3, _UTF-8'group-1')]) LogicalTableScan(table=[[default, userGroups]]) ``` ### Workers plan {% hint style="info" %} There have been some discussion about how to name this explain mode and it may change in future versions. The term *worker* is leaking an implementation detail that is not explained anywhere else in the user documentation. {% endhint %} The workers plan is a detailed representation of the query execution plan that includes information on how the query is distributed among different servers and workers inside them. This plan does not include the segment specific information, like data distribution, indexes, etc. and it is probably the less[^4] useful of the plans for normal use cases. Their main use case is to try to reduce data shuffling between workers by verifying that, for example, a join is executed in colocated fashion. #### Example ```sql EXPLAIN IMPLEMENTATION PLAN FOR SELECT DISTINCT deviceOS, groupUUID FROM userAttributes AS a JOIN userGroups AS g ON a.userUUID = g.userUUID WHERE g.groupUUID = 'group-1' LIMIT 100 ``` Returns: ``` 0]@192.168.0.98:54196|[0] MAIL_RECEIVE(BROADCAST_DISTRIBUTED) ├── [1]@192.168.0.98:54227|[3] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.0.98:54196|[0]} (Subtree Omitted) ├── [1]@192.168.0.98:54220|[2] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.0.98:54196|[0]} (Subtree Omitted) ├── [1]@192.168.0.98:54214|[1] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.0.98:54196|[0]} (Subtree Omitted) └── [1]@192.168.0.98:54206|[0] MAIL_SEND(BROADCAST_DISTRIBUTED)->{[0]@192.168.0.98:54196|[0]} └── [1]@192.168.0.98:54206|[0] SORT LIMIT 100 └── [1]@192.168.0.98:54206|[0] MAIL_RECEIVE(HASH_DISTRIBUTED) ├── [2]@192.168.0.98:54227|[3] MAIL_SEND(HASH_DISTRIBUTED)->{[1]@192.168.0.98:54207|[0],[1]@192.168.0.98:54215|[1],[1]@192.168.0.98:54221|[2],[1]@192.168.0.98:54228|[3]} (Subtree Omitted) ├── [2]@192.168.0.98:54220|[2] MAIL_SEND(HASH_DISTRIBUTED)->{[1]@192.168.0.98:54207|[0],[1]@192.168.0.98:54215|[1],[1]@192.168.0.98:54221|[2],[1]@192.168.0.98:54228|[3]} (Subtree Omitted) ├── [2]@192.168.0.98:54214|[1] MAIL_SEND(HASH_DISTRIBUTED)->{[1]@192.168.0.98:54207|[0],[1]@192.168.0.98:54215|[1],[1]@192.168.0.98:54221|[2],[1]@192.168.0.98:54228|[3]} (Subtree Omitted) └── [2]@192.168.0.98:54206|[0] MAIL_SEND(HASH_DISTRIBUTED)->{[1]@192.168.0.98:54207|[0],[1]@192.168.0.98:54215|[1],[1]@192.168.0.98:54221|[2],[1]@192.168.0.98:54228|[3]} └── [2]@192.168.0.98:54206|[0] SORT LIMIT 100 └── [2]@192.168.0.98:54206|[0] AGGREGATE_FINAL └── [2]@192.168.0.98:54206|[0] MAIL_RECEIVE(HASH_DISTRIBUTED) ├── [3]@192.168.0.98:54227|[3] MAIL_SEND(HASH_DISTRIBUTED)->{[2]@192.168.0.98:54207|[0],[2]@192.168.0.98:54215|[1],[2]@192.168.0.98:54221|[2],[2]@192.168.0.98:54228|[3]} (Subtree Omitted) ├── [3]@192.168.0.98:54220|[2] MAIL_SEND(HASH_DISTRIBUTED)->{[2]@192.168.0.98:54207|[0],[2]@192.168.0.98:54215|[1],[2]@192.168.0.98:54221|[2],[2]@192.168.0.98:54228|[3]} (Subtree Omitted) ├── [3]@192.168.0.98:54214|[1] MAIL_SEND(HASH_DISTRIBUTED)->{[2]@192.168.0.98:54207|[0],[2]@192.168.0.98:54215|[1],[2]@192.168.0.98:54221|[2],[2]@192.168.0.98:54228|[3]} (Subtree Omitted) └── [3]@192.168.0.98:54206|[0] MAIL_SEND(HASH_DISTRIBUTED)->{[2]@192.168.0.98:54207|[0],[2]@192.168.0.98:54215|[1],[2]@192.168.0.98:54221|[2],[2]@192.168.0.98:54228|[3]} └── [3]@192.168.0.98:54206|[0] AGGREGATE_LEAF └── [3]@192.168.0.98:54206|[0] JOIN ├── [3]@192.168.0.98:54206|[0] MAIL_RECEIVE(HASH_DISTRIBUTED) │ ├── [4]@192.168.0.98:54227|[1] MAIL_SEND(HASH_DISTRIBUTED)->{[3]@192.168.0.98:54207|[0],[3]@192.168.0.98:54215|[1],[3]@192.168.0.98:54221|[2],[3]@192.168.0.98:54228|[3]} (Subtree Omitted) │ └── [4]@192.168.0.98:54214|[0] MAIL_SEND(HASH_DISTRIBUTED)->{[3]@192.168.0.98:54207|[0],[3]@192.168.0.98:54215|[1],[3]@192.168.0.98:54221|[2],[3]@192.168.0.98:54228|[3]} │ └── [4]@192.168.0.98:54214|[0] PROJECT │ └── [4]@192.168.0.98:54214|[0] TABLE SCAN (userAttributes) null └── [3]@192.168.0.98:54206|[0] MAIL_RECEIVE(HASH_DISTRIBUTED) ├── [5]@192.168.0.98:54227|[1] MAIL_SEND(HASH_DISTRIBUTED)->{[3]@192.168.0.98:54207|[0],[3]@192.168.0.98:54215|[1],[3]@192.168.0.98:54221|[2],[3]@192.168.0.98:54228|[3]} (Subtree Omitted) └── [5]@192.168.0.98:54214|[0] MAIL_SEND(HASH_DISTRIBUTED)->{[3]@192.168.0.98:54207|[0],[3]@192.168.0.98:54215|[1],[3]@192.168.0.98:54221|[2],[3]@192.168.0.98:54228|[3]} └── [5]@192.168.0.98:54214|[0] PROJECT └── [5]@192.168.0.98:54214|[0] FILTER └── [5]@192.168.0.98:54214|[0] TABLE SCAN (userGroups) null ``` ## Interpreting multi-stage explain plans Multi-stage plans are more complex than single-stage plans. This section explains how to interpret them. You can use the `EXPLAIN PLAN` syntax to obtain the logical plan of a query. There are different formats for the output, but all of them represent the logical plan of the query. The query ```sql explain plan for select customer.c_address, orders.o_shippriority from customer join orders on customer.c_custkey = orders.o_custkey limit 10 ``` Can produce the following output: ``` LogicalSort(offset=[0], fetch=[10]) PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false]) LogicalSort(fetch=[10]) LogicalProject(c_address=[$0], o_shippriority=[$3]) LogicalJoin(condition=[=($1, $2)], joinType=[inner]) PinotLogicalExchange(distribution=[hash[1]]) LogicalProject(c_address=[$4], c_custkey=[$6]) LogicalTableScan(table=[[default, customer]]) PinotLogicalExchange(distribution=[hash[0]]) LogicalProject(o_custkey=[$5], o_shippriority=[$10]) LogicalTableScan(table=[[default, orders]]) ``` Each node in the tree represents an operation, and each operator has attributes. For example the `LogicalJoin` operator has a `condition` attribute that specifies the join condition and a `joinType`. ### Understanding indexed references Expressions like `$2` are indexed references into the input row for each operator. To understand them, look at the operator's children to see which attributes are being referenced, usually starting from the leaf operators. For example, `LogicalTableScan` always returns the whole row of the table, so its attributes are the columns of the table: ``` PinotLogicalExchange(distribution=[hash[0]]) LogicalProject(o_custkey=[$5], o_shippriority=[$10]) LogicalTableScan(table=[[default, orders]]) ``` The `LogicalProject` operator selects columns `o_custkey` and `o_shippriority` (at positions `$5` and `$10` in the table row) and generates a row with two columns. The `PinotLogicalExchange` distributes rows using `hash[0]`, meaning the hash of the first column from `LogicalProject` — which is `o_custkey`. ### Virtual rows in joins The `LogicalJoin` operator receives rows from two upstream stages. The virtual row seen by the join is the concatenation of the left-hand side plus the right-hand side. In the example above, the left stage sends `[c_address, c_custkey]` and the right stage sends `[o_custkey, o_shippriority]`. The join sees a row with columns `[c_address, c_custkey, o_custkey, o_shippriority]`. The condition `=($1, $2)` joins on `c_custkey` and `o_custkey`. The join passes through all columns unchanged, so its downstream `LogicalProject` selecting `$0` and `$3` produces `[c_address, o_shippriority]`. ### LogicalSort without ORDER BY A `LogicalSort` operator can appear even when the SQL query has no `ORDER BY`. In relational algebra, a sort node is used to express `LIMIT`. When no sort condition is specified, no actual sorting is performed — only the row limit is applied. ## Explain on single stage query engine {% hint style="info" %} Explain plan for single stage query engine is described in deep in [explain-plan.md](/build-with-pinot/querying-and-sql/query-execution-controls/explain-plan) {% endhint %} Explain plan for single stage query engine is simpler and less customized, but returns the information in a tabular format. For example, the query `EXPLAIN PLAN FOR SELECT playerID, playerName FROM baseballStats`. \ Returns the following table: ``` +---------------------------------------------|------------|---------| | Operator | Operator_Id|Parent_Id| +---------------------------------------------|------------|---------| |BROKER_REDUCE(limit:10) | 1 | 0 | |COMBINE_SELECT | 2 | 1 | |PLAN_START(numSegmentsForThisPlan:1) | -1 | -1 | |SELECT(selectList:playerID, playerName) | 3 | 2 | |TRANSFORM_PASSTHROUGH(playerID, playerName) | 4 | 3 | |PROJECT(playerName, playerID) | 5 | 4 | |DOC_ID_SET | 6 | 5 | |FILTER_MATCH_ENTIRE_SEGMENT(docs:97889) | 7 | 6 | +---------------------------------------------|------------|---------| ``` Where `Operator` column describes the operator that Pinot will run whereas the `Operator_Id` and `Parent_Id` columns show the parent-child relationship between operators, which forms the execution tree. For example, the plan above should be understood as: ``` BROKER_REDUCE(limit:10) └── COMBINE_SELECT └── PLAN_START(numSegmentsForThisPlan:1) └── SELECT(selectList:playerID, playerName) └── TRANSFORM_PASSTHROUGH(playerID, playerName) └── PROJECT(playerName, playerID) └── DOC_ID_SET └── FILTER_MATCH_ENTIRE_SEGMENT(docs:97889) ``` \\ [^1]: 'different query plans' ? [^2]: focused? [^3]: different plans? [^4]: least? --- [Next Page](/llms-full.txt/1)