Outcome

By the end of this page you will understand what Apache Pinot is, what problems it solves, and whether it is the right tool for your use case.

Prerequisites

None. This is the starting point of the onboarding path.

What Apache Pinot does

Apache Pinot is a real-time distributed online analytical processing (OLAP) datastore. It ingests data from streaming sources (such as Apache Kafka and Amazon Kinesis) and batch sources (such as Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage) and makes that data immediately available for analytic queries with sub-second latency.

Key capabilities

• Ultra-low-latency analytics -- Queries return in milliseconds, even at hundreds of thousands of queries per second.

• Columnar storage with smart indexing -- Purpose-built storage format with inverted, sorted, range, text, and other indexes to accelerate query patterns.

• Horizontal scaling -- Scale out by adding nodes with no upper bound on cluster size.

When to use Pinot

User-facing real-time analytics

Pinot was built at LinkedIn to power interactive analytics features such as Who Viewed Profile and Company Analytics. UberEats Restaurant Manager is another production example. These applications serve personalized analytics to every end user, generating hundreds of thousands of queries per second with strict latency requirements.

Real-time dashboards

Pinot supports slice-and-dice, drill-down, roll-up, and pivot operations on high-dimensional data. Connect business intelligence tools such as Apache Superset, Tableau, or PowerBI to Pinot to build live dashboards over streaming data.

Enterprise analytics

Pinot works well as a highly scalable platform for business intelligence. It converges the capabilities of a big data platform with the traditional role of a data warehouse, making it suitable for analysis and reporting at scale.

Aggregate store for microservices

Application developers can use Pinot as an aggregate store that consumes events from streaming sources and exposes them through SQL. This is useful for building a unified, queryable view across a microservice architecture. Query models are eventually consistent, as with all aggregate stores.

When NOT to use Pinot

Verify

You now know:

• What Apache Pinot is and how it differs from transactional databases.

• The four main categories of use cases where Pinot excels.

• When a simpler tool would be a better choice.

Next step

Continue to the 10-minute quickstart to launch a local Pinot cluster and run your first query:

Outcome

By the end of this guide you will have a fully functional Apache Pinot cluster running locally with sample data loaded, ready to query.

Prerequisites

• installed and running

• Recommended resources: 8 CPUs, 16 GB RAM

Steps

1. Set the Pinot version

See the page for the current stable release.

2. Start Pinot with sample data

This single command starts ZooKeeper, Controller, Broker, Server, and Minion, then loads a baseball statistics dataset.

Verify

1. Open the Pinot Query Console at .

2. Run a sample query:

You should see results returned within milliseconds.

Next step

This quickstart bundles everything in a single process for convenience. For a list of all available quickstart types (batch, streaming, hybrid, and more), see .

Ready for a production-style setup? Continue to .

Outcome

Select the right installation method for your use case and deploy a Pinot cluster.

Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including:

• Storing data in columnar form to support high-performance scanning

• Sharding of data to scale both storage and computation

Query Pinot using supported syntax.

The segment threshold determines when a segment is committed in real-time tables.

When data is first ingested from a streaming provider like Kafka, Pinot stores the data in a consuming segment.

This segment is on the disk of the server(s) processing a particular partition from the streaming provider.

However, it's not until a segment is committed that the segment is written to the deep store. The segment threshold decides when that should happen.

Why is the segment threshold important?

The segment threshold is important because it ensures segments are a reasonable size.

When queries are processed, smaller segments may increase query latency due to more overhead (number of threads spawned, meta data processing, and so on).

Larger segments may cause servers to run out of memory. When a server is restarted, the consuming segment must start consuming from the first row again, causing a lag between Pinot and the streaming provider.

Mark Needham explains the segment threshold

Outcome

Create an Azure Kubernetes Service cluster with the required tooling, ready to deploy Apache Pinot.

Prerequisites

• An Azure account

• The following CLI tools installed (see steps below)

Steps

1. Install tooling

kubectl

Verify:

Helm

Verify:

Azure CLI

Follow the or run:

2. Log in to Azure

3. Create a resource group

4. Create an AKS cluster

The following creates a 3-node cluster named pinot-quickstart:

5. Connect to the cluster

Verify

You should see your worker nodes listed and in Ready status.

Cleaning up

To delete the cluster when you are done:

Next step

Your cluster is ready. Continue to to deploy Pinot.

Outcome

Start a multi-component Pinot cluster directly on your machine without containers.

Prerequisites

• JDK 11 or 21 (JDK 17 should work but is not officially supported)

• Apache Maven 3.6+ (only if building from source)

Steps

1. Download or build Apache Pinot

2. Start ZooKeeper

3. Start Pinot Controller

4. Start Pinot Broker

5. Start Pinot Server

6. Start Pinot Minion (optional)

7. Start Kafka (optional)

Only needed if you plan to ingest real-time streaming data.

Verify

Check that the Controller is healthy:

The response should return OK. You can also open the Pinot Query Console at .

Next step

Your cluster is running. Continue to to load data.

Outcome

Provision a managed Kubernetes cluster on AWS, GCP, or Azure that is ready for a Pinot deployment.

Overview

These guides walk you through creating a managed Kubernetes cluster on your cloud provider. Once the cluster is running, you will use the page to deploy Pinot onto it.

Cloud providers

Next step

Once your cluster is ready, follow the to deploy Pinot.

Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including:

• Storing data in columnar form to support high-performance scanning

• Sharding of data to scale both storage and computation

• A distributed architecture designed to scale capacity linearly

• A tabular data model read by SQL queries

To learn about Pinot components, terminology, and gain a conceptual understanding of how data is stored in Pinot, review the following sections:

Outcome

Create a Google Kubernetes Engine cluster with the required tooling, ready to deploy Apache Pinot.

Pinot servers provide the primary storage for and perform the computation required to execute queries. A production Pinot cluster contains many servers. In general, the more servers, the more data the cluster can retain in tables, the lower latency the cluster can deliver on queries, and the more concurrent queries the cluster can process.

Servers are typically segregated into real-time and offline workloads, with "real-time" servers hosting only real-time tables, and "offline" servers hosting only offline tables. This is a ubiquitous operational convention, not a difference or an explicit configuration in the server process itself. There are two types of servers:

Offline

Pinot brokers take query requests from client processes, scatter them to applicable servers, gather the results, and return results to the client. The controller shares cluster metadata with the brokers, which allows the brokers to create a plan for executing the query involving a minimal subset of servers with the source data and, when required, other servers to shuffle and consolidate results.

A production Pinot cluster contains many brokers. In general, the more brokers, the more concurrent queries a cluster can process, and the lower latency it can deliver on queries.

Pinot brokers are modeled as Helix spectators. They need to know the location of each segment of a table (and each replica of the segments) and route requests to the appropriate server that hosts the segments of the table being queried.

The broker ensures that all the rows of the table are queried exactly once so as to return correct, consistent results for a query. The brokers may optimize to prune some of the segments

Ingestion is where Pinot tables become real. Start here to choose the right path for batch or stream data, then refine the design with upsert, dedup, file format, filesystem, transform, and aggregation decisions.

The detailed controller and table-config material belongs in . This section stays focused on data flow and operational choices.

Start Here

This guide walks you through loading streaming data into a Pinot cluster running in Kubernetes. Make sure you have completed first.

Load data into Pinot using Kafka

Segments in Pinot tables have a retention time, after which the segments are deleted. Typically, offline tables retain segments for a longer period of time than real-time tables.

The removal of segments is done by the retention manager. By default, the retention manager runs once every 6 hours.

The retention manager purges two types of segments:

• Expired segments: Segments whose end time has exceeded the retention period.

Stream ingestion keeps Pinot close to the source of truth. Use it when rows should be queryable soon after they are emitted, and when the system needs a steady flow rather than periodic batch loads.

Core decisions

Pick the stream connector and partitioning strategy.

Choose how Pinot should flush, commit, and complete segments.

Learn about time boundaries in hybrid tables. Hybrid tables are when we have offline and real-time tables with the same name.

When querying these tables, the Pinot broker decides which records to read from the offline table and which to read from the real-time table. It does this using the time boundary.

How is the time boundary determined?

The time boundary is determined by looking at the maximum end time of the offline segments and the segment ingestion frequency specified for the offline table.

Pinot schema evolution is intentionally narrow. The safe path is to add columns, reload the affected segments, and backfill only when the table type and data flow support it. If the change is more invasive than that, create a new table instead of forcing the old one to stretch.

What is safe

Additive schema changes are the normal path. New columns can be introduced without rewriting the whole table, as long as the ingestion flow and segment reload behavior are understood.

Batch ingestion builds Pinot segments outside the cluster and pushes them into Pinot after the data is already shaped. Use it when the data changes in larger chunks, when you need deterministic backfills, or when the pipeline already produces files or segment artifacts.

The most important design choice is not the framework, but the output contract: what the schema looks like, what the table expects, and where the segments land.

Common batch paths

Spark-based ingestion.

Apache Pinot™ is a real-time distributed online analytical processing (OLAP) datastore. Use Pinot to ingest and immediately query data from streaming or batch data sources (including, Apache Kafka, Amazon Kinesis, Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage).

Apache Pinot includes the following:

Use this section to decide how to query Pinot, how much SQL support you need, which query engine to use, and where to look for execution controls such as quotas, cancellation, and cursors. Narrative guidance lives here. Dense syntax and endpoint detail is linked where needed.

Start here

Prerequisite

Apache Pinot is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics. It ingests data from streaming and batch sources and makes it queryable in under a second. This guide walks you through everything you need to go from first contact to a working Pinot deployment.

Onboarding path

Follow these pages in order for a complete introduction:

For an overview of when to use the multi-stage engine (MSE) versus the single-stage engine (SSE), see . This section provides a deep dive into MSE internals. Most of the concepts explained here are related to the engine's execution model and are not required for writing queries. However, understanding them can help you take advantage of MSE's capabilities and troubleshoot issues.

Outcome

Run your first SQL queries against Pinot and understand the query interface.

Prerequisites

• You have completed either or . The transcript table exists and contains data.

• The Pinot cluster is running (Controller on port 9000, Broker on port 8099).

Steps

1. Open the Query Console

Navigate to in your browser. Click Query Console in the left sidebar. You should see the transcript table listed in the table explorer on the left.

2. Run a simple SELECT

Paste the following query into the query editor and click Run Query:

The results panel shows all columns in the transcript table -- studentID, firstName, lastName, gender, subject, score, and timestamp. The rows returned come from whichever data you loaded (batch, stream, or both). LIMIT 10 caps the result set so the response is fast.

3. Run an aggregation

This query calculates the average score per subject and sorts the results from highest to lowest. Pinot executes aggregations directly on each server's segment data and merges the results at the Broker, making GROUP BY queries fast even on large datasets.

4. Run a count

This returns the total number of rows in the table. The exact count depends on which ingestion steps you completed:

• Batch ingest only -- 4 rows

• Stream ingest only -- the number of events you published (up to 12 in the tutorial)

• Both -- the combined total

5. Run a filter

This filters rows to show only students with a score above 3.5. Pinot pushes filter predicates down to the servers so only matching rows are scanned and returned.

6. Try the REST API

The Query Console UI is convenient for exploration, but production applications query Pinot through its REST API. Open a terminal and run:

Port 8099 is the Broker, which handles all query requests. The Query Console UI uses the same API under the hood. The response is a JSON object containing the result rows, schema, and query execution metadata.

Verify

All five queries return results without errors. You have successfully completed the end-to-end onboarding flow: you set up a Pinot cluster, defined a schema and table, loaded data, and queried it through both the UI and the REST API.

What's next

You have finished the linear Start Here path. From here, explore the areas most relevant to your use case:

• -- the full SQL reference for Pinot's query language

• -- enable JOINs and complex queries across tables

• -- understand how queries flow from Broker to Server and back

Outcome

Deploy a production-ready Pinot cluster on Kubernetes with Helm charts.

Prerequisites

• A running Kubernetes cluster. Options include:

  • -- start with sufficient resources: minikube start --vm=true --cpus=4 --memory=8g --disk-size=50g

Steps

1. Add the Pinot Helm repository

2. Create a namespace

3. Install Pinot

Verify

Check the deployment status:

All pods should reach Running status. You can port-forward the Controller to access the UI:

Then open .

Loading data

For stream ingestion on Kubernetes, see the . For batch data loading and table creation, continue with the onboarding path below.

Deleting the cluster

To remove Pinot from your cluster:

Next step

Your cluster is running. Continue to to load data.

Introduction

Pinot batch ingestion involves two parts: routine ingestion job(hourly/daily) and backfill. Here are some examples to show how routine batch ingestion works in Pinot offline table:

• Batch Ingestion Overview

High-level description

1. Organize raw data into buckets (eg: /var/pinot/airlineStats/rawdata/2014/01/01). Each bucket typically contains several files (eg: /var/pinot/airlineStats/rawdata/2014/01/01/airlineStats_data_2014-01-01_0.avro)

2. Run a Pinot batch ingestion job, which points to a specific date folder like ‘/var/pinot/airlineStats/rawdata/2014/01/01’. The segment generation job will convert each such avro file into a Pinot segment for that day and give it a unique name.

3. Run Pinot segment push job to upload those segments with those uniques names via a Controller API

This newly uploaded data can now be queried in Pinot. However, sometimes users will make changes to the raw data which need to be reflected in Pinot. This process is known as 'Backfill'.

How to backfill data in Pinot

Pinot supports data modification only at the segment level, which means you must update entire segments for doing backfills. The high level idea is to repeat steps 2 (segment generation) and 3 (segment upload) mentioned above:

• Backfill jobs must run at the same granularity as the daily job. E.g., if you need to backfill data for 2014/01/01, specify that input folder for your backfill job (e.g.: ‘/var/pinot/airlineStats/rawdata/2014/01/01’)

• The backfill job will then generate segments with the same name as the original job (with the new data).

• When uploading those segments to Pinot, the controller will replace the old segments with the new ones (segment names act like primary keys within Pinot) one by one.

Edge case example

Backfill jobs expect the same number of (or more) data files on the backfill date. So the segment generation job will create the same number of (or more) segments than the original run.

For example, assuming table airlineStats has 2 segments(airlineStats_2014-01-01_2014-01-01_0, airlineStats_2014-01-01_2014-01-01_1) on date 2014/01/01 and the backfill input directory contains only 1 input file. Then the segment generation job will create just one segment: airlineStats_2014-01-01_2014-01-01_0. After the segment push job, only segment airlineStats_2014-01-01_2014-01-01_0 got replaced and stale data in segment airlineStats_2014-01-01_2014-01-01_1 are still there.

If the raw data is modified in such a way that the original time bucket has fewer input files than the first ingestion run, backfill will fail.

Outcome

Know which Pinot version to use and how to pin versions in examples.

This page is the single source of truth for version information across the Start Here guide and the wider documentation. When following tutorials or code samples, make sure the version you use matches your installed release.

Current stable release

Using PINOT_VERSION in examples

Most code samples in these docs set a PINOT_VERSION environment variable near the top of each snippet. Always verify that the value matches your installed version:

Once the variable is set, every command in the tutorial that references ${PINOT_VERSION} will use the correct value automatically.

Compatibility notes

If you are running JDK 8 and cannot upgrade, use Pinot 0.12.1. For all new deployments, JDK 11 or 21 is recommended.

Release links

Older versions

Older Pinot binaries are archived at .

The deep store (or deep storage) is the permanent store for segment files.

It is used for backup and restore operations. New server nodes in a cluster will pull down a copy of segment files from the deep store. If the local segment files on a server gets damaged in some way (or accidentally deleted), a new copy will be pulled down from the deep store on server restart.

The deep store stores a compressed version of the segment files and it typically won't include any indexes. These compressed files can be stored on a local file system or on a variety of other file systems. For more details on supported file systems, see File Systems.

Note: Deep store by itself is not sufficient for restore operations. Pinot stores metadata such as table config, schema, segment metadata in Zookeeper. For restore operations, both Deep Store as well as Zookeeper metadata are required.

How do segments get into the deep store?

There are several different ways that segments are persisted in the deep store.

For offline tables, the batch ingestion job writes the segment directly into the deep store, as shown in the diagram below:

The ingestion job then sends a notification about the new segment to the controller, which in turn notifies the appropriate server to pull down that segment.

For real-time tables, by default, a segment is first built-in memory by the server. It is then uploaded to the lead controller (as part of the Segment Completion Protocol sequence), which writes the segment into the deep store, as shown in the diagram below:

Having all segments go through the controller can become a system bottleneck under heavy load, in which case you can use the peer download policy, as described in .

When using this configuration, the server will directly write a completed segment to the deep store, as shown in the diagram below:

Configuring the deep store

For hands-on examples of how to configure the deep store, see the following tutorials:

A Pinot schema defines the columns that exist in a table and how Pinot should treat them. The important part is not only the column list, but also the shape of the table: which fields are dimensions, metrics, and time fields, how nulls behave, and whether the table is built for offline, realtime, or hybrid ingestion.

Pinot stores schema and table metadata separately, but the two should be designed together. Keep the schema narrow enough to match the data you actually query, and keep the table config dense enough for reference pages rather than this narrative overview.

What to design

The schema answers four practical questions:

• What columns exist?

• What data type does each column use?

• Which columns are dimensions, metrics, or date-time fields?

Good defaults

Use column names that are stable and business-facing. Prefer simple types that match the source data. Add only the fields you need at query time, because schema changes are additive and should be deliberate.

For time columns, keep one primary time field in mind for retention and hybrid-table boundary behavior. For null handling, decide early whether the table needs column-based or table-based semantics.

Example schema

When to use the reference pages

Use the when you need the exact JSON fields, validation rules, or date-time field formats. Use the when you need indexing, retention, or routing configuration.

What this page covered

This page covered the parts of Pinot schema design that shape ingestion and query behavior.

Next step

Read if one query name should route to multiple physical tables.

Related pages

Pinot supports several source formats and deep-storage choices. Pick these early, because they affect how segments are produced, moved, and recovered.

Source formats

Use the original format docs when you need the exact supported file types or loader behavior. The main landing page is Supported Data Formats.

Filesystems and deep storage

Choose the deep-storage backend that matches your operational environment. The detailed filesystem docs still live under .

Keep it simple

Do not mix format decisions with schema design. The schema says what the data means; the filesystem says where segments survive after Pinot produces them.

What this page covered

This page covered how source formats and deep storage fit into the ingestion design.

Next step

Read if data needs cleanup or pre-aggregation before query time.

Related pages

Row Expression

ROW()

Description:

ROW value expressions is supported in Pinot in comparison contexts, enabling efficient keyset pagination queries. ROW expressions allow users to write cleaner multi-column comparisons like WHERE (col1, col2, col3) > (val1, val2, val3) instead of verbose nested conditions. The evaluation of row expression is based on the () for the comparison operators.

Syntax:

Pinot supports implicit ROW-style expressions in comparison predicates using a parenthesized list of expressions on both sides of the comparator:

WHERE (col1, col2, col3) > (val1, val2, val3)

Supported comparison operators:

Note: Explicit use of the ROW() keyword (e.g., WHERE ROW(col1, col2) = ROW(1, 2)) is not yet supported due to the current SQL parser configuration (SqlConformanceEnum.BABEL). Future improvements may enable explicit row value constructors.

Note:

• ROW comparisons are lexicographic, not element-wise

• Pinot does not materialize row types — it rewrites comparisons at planning time

• Rewrite complexity grows linearly with the number of columns

Sample Example Usage:

Equality (=)

is rewritten to

Greater Than (>)

is rewritten to

Less Than (<)

is rewritten to

Use this section when you are designing tables, ingesting data, querying Pinot, choosing indexes, or connecting Pinot to applications and tools. The goal here is to help you decide what to do next and then take you to the right detailed docs without forcing you through raw reference first.

Core build workflows

When to use Reference

If you already know the exact property, endpoint, or plugin you need, jump to the section. Build-focused pages in this section explain how pieces fit together. Reference pages stay dense on purpose.

What this page covered

This page introduced the task-oriented Build with Pinot structure and pointed to the main workflows for modeling, ingestion, querying, indexing, and integration.

Next step

Start with the workflow that matches your immediate task, such as or .

Related pages

Pinot works best when the table shape is clear before data lands. Start here to understand the structure that every ingestion and query decision depends on: schema design, table composition, logical-table layout, and how schemas evolve without breaking existing pipelines.

If you need dense JSON config or controller endpoints, jump to the Reference section instead. This section stays narrative and decision-oriented.

Start Here

Related Existing Docs

What this page covered

This landing page defines the scope of Pinot data modeling and points to the core pages that matter first.

Next step

Read to lock in the table structure before designing ingestion.

Related pages

Logical tables are a naming and routing layer on top of physical tables. They let you split data by region, age, or operating mode while keeping one user-facing table name.

Use a logical table when the split is an implementation detail, not part of the query contract. Keep the physical tables aligned on schema, and use a reference physical table only as a metadata anchor.

When they help

Logical tables are most useful when you need one of these patterns:

Different physical tables per region or business unit.

Separate offline and realtime tables that still answer one business question.

Time-sliced tables that should be queried together.

Design rules

Keep the underlying schemas aligned. Keep the logical name stable. Prefer this pattern only when the underlying split is operationally meaningful; do not use it to hide a modeling problem that should instead be solved with cleaner ingestion.

For hybrid-style layouts, make the time boundary explicit so Pinot does not double count overlapping data.

Example pattern

Learn more

The original logical-table walkthrough lives in .

What this page covered

This page covered when to use logical tables and how they hide physical table splits from readers.

Next step

Read if the schema needs to grow after the table is already in production.

Related pages

Ingestion transformations clean up source records before they become Pinot rows. Ingestion aggregations reduce repeated values into fewer rows when a realtime table can safely store the summarized shape instead of the raw event stream.

Transformations

Use transformations to rename, reshape, extract, filter, or derive fields while ingesting. Keep the logic close to the table so the pipeline stays understandable.

The detailed examples still live in Ingestion Transformations.

Aggregations

Use ingestion aggregation when the use case only needs summarized realtime data. This can reduce storage and improve query performance, but it changes the data you keep, so use it only when raw rows are not needed later.

The detailed examples still live in .

What this page covered

This page covered when to transform or aggregate data during ingestion instead of waiting for query time.

Next step

Read the reference pages if you need the exact ingestion-config fields or table-config JSON.

Related pages

Outcome

Create an Amazon EKS cluster with the required tooling, ready to deploy Apache Pinot.

Pinot supports upsert on OFFLINE tables in builds that include .

Use it for batch corrections, replays, and late-arriving records.

For a full overview of upsert features (comparison columns, delete columns, TTL, metadata management), see the main page. This page covers the OFFLINE-specific configuration and differences.

FileSystem is an abstraction provided by Pinot to access data stored in distributed file systems (DFS).

Pinot uses distributed file systems for the following purposes:

• Batch ingestion job: To read the input data (CSV, Avro, Thrift, etc.) and to write generated segments to DFS.

Pinot uses the Apache Calcite SQL parser with the MYSQL_ANSI dialect. This page is the practical overview: it explains the syntax patterns most people use every day and points to the deeper reference when you need the full operator list.

Core rules

Deep dive into stages

As explained in the reference documentation, the multi-stage query engine breaks down a query into multiple stages. Each stage corresponds to a subset of the query plan and is executed independently. Stages are connected in a tree-like structure where the output of one stage is the input to another stage. The stage that is at the root of the tree sends the final results to the client. The stages that are at the leaves of the tree read from the tables. The intermediate stages process the data and send it to the next stage.

When the broker receives a query, it generates a query plan. This is a tree-like structure where each node is an operator. The plan is then optimized, moving and changing nodes to generate a plan that is semantically equivalent (it returns the same rows) but more efficient. During this phase the broker colors the nodes of the plan, assigning them to a stage. The broker also assigns a parallelism to each stage and defines which servers are going to execute each stage. For example, if a stage has a parallelism of 10, then at most 10 servers will execute that stage in parallel. One single server can execute multiple stages in parallel and it can even execute multiple instances of the same stage in parallel.

Upsert and dedup are for tables that ingest repeated keys. Use them when the current value matters more than the raw event history, or when duplicate events should not fan out into duplicate query results.

Choose the right behavior

Use upsert when newer rows should replace older rows for the same primary key.

Use dedup when repeated records should be filtered out and only the first or unique representation should remain.

Every table is associated with a tenant, or a logical namespace that restricts where the cluster processes queries on the table. A Pinot tenant takes the form of a text tag in the logical tenant namespace. Physical cluster hardware resources (i.e., brokers and servers) are also associated with a tenant tag in the common tenant namespace. Tables of a particular tenant tag will only be scheduled for storage and query processing on hardware resources that belong to the same tenant tag. This lets Pinot cluster operators assign specified workloads to certain hardware resources, preventing data in separate workloads from being stored or processed on the same physical hardware.

By default, all tables, brokers, and servers belong to a tenant called DefaultTenant, but you can configure multiple tenants in a Pinot cluster. If the cluster is planned to have multiple tenants, consider setting cluster.tenant.isolation.enable=false so that servers and brokers won't be tagged with DefaultTenant automatically while added into the cluster.

To support multi-tenancy, Pinot has first-class support for tenants. Every table is associated with a server tenant and a broker tenant, which controls the nodes used by the table as servers and brokers. Multi-tenancy lets Pinot group all tables belonging to a particular use case under a single tenant name.

The concept of tenants is very important when the multiple use cases are using Pinot and there is a need to provide quotas or some sort of isolation across tenants. For example, consider we have two tables Table A and Table B in the same Pinot cluster.

We can configure Table A with server tenant Tenant A and Table B with server tenant Tenant B. We can tag some of the server nodes for Tenant A and some for Tenant B. This will ensure that segments of Table A only reside on servers tagged with Tenant A, and segment of Table B only reside on servers tagged with Tenant B. The same isolation can be achieved at the broker level, by configuring broker tenants to the tables.

No need to create separate clusters for every table or use case!

Tenant configuration

This tenant is defined in the section of the table config.

This section contains two main fields broker and server , which decide the tenants used for the broker and server components of this table.

In the above example:

• The table will be served by brokers that have been tagged as brokerTenantName_BROKER in Helix.

• If this were an offline table, the offline segments for the table will be hosted in Pinot servers tagged in Helix as serverTenantName_OFFLINE

Create a tenant

Broker tenant

Here's a sample broker tenant config. This will create a broker tenant sampleBrokerTenant by tagging three untagged broker nodes as sampleBrokerTenant_BROKER.

To create this tenant use the following command. The creation will fail if number of untagged broker nodes is less than numberOfInstances.

Check out the table config in the to make sure it was successfully uploaded.

Server tenant

Here's a sample server tenant config. This will create a server tenant sampleServerTenant by tagging 1 untagged server node as sampleServerTenant_OFFLINE and 1 untagged server node as sampleServerTenant_REALTIME.

To create this tenant use the following command. The creation will fail if number of untagged server nodes is less than offlineInstances + realtimeInstances.

Check out the table config in the to make sure it was successfully uploaded.

Outcome

By the end of this page you will have a Pinot schema and an offline table called transcript registered in your cluster, ready to receive data.

Prerequisites

• A running Pinot cluster. See the install guides for or .

• For Docker users: the cluster must be on the pinot-demo network.

• Confirm your Pinot version. See the page and set the PINOT_VERSION environment variable:

Steps

1. Understand schemas

A Pinot schema defines every column in your table and assigns each one a column type. There are three column types:

Every table must have a schema before it can accept data. The schema tells Pinot how to interpret, index, and store each field.

2. Create the data directory

3. Save the sample CSV data

Create the file /tmp/pinot-quick-start/rawdata/transcript.csv with the following contents:

In this dataset, studentID, firstName, lastName, gender, and subject are dimensions, score is a metric, and timestampInEpoch is the datetime column.

4. Save the schema

Create the file /tmp/pinot-quick-start/transcript-schema.json:

5. Understand table configs

A table config tells Pinot how to manage the table at runtime -- which columns to index, how many replicas to keep, which tenants to assign, and whether the table is OFFLINE (batch) or REALTIME (streaming). You pair one table config with one schema.

6. Save the offline table config

Create the file /tmp/pinot-quick-start/transcript-table-offline.json:

7. Upload the schema and table config

Verify

1. Open the Pinot Data Explorer at .

2. Navigate to the Tables tab.

3. Confirm you see transcript_OFFLINE listed.

If the table appears, the schema and table config were registered successfully.

Next step

You now have an empty table. Continue to to import the CSV data into your transcript table.

A Pinot cluster is a collection of the software processes and hardware resources required to ingest, store, and process data. For detail about Pinot cluster components, see Physical architecture.

A Pinot cluster consists of the following processes, which are typically deployed on separate hardware resources in production. In development, they can fit comfortably into Docker containers on a typical laptop:

• Controller: Maintains cluster metadata and manages cluster resources.

• Zookeeper: Manages the Pinot cluster on behalf of the controller. Provides fault-tolerant, persistent storage of metadata, including table configurations, schemas, segment metadata, and cluster state.

• Broker: Accepts queries from client processes and forwards them to servers for processing.

• Server: Provides storage for segment files and compute for query processing.

• (Optional) Minion: Computes background tasks other than query processing, minimizing impact on query latency. Optimizes segments, and builds additional indexes to ensure performance (even if data is deleted).

The simplest possible Pinot cluster consists of four components: a server, a broker, a controller, and a Zookeeper node. In production environments, these components typically run on separate server instances, and scale out as needed for data volume, load, availability, and latency. Pinot clusters in production range from fewer than ten total instances to more than 1,000.

Pinot uses as a distributed metadata store and for cluster management.

Helix is a cluster management solution that maintains a persistent, fault-tolerant map of the intended state of the Pinot cluster. Helix constantly monitors the cluster to ensure that the right hardware resources are allocated for the present configuration. When the configuration changes, Helix schedules or decommissions hardware resources to reflect the new configuration. When elements of the cluster change state catastrophically, Helix schedules hardware resources to keep the actual cluster consistent with the ideal represented in the metadata. From a physical perspective, Helix takes the form of a controller process plus agents running on servers and brokers.

Cluster configuration

For details of cluster configuration settings, see .

Cluster components

Helix divides nodes into logical components based on their responsibilities:

Participant

Participants are the nodes that host distributed, partitioned resources

Pinot servers are modeled as participants. For details about server nodes, see .

Spectator

Spectators are the nodes that observe the current state of each participant and use that information to access the resources. Spectators are notified of state changes in the cluster (state of a participant, or that of a partition in a participant).

Pinot brokers are modeled as spectators. For details about broker nodes, see .

Controller

The node that observes and controls the Participant nodes. It is responsible for coordinating all transitions in the cluster and ensuring that state constraints are satisfied while maintaining cluster stability.

Pinot controllers are modeled as controllers. For details about controller nodes, see .

Logical view

Another way to visualize the cluster is a logical view, where:

• A cluster contains

• Tenants contain

• Tables contain

Set up a Pinot cluster

Typically, there is only one cluster per environment/data center. There is no need to create multiple Pinot clusters because Pinot supports .

To set up a cluster, see one of the following guides:

The Pinot controller schedules and reschedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, the Pinot controller schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (for example, ingest of real-time tables and offline tables). The Pinot controller can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration.

The controller exposes a REST API endpoint for cluster-wide administrative operations as well as a web-based query console to execute interactive SQL queries and perform simple administrative tasks.

The Pinot controller is responsible for the following:

• Maintaining global metadata (e.g., configs and schemas) of the system with the help of Zookeeper which is used as the persistent metadata store.

• Hosting the Helix Controller and managing other Pinot components (brokers, servers, minions)

• Maintaining the mapping of which servers are responsible for which segments. This mapping is used by the servers to download the portion of the segments that they are responsible for. This mapping is also used by the broker to decide which servers to route the queries to.

• Serving admin endpoints for viewing, creating, updating, and deleting configs, which are used to manage and operate the cluster.

• Serving endpoints for segment uploads, which are used in offline data pushes. They are responsible for initializing real-time consumption and coordination of persisting real-time segments into the segment store periodically.

• Undertaking other management activities such as managing retention of segments, validations.

For redundancy, there can be multiple instances of Pinot controllers. Pinot expects that all controllers are configured with the same back-end storage system so that they have a common view of the segments (e.g. NFS). Pinot can use other storage systems such as HDFS or .

Running the periodic task manually

The controller runs several periodic tasks in the background, to perform activities such as management and validation. Each periodic task has to define the run frequency and default frequency. Each task runs at its own schedule or can also be triggered manually if needed. The task runs on the lead controller for each table.

For period task configuration details, see .

Use the GET /periodictask/names API to fetch the names of all the periodic tasks running on your Pinot cluster.

To manually run a named periodic task, use the GET /periodictask/run API:

The Log Request Id (api-09630c07) can be used to search through pinot-controller log file to see log entries related to execution of the Periodic task that was manually run.

If tableName (and its type OFFLINE or REALTIME) is not provided, the task will run against all tables.

Starting a controller

Make sure you've . If you're using Docker, make sure to . To start a controller:

Apache Pinot provides multiple Kafka connector versions to match different Kafka broker deployments. Choose the connector that matches your Kafka cluster version.

Available Connectors

Kafka 4.0 Connector

The Kafka 4.0 connector (pinot-kafka-4.0) supports Apache Kafka 4.x brokers running in KRaft mode (ZooKeeper-free). It uses pure Java Kafka clients with no Scala dependency, resulting in a smaller deployment footprint.

When to use Kafka 4.0

• Your Kafka cluster runs Kafka 4.0+ with KRaft mode

• You want to eliminate the Scala transitive dependency

• You are deploying new Pinot clusters against modern Kafka infrastructure