Only this pageAll pages
Powered by GitBook
Couldn't generate the PDF for 463 pages, generation stopped at 100.
Extend with 50 more pages.
1 of 100

release-1.3.0

Loading...

Basics

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Concepts

Explore the fundamental concepts of Apache Pinot™ as a distributed OLAP database.

Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including:

  • Storing data in columnar form to support high-performance scanning

  • Sharding of data to scale both storage and computation

  • A distributed architecture designed to scale capacity linearly

  • A tabular data model read by SQL queries

To learn about Pinot components, terminology, and gain a conceptual understanding of how data is stored in Pinot, review the following sections:

  • Pinot storage model

  • Pinot architecture

  • Pinot components

Introduction

Apache Pinot is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics, and perfect for user-facing analytical workloads.

Apache Pinot™ is a real-time distributed online analytical processing (OLAP) datastore. Use Pinot to ingest and immediately query data from streaming or batch data sources (including, Apache Kafka, Amazon Kinesis, Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage).

We'd love to hear from you! Join us in our Slack channel to ask questions, troubleshoot, and share feedback.

Apache Pinot includes the following:

  • Ultra low-latency analytics even at extremely high throughput.

  • Columnar data store with several smart indexing and pre-aggregation techniques.

  • Scaling up and out with no upper bound.

  • Consistent performance based on the size of your cluster and an expected query per second (QPS) threshold.

It's perfect for user-facing real-time analytics and other analytical use cases, including internal dashboards, anomaly detection, and ad hoc data exploration.

User-facing real-time analytics

User-facing analytics refers to the analytical tools exposed to the end users of your product. In a user-facing analytics application, all users receive personalized analytics on their devices, resulting in hundreds of thousands of queries per second. Queries triggered by apps may grow quickly in proportion to the number of active users on the app, as many as millions of events per second. Data generated in Pinot is immediately available for analytics in latencies under one second.

User-facing real-time analytics requires the following:

  • Fresh data. The system needs to be able to ingest data in real time and make it available for querying, also in real time.

  • Support for high-velocity, highly dimensional event data from a wide range of actions and from multiple sources.

  • Low latency. Queries are triggered by end users interacting with apps, resulting in hundreds of thousands of queries per second with arbitrary patterns.

  • Reliability and high availability.

  • Scalability.

  • Low cost to serve.

Why Pinot?

Pinot is designed to execute OLAP queries with low latency. It works well where you need fast analytics, such as aggregations, on both mutable and immutable data.

User-facing, real-time analytics

Pinot was originally built at LinkedIn to power rich interactive real-time analytics applications, such as Who Viewed Profile, Company Analytics, Talent Insights, and many more. UberEats Restaurant Manager is another example of a user-facing analytics app built with Pinot.

Real-time dashboards for business metrics

Pinot can perform typical analytical operations such as slice and dice, drill down, roll up, and pivot on large scale multi-dimensional data. For instance, at LinkedIn, Pinot powers dashboards for thousands of business metrics. Connect various business intelligence (BI) tools such as Superset, Tableau, or PowerBI to visualize data in Pinot.

Enterprise business intelligence

For analysts and data scientists, Pinot works well as a highly-scalable data platform for business intelligence. Pinot converges big data platforms with the traditional role of a data warehouse, making it a suitable replacement for analysis and reporting.

Enterprise application development

For application developers, Pinot works well as an aggregate store that sources events from streaming data sources, such as Kafka, and makes it available for a query using SQL. You can also use Pinot to aggregate data across a microservice architecture into one easily queryable view of the domain.

Pinot tenants prevent any possibility of sharing ownership of database tables across microservice teams. Developers can create their own query models of data from multiple systems of record depending on their use case and needs. As with all aggregate stores, query models are eventually consistent.

Get started

If you're new to Pinot, take a look at our Getting Started guide:

To start importing data into Pinot, see how to import batch and stream data:

To start querying data in Pinot, check out our Query guide:

Learn

For a conceptual overview that explains how Pinot works, check out the Concepts guide:

To understand the distributed systems architecture that explains Pinot's operating model, take a look at our basic architecture section:

0.12.1

Summary

This is a bug-fixing release contains:

  • use legacy case-when format ()

The release is based on the release 0.12.0 with the following cherry-picks:

https://github.com/apache/pinot/pull/10291
6f5a8fc883e1d576117fdb92f09103067672aaca

Segment threshold

Learn how segment thresholds work in Pinot.

The segment threshold determines when a segment is committed in real-time tables.

When data is first ingested from a streaming provider like Kafka, Pinot stores the data in a consuming segment.

This segment is on the disk of the server(s) processing a particular partition from the streaming provider.

However, it's not until a segment is committed that the segment is written to the deep store. The segment threshold decides when that should happen.

Why is the segment threshold important?

The segment threshold is important because it ensures segments are a reasonable size.

When queries are processed, smaller segments may increase query latency due to more overhead (number of threads spawned, meta data processing, and so on).

Larger segments may cause servers to run out of memory. When a server is restarted, the consuming segment must start consuming from the first row again, causing a lag between Pinot and the streaming provider.

Mark Needham explains the segment threshold

Segment retention

In this Apache Pinot concepts guide, we'll learn how segment retention works.

Segments in Pinot tables have a retention time, after which the segments are deleted. Typically, offline tables retain segments for a longer period of time than real-time tables.

The removal of segments is done by the retention manager. By default, the retention manager runs once every 6 hours.

The retention manager purges two types of segments:

  • Expired segments: Segments whose end time has exceeded the retention period.

  • Replaced segments: Segments that have been replaced as part of the merge rollup task.

There are a couple of scenarios where segments in offline tables won't be purged:

  • If the segment doesn't have an end time. This would happen if the segment doesn't contain a time column.

  • If the segment's table has a segmentIngestionType of REFRESH.

If the retention period isn't specified, segments aren't purged from tables.

The retention manager initially moves these segments into a Deleted Segments area, from where they will eventually be permanently removed.

From Query Console

Insert a file into Pinot from Query Console

This feature is supported after the 0.11.0 release. Reference PR: https://github.com/apache/pinot/pull/8557

Prerequisite

  • Ensure you have available Pinot Minion instances deployed within the cluster.

  • Pinot version is 0.11.0 or above

How it works

  1. Parse the query with the table name and directory URI along with a list of options for the ingestion job.

  2. Call controller minion task execution API endpoint to schedule the task on minion

  3. Response has the schema of table name and task job id.

Usage Syntax

INSERT INTO [database.]table FROM FILE dataDirURI OPTION ( k=v ) [, OPTION (k=v)]*

Example

SET taskName = 'myTask-s3';
SET input.fs.className = 'org.apache.pinot.plugin.filesystem.S3PinotFS';
SET input.fs.prop.accessKey = 'my-key';
SET input.fs.prop.secretKey = 'my-secret';
SET input.fs.prop.region = 'us-west-2';
INSERT INTO "baseballStats"
FROM FILE 's3://my-bucket/public_data_set/baseballStats/rawdata/'

Insert Rows into Pinot

We are actively developing this feature...

The details will be revealed soon.

FST index

The FST index supports regex queries on text. Decreases on-disk index by 4-6 times.

  • Only supports regex queries

  • Only supported on stored or completed Pinot segments (no consuming segments).

  • Only supported on dictionary-encoded columns.

  • Works better for prefix queries

Note: Lucene is case sensitive as such when using FST index based column(s) in query, user needs to ensure this is taken into account. For e.g Select * from table T where colA LIKE %Value% which has a FST index on colA will only return rows containing string "Value" but not "value".

For more information on the FST construction and code, see Lucene documentation.

Enable the FST index

To enable the FST index on a dictionary-encoded column, include the following configuration:

"fieldConfigList":[
{
"name":"text_col_1",
"encodingType":"DICTIONARY",
"indexType":"FST"
}
]

The FST index generates one FST index file (.lucene.fst). If the inverted index is enabled, this is further able to take advantage of that.

For more information about enabling the FST index, see ways to enable indexes.

0.9.1

Summary

This release fixes the major issue of CVE-2021-44228 and a major bug fixing of pinot admin exit code issue(#7798).

The release is based on the release 0.9.0 with the following cherry-picks:

e44d2e4 af2858a

Getting Started
Import Data
Query
Concepts
Architecture

Frequently Asked Questions (FAQs)

This page lists pages with frequently asked questions with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, make a pull request.

Components

Discover the core components of Apache Pinot, enabling efficient data processing and analytics. Unleash the power of Pinot's building blocks for high-performance data-driven applications.

Apache Pinot™ is a database designed to deliver highly concurrent, ultra-low-latency queries on large datasets through a set of common data model abstractions. Delivering on these goals requires several foundational architectural commitments, including:

  • Storing data in columnar form to support high-performance scanning

  • Sharding of data to scale both storage and computation

  • A distributed architecture designed to scale capacity linearly

  • A tabular data model read by SQL queries

Components

Learn about the major components and logical abstractions used in Pinot.

Operator reference

Developer reference

Broker

Discover how Apache Pinot's broker component optimizes query processing, data retrieval, and enhances data-driven applications.

Pinot brokers take query requests from client processes, scatter them to applicable servers, gather the results, and return results to the client. The controller shares cluster metadata with the brokers, which allows the brokers to create a plan for executing the query involving a minimal subset of servers with the source data and, when required, other servers to shuffle and consolidate results.

A production Pinot cluster contains many brokers. In general, the more brokers, the more concurrent queries a cluster can process, and the lower latency it can deliver on queries.

Pinot brokers are modeled as Helix spectators. They need to know the location of each segment of a table (and each replica of the segments) and route requests to the appropriate server that hosts the segments of the table being queried.

The broker ensures that all the rows of the table are queried exactly once so as to return correct, consistent results for a query. The brokers may optimize to prune some of the segments as long as accuracy is not sacrificed.

Helix provides the framework by which spectators can learn the location in which each partition of a resource (i.e. participant) resides. The brokers use this mechanism to learn the servers that host specific segments of a table.

In the case of hybrid tables, the brokers ensure that the overlap between real-time and offline segment data is queried exactly once, by performing offline and real-time federation.

Let's take this example, we have real-time data for five days - March 23 to March 27, and offline data has been pushed until Mar 25, which is two days behind real-time. The brokers maintain this time boundary.

Suppose, we get a query to this table : select sum(metric) from table. The broker will split the query into 2 queries based on this time boundary – one for offline and one for real-time. This query becomes select sum(metric) from table_REALTIME where date >= Mar 25 and select sum(metric) from table_OFFLINE where date < Mar 25

The broker merges results from both these queries before returning the result to the client.

Starting a broker

Make sure you've . If you're using Docker, make sure to . To start a broker:

Time boundary

Learn about time boundaries in hybrid tables.

Learn about time boundaries in hybrid tables. Hybrid tables are when we have offline and real-time tables with the same name.

When querying these tables, the Pinot broker decides which records to read from the offline table and which to read from the real-time table. It does this using the time boundary.

How is the time boundary determined?

The time boundary is determined by looking at the maximum end time of the offline segments and the segment ingestion frequency specified for the offline table.

If it's set to hourly, then:

Otherwise:

It is possible to force the hybrid table to use max(all offline segments' end time) by calling the API (V 0.12.0+)

Note that this will not automatically update the time boundary as more segments are added to the offline table, and must be called each time a segment with more recent end time is uploaded to the offline table. You can revert back to using the derived time boundary by calling API:

Querying

When a Pinot broker receives a query for a hybrid table, the broker sends a time boundary annotated version of the query to the offline and real-time tables.

For example, if we executed the following query:

The broker would send the following query to the offline table:

And the following query to the real-time table:

The results of the two queries are merged by the broker before being returned to the client.

Troubleshooting Pinot

Find debug information in Pinot

Pinot offers various ways to assist with troubleshooting and debugging problems that might happen.

Start with the which will surface many of the commonly occurring problems. The debug api provides information such as tableSize, ingestion status, and error messages related to state transition in server.

The table debug API can be invoked via the Swagger UI, as in the following image:

It can also be invoked directly by accessing the URL as follows. The api requires the tableName, and can optionally take tableType (offline|realtime) and verbosity level.

Pinot also provides a variety of operational metrics that can be used for creating dashboards, alerting and .

Finally, all pinot components log debug information related to error conditions.

Debug a slow query or a query which keeps timing out

Use the following steps:

  1. If the query executes, look at the query result. Specifically look at numEntriesScannedInFilter and numDocsScanned.

    1. If numEntriesScannedInFilter is very high, consider adding indexes for the corresponding columns being used in the filter predicates. You should also think about partitioning the incoming data based on the dimension most heavily used in your filter queries.

    2. If numDocsScanned is very high, that means the selectivity for the query is low and lots of documents need to be processed after the filtering. Consider refining the filter to increase the selectivity of the query.

  2. If the query is not executing, you can extend the query timeout by appending a timeoutMs parameter to the query, for example, select * from mytable limit 10 option(timeoutMs=60000). Then repeat step 1, as needed.

  3. Look at garbage collection (GC) stats for the corresponding Pinot servers. If a particular server seems to be running full GC all the time, you can do a couple of things such as

    1. Increase Java Virtual Machine (JVM) heap (java -Xmx<size>).

    2. Consider using off-heap memory for segments.

    3. Decrease the total number of segments per server (by partitioning the data in a more efficient way).

General

This page has a collection of frequently asked questions of a general nature with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, .

How does Apache Pinot use deep storage?

When data is pushed to Apache Pinot, Pinot makes a backup copy of the data and stores it on the configured deep-storage (S3/GCP/ADLS/NFS/etc). This copy is stored as tar.gz Pinot segments. Note, that Pinot servers keep a (untarred) copy of the segments on their local disk as well. This is done for performance reasons.

How does Pinot use Zookeeper?

Pinot uses Apache Helix for cluster management, which in turn is built on top of Zookeeper. Helix uses Zookeeper to store the cluster state, including Ideal State, External View, Participants, and so on. Pinot also uses Zookeeper to store information such as Table configurations, schemas, Segment Metadata, and so on.

Why am I getting "Could not find or load class" error when running Quickstart using 0.8.0 release?

Check the JDK version you are using. You may be getting this error if you are using an older version than the current Pinot binary release was built on. If so, you have two options: switch to the same JDK release as Pinot was built with or download the for the Pinot release and it locally.

How to change TimeZone when running Pinot?

Pinot uses the local timezone by default. To change the timezone, set the pinot.timezone value in the .conf config file. It is set once for all Pinot components (Controller, Broker, Server, Minion). See the following sample configuration:

Reload a table segment

Reload a table segment in Apache Pinot.

When Pinot writes data to segments in a table, it saves those segments to a deep store location specified in your , such as a storage drive or Amazon S3 bucket.

If a new column is added to your table or schema configuration during ingestion, incorrect data may appear in the consuming segment(s). To ensure accurate values are reloaded, see how to .

Use the Pinot Controller API to reload segments

To reload all segments from a table, use:

To reload a specific segment from a table, use:

A successful API call returns the following response:

Use the Pinot Admin Console to reload segments

To use the Pinot Admin Console, do the following:

  1. From the left navigation menu, select Cluster Manager.

  2. Under TENANTS, select the Tenant Name.

  3. From the list of tables in the tenant, select the Table Name.

  4. Do one of the following:

    • To reload all segments, under OPERATIONS, click Reload All Segments.

    • To reload a specific segment, under SEGMENTS, select the Segment Name, and then in the new OPERATIONS section, select Reload Segment.

Range index

This page describes configuring the range index for Apache Pinot

Range indexing allows you to get better performance for queries that involve filtering over a range.

It would be useful for a query like the following:

A range index is a variant of an , where instead of creating a mapping from values to columns, we create mapping of a range of values to columns. You can use the range index by setting the following config in the .

Range index is supported for dictionary encoded columns of any type as well as raw encoded columns of a numeric type. Note that the range index can also be used on a dictionary encoded time column using STRING type, since Pinot only supports datetime formats that are in lexicographical order.

A good thumb rule is to use a range index when you want to apply range predicates on metric columns that have a very large number of unique values. This is because using an inverted index for such columns will create a very large index that is inefficient in terms of storage and performance.

0.9.3

Summary

This is a bug fixing release contains:

  • Update Log4j to 2.17.0 to address ()

The release is based on the release 0.9.2 with the following cherry-picks:

General
Pinot On Kubernetes FAQ
Ingestion FAQ
Query FAQ
Operations FAQ
timeBoundary = Maximum end time of offline segments - 1 hour
timeBoundary = Maximum end time of offline segments - 1 day
curl -X POST \
  "http://localhost:9000/tables/{tableName}/timeBoundary" \
  -H "accept: application/json"
curl -X DELETE \
  "http://localhost:9000/tables/{tableName}/timeBoundary" \
  -H "accept: application/json"
SELECT count(*)
FROM events
SELECT count(*)
FROM events_OFFLINE
WHERE timeColumn <= $timeBoundary
SELECT count(*)
FROM events_REALTIME
WHERE timeColumn > $timeBoundary
pinot.timezone=UTC
make a pull request
source code
build
POST /segments/{tableName}/reload
POST /segments/{tableName}/{segmentName}/reload
{
    "status": "200"
}
table configuration
add a new column during ingestion
SELECT COUNT(*) 
FROM baseballStats 
WHERE hits > 11
{
    "tableIndexConfig": {
        "rangeIndexColumns": [
            "column_name",
            ...
        ],
        ...
    }
}
inverted index
table configuration
CVE-2021-45105
#7933
93c0404

Backfill Data

Batch ingestion of backfill data into Apache Pinot.

Introduction

Pinot batch ingestion involves two parts: routine ingestion job(hourly/daily) and backfill. Here are some examples to show how routine batch ingestion works in Pinot offline table:

  • Batch Ingestion Overview

  • Batch Ingestion in Practice

High-level description

  1. Organize raw data into buckets (eg: /var/pinot/airlineStats/rawdata/2014/01/01). Each bucket typically contains several files (eg: /var/pinot/airlineStats/rawdata/2014/01/01/airlineStats_data_2014-01-01_0.avro)

  2. Run a Pinot batch ingestion job, which points to a specific date folder like ‘/var/pinot/airlineStats/rawdata/2014/01/01’. The segment generation job will convert each such avro file into a Pinot segment for that day and give it a unique name.

  3. Run Pinot segment push job to upload those segments with those uniques names via a Controller API

IMPORTANT: The segment name is the unique identifier used to uniquely identify that segment in Pinot. If the controller gets an upload request for a segment with the same name - it will attempt to replace it with the new one.

This newly uploaded data can now be queried in Pinot. However, sometimes users will make changes to the raw data which need to be reflected in Pinot. This process is known as 'Backfill'.

How to backfill data in Pinot

Pinot supports data modification only at the segment level, which means you must update entire segments for doing backfills. The high level idea is to repeat steps 2 (segment generation) and 3 (segment upload) mentioned above:

  • Backfill jobs must run at the same granularity as the daily job. E.g., if you need to backfill data for 2014/01/01, specify that input folder for your backfill job (e.g.: ‘/var/pinot/airlineStats/rawdata/2014/01/01’)

  • The backfill job will then generate segments with the same name as the original job (with the new data).

  • When uploading those segments to Pinot, the controller will replace the old segments with the new ones (segment names act like primary keys within Pinot) one by one.

Edge case example

Backfill jobs expect the same number of (or more) data files on the backfill date. So the segment generation job will create the same number of (or more) segments than the original run.

For example, assuming table airlineStats has 2 segments(airlineStats_2014-01-01_2014-01-01_0, airlineStats_2014-01-01_2014-01-01_1) on date 2014/01/01 and the backfill input directory contains only 1 input file. Then the segment generation job will create just one segment: airlineStats_2014-01-01_2014-01-01_0. After the segment push job, only segment airlineStats_2014-01-01_2014-01-01_0 got replaced and stale data in segment airlineStats_2014-01-01_2014-01-01_1 are still there.

If the raw data is modified in such a way that the original time bucket has fewer input files than the first ingestion run, backfill will fail.

Dimension table

Batch ingestion of data into Apache Pinot using dimension tables.

Dimension tables are a special kind of offline tables from which data can be looked up via the lookup UDF, providing join-like functionality.

Dimension tables are replicated on all the hosts for a given tenant to allow faster lookups. When a table is marked as a dimension table, it will be replicated on all the hosts, which means that these tables must be small in size.

A dimension table cannot be part of a hybrid table.

Configure dimension tables using following properties in the table configuration:

  • isDimTable: Set to true.

  • ingestionConfig.batchIngestionConfig.segmentIngestionType: Set to REFRESH.

  • dimensionTableConfig.disablePreload: By default, dimension tables are preloaded to allow for fast lookups. Set to true to trade off speed for memory by storing only the segment reference and docID. Otherwise, the whole row is stored in the Dimension table hash map.

  • controller.dimTable.maxSize: Determines the maximum size quota for a dimension table in a cluster. Table creation will fail if the storage quota exceeds this maximum size.

  • dimensionFieldSpecs: To look up dimension values, dimension tables need a primary key. For details, see dimensionFieldSpecs.

Example dimension table configuration

{
  "OFFLINE": {
    "tableName": "dimBaseballTeams_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
    },
    "ingestionConfig": {
      "batchIngestionConfig": {
        "segmentIngestionType": "REFRESH"
      }
    }
    "quota": {
      "storage": "200M"
    },
    "isDimTable": true,
    "dimensionTableConfig": {
      "disablePreload": true
    }
  }
}

Example table schema configuration

{
  "dimensionFieldSpecs": [
    {
      "dataType": "STRING",
      "name": "teamID"
    },
    {
      "dataType": "STRING",
      "name": "teamName"
    }
  ],
  "schemaName": "dimBaseballTeams",
  "primaryKeyColumns": ["teamID"]
}

File Systems

This section contains a collection of short guides to show you how to import data from a Pinot-supported file system.

FileSystem is an abstraction provided by Pinot to access data stored in distributed file systems (DFS).

Pinot uses distributed file systems for the following purposes:

  • Batch ingestion job: To read the input data (CSV, Avro, Thrift, etc.) and to write generated segments to DFS.

  • Controller: When a segment is uploaded to the controller, the controller saves it in the configured DFS.

  • Server:- When a server(s) is notified of a new segment, the server copies the segment from remote DFS to their local node using the DFS abstraction.

Supported file systems

Pinot lets you choose a distributed file system provider. The following file systems are supported by Pinot:

  • Amazon S3

  • Google Cloud Storage

  • HDFS

  • Azure Data Lake Storage

Enabling a file system

To use a distributed file system, you need to enable plugins. To do that, specify the plugin directory and include the required plugins:

-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-plugin-to-include-1,pinot-plugin-to-include-2

You can change the file system in the controller and server configuration. In the following configuration example, the URI is s3://bucket/path/to/file and scheme refers to the file system URI prefix s3.

#CONTROLLER

pinot.controller.storage.factory.class.[scheme]=className of the pinot file system
pinot.controller.segment.fetcher.protocols=file,http,[scheme]
pinot.controller.segment.fetcher.[scheme].class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
#SERVER

pinot.server.storage.factory.class.[scheme]=className of the Pinot file system
pinot.server.segment.fetcher.protocols=file,http,[scheme]
pinot.server.segment.fetcher.[scheme].class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

You can also change the file system during ingestion. In the ingestion job spec, specify the file system with the following configuration:

pinotFSSpecs
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS

Native text index

This page talks about native text indices and corresponding search functionality in Apache Pinot.

Experimental

This index is experimental and should only be used for testing. It is not recommended for use in production.

Instead, use this text index.

Native text index

Pinot supports text indexing and search by building Lucene indices as sidecars to the main Pinot segments. While this is a great technique, it essentially limits the avenues of optimizations that can be done for Pinot specific use cases of text search.

How is Pinot different?

Pinot, like any other database/OLAP engine, does not need to conform to the entire full text search domain-specific language (DSL) that is traditionally used by full-text search (FTS) engines like ElasticSearch and Solr. In traditional SQL text search use cases, the majority of text searches belong to one of three patterns: prefix wildcard queries (like pino*), postfix or suffix wildcard queries (like *inot), and term queries (like pinot).

Native text indices in Pinot

In Pinot, native text indices are built from the ground up. They use a custom text-indexing engine, coupled with Pinot's powerful inverted indices, to provide a fast text search experience.

The benefits are that native text indices are 80-120% faster than Lucene-based indices for the text search use cases mentioned above. They are also 40% smaller on disk.

Native text indices support real-time text search. For REALTIME tables, native text indices allow data to be indexed in memory in the text index, while concurrently supporting text searches on the same index.

Historically, most text indices depend on the in-memory text index being written to first and then sealed, before searches are possible. This limits the freshness of the search, being near-real-time at best.

Native text indices come with a custom in-memory text index, which allows for real-time indexing and search.

Searching Native Text Indices

The function, TEXT\_CONTAINS, supports text search on native text indices.

SELECT COUNT(*) FROM Foo WHERE TEXT_CONTAINS (<column_name>, <search_expression>)

Examples:

SELECT COUNT(*) FROM Foo WHERE TEXT_CONTAINS (<column_name>, "foo.*")
SELECT COUNT(*) FROM Foo WHERE TEXT_CONTAINS (<column_name>, ".*bar")
SELECT COUNT(*) FROM Foo WHERE TEXT_CONTAINS (<column_name>, "foo")

TEXT\_CONTAINS can be combined using standard boolean operators

SELECT COUNT(*) FROM Foo WHERE TEXT_CONTAINS ("col1", "foo") AND TEXT_CONTAINS ("col2", "bar")

Note: TEXT\_CONTAINS supports regex and term queries and will work only on native indices. TEXT\_CONTAINS supports standard regex patterns (as used by LIKE in SQL Standard), so there might be some syntatical differences from Lucene queries.

Creating Native Text Indices

Native text indices are created using field configurations. To indicate that an index type is native, specify it using properties in the field configuration:

"fieldConfigList":[
  {
     "name":"text_col_1",
     "encodingType":"RAW",
     "indexTypes": ["TEXT"],
     "properties":{"fstType":"native"}
  }
]

0.9.2

Summary

This is a bug fixing release contains:

  • Upgrade log4j to 2.16.0 to fix CVE-2021-45046 (#7903)

  • Upgrade swagger-ui to 3.23.11 to fix CVE-2019-17495 (#7902)

  • Fix the bug that RealtimeToOfflineTask failed to progress with large time bucket gaps (#7814).

The release is based on the release 0.9.1 with the following cherry-picks:

9ed6498 50e1613 767aa8a

docker run \
    --network=pinot-demo \
    --name pinot-broker \
    -d ${PINOT_IMAGE} StartBroker \
    -zkAddress pinot-zookeeper:2181
bin/pinot-admin.sh StartBroker \
  -zkAddress localhost:2181 \
  -clusterName PinotCluster \
  -brokerPort 7000
set up Zookeeper
pull the
Pinot Docker image
Broker interaction with other components
curl -X GET "http://localhost:9000/debug/tables/airlineStats?verbosity=0" -H "accept: application/json"
debug API
monitoring
Swagger - Table Debug Api
Cluster
Controller
Broker
Server
Minion
Tenant
Table
Schema
Segment

Server

Uncover the efficient data processing and storage capabilities of Apache Pinot's server component, optimizing performance for data-driven applications.

Pinot servers provide the primary storage for segments and perform the computation required to execute queries. A production Pinot cluster contains many servers. In general, the more servers, the more data the cluster can retain in tables, the lower latency the cluster can deliver on queries, and the more concurrent queries the cluster can process.

Servers are typically segregated into real-time and offline workloads, with "real-time" servers hosting only real-time tables, and "offline" servers hosting only offline tables. This is a ubiquitous operational convention, not a difference or an explicit configuration in the server process itself. There are two types of servers:

Offline

Offline servers are responsible for downloading segments from the segment store, to host and serve queries off. When a new segment is uploaded to the controller, the controller decides the servers (as many as replication) that will host the new segment and notifies them to download the segment from the segment store. On receiving this notification, the servers download the segment file and load the segment onto the server, to server queries off them.

Real-time

Real-time servers directly ingest from a real-time stream (such as Kafka or EventHubs). Periodically, they make segments of the in-memory ingested data, based on certain thresholds. This segment is then persisted onto the segment store.

Pinot servers are modeled as Helix participants, hosting Pinot tables (referred to as resources in Helix terminology). Segments of a table are modeled as Helix partitions (of a resource). Thus, a Pinot server hosts one or more Helix partitions of one or more helix resources (i.e. one or more segments of one or more tables).

Starting a server

Make sure you've set up Zookeeper. If you're using Docker, make sure to pull the Pinot Docker image. To start a server:

Usage: StartServer
	-serverHost               <String>                      : Host name for controller. (required=false)
	-serverPort               <int>                         : Port number to start the server at. (required=false)
	-serverAdminPort          <int>                         : Port number to serve the server admin API at. (required=false)
	-dataDir                  <string>                      : Path to directory containing data. (required=false)
	-segmentDir               <string>                      : Path to directory containing segments. (required=false)
	-zkAddress                <http>                        : Http address of Zookeeper. (required=false)
	-clusterName              <String>                      : Pinot cluster name. (required=false)
	-configFileName           <Config File Name>            : Broker Starter Config file. (required=false)
	-help                                                   : Print this message. (required=false)
docker run \
    --network=pinot-demo \
    --name pinot-server \
    -d ${PINOT_IMAGE} StartServer \
    -zkAddress pinot-zookeeper:2181
bin/pinot-admin.sh StartServer \
    -zkAddress localhost:2181

Import Data

This page lists options for importing data into Apache Pinot™ with links to detailed instructions with examples.

There are multiple options for importing data into Apache Pinot™. The pages in this section provide step-by-step instructions for importing records into Pinot, supported by our plugin architecture. The intent is to get you up and running with imported data as quickly as possible.

Pinot supports multiple file input formats without needing to change anything other than the file name. Each example imports a ready-made dataset so you can see how things work without needing to find or create your own dataset.

Pinot Batch Ingestion

These guides show you how to import data from popular big data platforms.

Pinot Stream Ingestion

This guide shows you how to import data using stream ingestion from Apache Kafka topics.

This guide shows you how to import data using stream ingestion with upsert.

This guide shows you how to import data using stream ingestion with deduplication.

This guide shows you how to import data using stream ingestion with CLP.

Pinot file systems

By default, Pinot does not come with a storage layer, so all the data sent won't be stored in case of system crash. In order to persistently store the generated segments, you will need to change controller and server configs to add a deep storage. See File systems for all the info and related configs.

These guides show you how to import data and persist it in these file systems.

Pinot input formats

This guide shows you how to import data from various Pinot-supported input formats.

This guide shows you how to handle the complex type in the ingested data, such as map and array.

This guide shows additional examples on how to work with complex types.

This guide shows you how to handle records with dynamic schemas, like JSON log events.

Reloading and uploading existing Pinot segments

This guide shows you how to reload Pinot segments from your deep store.

This guide shows you how to upload Pinot segments from an old, closed Pinot instance.

Inverted index

This page describes configuring the inverted index for Apache Pinot

We can define the forward index as a mapping from document IDs (also known as rows) to values. Similarly, an inverted index establishes a mapping from values to a set of document IDs, making it the "inverted" version of the forward index. When you frequently use a column for filtering operations like EQ (equal), IN (membership check), GT (greater than), etc., incorporating an inverted index can significantly enhance query performance.

Pinot supports two distinct types of inverted indexes: bitmap inverted indexes and sorted inverted indexes. Bitmap inverted indexes represent the actual inverted index type, whereas the sorted type is automatically available when the column is sorted. Both types of indexes necessitate the enabling of a dictionary for the respective column.

Bitmap inverted index

When a column is not sorted, and an inverted index is enabled for that column, Pinot maintains a mapping from each value to a bitmap of rows. This design ensures that value lookup operations take constant time, providing efficient querying capabilities.

When an inverted index is enabled for a column, Pinot maintains a map from each value to a bitmap of rows, which makes value lookup take constant time. If you have a column that is frequently used for filtering, adding an inverted index will improve performance greatly. You can create an inverted index on a multi-value column.

Inverted indexes are disabled by default and can be enabled for a column by specifying the configuration within the table configuration:

inverted index defined in tableConfig
{
  "fieldConfigList": [
    {
      "name": "theColumnName",
      "indexes": {
        "inverted": {}
      }
    }
  ],
  ...
}

The older way to configure inverted indexes can also be used, although it is not actually recommended:

old way to define inverted index in tableConfig
{
    "tableIndexConfig": {
        "invertedIndexColumns": [
            "theColumnName",
            ...
        ],
        ...
    }
}

When the index is created

By default, bitmap inverted indexes are not generated when the segment is initially created; instead, they are created when the segment is loaded by Pinot. This behavior is governed by the table configuration option indexingConfig.createInvertedIndexDuringSegmentGeneration, which is set to false by default.

Sorted inverted index

As explained in the forward index section, a column that is both sorted and equipped with a dictionary is encoded in a specialized manner that serves the purpose of implementing both forward and inverted indexes. Consequently, when these conditions are met, an inverted index is effectively created without additional configuration, even if the configuration suggests otherwise. This sorted version of the forward index offers a lookup time complexity of log(n) and leverages data locality.

For instance, consider the following example: if a query includes a filter on the memberId column, Pinot will perform a binary search on memberId values to find the range pair of docIds for corresponding filtering value. If the query needs to scan values for other columns after filtering, values within the range docId pair will be located together, which means we can benefit from data locality.

_images/sorted-inverted.png

A sorted inverted index indeed offers superior performance compared to a bitmap inverted index, but it's important to note that it can only be applied to sorted columns. In cases where query performance with a regular inverted index is unsatisfactory, especially when a large portion of queries involve filtering on the same column (e.g., _memberId_), using a sorted index can substantially enhance query performance.

Release notes

The following summarizes Apache Pinot™ releases, from the latest one to the earliest one.

Note

Before upgrading from one version to another one, read the release notes. While the Pinot committers strive to keep releases backward-compatible and introduce new features in a compatible manner, your environment may have a unique combination of configurations/data/schema that may have been somehow overlooked. Before you roll out a new release of Pinot on your cluster, it is best that you run the compatibility test suite that Pinot provides. The tests can be easily customized to suit the configurations and tables in your pinot cluster(s). As a good practice, you should build your own test suite, mirroring the table configurations, schema, sample data, and queries that are used in your cluster.

1.3.0 (February 2025)

1.2.0 (August 2024)

1.1.0 (March 2024)

1.0.0 (September 2023)

0.12.1 (March 2023)

0.12.0 (December 2022)

0.11.0 (September 2022)

0.10.0 (March 2022)

0.9.3 (December 2021)

0.9.2 (December 2021)

0.9.1 (December 2021)

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

Running on public clouds

This page links to multiple quick start guides for deploying Pinot to different public cloud providers.

These quickstart guides show you how to run an Apache Pinot cluster using Kubernetes on different public cloud providers.

Cluster

Learn to build and manage Apache Pinot clusters, uncovering key components for efficient data processing and optimized analysis.

A Pinot cluster is a collection of the software processes and hardware resources required to ingest, store, and process data. For detail about Pinot cluster components, see .

A Pinot cluster consists of the following processes, which are typically deployed on separate hardware resources in production. In development, they can fit comfortably into Docker containers on a typical laptop:

  • Controller: Maintains cluster metadata and manages cluster resources.

  • Zookeeper: Manages the Pinot cluster on behalf of the controller. Provides fault-tolerant, persistent storage of metadata, including table configurations, schemas, segment metadata, and cluster state.

  • Broker: Accepts queries from client processes and forwards them to servers for processing.

  • Server: Provides storage for segment files and compute for query processing.

  • (Optional) Minion: Computes background tasks other than query processing, minimizing impact on query latency. Optimizes segments, and builds additional indexes to ensure performance (even if data is deleted).

The simplest possible Pinot cluster consists of four components: a server, a broker, a controller, and a Zookeeper node. In production environments, these components typically run on separate server instances, and scale out as needed for data volume, load, availability, and latency. Pinot clusters in production range from fewer than ten total instances to more than 1,000.

Pinot uses as a distributed metadata store and for cluster management.

Helix is a cluster management solution that maintains a persistent, fault-tolerant map of the intended state of the Pinot cluster. Helix constantly monitors the cluster to ensure that the right hardware resources are allocated for the present configuration. When the configuration changes, Helix schedules or decommissions hardware resources to reflect the new configuration. When elements of the cluster change state catastrophically, Helix schedules hardware resources to keep the actual cluster consistent with the ideal represented in the metadata. From a physical perspective, Helix takes the form of a controller process plus agents running on servers and brokers.

Cluster configuration

For details of cluster configuration settings, see .

Cluster components

Helix divides nodes into logical components based on their responsibilities:

Participant

Participants are the nodes that host distributed, partitioned resources

Pinot servers are modeled as participants. For details about server nodes, see .

Spectator

Spectators are the nodes that observe the current state of each participant and use that information to access the resources. Spectators are notified of state changes in the cluster (state of a participant, or that of a partition in a participant).

Pinot brokers are modeled as spectators. For details about broker nodes, see .

Controller

The node that observes and controls the Participant nodes. It is responsible for coordinating all transitions in the cluster and ensuring that state constraints are satisfied while maintaining cluster stability.

Pinot controllers are modeled as controllers. For details about controller nodes, see .

Logical view

Another way to visualize the cluster is a logical view, where:

  • A cluster contains

  • Tenants contain

  • Tables contain

Set up a Pinot cluster

Typically, there is only one cluster per environment/data center. There is no need to create multiple Pinot clusters because Pinot supports .

To set up a cluster, see one of the following guides:

Deep Store

Leverage Apache Pinot's deep store component for efficient large-scale data storage and management, enabling impactful data processing and analysis.

The deep store (or deep storage) is the permanent store for files.

It is used for backup and restore operations. New nodes in a cluster will pull down a copy of segment files from the deep store. If the local segment files on a server gets damaged in some way (or accidentally deleted), a new copy will be pulled down from the deep store on server restart.

The deep store stores a compressed version of the segment files and it typically won't include any indexes. These compressed files can be stored on a local file system or on a variety of other file systems. For more details on supported file systems, see .

Note: Deep store by itself is not sufficient for restore operations. Pinot stores metadata such as table config, schema, segment metadata in Zookeeper. For restore operations, both Deep Store as well as Zookeeper metadata are required.

How do segments get into the deep store?

There are several different ways that segments are persisted in the deep store.

For offline tables, the batch ingestion job writes the segment directly into the deep store, as shown in the diagram below:

The ingestion job then sends a notification about the new segment to the controller, which in turn notifies the appropriate server to pull down that segment.

For real-time tables, by default, a segment is first built-in memory by the server. It is then uploaded to the lead controller (as part of the Segment Completion Protocol sequence), which writes the segment into the deep store, as shown in the diagram below:

Having all segments go through the controller can become a system bottleneck under heavy load, in which case you can use the peer download policy, as described in .

When using this configuration, the server will directly write a completed segment to the deep store, as shown in the diagram below:

Configuring the deep store

For hands-on examples of how to configure the deep store, see the following tutorials:

Running on Azure

This quickstart guide helps you get started running Pinot on Microsoft Azure.

In this quickstart guide, you will set up a Kubernetes Cluster on

1. Tooling Installation

1.1 Install Kubectl

Follow this link () to install kubectl.

For Mac users

Check kubectl version after installation.

Quickstart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12

1.2 Install Helm

To install Helm, see .

For Mac users

Check helm version after installation.

This quickstart provides helm supports for helm v3.0.0 and v2.12.1. Pick the script based on your helm version.

1.3 Install Azure CLI

Follow this link () to install Azure CLI.

For Mac users

2. (Optional) Log in to your Azure account

This script will open your default browser to sign-in to your Azure Account.

3. (Optional) Create a Resource Group

Use the following script create a resource group in location eastus.

4. (Optional) Create a Kubernetes cluster(AKS) in Azure

This script will create a 3 node cluster named pinot-quickstart for demo purposes.

Modify the parameters in the following example command with your resource group and cluster details:

Once the command succeeds, the cluster is ready to be used.

5. Connect to an existing cluster

Run the following command to get the credential for the cluster pinot-quickstart that you just created:

To verify the connection, run the following:

6. Pinot quickstart

Follow this to deploy your Pinot demo.

7. Delete a Kubernetes Cluster

Pinot On Kubernetes FAQ

This page has a collection of frequently asked questions about Pinot on Kubernetes with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, .

How to increase server disk size on AWS

The following is an example using Amazon Elastic Kubernetes Service (Amazon EKS).

1. Update Storage Class

In the Kubernetes (k8s) cluster, check the storage class: in Amazon EKS, it should be gp2.

Then update StorageClass to ensure:

Once StorageClass is updated, it should look like this:

2. Update PVC

Once the storage class is updated, then we can update the PersistentVolumeClaim (PVC) for the server disk size.

Now we want to double the disk size for pinot-server-3.

The following is an example of current disks:

The following is the output of data-pinot-server-3:

Now, let's change the PVC size to 2T by editing the server PVC.

Once updated, the specification's PVC size is updated to 2T, but the status's PVC size is still 1T.

3. Restart pod to let it reflect

Restart the pinot-server-3 pod:

Recheck the PVC size:

Vector index

Overview

Apache Pinot now supports a Vector Index for efficient similarity searches over high-dimensional vector embeddings. This feature introduces the capability to store and query float array columns (multi-valued) using a vector similarity algorithm.

Key Features

  • Vector Index is implemented using HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor (ANN) search.

  • Adds support for a predicate and function:

    • VECTOR_SIMILARITY(v1, v2, [optional topK]) to retrieve the topK closest vectors based on similarity.

    • The similarity function can be used as part of a query to filter and rank results.

Examples

Below is an example schema designed for a use case involving product reviews with vector embeddings for each review.

Schema

In this schema:

• The embedding column is a multi-valued float array designed to store high-dimensional vector embeddings (e.g., 1536 dimensions from an NLP model).

• Other fields, such as ProductId, UserId, and Text, store metadata and review text.

Table Config

To enable the Vector Index, configure the table with the appropriate fieldConfigList. The embedding column is specified to use the Vector Index with HNSW for similarity searches.

Explanation of Properties:

  1. vectorIndexType:

Specifies the type of vector index to use. Currently supports HNSW.

  1. vectorDimension:

Defines the dimensionality of the vectors stored in the column. (e.g., 1536 for typical embeddings from models like OpenAI or BERT).

  1. vectorDistanceFunction:

Specifies the distance metric for similarity computation. Options include:

  • INNER_PRODUCT:

    • Computes the inner product (dot product) of the two vectors.

    • Typically used when vectors are normalized and higher scores indicate greater similarity.

  • L2:

    • Measures the Euclidean distance between vectors.

    • Suitable for tasks where spatial closeness in high-dimensional space indicates similarity.

  • L1:

    • Measures the Manhattan distance between vectors (sum of absolute differences of coordinates).

    • Useful for some scenarios where simpler distance metrics are preferred.

  • COSINE:

    • Measures cosine similarity, which considers the angle between vectors.

    • Ideal for normalized vectors where orientation matters more than magnitude.

  1. version:

Specifies the version of the Vector Index implementation.

Query

VECTOR_SIMILARITY:

A predicate that retrieves the top k closest vectors to the query vector.

Inputs:

  • embedding: The vector column.

  • Query vector (literal array).

  • Optional topK parameter (default: 10).

Spark
Hadoop
Ingest streaming data from Apache Kafka
Stream ingestion with Upsert
Stream ingestion with Dedup
Stream ingestion with CLP
Amazon S3
Azure Data Lake Storage
Google Cloud Storage
HDFS
Input formats
Complex Type (Array, Map) Handling
Complex Type Examples
Ingest records with dynamic schemas
Reload a table segment
Upload a table segment
1.3.0
1.2.0
1.1.0
1.0.0
0.12.1
0.12.0
0.11.0
0.10.0
0.9.3
0.9.2
0.9.1
0.9.0
0.8.0
0.7.1
0.6.0
0.5.0
0.4.0
0.3.0
0.2.0
0.1.0
Running on Azure
Running on GCP
Running on AWS
{
  "metricFieldSpecs": [],
  "dimensionFieldSpecs": [
    {
      "dataType": "STRING",
      "name": "ProductId"
    },
    {
      "dataType": "STRING",
      "name": "UserId"
    },
    {
      "dataType": "INT",
      "name": "Score"
    },
    {
      "dataType": "STRING",
      "name": "Summary"
    },
    {
      "dataType": "STRING",
      "name": "Text"
    },
    {
      "dataType": "STRING",
      "name": "combined"
    },
    {
      "dataType": "INT",
      "name": "n_tokens"
    },
    {
      "dataType": "FLOAT",
      "name": "embedding",
      "singleValueField": false
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "ts",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:TIMESTAMP",
      "granularity": "1:SECONDS"
    }
  ],
  "schemaName": "fineFoodReviews"
}
{
  ...
  "fieldConfigList": [
    {
      "encodingType": "RAW",
      "indexType": "VECTOR",
      "name": "embedding",
      "properties": {
        "vectorIndexType": "HNSW",
        "vectorDimension": 1536,
        "vectorDistanceFunction": "COSINE",
        "version": 1
      }
    }
  ]
}
SELECT ProductId, 
       UserId, 
       l2_distance(embedding, ARRAY[-0.0013143676, -0.011042999, ...]) AS l2_dist, 
       n_tokens, 
       combined
FROM fineFoodReviews
WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.0013143676, -0.011042999, ...], 5)  
ORDER BY l2_dist ASC 
LIMIT 10;
brew install kubernetes-cli
kubectl version
brew install kubernetes-helm
helm version
brew update && brew install azure-cli
az login
AKS_RESOURCE_GROUP=pinot-demo
AKS_RESOURCE_GROUP_LOCATION=eastus
az group create --name ${AKS_RESOURCE_GROUP} \
                --location ${AKS_RESOURCE_GROUP_LOCATION}
AKS_RESOURCE_GROUP=pinot-demo
AKS_CLUSTER_NAME=pinot-quickstart
az aks create --resource-group ${AKS_RESOURCE_GROUP} \
              --name ${AKS_CLUSTER_NAME} \
              --node-count 3
AKS_RESOURCE_GROUP=pinot-demo
AKS_CLUSTER_NAME=pinot-quickstart
az aks get-credentials --resource-group ${AKS_RESOURCE_GROUP} \
                       --name ${AKS_CLUSTER_NAME}
kubectl get nodes
AKS_RESOURCE_GROUP=pinot-demo
AKS_CLUSTER_NAME=pinot-quickstart
az aks delete --resource-group ${AKS_RESOURCE_GROUP} \
              --name ${AKS_CLUSTER_NAME}
Azure Kubernetes Service (AKS)
https://kubernetes.io/docs/tasks/tools/install-kubectl
Installing Helm
https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
Kubernetes quickstart

Running on GCP

This quickstart guide helps you get started running Pinot on Google Cloud Platform (GCP).

In this quickstart guide, you will set up a Kubernetes Cluster on Google Kubernetes Engine(GKE)

1. Tooling Installation

1.1 Install Kubectl

Follow this link (https://kubernetes.io/docs/tasks/tools/install-kubectl) to install kubectl.

For Mac users

brew install kubernetes-cli

Check kubectl version after installation.

kubectl version

Quickstart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12

1.2 Install Helm

Follow this link (https://helm.sh/docs/using_helm/#installing-helm) to install helm.

For Mac users

brew install kubernetes-helm

Check helm version after installation.

helm version

This quickstart provides helm supports for helm v3.0.0 and v2.12.1. Choose the script based on your helm version.

1.3 Install Google Cloud SDK

To install Google Cloud SDK, see Install the gcloud CLI

1.3.1 For Mac users

  • Install Google Cloud SDK

curl https://sdk.cloud.google.com | bash

Restart your shell

exec -l $SHELL

2. (Optional) Initialize Google Cloud Environment

gcloud init

3. (Optional) Create a Kubernetes cluster(GKE) in Google Cloud

This script will create a 3 node cluster named pinot-quickstart in us-west1-b with n1-standard-2 machines for demo purposes.

Modify the parameters in the following example command with your gcloud details:

GCLOUD_PROJECT=[your gcloud project name]
GCLOUD_ZONE=us-west1-b
GCLOUD_CLUSTER=pinot-quickstart
GCLOUD_MACHINE_TYPE=n1-standard-2
GCLOUD_NUM_NODES=3
gcloud container clusters create ${GCLOUD_CLUSTER} \
  --num-nodes=${GCLOUD_NUM_NODES} \
  --machine-type=${GCLOUD_MACHINE_TYPE} \
  --zone=${GCLOUD_ZONE} \
  --project=${GCLOUD_PROJECT}

Use the following command do monitor cluster status:

gcloud compute instances list

Once the cluster is in RUNNING status, it's ready to be used.

4. Connect to an existing cluster

Run the following command to get the credential for the cluster pinot-quickstart that you just created:

GCLOUD_PROJECT=[your gcloud project name]
GCLOUD_ZONE=us-west1-b
GCLOUD_CLUSTER=pinot-quickstart
gcloud container clusters get-credentials ${GCLOUD_CLUSTER} --zone ${GCLOUD_ZONE} --project ${GCLOUD_PROJECT}

To verify the connection, run the following:

kubectl get nodes

5. Pinot quickstart

Follow this Kubernetes quickstart to deploy your Pinot demo.

6. Delete a Kubernetes Cluster

GCLOUD_ZONE=us-west1-b
gcloud container clusters delete pinot-quickstart --zone=${GCLOUD_ZONE}

Flink

Batch ingestion of data into Apache Pinot using Apache Flink.

Pinot supports Apache Flink as a processing framework to push segment files to the database.

Pinot distribution contains an Apache Flink SinkFunction that can be used as part of the Apache Flink application (Streaming or Batch) to directly write into a designated Pinot database.

Example

Flink application

Here is an example code snippet to show how to utilize the PinotSinkFunction in a Flink streaming application:

// some environmental setup
StreamExecutionEnvironment execEnv = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<Row> srcRows = execEnv.addSource(new FlinkKafkaConsumer<Row>(...));
RowTypeInfo typeInfo = new RowTypeInfo(
  new TypeInformation[]{Types.FLOAT, Types.FLOAT, Types.STRING, Types.STRING},
  new String[]{"lon", "lat", "address", "name"});


// add processing logic for the data stream for example:
DataStream<Row> processedRows = srcRow.keyBy(r -> r.getField(0));
...

// configurations for PinotSinkFunction
Schema pinotSchema = ...
TableConfig pinotTableConfig = ...
processedRows.addSink(new PinotSinkFunction<>(
  new FlinkRowGenericRowConverter(typeInfo), 
  pinotTableConfig,
  pinotSchema);

// execute the program
execEnv.execute();

As in the example shown above, the only required information from the Pinot side is the table schema and the table config.

For a more detailed executable, refer to the quick start example.

Table Config

PinotSinkFunction uses mostly the TableConfig object to infer the batch ingestion configuration to start a SegmentWriter and SegmentUploader to communicate with the Pinot cluster.

Note that even though in the above example Flink application is running in streaming mode, the data is still batch together and flush/upload to Pinot once the flush threshold is reached. It is not a direct streaming write into Pinot.

Here is an example table config

{
  "tableName" : "tbl_OFFLINE",
  "tableType" : "OFFLINE",
  "segmentsConfig" : {
    // ...
  },
  "tenants" : {
    // ...
  },
  "tableIndexConfig" : {
    // ....
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "HOURLY", 
      "batchConfigMaps": [
        {
          "outputDirURI": "file://path/to/flink/segmentwriter/output/dir",
          "overwriteOutput": "false",
          "push.controllerUri": "https://target.pinot.cluster.controller.url"
        }
      ]
    }
  }
}

the only required configurations are:

  • "outputDirURI": where PinotSinkFunction should write the constructed segment file to

  • "push.controllerUri": which Pinot cluster (controller) URL PinotSinkFunction should communicate with.

The rest of the configurations are standard for any Pinot table.

Google Cloud Storage

This guide shows you how to import data from GCP (Google Cloud Platform).

Enable the Google Cloud Storage using the pinot-gcs plugin. In the controller or server, add the config:

-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcs

By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify -Dplugins.include, you need to put all the plugins you want to use, e.g. pinot-json, pinot-avro , pinot-kafka-2.0...

GCP file systems provides the following options:

  • projectId - The name of the Google Cloud Platform project under which you have created your storage bucket.

  • gcpKey - Location of the json file containing GCP keys. You can refer Creating and managing service account keys to download the keys.

Each of these properties should be prefixed by pinot.[node].storage.factory.class.gs. where node is either controller or server depending on the configuration, like this:

pinot.controller.storage.factory.class.gs.projectId=test-project

Examples

Job spec

executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: 'gs://my-bucket/path/to/input/directory/'
outputDirURI: 'gs://my-bucket/path/to/output/directory/'
overwriteOutput: true
pinotFSSpecs:
    - scheme: gs
      className: org.apache.pinot.plugin.filesystem.GcsPinotFS
      configs:
        projectId: 'my-project'
        gcpKey: 'path-to-gcp json key file'
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'students'
pinotClusterSpecs:
    - controllerURI: 'http://localhost:9000'

Controller config

controller.data.dir=gs://path/to/data/directory/
controller.local.temp.dir=/path/to/local/temp/directory
controller.enable.split.commit=true
pinot.controller.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=my-project
pinot.controller.storage.factory.gs.gcpKey=path/to/gcp/key.json
pinot.controller.segment.fetcher.protocols=file,http,gs
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

Server config

pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.server.storage.factory.gs.projectId=my-project
pinot.server.storage.factory.gs.gcpKey=path/to/gcp/key.json
pinot.server.segment.fetcher.protocols=file,http,gs
pinot.server.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

Minion config

pinot.minion.storage.factory.class.gs=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.minion.storage.factory.gs.projectId=my-project
pinot.minion.storage.factory.gs.gcpKey=path/to/gcp/key.json
pinot.minion.segment.fetcher.protocols=file,http,gs
pinot.minion.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
allowVolumeExpansion: true
kubectl edit pvc data-pinot-server-3 -n pinot
make a pull request
PVC data-pinot-server-3
Physical architecture
Apache Zookeeper
Apache Helix
Cluster configuration reference
Server
Broker
Controller
tenants
tables
segments
tenants
Running Pinot in Docker
Running Pinot locally
segment
server
File Systems
Decoupling Controller from the Data Path
Use OSS as Deep Storage for Pinot
Use S3 as Deep Storage for Pinot
Batch job writing a segment into the deep store
Server sends segment to Controller, which writes segments into the deep store
Server writing a segment into the deep store

Pinot Data Explorer

Pinot Data Explorer is a user-friendly interface in Apache Pinot for interactive data exploration, querying, and visualization.

Once you have set up a cluster, you can start exploring the data and the APIs using the Pinot Data Explorer.

Navigate to http://localhost:9000 in your browser to open the Data Explorer UI.

Cluster Manager

The first screen that you'll see when you open the Pinot Data Explorer is the Cluster Manager. The Cluster Manager provides a UI to operate and manage your cluster.

Pinot Cluster Manager

If you want to view the contents of a server, click on its instance name. You'll then see the following:

Pinot Server

To view the baseballStats table, click on its name, which will show the following screen:

baseballStats Table

From this screen, we can edit or delete the table, edit or adjust its schema, as well as several other operations.

For example, if we want to add yearID to the list of inverted indexes, click on Edit Table, add the extra column, and click Save:

Edit Table

Query Console

Let's run some queries on the data in the Pinot cluster. Navigate to Query Console to see the querying interface.

We can see our baseballStats table listed on the left (you will see meetupRSVP or airlineStats if you used the streaming or the hybrid quick start). Click on the table name to display all the names along with the data types of the columns of the table.

You can also execute a sample query select * from baseballStats limit 10 by typing it in the text box and clicking the Run Query button.

Cmd + Enter can also be used to run the query when focused on the console.

Here are some sample queries you can try:

select playerName, max(hits) 
from baseballStats 
group by playerName 
order by max(hits) desc
select sum(hits), sum(homeRuns), sum(numberOfGames) 
from baseballStats 
where yearID > 2010
select * 
from baseballStats 
order by league

Pinot supports a subset of standard SQL. For more information, see Pinot Query Language.

Rest API

The Pinot Admin UI contains all the APIs that you will need to operate and manage your cluster. It provides a set of APIs for Pinot cluster management including health check, instances management, schema and table management, data segments management.

Let's check out the tables in this cluster by going to Table -> List all tables in cluster, click Try it out, and then click Execute. We can see thebaseballStats table listed here. We can also see the exact cURL call made to the controller API.

List all tables in cluster

You can look at the configuration of this table by going to Tables -> Get/Enable/Disable/Drop a table, click Try it out, type baseballStats in the table name, and then click Execute.

Let's check out the schemas in the cluster by going to Schema -> List all schemas in the cluster, click Try it out, and then click Execute. We can see a schema called baseballStats in this list.

List all schemas in the cluster

Take a look at the schema by going to Schema -> Get a schema, click Try it out, type baseballStats in the schema name, and then click Execute.

baseballStats Schema

Finally, let's check out the data segments in the cluster by going to Segment -> List all segments, click Try it out, type in baseballStats in the table name, and then click Execute. There's 1 segment for this table, called baseballStats_OFFLINE_0.

To learn how to upload your own data and schema, see Batch Ingestion or Stream ingestion.

Segment compaction on upserts

Use segment compaction on upsert-enabled real-time tables.

Overview of segment compaction

Compacting a segment replaces the completed segment with a compacted segment that only contains the latest version of records. For more information about how to use upserts on a real-time table in Pinot, see Stream Ingestion with Upsert.

The Pinot upsert feature stores all versions of the record ingested into immutable segments on disk. Even though the previous versions are not queried, they continue to add to the storage overhead. To remove older records (no longer used in query results) and reclaim storage space, we need to compact Pinot segments periodically. Segment compaction is done via a new minion task. To schedule Pinot tasks periodically, see the Minion documentation.

Compact segments on upserts in a real-time table

To compact segments on upserts, complete the following steps:

  1. Ensure task scheduling is enabled and a minion is available.

  2. Add the following to your table configuration. These configurations (except schedule)determine which segments to compact.

"task": {
  "taskTypeConfigsMap": {
    "UpsertCompactionTask": {
      "schedule": "0 */5 * ? * *",
      "bufferTimePeriod": "7d",
      "invalidRecordsThresholdPercent": "30",
      "invalidRecordsThresholdCount": "100000",
      "tableMaxNumTasks": "100",
      "validDocIdsType": "SNAPSHOT"
    }
  }
}
  • bufferTimePeriod: To compact segments once they are complete, set to “0d”. To delay compaction (as the configuration above shows by 7 days ("7d")), specify the number of days to delay compaction after a segment completes.

  • invalidRecordsThresholdPercent (Optional) Limits the older records allowed in the completed segment as a percentage of the total number of records in the segment. In the example above, the completed segment may be selected for compaction when 30% of the records in the segment are old.

  • invalidRecordsThresholdCount (Optional) Limits the older records allowed in the completed segment by record count. In the example above, if the segment contains more than 100K records, it may be selected for compaction.

  • tableMaxNumTasks (Optional) Limits the number of tasks allowed to be scheduled.

  • validDocIdsType (Optional) Specifies the source of validDocIds to fetch when running the data compaction. The valid types are SNAPSHOT, IN_MEMORY, IN_MEMORY_WITH_DELETE

    • SNAPSHOT: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.

    • IN_MEMORY: This indicates that the validDocIds bitmap is loaded from the real-time server's in-memory.

    • IN_MEMORY_WITH_DELETE: This indicates that the validDocIds bitmap is read from the real-time server's in-memory. The valid document ids here does take account into the deleted records. UpsertConfig's deleteRecordColumn must be provided for this type.

WARNING Using in-memory based validDocids type (IN_MEMORY, IN_MEMORY_WITH_DELETE) is dangerous as it will not guarantee us the consistency in some edge cases (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).

Because segment compaction is an expensive operation, we do not recommend setting invalidRecordsThresholdPercent and invalidRecordsThresholdCount too low (close to 1). By default, all configurations above are 0, so no thresholds are applied.

Example

The following example includes a dataset with 24M records and 240K unique keys that have each been duplicated 100 times. After ingesting the data, there are 6 segments (5 completed segments and 1 consuming segment) with a total estimated size of 22.8MB.

Example dataset

Submitting the query “set skipUpsert=true; select count(*) from transcript_upsert” before compaction produces 24,000,000 results:

Results before segment compaction

After the compaction tasks are complete, the Minion Task Manager UI reports the following.

Minion compaction task completed

Segment compactions generates a task for each segment to compact. Five tasks were generated in this case because 90% of the records (3.6–4.5M records) are considered ready for compaction in the completed segments, exceeding the configured thresholds.

If a completed segment only contains old records, Pinot immediately deletes the segment (rather than creating a task to compact it).

Submitting the query again shows the count matches the set of 240K unique keys.

Results after segment compaction

Once segment compaction has completed, the total number of segments remain the same and the total estimated size drops to 2.77MB.

To further improve query latency, merge small segments into larger one.

Timestamp index

Use a timestamp index to speed up your time query with different granularities

This feature is supported from Pinot 0.11+.

Background

The TIMESTAMP data type introduced in the Pinot 0.8.0 release stores value as millisecond epoch long value.

Typically, users won't need this low level granularity for analytics queries. Scanning the data and time value conversion can be costly for big data.

A common query pattern for timestamp columns is filtering on a time range and then grouping by using different time granularities(days/month/etc).

Typically, this requires the query executor to extract values, apply the transform functions then do filter/groupBy, with no leverage on the dictionary or index.

This was the inspiration for the Pinot timestamp index, which is used to improve the query performance for range query and group by queries on TIMESTAMP columns.

Supported data type

A TIMESTAMP index can only be created on the TIMESTAMP data type.

Timestamp Index

You can configure the granularity for a Timestamp data type column. Then:

  1. Pinot will pre-generate one column per time granularity using a forward index and range index. The naming convention is $${ts_column_name}$${ts_granularity}, where the timestamp column ts with granularities DAY, MONTH will have two extra columns generated: $ts$DAY and $ts$MONTH.

  2. Query overwrite for predicate and selection/group by: 2.1 GROUP BY: Functions like dateTrunc('DAY', ts) will be translated to use the underly column $ts$DAY to fetch data. 2.2 PREDICATE: range index is auto-built for all granularity columns.

Example query usage:

select count(*), 
       datetrunc('WEEK', ts) as tsWeek 
from airlineStats 
WHERE datetrunc('WEEK', ts) > fromDateTime('2014-01-16', 'yyyy-MM-dd') 
group by tsWeek
limit 10

Some preliminary benchmarking shows the query performance across 2.7 billion records improved from 45 secs to 4.2 secs using a timestamp index and a query like this:

select dateTrunc('YEAR', event_time) as y, 
       dateTrunc('MONTH', event_time) as m,  
       sum(pull_request_commits) 
from githubEvents 
group by y, m
limit 1000
Option(timeoutMs=3000000)
Without Timestamp Index

vs.

With Timestamp Index

Usage

The timestamp index is configured on a per column basis inside the fieldConfigList section in the table configuration.

Specify the timestampConfig field. This object must contain a field called granularities, which is an array with at least one of the following values:

  • MILLISECOND

  • SECOND

  • MINUTE

  • HOUR

  • DAY

  • WEEK

  • MONTH

  • QUARTER

  • YEAR

Sample config:

{
  "fieldConfigList": [
    {
      "name": "ts",
      "timestampConfig": {
        "granularities": [
          "DAY",
          "WEEK",
          "MONTH"
        ]
      }
    }
    ...
  ]
  ...
}

Getting Started

This section contains quick start guides to help you get up and running with Pinot.

Running Pinot

To simplify the getting started experience, Apache Pinot™ ships with quick start guides that launch Pinot components in a single process and import pre-built datasets.

For a full list of these guides, see Quick Start Examples.

Deploy to a public cloud

Data import examples

Getting data into Pinot is easy. Take a look at these two quick start guides which will help you get up and running with sample data for offline and real-time tables.

Tenant

Discover the tenant component of Apache Pinot, which facilitates efficient data isolation and resource management within Pinot clusters.

Every table is associated with a tenant, or a logical namespace that restricts where the cluster processes queries on the table. A Pinot tenant takes the form of a text tag in the logical tenant namespace. Physical cluster hardware resources (i.e., and ) are also associated with a tenant tag in the common tenant namespace. Tables of a particular tenant tag will only be scheduled for storage and query processing on hardware resources that belong to the same tenant tag. This lets Pinot cluster operators assign specified workloads to certain hardware resources, preventing data in separate workloads from being stored or processed on the same physical hardware.

By default, all tables, brokers, and servers belong to a tenant called DefaultTenant, but you can configure multiple tenants in a Pinot cluster.

To support multi-tenancy, Pinot has first-class support for tenants. Every table is associated with a server tenant and a broker tenant, which controls the nodes used by the table as servers and brokers. Multi-tenancy lets Pinot group all tables belonging to a particular use case under a single tenant name.

The concept of tenants is very important when the multiple use cases are using Pinot and there is a need to provide quotas or some sort of isolation across tenants. For example, consider we have two tables Table A and Table B in the same Pinot cluster.

We can configure Table A with server tenant Tenant A and Table B with server tenant Tenant B. We can tag some of the server nodes for Tenant A and some for Tenant B. This will ensure that segments of Table A only reside on servers tagged with Tenant A, and segment of Table B only reside on servers tagged with Tenant B. The same isolation can be achieved at the broker level, by configuring broker tenants to the tables.

No need to create separate clusters for every table or use case!

Tenant configuration

This tenant is defined in the section of the table config.

This section contains two main fields broker and server , which decide the tenants used for the broker and server components of this table.

In the above example:

  • The table will be served by brokers that have been tagged as brokerTenantName_BROKER in Helix.

  • If this were an offline table, the offline segments for the table will be hosted in Pinot servers tagged in Helix as serverTenantName_OFFLINE

  • If this were a real-time table, the real-time segments (both consuming as well as completed ones) will be hosted in pinot servers tagged in Helix as serverTenantName_REALTIME.

Create a tenant

Broker tenant

Here's a sample broker tenant config. This will create a broker tenant sampleBrokerTenant by tagging three untagged broker nodes as sampleBrokerTenant_BROKER.

To create this tenant use the following command. The creation will fail if number of untagged broker nodes is less than numberOfInstances.

Follow instructions in to get Pinot locally, and then

Check out the table config in the to make sure it was successfully uploaded.

Server tenant

Here's a sample server tenant config. This will create a server tenant sampleServerTenant by tagging 1 untagged server node as sampleServerTenant_OFFLINE and 1 untagged server node as sampleServerTenant_REALTIME.

To create this tenant use the following command. The creation will fail if number of untagged server nodes is less than offlineInstances + realtimeInstances.

Follow instructions in to get Pinot locally, and then

Check out the table config in the to make sure it was successfully uploaded.

Controller

Discover the controller component of Apache Pinot, enabling efficient data and query management.

The Pinot controller schedules and reschedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, the Pinot controller schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (for example, ingest of and ). The Pinot controller can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration.

The controller exposes a for cluster-wide administrative operations as well as a web-based query console to execute interactive SQL queries and perform simple administrative tasks.

The Pinot controller is responsible for the following:

  • Maintaining global metadata (e.g., configs and schemas) of the system with the help of Zookeeper which is used as the persistent metadata store.

  • Hosting the Helix Controller and managing other Pinot components (brokers, servers, minions)

  • Maintaining the mapping of which servers are responsible for which segments. This mapping is used by the servers to download the portion of the segments that they are responsible for. This mapping is also used by the broker to decide which servers to route the queries to.

  • Serving admin endpoints for viewing, creating, updating, and deleting configs, which are used to manage and operate the cluster.

  • Serving endpoints for segment uploads, which are used in offline data pushes. They are responsible for initializing real-time consumption and coordination of persisting real-time segments into the segment store periodically.

  • Undertaking other management activities such as managing retention of segments, validations.

For redundancy, there can be multiple instances of Pinot controllers. Pinot expects that all controllers are configured with the same back-end storage system so that they have a common view of the segments (e.g. NFS). Pinot can use other storage systems such as HDFS or .

Running the periodic task manually

The controller runs several periodic tasks in the background, to perform activities such as management and validation. Each periodic task has to define the run frequency and default frequency. Each task runs at its own schedule or can also be triggered manually if needed. The task runs on the lead controller for each table.

For period task configuration details, see .

Use the GET /periodictask/names API to fetch the names of all the periodic tasks running on your Pinot cluster.

To manually run a named periodic task, use the GET /periodictask/run API:

The Log Request Id (api-09630c07) can be used to search through pinot-controller log file to see log entries related to execution of the Periodic task that was manually run.

If tableName (and its type OFFLINE or REALTIME) is not provided, the task will run against all tables.

Starting a controller

Make sure you've . If you're using Docker, make sure to . To start a controller:

Running on AWS

This quickstart guide helps you get started running Pinot on Amazon Web Services (AWS).

In this quickstart guide, you will set up a Kubernetes Cluster on

1. Tooling Installation

1.1 Install Kubectl

To install kubectl, see .

For Mac users

Check kubectl version after installation.

Quickstart scripts are tested under kubectl client version v1.16.3 and server version v1.13.12

1.2 Install Helm

Follow this link () to install helm.

For Mac users

Check helm version after installation.

This quickstart provides helm supports for helm v3.0.0 and v2.12.1. Pick the script based on your helm version.

1.3 Install AWS CLI

Follow this link () to install AWS CLI.

For Mac users

1.4 Install Eksctl

Follow this link () to install AWS CLI.

For Mac users

2. (Optional) Log in to your AWS account

For first-time AWS users, register your account at .

Once you have created the account, go to to create a user and create access keys under Security Credential tab.

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY will override the AWS configuration stored in file ~/.aws/credentials

3. (Optional) Create a Kubernetes cluster(EKS) in AWS

The script below will create a 1 node cluster named pinot-quickstart in us-west-2 with a t3.xlarge machine for demo purposes:

For k8s 1.23+, run the following commands to allow the containers to provision their storage:

Use the following command to monitor the cluster status:

Once the cluster is in ACTIVE status, it's ready to be used.

4. Connect to an existing cluster

Run the following command to get the credential for the cluster pinot-quickstart that you just created:

To verify the connection, run the following:

5. Pinot quickstart

Follow this to deploy your Pinot demo.

6. Delete a Kubernetes Cluster

HDFS as Deep Storage

This guide shows how to set up HDFS as deep storage for a Pinot segment.

To use HDFS as deep storage you need to include HDFS dependency jars and plugins.

Server Setup

Configuration

Executable

Controller Setup

Configuration

Executable

Broker Setup

Configuration

Executable

Troubleshooting

If you receive an error that says No FileSystem for scheme"hdfs", the problem is likely to be a class loading issue.

To fix, try adding the following property to core-site.xml:

fs.hdfs.impl org.apache.hadoop.hdfs.DistributedFileSystem

And then export /opt/pinot/lib/hadoop-common-<release-version>.jar in the classpath.

Hadoop

Batch ingestion of data into Apache Pinot using Apache Hadoop.

Segment Creation and Push

Pinot supports as a processor to create and push segment files to the database. Pinot distribution is bundled with the Spark code to process your files and convert and upload them to Pinot.

You can follow the to build Pinot from source. The resulting JAR file can be found in pinot/target/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar

Next, you need to change the execution config in the job spec to the following -

You can check out the sample job spec here.

Finally execute the hadoop job using the command -

Ensure environment variables PINOT_ROOT_DIR and PINOT_VERSION are set properly.

Data Preprocessing before Segment Creation

We’ve seen some requests that data should be massaged (like partitioning, sorting, resizing) before creating and pushing segments to Pinot.

The MapReduce job called SegmentPreprocessingJob would be the best fit for this use case, regardless of whether the input data is of AVRO or ORC format.

Check the below example to see how to use SegmentPreprocessingJob.

In Hadoop properties, set the following to enable this job:

In table config, specify the operations in preprocessing.operations that you'd like to enable in the MR job, and then specify the exact configs regarding those operations:

preprocessing.num.reducers

Minimum number of reducers. Optional. Fetched when partitioning gets disabled and resizing is enabled. This parameter is to avoid having too many small input files for Pinot, which leads to the case where Pinot server is holding too many small segments, causing too many threads.

preprocessing.max.num.records.per.file

Maximum number of records per reducer. Optional.Unlike, “preprocessing.num.reducers”, this parameter is to avoid having too few large input files for Pinot, which misses the advantage of muti-threading when querying. When not set, each reducer will finally generate one output file. When set (e.g. M), the original output file will be split into multiple files and each new output file contains at most M records. It does not matter whether partitioning is enabled or not.

For more details on this MR job, refer to this .

Configure indexes

Learn how to apply indexes to a Pinot table. This guide assumes that you have followed the guide.

Pinot supports a series of different indexes that can be used to optimize query performance. In this guide, we'll learn how to add indexes to the events table that we set up in the guide.

Why do we need indexes?

If no indexes are applied to the columns in a Pinot segment, the query engine needs to scan through every document, checking whether that document meets the filter criteria provided in a query. This can be a slow process if there are a lot of documents to scan.

When indexes are applied, the query engine can more quickly work out which documents satisfy the filter criteria, reducing the time it takes to execute the query.

What indexes does Pinot support?

By default, Pinot creates a forward index for every column. The forward index generally stores documents in insertion order.

However, before flushing the segment, Pinot does a single pass over every column to see whether the data is sorted. If data is sorted, Pinot creates a sorted (forward) index for that column instead of the forward index.

For real-time tables you can also explicitly tell Pinot that one of the columns should be sorted. For more details, see the [Sorted Index Documentation](https://docs.pinot.apache.org/basics/indexing/forward-index#real-time-tables).

For filtering documents within a segment, Pinot supports the following indexing techniques:

  • Inverted index: Used for exact lookups.

  • Range index - Used for range queries.

  • Text index - Used for phrase, term, boolean, prefix, or regex queries.

  • Geospatial index - Based on H3, a hexagon-based hierarchical gridding. Used for finding points that exist within a certain distance from another point.

  • JSON index - Used for querying columns in JSON documents.

  • Star-Tree index - Pre-aggregates results across multiple columns.

View events table

Let's see how we can apply these indexing techniques to our data. To recap, the events table has the following fields:

Date Time Fields
Dimensions Fields
Metric Fields

We might want to write queries that filter on the ts and uuid columns, so these are the columns on which we would want to configure indexes.

Since the data we're ingesting into the Kafka topic is all implicitly ordered by timestamp, this means that the ts column already has a sorted index. This means that any queries that filter on this column are already optimised.

So that leaves us with the uuid column.

Add an inverted index

We're going to add an inverted index to the uuid column so that queries that filter on that column will return quicker. We need to add the following line:

To the tableIndexConfig section.

Copy the following to the clipboard:

/tmp/pinot/table-config-stream.json

Navigate to , click on Edit Table, paste the next table config, and then click Save.

Once you've done that, you'll need to click Reload All Segments and then Yes to apply the indexing change to all segments.

Check the index has been applied

We can check that the index has been applied to all our segments by querying Pinot's REST API. You can find Swagger documentation at .

The following query will return the indexes defined on the uuid column:

Output

We're using the to extract the fields that we're interested in.

We can see from looking at the inverted-index property that the index has been applied.

Querying

You can now run some queries that filter on the uuid column, as shown below:

You'll need to change the actual uuid value to a value that exists in your database, because the UUIDs are generated randomly by our script.

Stream ingestion with Dedup

Deduplication support in Apache Pinot.

Pinot provides native support for deduplication (dedup) during the real-time ingestion (v0.11.0+).

Prerequisites for enabling dedup

To enable dedup on a Pinot table, make the following table configuration and schema changes:

Define the primary key in the schema

To be able to dedup records, a primary key is needed to uniquely identify a given record. To define a primary key, add the field primaryKeyColumns to the schema definition.

Note this field expects a list of columns, as the primary key can be composite.

While ingesting a record, if its primary key is found to be already present, the record will be dropped.

Partition the input stream by the primary key

An important requirement for the Pinot dedup table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the API. If the original stream is not partitioned, then a streaming processing job (e.g. Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion.

Use strictReplicaGroup for routing

The dedup Pinot table can use only the low-level consumer for the input streams. As a result, it uses the for the segments. Moreover, dedup poses the additional requirement that all segments of the same partition must be served from the same server to ensure the data consistency across the segments. Accordingly, it requires strictReplicaGroup as the routing strategy. To use that, configure instanceSelectorType in Routing as the following:

instance assignment is persisted. Note that numInstancesPerPartition should always be 1 in replicaGroupPartitionConfig.

Other limitations

  • The high-level consumer is not allowed for the input stream ingestion, which means stream.kafka.consumer.type must be lowLevel.

  • The incoming stream must be partitioned by the primary key such that, all records with a given primaryKey must be consumed by the same Pinot server instance.

Enable dedup in the table configurations

To enable dedup for a REALTIME table, add the following to the table config.

Supported values for hashFunction are NONE, MD5 and MURMUR3, with the default being NONE.

Metadata TTL

Server stores the existing primary keys in dedup metadata map kept on JVM heap. As the dedup metadata grows, the heap memory pressure increases, which may affect the performance of ingestion and queries. One can set a positive metadata TTL to enable the TTL mechanism to keep the metadata size bounded. By default, the table's time colum is used as the dedup time column. The time unit of TTL is the same as the dedup time column. The TTL should be set long enough so that new records can be deduplicated before the primary keys gets removed.

Enable preload for faster server restarts

When ingesting new records, the server has to read the metadata map to check for duplicates. But when server restarts, the documents in existing segments are all unique as ensured by the dedup logic during real-time ingestion. So we can do write-only to bootstrap the metadata map faster.

The feature also requires you to specify pinot.server.instance.max.segment.preload.threads: N in the server config where N should be replaced with the number of threads that should be used for preload. It's 0 by default to disable the preloading feature. This preloading thread pool is shared with .

Best practices

Unlike other real-time tables, Dedup table takes up more memory resources as it needs to bookkeep the primary key and its corresponding segment reference, in memory. As a result, it's important to plan the capacity beforehand, and monitor the resource usage. Here are some recommended practices of using Dedup table.

  • Create the Kafka topic with more partitions. The number of Kafka partitions determines the partition numbers of the Pinot table. The more partitions you have in the Kafka topic, more Pinot servers you can distribute the Pinot table to and therefore more you can scale the table horizontally.

  • Dedup table maintains an in-memory map from the primary key to the segment reference. So it's recommended to use a simple primary key type and avoid composite primary keys to save the memory cost. In addition, consider the hashFunction config in the Dedup config, which can be MD5 or MURMUR3, to store the 128-bit hashcode of the primary key instead. This is useful when your primary key takes more space. But keep in mind, this hash may introduce collisions, though the chance is very low.

  • Monitoring: Set up a dashboard over the metric pinot.server.dedupPrimaryKeysCount.tableName to watch the number of primary keys in a table partition. It's useful for tracking its growth which is proportional to the memory usage growth.

  • Capacity planning: It's useful to plan the capacity beforehand to ensure you will not run into resource constraints later. A simple way is to measure the amount of the primary keys in the Kafka throughput per partition and time the primary key space cost to approximate the memory usage. A heap dump is also useful to check the memory usage so far on an dedup table instance.

Azure Data Lake Storage

This guide shows you how to import data from files stored in Azure Data Lake Storage Gen2 (ADLS Gen2)

Enable the Azure Data Lake Storage using the pinot-adls plugin. In the controller or server, add the config:

By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify -Dplugins.include, you need to put all the plugins you want to use, e.g. pinot-json, pinot-avro , pinot-kafka-2.0...

Azure Blob Storage provides the following options:

  • accountName: Name of the Azure account under which the storage is created.

  • accessKey: Access key required for the authentication.

  • fileSystemName: Name of the file system to use, for example, the container name (similar to the bucket name in S3).

  • enableChecksum: Enable MD5 checksum for verification. Default is false.

Each of these properties should be prefixed by pinot.[node].storage.factory.class.adl2. where node is either controller or server depending on the config, like this:

Examples

Job spec

Controller config

Server config

Minion config

HDFS

This guide shows you how to import data from HDFS.

Enable the using the pinot-hdfs plugin. In the controller or server, add the config:

By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify -Dplugins.include, you need to put all the plugins you want to use, e.g. pinot-json, pinot-avro , pinot-kafka-2.0...

HDFS implementation provides the following options:

  • hadoop.conf.path: Absolute path of the directory containing Hadoop XML configuration files, such as hdfs-site.xml, core-site.xml .

  • hadoop.write.checksum: Create checksum while pushing an object. Default is false

  • hadoop.kerberos.principle

  • hadoop.kerberos.keytab

Each of these properties should be prefixed by pinot.[node].storage.factory.class.hdfs. where node is either controller or server depending on the config

The kerberos configs should be used only if your Hadoop installation is secured with Kerberos. Refer to the for information on how to secure Hadoop using Kerberos.

You must provide proper Hadoop dependencies jars from your Hadoop installation to your Pinot startup scripts.

Push HDFS segment to Pinot Controller

To push HDFS segment files to Pinot controller, send the HDFS path of your newly created segment files to the Pinot Controller. The controller will download the files.

This curl example requests tells the controller to download segment files to the proper table:

Examples

Job spec

Standalone Job:

Hadoop Job:

Controller config

Server config

Minion config

Upload a table segment

Upload a table segment in Apache Pinot.

This procedure uploads one or more table segments that have been stored as Pinot segment binary files outside of Apache Pinot, such as if you had to close an original Pinot cluster and create a new one.

Choose one of the following:

  • If your data is in a location that uses HDFS, create a segment fetcher.

  • If your data is on a host where you have SSH access, use the Pinot Admin script.

Before you upload, do the following:

  1. or confirm one exists that matches the segment you want to upload.

  2. or confirm one exists that matches the segment you want to upload.

  3. (If needed) Upload the schema and table configs.

Create a segment fetcher

If the data is in a location using HDFS, you can create a , which will push segment files from external systems such as those running Hadoop or Spark. It is possible to with an external jar by implementing a class that extends this interface.

Use the Pinot Admin script to upload segments

To do this, you need to create a JobSpec configuration file. For details, see . This file defines the job, including things like the job type, the input directory or URI, and the table name that the segments will be connected to.

You can upload a Pinot segment using several methods:

  • Segment tar push

  • Segment URI push

  • Segment metadata push

Segment tar push

This is the original and default push mechanism. It requires the segment to be stored locally, or that the segment can be opened as an InputStream on PinotFS, so we can stream the entire segment tar file to the controller.

The push job will upload the entire segment tar file to the Pinot controller.

The Pinot controller will save the segment into the controller segment directory (Local or any PinotFS), then extract segment metadata, and add the segment to the table.

While you can create a JobSpec for this job, in simple instances you can push without one.

Upload segment files to your Pinot server from controller using the Pinot Admin script as follows:

All options should be prefixed with - (hyphen)

Option
Description

Segment URI push

This push mechanism requires the segment tar file stored on a deep store with a globally accessible segment tar URI.

URI push is lightweight on the client-side, and the controller side requires equivalent work as the tar push.

The push job posts this segment tar URI to the Pinot controller.

The Pinot controller saves the segment into the controller segment directory (local or any PinotFS), then extracts segment metadata, and adds the segment to the table.

Upload segment files to your Pinot server using the JobSpec you create and the Pinot Admin script as follows:

Segment metadata push

This push mechanism also requires the segment tar file stored on a deep store with a globally accessible segment tar URI.

Metadata push is lightweight on the controller side. There is no deep store download involved from the controller side.

The push job downloads the segment based on URI, then extracts metadata, and upload metadata to the Pinot controller.

The Pinot controller adds the segment to the table based on the metadata.

Upload segment metadata to your Pinot server using the JobSpec you create and the Pinot Admin script as follows:

Running Pinot locally
Running Pinot in Docker
Running in Kubernetes
Running on Azure
Running on GCP
Running on AWS
Batch import example
Stream ingestion example
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/path/to/hadoop/conf/directory/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle=<your kerberos principal>
pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab=<your kerberos keytab>
pinot.set.instance.id.to.hostname=true
pinot.server.instance.dataDir=/path/in/local/filesystem/for/pinot/data/server/index
pinot.server.instance.segmentTarDir=/path/in/local/filesystem/for/pinot/data/server/segment
pinot.server.grpc.enable=true
pinot.server.grpc.port=8090
export HADOOP_HOME=/path/to/hadoop/home
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export GC_LOG_LOCATION=/path/to/gc/log/file
export PINOT_VERSION=0.10.0
export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/
export SERVER_CONF_DIR=/path/to/pinot/conf/dir/
export ZOOKEEPER_ADDRESS=localhost:2181


export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"
export JAVA_OPTS="-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-server.log"
${PINOT_DISTRIBUTION_DIR}/bin/start-server.sh  -zkAddress ${ZOOKEEPER_ADDRESS} -configFileName ${SERVER_CONF_DIR}/server.conf
controller.data.dir=hdfs://path/in/hdfs/for/controller/segment
controller.local.temp.dir=/tmp/pinot/
controller.zk.str=<ZOOKEEPER_HOST:ZOOKEEPER_PORT>
controller.enable.split.commit=true
controller.access.protocols.http.port=9000
controller.helix.cluster.name=PinotCluster
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=/path/to/hadoop/conf/directory/
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=<your kerberos principal>
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=<your kerberos keytab>
controller.vip.port=9000
controller.port=9000
pinot.set.instance.id.to.hostname=true
pinot.server.grpc.enable=true
export HADOOP_HOME=/path/to/hadoop/home
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export GC_LOG_LOCATION=/path/to/gc/log/file
export PINOT_VERSION=0.10.0
export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/
export SERVER_CONF_DIR=/path/to/pinot/conf/dir/
export ZOOKEEPER_ADDRESS=localhost:2181


export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"
export JAVA_OPTS="-Xms8G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-controller.log"
${PINOT_DISTRIBUTION_DIR}/bin/start-controller.sh -configFileName ${SERVER_CONF_DIR}/controller.conf
pinot.set.instance.id.to.hostname=true
pinot.server.grpc.enable=true
export HADOOP_HOME=/path/to/hadoop/home
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export GC_LOG_LOCATION=/path/to/gc/log/file
export PINOT_VERSION=0.10.0
export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin/
export SERVER_CONF_DIR=/path/to/pinot/conf/dir/
export ZOOKEEPER_ADDRESS=localhost:2181


export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"
export JAVA_OPTS="-Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-broker.log"
${PINOT_DISTRIBUTION_DIR}/bin/start-broker.sh -zkAddress ${ZOOKEEPER_ADDRESS} -configFileName  ${SERVER_CONF_DIR}/broker.conf
-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-adls
pinot.controller.storage.factory.class.adl2.accountName=test-user
executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: 'adl2://path/to/input/directory/'
outputDirURI: 'adl2://path/to/output/directory/'
overwriteOutput: true
pinotFSSpecs:
    - scheme: adl2
      className: org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
      configs:
        accountName: 'my-account'
        accessKey: 'foo-bar-1234'
        fileSystemName: 'fs-name'
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'students'
pinotClusterSpecs:
    - controllerURI: 'http://localhost:9000'
controller.data.dir=adl2://path/to/data/directory/
controller.local.temp.dir=/path/to/local/temp/directory
controller.enable.split.commit=true
pinot.controller.storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
pinot.controller.storage.factory.adl2.accountName=my-account
pinot.controller.storage.factory.adl2.accessKey=foo-bar-1234
pinot.controller.storage.factory.adl2.fileSystemName=fs-name
pinot.controller.segment.fetcher.protocols=file,http,adl2
pinot.controller.segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
pinot.server.storage.factory.adl2.accountName=my-account
pinot.server.storage.factory.adl2.accessKey=foo-bar-1234
pinot.server.storage.factory.adl2.fileSystemName=fs-name
pinot.server.segment.fetcher.protocols=file,http,adl2
pinot.server.segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
storage.factory.class.adl2=org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS
storage.factory.adl2.accountName=my-account
storage.factory.adl2.fileSystemName=fs-name
storage.factory.adl2.accessKey=foo-bar-1234
segment.fetcher.protocols=file,http,adl2
segment.fetcher.adl2.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
curl -X GET "http://localhost:9000/periodictask/names" -H "accept: application/json"

[
  "RetentionManager",
  "OfflineSegmentIntervalChecker",
  "RealtimeSegmentValidationManager",
  "BrokerResourceValidationManager",
  "SegmentStatusChecker",
  "SegmentRelocator",
  "StaleInstancesCleanupTask",
  "TaskMetricsEmitter"
]
curl -X GET "http://localhost:9000/periodictask/run?taskname=SegmentStatusChecker&tableName=jsontypetable&type=OFFLINE" -H "accept: application/json"

{
  "Log Request Id": "api-09630c07",
  "Controllers notified": true
}
docker run \
    --network=pinot-demo \
    --name pinot-controller \
    -p 9000:9000 \
    -d ${PINOT_IMAGE} StartController \
    -zkAddress pinot-zookeeper:2181
bin/pinot-admin.sh StartController \
  -zkAddress localhost:2181 \
  -clusterName PinotCluster \
  -controllerPort 9000
real-time tables
offline tables
REST API endpoint
ADLS
its own configuration
Controller configuration reference
set up Zookeeper
pull the Pinot Docker image
brew install kubernetes-cli
kubectl version
brew install kubernetes-helm
helm version
curl "https://d1vvhvl2y92vvt.cloudfront.net/awscli-exe-macos.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
brew tap weaveworks/tap
brew install weaveworks/tap/eksctl
aws configure
EKS_CLUSTER_NAME=pinot-quickstart
eksctl create cluster \
--name ${EKS_CLUSTER_NAME} \
--version 1.16 \
--region us-west-2 \
--nodegroup-name standard-workers \
--node-type t3.xlarge \
--nodes 1 \
--nodes-min 1 \
--nodes-max 1
eksctl utils associate-iam-oidc-provider --region=us-east-2 --cluster=pinot-quickstart --approve

eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster pinot-quickstart \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

eksctl create addon --name aws-ebs-csi-driver --cluster pinot-quickstart --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_DriverRole --force
EKS_CLUSTER_NAME=pinot-quickstart
aws eks describe-cluster --name ${EKS_CLUSTER_NAME} --region us-west-2
EKS_CLUSTER_NAME=pinot-quickstart
aws eks update-kubeconfig --name ${EKS_CLUSTER_NAME}
kubectl get nodes
EKS_CLUSTER_NAME=pinot-quickstart
aws eks delete-cluster --name ${EKS_CLUSTER_NAME}
Amazon Elastic Kubernetes Service (Amazon EKS)
Install kubectl
https://helm.sh/docs/using_helm/#installing-helm
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html#install-tool-bundled
https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html#installing-eksctl
https://aws.amazon.com/
AWS Identity and Access Management (IAM)
Kubernetes quickstart
# executionFrameworkSpec: Defines ingestion jobs to be running.
executionFrameworkSpec:

    # name: execution framework name
  name: 'hadoop'

  # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner'

  # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentTarPushJobRunner'

  # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentUriPushJobRunner'

  # segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentMetadataPushJobRunner'

    # extraConfigs: extra configs for execution framework.
  extraConfigs:

    # stagingDir is used in distributed filesystem to host all the segments then move this directory entirely to output directory.
    stagingDir: your/local/dir/staging
export PINOT_VERSION=0.10.0
export PINOT_DISTRIBUTION_DIR=${PINOT_ROOT_DIR}/build/
export HADOOP_CLIENT_OPTS="-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml"

hadoop jar  \\
        ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \\
        org.apache.pinot.tools.admin.PinotAdministrator \\
        LaunchDataIngestionJob \\
        -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/hadoopIngestionJobSpec.yaml
enable.preprocessing = true
preprocess.path.to.output = <output_path>
{
    "OFFLINE": {
        "metadata": {
            "customConfigs": {
                “preprocessing.operations”: “resize, partition, sort”, // To enable the following preprocessing operations
                "preprocessing.max.num.records.per.file": "100",       // To enable resizing
                "preprocessing.num.reducers": "3"                      // To enable resizing
            }
        },
        ...
        "tableIndexConfig": {
            "aggregateMetrics": false,
            "autoGeneratedInvertedIndex": false,
            "bloomFilterColumns": [],
            "createInvertedIndexDuringSegmentGeneration": false,
            "invertedIndexColumns": [],
            "loadMode": "MMAP",
            "nullHandlingEnabled": false,
            "segmentPartitionConfig": {       // To enable partitioning
                "columnPartitionMap": {
                    "item": {
                        "functionName": "murmur",
                        "numPartitions": 4
                    }
                }
            },
            "sortedColumn": [                // To enable sorting
                "actorId"
            ],
            "streamConfigs": {}
        },
        "tableName": "tableName_OFFLINE",
        "tableType": "OFFLINE",
        "tenants": {
            ...
        }
    }
}
Apache Hadoop
wiki
document

ts

uuid

count

"invertedIndexColumns": ["uuid"]
{
  "tableName": "events",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "ts",
    "schemaName": "events",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "invertedIndexColumns": ["uuid"],
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "events",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
curl -X GET "http://localhost:9000/segments/events/metadata?columns=uuid" \
  -H "accept: application/json" 2>/dev/null | 
  jq '.[] | [.segmentName, .indexes]'
[
  "events__0__1__20220214T1106Z",
  {
    "uuid": {
      "bloom-filter": "NO",
      "dictionary": "YES",
      "forward-index": "YES",
      "inverted-index": "YES",
      "null-value-vector-reader": "NO",
      "range-index": "NO",
      "json-index": "NO"
    }
  }
]
[
  "events__0__0__20220214T1053Z",
  {
    "uuid": {
      "bloom-filter": "NO",
      "dictionary": "YES",
      "forward-index": "YES",
      "inverted-index": "YES",
      "null-value-vector-reader": "NO",
      "range-index": "NO",
      "json-index": "NO"
    }
  }
]
SELECT * 
FROM events 
WHERE uuid = 'f4a4f'
LIMIT 10
Ingest data from Apache Kafka
Ingest data from Apache Kafka
localhost:9000/#/tenants/table/events_REALTIME
localhost:9000/help
jq command line JSON processor
schemaWithPK.json
{
    "primaryKeyColumns": ["id"]
}
routing
{
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  }
}
tableConfigWithDedup.json
{ 
 ...
  "dedupConfig": { 
        "dedupEnabled": true, 
        "hashFunction": "NONE" 
   }, 
 ...
}
{ 
 ...
  "dedupConfig": { 
        "dedupEnabled": true, 
        "hashFunction": "NONE",
        "dedupTimeColumn": "mtime",
        "metadataTTL": 30000
   }, 
 ...
}
{ 
 ...
  "dedupConfig": { 
        "dedupEnabled": true, 
        "hashFunction": "NONE",
        "dedupTimeColumn": "mtime",
        "metadataTTL": 30000,
        "enablePreload": true
   }, 
 ...
}
send
partitioned replica-group assignment
upsert table's preloading
-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-hdfs
export HADOOP_HOME=/local/hadoop/
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"
curl -X POST -H "UPLOAD_TYPE:URI" -H "DOWNLOAD_URI:hdfs://nameservice1/hadoop/path/to/segment/file.
executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: 'hdfs:///path/to/input/directory/'
outputDirURI: 'hdfs:///path/to/output/directory/'
includeFileNamePath: 'glob:**/*.csv'
overwriteOutput: true
pinotFSSpecs:
    - scheme: hdfs
      className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
      configs:
        hadoop.conf.path: 'path/to/conf/directory/' 
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'students'
pinotClusterSpecs:
    - controllerURI: 'http://localhost:9000'
executionFrameworkSpec:
    name: 'hadoop'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.hadoop.HadoopSegmentUriPushJobRunner'
    extraConfigs:
      stagingDir: 'hdfs:///path/to/staging/directory/'
jobType: SegmentCreationAndTarPush
inputDirURI: 'hdfs:///path/to/input/directory/'
outputDirURI: 'hdfs:///path/to/output/directory/'
includeFileNamePath: 'glob:**/*.csv'
overwriteOutput: true
pinotFSSpecs:
    - scheme: hdfs
      className: org.apache.pinot.plugin.filesystem.HadoopPinotFS
      configs:
        hadoop.conf.path: '/etc/hadoop/conf/' 
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'students'
pinotClusterSpecs:
    - controllerURI: 'http://localhost:9000'
controller.data.dir=hdfs://path/to/data/directory/
controller.local.temp.dir=/path/to/local/temp/directory
controller.enable.split.commit=true
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory/
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=<your kerberos principal>
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=<your kerberos keytab>
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.segment.fetcher.hdfs.hadoop.kerberos.principle=<your kerberos principal>
pinot.server.segment.fetcher.hdfs.hadoop.kerberos.keytab=<your kerberos keytab>
storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
storage.factory.hdfs.hadoop.conf.path=path/to/conf/directory
segment.fetcher.protocols=file,http,hdfs
segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
segment.fetcher.hdfs.hadoop.kerberos.principle=<your kerberos principal>
segment.fetcher.hdfs.hadoop.kerberos.keytab=<your kerberos keytab>
Hadoop distributed file system (HDFS)
Hadoop in secure mode documentation
pinot-admin.sh AddTable \\
  -tableConfigFile /path/to/table-config.json \\
  -schemaFile /path/to/table-schema.json -exec
pinot-admin.sh UploadSegment \\
  -controllerHost localhost \\
  -controllerPort 9000 \\
  -segmentDir /path/to/local/dir \\
  -tableName myTable

controllerHost

Hostname or IP address of the controller

controllerPort

Port of the controller

segmentDir

Local directory containing segment files

tableName

Name of the table to push the segments into

pinot-admin.sh LaunchDataIngestionJob \\
    -jobSpecFile /file/location/my-job-spec.yaml
pinot-admin.sh LaunchDataIngestionJob \\
    -jobSpecFile /file/location/my-job-spec.yaml
Create a schema configuration
Create a table configuration
segment fetcher
implement your own segment fetcher for other systems
Ingestion job spec

Query FAQ

This page has a collection of frequently asked questions about queries with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, make a pull request.

Querying

I get the following error when running a query, what does it mean?

{'errorCode': 410, 'message': 'BrokerResourceMissingError'}

This implies that the Pinot Broker assigned to the table specified in the query was not found. A common root cause for this is a typo in the table name in the query. Another uncommon reason could be if there wasn't actually a broker with required broker tenant tag for the table.

What are all the fields in the Pinot query's JSON response?

See this page explaining the Pinot response format: https://docs.pinot.apache.org/users/api/querying-pinot-using-standard-sql/response-format.

SQL Query fails with "Encountered 'timestamp' was expecting one of..."

"timestamp" is a reserved keyword in SQL. Escape timestamp with double quotes.

select "timestamp" from myTable

Other commonly encountered reserved keywords are date, time, table.

Filtering on STRING column WHERE column = "foo" does not work?

For filtering on STRING columns, use single quotes:

SELECT COUNT(*) from myTable WHERE column = 'foo'

ORDER BY using an alias doesn't work?

The fields in the ORDER BY clause must be one of the group by clauses or aggregations, BEFORE applying the alias. Therefore, this will not work:

SELECT count(colA) as aliasA, colA from tableA GROUP BY colA ORDER BY aliasA

But, this will work:

SELECT count(colA) as sumA, colA from tableA GROUP BY colA ORDER BY count(colA)

Does pagination work in GROUP BY queries?

No. Pagination only works for SELECTION queries.

How do I increase timeout for a query ?

You can add this at the end of your query: option(timeoutMs=X). Tthe following example uses a timeout of 20 seconds for the query:

SELECT COUNT(*) from myTable option(timeoutMs=20000)

You can also use SET "timeoutMs" = 20000; SELECT COUNT(*) from myTable.

For changing the timeout on the entire cluster, set this property pinot.broker.timeoutMs in either broker configs or cluster configs (using the POST /cluster/configs API from Swagger).

How do I cancel a query?

Add these two configs for Pinot server and broker to start tracking of running queries. The query tracks are added and cleaned as query starts and ends, so should not consume much resource.

pinot.server.enable.query.cancellation=true // false by default
pinot.broker.enable.query.cancellation=true // false by default

Then use the Rest APIs on Pinot controller to list running queries and cancel them via the query ID and broker ID (as query ID is only local to broker), like in the following:

GET /queries: to show running queries as tracked by all brokers
Response example: `{
  "Broker_192.168.0.105_8000": {
    "7": "select G_old from baseballStats limit 10",
    "8": "select G_old from baseballStats limit 100"
  }
}`

DELETE /query/{brokerId}/{queryId}[?verbose=false/true]: to cancel a running query 
with queryId and brokerId. The verbose is false by default, but if set to true, 
responses from servers running the query also return.

Response example: `Cancelled query: 8 with responses from servers: 
{192.168.0.105:7501=404, 192.168.0.105:7502=200, 192.168.0.105:7500=200}`

How do I optimize my Pinot table for doing aggregations and group-by on high cardinality columns ?

In order to speed up aggregations, you can enable metrics aggregation on the required column by adding a metric field in the corresponding schema and setting aggregateMetrics to true in the table configuration. You can also use a star-tree index config for columns like these (see here for more about star-tree).

How do I verify that an index is created on a particular column ?

There are two ways to verify this:

  1. Log in to a server that hosts segments of this table. Inside the data directory, locate the segment directory for this table. In this directory, there is a file named index_map which lists all the indexes and other data structures created for each segment. Verify that the requested index is present here.

  2. During query: Use the column in the filter predicate and check the value of numEntriesScannedInFilter. If this value is 0, then indexing is working as expected (works for Inverted index).

Does Pinot use a default value for LIMIT in queries?

Yes, Pinot uses a default value of LIMIT 10 in queries. The reason behind this default value is to avoid unintentionally submitting expensive queries that end up fetching or processing a lot of data from Pinot. Users can always overwrite this by explicitly specifying a LIMIT value.

Does Pinot cache query results?

Pinot does not cache query results. Each query is computed in its entirety. Note though, running the same or similar query multiple times will naturally pull in segment pages into memory making subsequent calls faster. Also, for real-time systems, the data is changing in real-time, so results cannot be cached. For offline-only systems, caching layer can be built on top of Pinot, with invalidation mechanism built-in to invalidate the cache when data is pushed into Pinot.

I'm noticing that the first query is slower than subsequent queries. Why is that?

Pinot memory maps segments. It warms up during the first query, when segments are pulled into the memory by the OS. Subsequent queries will have the segment already loaded in memory, and hence will be faster. The OS is responsible for bringing the segments into memory, and also removing them in favor of other segments when other segments not already in memory are accessed.

How do I determine if the star-tree index is being used for my query?

The query execution engine will prefer to use the star-tree index for all queries where it can be used. The criteria to determine whether the star-tree index can be used is as follows:

  • All aggregation function + column pairs in the query must exist in the star-tree index.

  • All dimensions that appear in filter predicates and group-by should be star-tree dimensions.

For queries where above is true, a star-tree index is used. For other queries, the execution engine will default to using the next best index available.

Ingest streaming data from Amazon Kinesis

This guide shows you how to ingest a stream of records from an Amazon Kinesis topic into a Pinot table.

To ingest events from an Amazon Kinesis stream into Pinot, set the following configs into your table config:

{
  "tableName": "kinesisTable",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kinesis",
      "stream.kinesis.topic.name": "<your kinesis stream name>",
      "region": "<your region>",
      "accessKey": "<your access key>",
      "secretKey": "<your secret key>",
      "shardIteratorType": "AFTER_SEQUENCE_NUMBER",
      "stream.kinesis.consumer.type": "lowlevel",
      "stream.kinesis.fetch.timeout.millis": "30000",
      "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
      "realtime.segment.flush.threshold.rows": "1000000",
      "realtime.segment.flush.threshold.time": "6h"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}

where the Kinesis specific properties are:

Property
Description

streamType

This should be set to "kinesis"

stream.kinesis.topic.name

Kinesis stream name

region

Kinesis region e.g. us-west-1

accessKey

Kinesis access key

secretKey

Kinesis secret key

shardIteratorType

Set to LATEST to consume only new records, TRIM_HORIZON for earliest sequence number_,_ AT___SEQUENCE_NUMBER and AFTER_SEQUENCE_NUMBER to start consumptions from a particular sequence number

maxRecordsToFetch

... Default is 20.

Kinesis supports authentication using the DefaultCredentialsProviderChain. The credential provider looks for the credentials in the following order:

  • Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)

  • Java System Properties - aws.accessKeyId and aws.secretKey

  • Web Identity Token credentials from the environment or container

  • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI

  • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and security manager has permission to access the variable,

  • Instance profile credentials delivered through the Amazon EC2 metadata service

You must provide all read access level permissions for Pinot to work with an AWS Kinesis data stream. See the AWS documentation for details.

Although you can also specify the accessKey and secretKey in the properties above, we don't recommend this insecure method. We recommend using it only for non-production proof-of-concept (POC) setups. You can also specify other AWS fields such as AWS_SESSION_TOKEN as environment variables and config and it will work.

Resharding

In Kinesis, whenever you reshard a stream, it is done via split or merge operations on shards. If you split a shard, the shard closes and creates 2 new children shards. So if you started with shard0, and then split it, it would result in shard1 and shard2. Similarly, if you merge 2 shards, both those will close and create a child shard. So in the same example, if you merge shards 1 and 2, you'll end up with shard3 as the active shard, while shard0, shard1, shard2 will remain closed forever.

Please check out this recipe for more details: https://dev.startree.ai/docs/pinot/recipes/github-events-stream-kinesis#resharding-kinesis-stream

In Pinot, resharding of any stream is detected by periodic task RealtimeValidationManager: https://docs.pinot.apache.org/configuration-reference/controller#realtimesegmentvalidationmanager. This runs hourly. If you rehsard, your new shards will not get detected unless:

  1. We finish ingesting from parent shards completely

  2. And after 1, the RealtimeValidationManager runs

You will see a period where the ideal state will show all segments ONLINE, as parents have naturally completed ingesting, and we're waiting for RealtimeValidationManager to kickstart the ingestion from children.

If you need the ingestion to happen sooner, you can manually invoke the RealtimeValidationManager: https://docs.pinot.apache.org/basics/concepts/components/cluster/controller#running-the-periodic-task-manually

Limitations

  1. ShardID is of the format "shardId-000000000001". We use the numeric part as partitionId. Our partitionId variable is integer. If shardIds grow beyond Integer.MAX\_VALUE, we will overflow into the partitionId space.

  2. Segment size based thresholds for segment completion will not work. It assumes that partition "0" always exists. However, once the shard 0 is split/merged, we will no longer have partition 0.

Stream ingestion with CLP

Support for encoding fields with CLP during ingestion.

This is an experimental feature. Configuration options and usage may change frequently until it is stabilized.

When performing stream ingestion of JSON records using Kafka, users can encode specific fields with CLP by using a CLP-specific StreamMessageDecoder.

CLP is a compressor designed to encode unstructured log messages in a way that makes them more compressible while retaining the ability to search them. It does this by decomposing the message into three fields:

  • the message's static text, called a log type;

  • repetitive variable values, called dictionary variables; and

  • non-repetitive variable values (called encoded variables since we encode them specially if possible).

Searches are similarly decomposed into queries on the individual fields.

Although CLP is designed for log messages, other unstructured text like file paths may also benefit from its encoding.

For example, consider this JSON record:

{
  "timestamp": 1672531200000,
  "message": "INFO Task task_12 assigned to container: [ContainerID:container_15], operation took 0.335 seconds. 8 tasks remaining.",
  "logPath": "/mnt/data/application_123/container_15/stdout"
}

If the user specifies the fields message and logPath should be encoded with CLP, then the StreamMessageDecoder will output:

{
  "timestamp": 1672531200000,
  "message_logtype": "INFO Task \\x12 assigned to container: [ContainerID:\\x12], operation took \\x13 seconds. \\x11 tasks remaining.",
  "message_dictionaryVars": [
    "task_12",
    "container_15"
  ],
  "message_encodedVars": [
    1801439850948198735,
    8
  ],
  "logPath_logtype": "/mnt/data/\\x12/\\x12/stdout",
  "logPath_dictionaryVars": [
    "application_123",
    "container_15"
  ],
  "logPath_encodedVars": []
}

In the fields with the _logtype suffix, \x11 is a placeholder for an integer variable, \x12 is a placeholder for a dictionary variable, and \x13 is a placeholder for a float variable. In message_encoedVars, the float variable 0.335 is encoded as an integer using CLP's custom encoding.

All remaining fields are processed in the same way as they are in org.apache.pinot.plugin.inputformat.json.JSONRecordExtractor. Specifically, fields in the table's schema are extracted from each record and any remaining fields are dropped.

Configuration

Table Index

Assuming the user wants to encode message and logPath as in the example, they should change/add the following settings to their tableIndexConfig (we omit irrelevant settings for brevity):

{
  "tableIndexConfig": {
    "streamConfigs": {
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.clplog.CLPLogMessageDecoder",
      "stream.kafka.decoder.prop.fieldsForClpEncoding": "message,logPath"
    },
    "varLengthDictionaryColumns": [
      "message_logtype",
      "message_dictionaryVars",
      "logPath_logtype",
      "logPath_dictionaryVars"
    ]
  }
}
  • stream.kafka.decoder.prop.fieldsForClpEncoding is a comma-separated list of names for fields that should be encoded with CLP.

  • We use variable-length dictionaries for the logtype and dictionary variables since their length can vary significantly.

Schema

For the table's schema, users should configure the CLP-encoded fields as follows (we omit irrelevant settings for brevity):

{
  "dimensionFieldSpecs": [
    {
      "name": "message_logtype",
      "dataType": "STRING",
      "maxLength": 2147483647
    },
    {
      "name": "message_encodedVars",
      "dataType": "LONG",
      "singleValueField": false
    },
    {
      "name": "message_dictionaryVars",
      "dataType": "STRING",
      "maxLength": 2147483647,
      "singleValueField": false
    },
    {
      "name": "message_logtype",
      "dataType": "STRING",
      "maxLength": 2147483647
    },
    {
      "name": "message_encodedVars",
      "dataType": "LONG",
      "singleValueField": false
    },
    {
      "name": "message_dictionaryVars",
      "dataType": "STRING",
      "maxLength": 2147483647,
      "singleValueField": false
    }
  ]
}
  • We use the maximum possible length for the logtype and dictionary variable columns.

  • The dictionary and encoded variable columns are multi-valued columns.

Searching and decoding CLP-encoded fields

To decode CLP-encoded fields, use CLPDECODE.

To search CLP-encoded fields, you can combine CLPDECODE with LIKE. Note, this may decrease performance when querying a large number of rows.

We are working to integrate efficient searches on CLP-encoded columns as another UDF. The development of this feature is being tracked in this design doc.

What is Apache Pinot? (and User-Facing Analytics) by Tim Berglund
"tenants": {
  "broker": "brokerTenantName",
  "server": "serverTenantName"
}
sample-broker-tenant.json
{
     "tenantRole" : "BROKER",
     "tenantName" : "sampleBrokerTenant",
     "numberOfInstances" : 3
}
bin/pinot-admin.sh AddTenant \
    -name sampleBrokerTenant 
    -role BROKER 
    -instanceCount 3 -exec
curl -i -X POST -H 'Content-Type: application/json' -d @sample-broker-tenant.json localhost:9000/tenants
sample-server-tenant.json
{
     "tenantRole" : "SERVER",
     "tenantName" : "sampleServerTenant",
     "offlineInstances" : 1,
     "realtimeInstances" : 1
}
bin/pinot-admin.sh AddTenant \
    -name sampleServerTenant \
    -role SERVER \
    -offlineInstanceCount 1 \
    -realtimeInstanceCount 1 -exec
curl -i -X POST -H 'Content-Type: application/json' -d @sample-server-tenant.json localhost:9000/tenants
brokers
servers
tenants
Getting Pinot
Rest API
Getting Pinot
Rest API
Defining tenants for tables
Table isolation using tenants

Schema

Explore the Schema component in Apache Pinot, vital for defining the structure and data types of Pinot tables, enabling efficient data processing and analysis.

Each table in Pinot is associated with a schema. A schema defines:

  • Fields in the table with their data types.

  • Whether the table uses column-based or table-based null handling. For more information, see Null value support.

The schema is stored in Zookeeper along with the table configuration.

Schema naming in Pinot follows typical database table naming conventions, such as starting names with a letter, not ending with an underscore, and using only alphanumeric characters

Categories

A schema also defines what category a column belongs to. Columns in a Pinot table can be categorized into three categories:

Category
Description

Dimension

Dimension columns are typically used in slice and dice operations for answering business queries. Some operations for which dimension columns are used:

  • GROUP BY - group by one or more dimension columns along with aggregations on one or more metric columns

  • Filter clauses such as WHERE

Metric

These columns represent the quantitative data of the table. Such columns are used for aggregation. In data warehouse terminology, these can also be referred to as fact or measure columns.

Some operation for which metric columns are used:

  • Aggregation - SUM, MIN, MAX, COUNT, AVG etc

  • Filter clause such as WHERE

DateTime

This column represents time columns in the data. There can be multiple time columns in a table, but only one of them can be treated as primary. The primary time column is the one that is present in the . The primary time column is used by Pinot to maintain the time boundary between offline and real-time data in a hybrid table and for retention management. A primary time column is mandatory if the table's push type is APPEND and optional if the push type is REFRESH .

Common operations that can be done on time column:

  • GROUP BY

  • Filter clauses such as WHERE

Pinot does not enforce strict rules on which of these categories columns belong to, rather the categories can be thought of as hints to Pinot to do internal optimizations.

For example, metrics may be stored without a dictionary and can have a different default null value.

The categories are also relevant when doing segment merge and rollups. Pinot uses the dimension and time fields to identify records against which to apply merge/rollups.

Metrics aggregation is another example where Pinot uses dimensions and time are used as the key, and automatically aggregates values for the metric columns.

For configuration details, see Schema configuration reference.

Date and time fields

Since Pinot doesn't have a dedicated DATETIME datatype support, you need to input time in either STRING, LONG, or INT format. However, Pinot needs to convert the date into an understandable format such as epoch timestamp to do operations. You can refer to DateTime field spec configs for more details on supported formats.

Creating a schema

First, Make sure your cluster is up and running.

Let's create a schema and put it in a JSON file. For this example, we have created a schema for flight data.

For more details on constructing a schema file, see the Schema configuration reference.

flights-schema.json
{
  "schemaName": "flights",
  "enableColumnBasedNullHandling": true,
  "dimensionFieldSpecs": [
    {
      "name": "flightNumber",
      "dataType": "LONG",
      "notNull": true
    },
    {
      "name": "tags",
      "dataType": "STRING",
      "singleValueField": false,
      "defaultNullValue": "null"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "price",
      "dataType": "DOUBLE",
      "notNull": true,
      "defaultNullValue": 0
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "millisSinceEpoch",
      "dataType": "LONG",
      "format": "EPOCH",
      "granularity": "15:MINUTES"
    },
    {
      "name": "hoursSinceEpoch",
      "dataType": "INT",
      "notNull": true,
      "format": "EPOCH|HOURS",
      "granularity": "1:HOURS"
    },
    {
      "name": "dateString",
      "dataType": "STRING",
      "format": "SIMPLE_DATE_FORMAT|yyyy-MM-dd",
      "granularity": "1:DAYS"
    }
  ]
}

Then, we can upload the sample schema provided above using either a Bash command or REST API call.

bin/pinot-admin.sh AddSchema -schemaFile flights-schema.json -exec

OR

bin/pinot-admin.sh AddTable -schemaFile flights-schema.json -tableFile flights-table.json -exec
curl -F [email protected]  localhost:9000/schemas

Check out the schema in the Rest API to make sure it was successfully uploaded

Pinot storage model

Apache Pinot™ uses a variety of terms which can refer to either abstractions that model the storage of data or infrastructure components that drive the functionality of the system, including:

  • to store data

  • to partition data

  • to isolate data

  • to manage data

Pinot has a distributed systems architecture that scales horizontally. Pinot expects the size of a table to grow infinitely over time. To achieve this, all data needs to be distributed across multiple nodes. Pinot achieves this by breaking data into smaller chunks known as (similar to shards/partitions in HA relational databases). Segments can also be seen as time-based partitions.

Table

Similar to traditional databases, Pinot has the concept of a —a logical abstraction to refer to a collection of related data. As is the case with relational database management systems (RDBMS), a table is a construct that consists of columns and rows (documents) that are queried using SQL. A table is associated with a , which defines the columns in a table as well as their data types.

As opposed to RDBMS schemas, multiple tables can be created in Pinot (real-time or batch) that inherit a single schema definition. Tables are independently configured for concerns such as indexing strategies, partitioning, tenants, data sources, and replication.

Pinot stores data in . A Pinot table is conceptually identical to a relational database table with rows and columns. Columns have the same name and data type, known as the table's .

Pinot schemas are defined in a JSON file. Because that schema definition is in its own file, multiple tables can share a single schema. Each table can have a unique name, indexing strategy, partitioning, data sources, and other metadata.

Pinot table types include:

  • real-time: Ingests data from a streaming source like Apache Kafka®

  • offline: Loads data from a batch source

  • hybrid: Loads data from both a batch source and a streaming source

Segment

Pinot tables are stored in one or more independent shards called . A small table may be contained by a single segment, but Pinot lets tables grow to an unlimited number of segments. There are different processes for creating segments (see ). Segments have time-based partitions of table data, and are stored on Pinot that scale horizontally as needed for both storage and computation.

Tenant

To support multi-tenancy, Pinot has first class support for tenants. A table is associated with a . This allows all tables belonging to a particular logical namespace to be grouped under a single tenant name and isolated from other tenants. This isolation between tenants provides different namespaces for applications and teams to prevent sharing tables or schemas. Development teams building applications do not have to operate an independent deployment of Pinot. An organization can operate a single cluster and scale it out as new tenants increase the overall volume of queries. Developers can manage their own schemas and tables without being impacted by any other tenant on a cluster.

Every table is associated with a , or a logical namespace that restricts where the cluster processes queries on the table. A Pinot tenant takes the form of a text tag in the logical tenant namespace. Physical cluster hardware resources (i.e., and ) are also associated with a tenant tag in the common tenant namespace. Tables of a particular tenant tag will only be scheduled for storage and query processing on hardware resources that belong to the same tenant tag. This lets Pinot cluster operators assign specified workloads to certain hardware resources, preventing data from separate workloads from being stored or processed on the same physical hardware.

By default, all tables, brokers, and servers belong to a tenant called DefaultTenant, but you can configure multiple tenants in a Pinot cluster.

Cluster

A Pinot is a collection of the software processes and hardware resources required to ingest, store, and process data. For detail about Pinot cluster components, see .

Physical architecture

A Pinot cluster consists of the following processes, which are typically deployed on separate hardware resources in production. In development, they can fit comfortably into Docker containers on a typical laptop.

  • Controller: Maintains cluster metadata and manages cluster resources.

  • Zookeeper: Manages the Pinot cluster on behalf of the controller. Provides fault-tolerant, persistent storage of metadata, including table configurations, schemas, segment metadata, and cluster state.

  • Broker: Accepts queries from client processes and forwards them to servers for processing.

  • Server: Provides storage for segment files and compute for query processing.

  • (Optional) Minion: Computes background tasks other than query processing, minimizing impact on query latency. Optimizes segments, and builds additional indexes to ensure performance (even if data is deleted).

The simplest possible Pinot cluster consists of four components: a server, a broker, a controller, and a Zookeeper node. In production environments, these components typically run on separate server instances, and scale out as needed for data volume, load, availability, and latency. Pinot clusters in production range from fewer than ten total instances to more than 1,000.

Pinot uses as a distributed metadata store and and for cluster management.

Helix is a cluster management solution created by the authors of Pinot. Helix maintains a persistent, fault-tolerant map of the intended state of the Pinot cluster. It constantly monitors the cluster to ensure that the right hardware resources are allocated to implement the present configuration. When the configuration changes, Helix schedules or decommissions hardware resources to reflect the new configuration. When elements of the cluster change state catastrophically, Helix schedules hardware resources to keep the actual cluster consistent with the ideal represented in the metadata. From a physical perspective, Helix takes the form of a controller process plus agents running on servers and brokers.

Controller

A is the core orchestrator that drives the consistency and routing in a Pinot cluster. Controllers are horizontally scaled as an independent component (container) and has visibility of the state of all other components in a cluster. The controller reacts and responds to state changes in the system and schedules the allocation of resources for tables, segments, or nodes. As mentioned earlier, Helix is embedded within the controller as an agent that is a participant responsible for observing and driving state changes that are subscribed to by other components.

The Pinot schedules and re-schedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, it schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (e.g., ingest of and ). It can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration.

The controller exposes a for cluster-wide administrative operations as well as a web-based query console to execute interactive SQL queries and perform simple administrative tasks.

Server

host segments (shards) that are scheduled and allocated across multiple nodes and routed on an assignment to a tenant (there is a single tenant by default). Servers are independent containers that scale horizontally and are notified by Helix through state changes driven by the controller. A server can either be a real-time server or an offline server.

A real-time and offline server have very different resource usage requirements, where real-time servers are continually consuming new messages from external systems (such as Kafka topics) that are ingested and allocated on segments of a tenant. Because of this, resource isolation can be used to prioritize high-throughput real-time data streams that are ingested and then made available for query through a broker.

Broker

Pinot take query requests from client processes, scatter them to applicable servers, gather the results, and return them to the client. The controller shares cluster metadata with the brokers that allows the brokers to create a plan for executing the query involving a minimal subset of servers with the source data and, when required, other servers to shuffle and consolidate results.

A production Pinot cluster contains many brokers. In general, the more brokers, the more concurrent queries a cluster can process, and the lower latency it can deliver on queries.

Pinot minion

Pinot minion is an optional component that can be used to run background tasks such as "purge" for GDPR (General Data Protection Regulation). As Pinot is an immutable aggregate store, records containing sensitive private data need to be purged on a request-by-request basis. Minion provides a solution for this purpose that complies with GDPR while optimizing Pinot segments and building additional indices that guarantees performance in the presence of the possibility of data deletion. One can also write a custom task that runs on a periodic basis. While it's possible to perform these tasks on the Pinot servers directly, having a separate process (Minion) lessens the overall degradation of query latency as segments are impacted by mutable writes.

A Pinot is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing minion tasks as directed by the controller. Examples of minon tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically.

Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function with or without minions, they are typically present to support routine tasks like batch data ingest.

\

Stream ingestion example

The Docker instructions on this page are still WIP

This example assumes you have set up your cluster using .

Data Stream

First, we need to set up a stream. Pinot has out-of-the-box real-time ingestion support for Kafka. Other streams can be plugged in for use, see .

Let's set up a demo Kafka cluster locally, and create a sample topic transcript-topic.

Start Kafka

Create a Kafka Topic

Start Kafka

Start Kafka cluster on port 9876 using the same Zookeeper from the quick-start examples.

Create a Kafka topic

Download the latest . Create a topic.

Creating a schema

If you followed , you have already pushed a schema for your sample table. If not, see to learn how to create a schema for your sample data.

Creating a table configuration

If you followed , you pushed an offline table and schema. To create a real-time table configuration for the sample use this table configuration for the transcript table. For a more detailed overview about table, see .

Uploading your schema and table configuration

Next, upload the table and schema to the cluster. As soon as the real-time table is created, it will begin ingesting from the Kafka topic.

Loading sample data into stream

Use the following sample JSON file for transcript table data in the following step.

Push the sample JSON file into the Kafka topic, using the Kafka script from the Kafka download.

Ingesting streaming data

As soon as data flows into the stream, the Pinot table will consume it and it will be ready for querying. Browse to the running in your Pinot instance (we use localhost in this link as an example) to examine the real-time data.

Amazon S3

This guide shows you how to import data from files stored in Amazon S3.

Enable the file system backend by including the pinot-s3 plugin. In the controller or server configuration, add the config:

By default Pinot loads all the plugins, so you can just drop this plugin there. Also, if you specify -Dplugins.include, you need to put all the plugins you want to use, e.g. pinot-json, pinot-avro , pinot-kafka-2.0...

You can configure the S3 file system using the following options:

Configuration
Description

Each of these properties should be prefixed by pinot.[node].storage.factory.s3. where node is either controller or server depending on the config

e.g.

S3 Filesystem supports authentication using the . The credential provider looks for the credentials in the following order -

  • Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)

  • Java System Properties - aws.accessKeyId and aws.secretKey

  • Web Identity Token credentials from the environment or container

  • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI

  • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and security manager has permission to access the variable,

  • Instance profile credentials delivered through the Amazon EC2 metadata service

You can also specify the accessKey and secretKey using the properties. However, this method is not secure and should be used only for POC setups.

Examples

Job spec

Controller config

Server config

Minion config

Bloom filter

This page describes configuring the Bloom filter for Apache Pinot

When a column is configured to use this filter, Pinot creates one Bloom filter per segment. The Bloom filter help to prune segments that do not contain any record matching an EQUALITY or IN predicate.

Note: Support for IN clause is limited to <= 10 values in the predicate, this is to ensure pruning overhead is minimal.

This is useful for query patterns like below where Bloom Filter is defined on playerID column in the table:

Details

A Bloom filter is a probabilistic data structure used to definitively determine if an element is not present in a dataset, but it cannot be employed to determine if an element is present in the dataset. This limitation arises because Bloom filters may produce false positives but never yield false negatives.

An intriguing aspect of these filters is the existence of a mathematical formula that establishes a relationship between their size, the cardinality of the dataset they index, and the rate of false positives.

In Pinot, this cardinality corresponds to the number of unique values expected within each segment. If necessary, the false positive rate and the index size can be configured.

Configuration

Bloom filters are deactivated by default, implying that columns will not be indexed unless they are explicitly configured within the .

There are 3 optional parameters to configure the Bloom filter:

Parameter
Default
Description

The lower the fpp (false positive probability), the greater the accuracy of the Bloom filter, but this reduction in fpp will also lead to an increase in the index size. It's important to note that maxSizeInBytes takes precedence over fpp. If maxSizeInBytes is set to a value greater than 0 and the calculated size of the Bloom filter, based on the specified fpp, exceeds this size limit, Pinot will adjust the fpp to ensure that the Bloom filter size remains within the specified limit.

Similar to other indexes, a Bloom filter can be explicitly deactivated by setting the special parameter disabled to true.

Example

For example the following table config enables the Bloom filter in the playerId column using the default values:

In case some parameter needs to be customized, they can be included in fieldConfigList.indexes.bloom. Remember that even the example customizes all parameters, you can just modify the ones you need.

Older configuration

Use default settings

To use default values, include the name of the column in tableIndexConfig.bloomFilterColumns.

For example:

Customized parameters

To specify custom parameters, add a new entry in tableIndexConfig.bloomFilterConfig object. The key should be the name of the column and the value should be an object similar to the one that can be used in the Bloom section of fieldConfigList.

For example:

segment config
-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-s3

region

The AWS Data center region in which the bucket is located

accessKey

(Optional) AWS access key required for authentication. This should only be used for testing purposes as we don't store these keys in secret.

secretKey

(Optional) AWS secret key required for authentication. This should only be used for testing purposes as we don't store these keys in secret.

endpoint

(Optional) Override endpoint for s3 client.

disableAcl

If this is set tofalse, bucket owner is granted full access to the objects created by pinot. Default value is true.

serverSideEncryption

(Optional) The server-side encryption algorithm used when storing this object in Amazon S3 (Now supports aws:kms), set to null to disable SSE.

ssekmsKeyId

(Optional, but required when serverSideEncryption=aws:kms) Specifies the AWS KMS key ID to use for object encryption. All GET and PUT requests for an object protected by AWS KMS will fail if not made via SSL or using SigV4.

ssekmsEncryptionContext

(Optional) Specifies the AWS KMS Encryption Context to use for object encryption. The value of this header is a base64-encoded UTF-8 string holding JSON with the encryption context key-value pairs.

pinot.controller.storage.factory.s3.region=ap-southeast-1
executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: 's3://pinot-bucket/pinot-ingestion/batch-input/'
outputDirURI: 's3://pinot-bucket/pinot-ingestion/batch-output/'
overwriteOutput: true
pinotFSSpecs:
    - scheme: s3
      className: org.apache.pinot.plugin.filesystem.S3PinotFS
      configs:
        region: 'ap-southeast-1'
recordReaderSpec:
    dataFormat: 'csv'
    className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
    configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
    tableName: 'students'
pinotClusterSpecs:
    - controllerURI: 'http://localhost:9000'
controller.data.dir=s3://path/to/data/directory/
controller.local.temp.dir=/path/to/local/temp/directory
controller.enable.split.commit=true
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=ap-southeast-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=ap-southeast-1
pinot.server.storage.factory.s3.httpclient.maxConnections=50
pinot.server.storage.factory.s3.httpclient.socketTimeout=30s
pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s
pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s
pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.minion.storage.factory.s3.region=ap-southeast-1
pinot.minion.segment.fetcher.protocols=file,http,s3
pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
Amazon S3
DefaultCredentialsProviderChain
SELECT COUNT(*) 
FROM baseballStats 
WHERE playerID = 12345

OR

SELECT COUNT(*) 
FROM baseballStats 
WHERE playerID IN(12345, 45668, 56789)

fpp

0.05

False positive probability of the Bloom filter (from 0 to 1).

maxSizeInBytes

0 (unlimited)

Maximum size of the Bloom filter.

loadOnHeap

false

Whether to load the Bloom filter using heap memory or off-heap memory.

Configured in tableConfig fieldConfigList
{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name": "playerID",
      "encodingType": "RAW",
      "indexes": {
        "bloom": {}
      }
    },
    ...
  ],
  ...
}
Configured in tableConfig fieldConfigList
{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name": "playerID",
      "encodingType": "RAW",
      "indexes": {
        "bloom": {
          "fpp": 0.01,
          "maxSizeInBytes": 1000000,
          "loadOnHeap": true
        }
      }
    },
    ...
  ],
  ...
}
Part of a tableConfig
{
  "tableName": "somePinotTable",
  "tableIndexConfig": {
    "bloomFilterColumns": [
      "playerID",
      ...
    ],
    ...
  },
  ...
}
Part of a tableConfig
{
  "tableIndexConfig": {
    "bloomFilterConfigs": {
      "playerID": {
        "fpp": 0.01,
        "maxSizeInBytes": 1000000,
        "loadOnHeap": true
      },
      ...
    },
    ...
  },
  ...
}
table configuration

Create and update a table configuration

Create and edit a table configuration in the Pinot UI or with the API.

In Apache Pinot, create a table by creating a JSON file, generally referred to as your table config. Update, add, or delete parameters as needed, and then reload the file.

Create a Pinot table configuration

Before you create a Pinot table configuration, you must first have a running Pinot cluster with broker and server tenants.

  1. Create a plaintext file table configuration locally using settings from the available properties for your use case. You may find it useful to download an example from the Pinot GitHub and then modify it. An example from among these is included at the end of this page in Example Pinot table config file

  2. Use the Pinot API to upload your table config file: POST @fileName.json URL:9000/tables

Update a Pinot table configuration

To modify your Pinot table configuration, use the Pinot UI or the API.

Any time you make a change to your table config, you may need to do one or more of the following, depending on the change.

Simple changes only require updating and saving your modified table configuration file. These include:

  • Changing the data or segment retention time

  • Changing the realtime consumption rate limiter settings

To update existing data and segments, after you update and save the changes to the table config file, do the following as applicable:

When you add or modify indexes or the table schema, perform a segment reload. To reload all segments:

  • In the Pinot UI, from the table page, click Reload All Segments.

  • Using the Pinot API, send POST /segments/{tableName}/reload.

When you re-partition data, perform a segment refresh. To refresh, replace an existing segment with a new one by uploading a segment reusing the existing filename. Use the Pinot API, send POST /segments?tableName={yourTableName}.

When you change the transform function used to populate a derived field or increase the number of partitions in an upsert-enabled table, perform a table re-bootstrap. One way to do this is to delete and recreate the table:

  • Using the Pinot API, first send DELETE /tables/{tableName} followed by POST /tables with the new table configuration.

When you change the stream topic or change the Kafka cluster containing the Kafka topic you want to consume from, perform a real-time ingestion pause and resume. To pause and resume real-time ingestion:

  • Using the Pinot API, first send POST /tables/{tableName}/pauseConsumption followed by POST /tables/{tableName}/resumeConsumption.

Update a Pinot table in the UI

To update a table configuration in the Pinot UI, do the following:

  1. In the Cluster Manager click the Tenant Name of the tenant that hosts the table you want to modify.

  2. Click the Table Name in the list of tables in the tenant.

  3. Click the Edit Table button. This creates a pop-up window containing the table configuration. Edit the contents in this window. Click Save when you are done.

Update a Pinot table using the API

To update a table configuration using the Pinot API, do the following:

  1. Get the current table configuration with GET /tables/{tableName}.

  2. Modify the file locally.

  3. Upload the edited file with PUT /table/{tableName} fileName.json.

Example Pinot table configuration file

This example comes from the Apache Pinot Quickstart Examples. This table configuration defines a table called airlineStats_OFFLINE, which you can interact with by running the example.

{
  "OFFLINE": {
    "tableName": "airlineStats_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "timeType": "DAYS",
      "replication": "1",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "timeColumnName": "DaysSinceEpoch",
      "segmentPushType": "APPEND",
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "loadMode": "MMAP",
      "enableDefaultStarTree": false,
      "starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "AirlineID",
            "Origin",
            "Dest"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [
            "COUNT__*",
            "MAX__ArrDelay"
          ],
          "maxLeafRecords": 10
        },
        {
          "dimensionsSplitOrder": [
            "Carrier",
            "CancellationCode",
            "Origin",
            "Dest"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [
            "MAX__CarrierDelay",
            "AVG__CarrierDelay"
          ],
          "maxLeafRecords": 10
        }
      ],
      "enableDynamicStarTreeCreation": true,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false,
      "optimizeDictionary": false,
      "optimizeDictionaryForMetrics": false,
      "noDictionarySizeRatioThreshold": 0
    },
    "metadata": {
      "customConfigs": {}
    },
    "fieldConfigList": [
      {
        "name": "ts",
        "encodingType": "DICTIONARY",
        "indexType": "TIMESTAMP",
        "indexTypes": [
          "TIMESTAMP"
        ],
        "timestampConfig": {
          "granularities": [
            "DAY",
            "WEEK",
            "MONTH"
          ]
        }
      }
    ],
    "ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "ts",
          "transformFunction": "fromEpochDays(DaysSinceEpoch)"
        },
        {
          "columnName": "tsRaw",
          "transformFunction": "fromEpochDays(DaysSinceEpoch)"
        }
      ],
      "continueOnError": false,
      "rowTimeValueCheck": false,
      "segmentTimeValueCheck": true
    },
    "tierConfigs": [
      {
        "name": "hotTier",
        "segmentSelectorType": "time",
        "segmentAge": "3130d",
        "storageType": "pinot_server",
        "serverTag": "DefaultTenant_OFFLINE"
      },
      {
        "name": "coldTier",
        "segmentSelectorType": "time",
        "segmentAge": "3140d",
        "storageType": "pinot_server",
        "serverTag": "DefaultTenant_OFFLINE"
      }
    ],
    "isDimTable": false
  }
}

Spark

Batch ingestion of data into Apache Pinot using Apache Spark.

Pinot supports Apache Spark (2.x and 3.x) as a processor to create and push segment files to the database. Pinot distribution is bundled with the Spark code to process your files and convert and upload them to Pinot.

To set up Spark, do one of the following:

  • Use the Spark-Pinot Connector. For more information, see the ReadMe.

  • Follow the instructions below.

You can follow the wiki to build Pinot from source. The resulting JAR file can be found in pinot/target/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar

If you do build Pinot from Source, you should consider opting into using the build-shaded-jar jar profile with -Pbuild-shaded-jar. While Pinot does not bundle spark into its jar, it does bundle certain hadoop libraries.

Next, you need to change the execution config in the job spec to the following:

# executionFrameworkSpec: Defines ingestion jobs to be running.
executionFrameworkSpec:

  # name: execution framework name
  name: 'spark'

  # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner'

  # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentTarPushJobRunner'

  # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface.
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentUriPushJobRunner'

  #segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.spi.ingestion.batch.runner.IngestionJobRunner interface
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentMetadataPushJobRunner'

  # extraConfigs: extra configs for execution framework.
  extraConfigs:

    # stagingDir is used in distributed filesystem to host all the segments then move this directory entirely to output directory.
    stagingDir: your/local/dir/staging

To run Spark ingestion, you need the following jars in your classpath

  • pinot-batch-ingestion-spark plugin jar - available in plugins-external directory in the package

  • pinot-all jar - available in lib directory in the package

These jars can be specified using spark.driver.extraClassPath or any other option.

spark.driver.extraClassPath =>
pinot-batch-ingestion-spark-${PINOT_VERSION}-shaded.jar:pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar

For loading any other plugins that you want to use, use:

spark.driver.extraJavaOptions =>
-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins

The complete spark-submit command should look like this:

export PINOT_VERSION=0.10.0
export PINOT_DISTRIBUTION_DIR=/path/to/apache-pinot-${PINOT_VERSION}-bin

spark-submit //
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand //
--master local --deploy-mode client //
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins" //
--conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
-conf "spark.executor.extraClassPath=${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile /path/to/spark_job_spec.yaml

Ensure environment variables PINOT_ROOT_DIR and PINOT_VERSION are set properly.

Note: You should change the master to yarn and deploy-mode to cluster for production environments.

We have stopped including spark-core dependency in our jars post 0.10.0 release. Users can try 0.11.0-SNAPSHOT and later versions of pinot-batch-ingestion-spark in case of any runtime issues. You can either build from source or download latest master build jars.

Running in Cluster Mode on YARN

If you want to run the spark job in cluster mode on YARN/EMR cluster, the following needs to be done -

  • Build Pinot from source with option -DuseProvidedHadoop

  • Copy Pinot binaries to S3, HDFS or any other distributed storage that is accessible from all nodes.

  • Copy Ingestion spec YAML file to S3, HDFS or any other distributed storage. Mention this path as part of --files argument in the command

  • Add --jars options that contain the s3/hdfs paths to all the required plugin and pinot-all jar

  • Point classPath to spark working directory. Generally, just specifying the jar names without any paths works. Same should be done for main jar as well as the spec YAML file

Example

spark-submit //
--class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand //
--master yarn --deploy-mode cluster //
--conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins" //
--conf "spark.driver.extraClassPath=pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
--conf "spark.executor.extraClassPath=pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar:pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" //
--jars "${PINOT_DISTRIBUTION_DIR}/plugins-external/pinot-batch-ingestion/pinot-batch-ingestion-spark-2.4/pinot-batch-ingestion-spark-2.4-${PINOT_VERSION}-shaded.jar,${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar"
--files s3://path/to/spark_job_spec.yaml
local://pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar -jobSpecFile spark_job_spec.yaml

For Spark 3.x, replace pinot-batch-ingestion-spark-2.4 with pinot-batch-ingestion-spark-3.2 in all places in the commands. Also, ensure the classpath in ingestion spec is changed from org.apache.pinot.plugin.ingestion.batch.spark. to org.apache.pinot.plugin.ingestion.batch.spark3.

FAQ

Q - I am getting the following exception - Class has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0

Since 0.8.0 release, Pinot binaries are compiled with JDK 11. If you are using Spark along with Hadoop 2.7+, you need to use the Java8 version of Pinot. Currently, you need to build jdk 8 version from source.

Q - I am not able to find pinot-batch-ingestion-spark jar.

For Pinot version prior to 0.10.0, the spark plugin is located in plugin dir of binary distribution. For 0.10.0 and later, it is located in pinot-external dir.

Q - Spark is not able to find the jars leading to java.nio.file.NoSuchFileException

This means the classpath for spark job has not been configured properly. If you are running spark in a distributed environment such as Yarn or k8s, make sure both spark.driver.classpath and spark.executor.classpath are set. Also, the jars in driver.classpath should be added to --jars argument in spark-submit so that spark can distribute those jars to all the nodes in your cluster. You also need to take provide appropriate scheme with the file path when running the jar. In this doc, we have used local:\\ but it can be different depending on your cluster setup.

Q - Spark job failing while pushing the segments.

It can be because of misconfigured controllerURI in job spec yaml file. If the controllerURI is correct, make sure it is accessible from all the nodes of your YARN or k8s cluster.

Q - My data gets overwritten during ingestion.

Set segmentPushType to APPEND in the tableConfig.

If already set to APPEND, this is likely due to a missing timeColumnName in your table config. If you can't provide a time column, use our segment name generation configs in ingestion spec. Generally using inputFile segment name generator should fix your issue.

Q - I am getting java.lang.RuntimeException: java.io.IOException: Failed to create directory: pinot-plugins-dir-0/plugins/*

Removing -Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins from spark.driver.extraJavaOptions should fix this. As long as plugins are mentioned in classpath and jars argument it should not be an issue.

Q - Getting Class not found: exception

Check if extraClassPath arguments contain all the plugin jars for both driver and executors. Also, all the plugin jars are mentioned in the --jars argument. If both of these are correct, check if the extraClassPath contains local filesystem classpaths and not s3 or hdfs or any other distributed file system classpaths.

/tmp/pinot-quick-start/transcript-table-realtime.json
{
  "tableName": "transcript",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "transcript-topic",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
docker run \
    --network=pinot-demo \
    -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \
    --name pinot-streaming-table-creation \
    apachepinot/pinot:latest AddTable \
    -schemaFile /tmp/pinot-quick-start/transcript-schema.json \
    -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \
    -controllerHost manual-pinot-controller \
    -controllerPort 9000 \
    -exec
bin/pinot-admin.sh AddTable \
    -schemaFile /tmp/pinot-quick-start/transcript-schema.json \
    -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \
    -exec
/tmp/pinot-quick-start/rawData/transcript.json
{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"Maths","score":3.8,"timestampInEpoch":1571900400000}
{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"History","score":3.5,"timestampInEpoch":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Maths","score":3.2,"timestampInEpoch":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Chemistry","score":3.6,"timestampInEpoch":1572418800000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Geography","score":3.8,"timestampInEpoch":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"English","score":3.5,"timestampInEpoch":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Maths","score":3.2,"timestampInEpoch":1572678000000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Physics","score":3.6,"timestampInEpoch":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"Maths","score":3.8,"timestampInEpoch":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"English","score":3.5,"timestampInEpoch":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"History","score":3.2,"timestampInEpoch":1572854400000}
{"studentID":212,"firstName":"Nick","lastName":"Young","gender":"Male","subject":"History","score":3.6,"timestampInEpoch":1572854400000}
bin/kafka-console-producer.sh \
    --broker-list localhost:9876 \
    --topic transcript-topic < /tmp/pinot-quick-start/rawData/transcript.json
Pinot in Docker
Pluggable Streams
docker run \
    --network pinot-demo --name=kafka \
    -e KAFKA_ZOOKEEPER_CONNECT=manual-zookeeper:2181/kafka \
    -e KAFKA_BROKER_ID=0 \
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
    -d bitnami/kafka:latest
docker exec \
  -t kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --zookeeper manual-zookeeper:2181/kafka \
  --partitions=1 --replication-factor=1 \
  --create --topic transcript-topic
bin/pinot-admin.sh  StartKafka -zkAddress=localhost:2123/kafka -port 9876
Kafka
Batch upload sample data
Creating a schema
Batch upload sample data
Table
Query Console
bin/kafka-topics.sh --create --bootstrap-server localhost:9876 --replication-factor 1 --partitions 1 --topic transcript-topic
Tables
Segments
Tenants
Clusters
segments
table
schema
tables
schema
segments
ingestion
servers
tenant
tenant
brokers
servers
cluster
Physical architecture
Apache Zookeeper
Apache Helix
controller
controller
real-time tables
offline tables
REST API endpoint
Servers
brokers
minion

Running Pinot locally

This quick start guide will help you bootstrap a Pinot standalone instance on your local machine.

In this guide, you'll learn how to download and install Apache Pinot as a standalone instance.

Download Apache Pinot

First, download the Pinot distribution for this tutorial. You can either download a packaged release or build a distribution from the source code.

Prerequisites

  • Install with JDK 11 or 21. JDK 17 should work, but it is not officially supported.

  • For JDK 8 support, Pinot 0.12.1 is the last version compilable from the source code.

  • Pinot 1.0+ doesn't support JDK 8 anymore, build with JDK 11+

Note that some installations of the JDK do not contain the JNI bindings necessary to run all tests. If you see an error like java.lang.UnsatisfiedLinkError while running tests, you might need to change your JDK.

Download the distribution or build from source by selecting one of the following tabs:

Download the latest binary release from , or use this command:

Extract the TAR file:

Navigate to the directory containing the launcher scripts:

You can also find older versions of Apache Pinot at . For example, to download Pinot 0.10.0, run the following command:

Follow these steps to checkout code from and build Pinot locally

Prerequisites

Install 3.6 or higher

Check out Pinot:

Build Pinot:

If you're building with JDK 8, add Maven option -Djdk.version=8.

Navigate to the directory containing the setup scripts. Note that Pinot scripts are located under pinot-distribution/target, not the target directory under root.

Pinot can also be installed on Mac OS using the Brew package manager. For instructions on installing Brew, see the .

Set up a cluster

Now that we've downloaded Pinot, it's time to set up a cluster. There are two ways to do this: through quick start or through setting up a cluster manually.

Quick start

Pinot comes with quick start commands that launch instances of Pinot components in the same process and import pre-built datasets.

For example, the following quick start command launches Pinot with a baseball dataset pre-loaded:

For a list of all the available quick start commands, see the .

Manual cluster

If you want to play with bigger datasets (more than a few megabytes), you can launch each component individually.

The video below is a step-by-step walk through for launching the individual components of Pinot and scaling them to multiple instances.

You can find the commands that are shown in this video in the .

The examples below assume that you are using Java 11+.

If you are using Java 8, add the following settings insideJAVA_OPTS. So, for example, instead of this:

Use the following:

Start Zookeeper

You can use to browse the Zookeeper instance.

Start Pinot Controller

Start Pinot Broker

Start Pinot Server

Start Pinot Minion

Start Kafka

Once your cluster is up and running, you can head over to to learn how to run queries against the data.

Setup cluster with config files

Users could start and customize the cluster by modifying the config files and start the components with config files:

Start a Pinot component in debug mode with IntelliJ

Set break points and inspect variables by starting a Pinot component with debug mode in IntelliJ.

The following example demonstrates server debugging:

  1. First, startzookeeper , controller, and broker using the .

  2. Then, use the following configuration under $PROJECT_DIR$\.run ) to start the server, replacing the metrics-core version and cluster name as needed. This is an example of how to use it.

0.4.0

0.4.0 release introduced the theta-sketch based distinct count function, an S3 filesystem plugin, a unified star-tree index implementation, migration from TimeFieldSpec to DateTimeFieldSpec, etc.

Summary

0.4.0 release introduced various new features, including the theta-sketch based distinct count aggregation function, an S3 filesystem plugin, a unified star-tree index implementation, deprecation of TimeFieldSpec in favor of DateTimeFieldSpec, etc. Miscellaneous refactoring, performance improvement and bug fixes were also included in this release. See details below.

Notable New Features

  • Made DateTimeFieldSpecs mainstream and deprecated TimeFieldSpec (#2756)

    • Used time column from table config instead of schema (#5320)

    • Included dateTimeFieldSpec in schema columns of Pinot Query Console #5392

    • Used DATE_TIME as the primary time column for Pinot tables (#5399)

  • Supported range queries using indexes (#5240)

  • Supported complex aggregation functions

    • Supported Aggregation functions with multiple arguments (#5261)

    • Added api in AggregationFunction to get compiled input expressions (#5339)

  • Added a simple PinotFS benchmark driver (#5160)

  • Supported default star-tree (#5147)

  • Added an initial implementation for theta-sketch based distinct count aggregation function (#5316)

    • One minor side effect: DataSchemaPruner won't work for DistinctCountThetaSketchAggregatinoFunction (#5382)

  • Added access control for Pinot server segment download api (#5260)

  • Added Pinot S3 Filesystem Plugin (#5249)

  • Text search improvement

    • Pruned stop words for text index (#5297)

    • Used 8byte offsets in chunk based raw index creator (#5285)

    • Derived num docs per chunk from max column value length for varbyte raw index creator (#5256)

    • Added inter segment tests for text search and fixed a bug for Lucene query parser creation (#5226)

    • Made text index query cache a configurable option (#5176)

    • Added Lucene DocId to PinotDocId cache to improve performance (#5177)

    • Removed the construction of second bitmap in text index reader to improve performance (#5199)

  • Tooling/usability improvement

    • Added template support for Pinot Ingestion Job Spec (#5341)

    • Allowed user to specify zk data dir and don't do clean up during zk shutdown (#5295)

    • Allowed configuring minion task timeout in the PinotTaskGenerator (#5317)

    • Update JVM settings for scripts (#5127)

    • Added Stream github events demo (#5189)

    • Moved docs link from gitbook to docs.pinot.apache.org (#5193)

  • Re-implemented ORCRecordReader (#5267)

  • Evaluated schema transform expressions during ingestion (#5238)

  • Handled count distinct query in selection list (#5223)

  • Enabled async processing in pinot broker query api (#5229)

  • Supported bootstrap mode for table rebalance (#5224)

  • Supported order-by on BYTES column (#5213)

  • Added Nightly publish to binary (#5190)

  • Shuffled the segments when rebalancing the table to avoid creating hotspot servers (#5197)

  • Supported built-in transform functions (#5312)

    • Added date time transform functions (#5326)

  • Deepstore by-pass in LLC: introduced segment uploader (#5277, #5314)

  • APIs Additions/Changes

    • Added a new server api for download of segments

      • /GET /segments/{tableNameWithType}/{segmentName}

  • Upgraded helix to 0.9.7 (#5411)

  • Added support to execute functions during query compilation (#5406)

  • Other notable refactoring

    • Moved table config into pinot-spi (#5194)

    • Cleaned up integration tests. Standardized the creation of schema, table config and segments (#5385)

    • Added jsonExtractScalar function to extract field from json object (#4597)

    • Added template support for Pinot Ingestion Job Spec #5372

    • Cleaned up AggregationFunctionContext (#5364)

    • Optimized real-time range predicate when cardinality is high (#5331)

    • Made PinotOutputFormat use table config and schema to create segments (#5350)

    • Tracked unavailable segments in InstanceSelector (#5337)

    • Added a new best effort segment uploader with bounded upload time (#5314)

    • In SegmentPurger, used table config to generate the segment (#5325)

    • Decoupled schema from RecordReader and StreamMessageDecoder (#5309)

    • Implemented ARRAYLENGTH UDF for multi-valued columns (#5301)

    • Improved GroupBy query performance (#5291)

    • Optimized ExpressionFilterOperator (#5132)

Major Bug Fixes

  • Do not release the PinotDataBuffer when closing the index (#5400)

  • Handled a no-arg function in query parsing and expression tree (#5375)

  • Fixed compatibility issues during rolling upgrade due to unknown json fields (#5376)

  • Fixed missing error message from pinot-admin command (#5305)

  • Fixed HDFS copy logic (#5218)

  • Fixed spark ingestion issue (#5216)

  • Fixed the capacity of the DistinctTable (#5204)

  • Fixed various links in the Pinot website

Work in Progress

  • Upsert: support overriding data in the real-time table (#4261).

    • Add pinot upsert features to pinot common (#5175)

  • Enhancements for theta-sketch, e.g. multiValue aggregation support, complex predicates, performance tuning, etc

Backward Incompatible Changes

  • TableConfig no longer support de-serialization from json string of nested json string (i.e. no \" inside the json) (#5194)

  • The following APIs are changed in AggregationFunction (use TransformExpressionTree instead of String as the key of blockValSetMap) (#5371):

void aggregate(int length, AggregationResultHolder aggregationResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);
void aggregateGroupBySV(int length, int[] groupKeyArray, GroupByResultHolder groupByResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);
void aggregateGroupByMV(int length, int[][] groupKeysArray, GroupByResultHolder groupByResultHolder, Map<TransformExpressionTree, BlockValSet> blockValSetMap);
brew install pinot
./bin/pinot-admin.sh QuickStart -type batch
export JAVA_OPTS="-Xms4G -Xmx8G"
export JAVA_OPTS="-Xms4G -Xmx8G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
./bin/pinot-admin.sh StartZookeeper \
  -zkPort 2191
export JAVA_OPTS="-Xms4G -Xmx8G"
./bin/pinot-admin.sh StartController \
    -zkAddress localhost:2191 \
    -controllerPort 9000
export JAVA_OPTS="-Xms4G -Xmx4G"
./bin/pinot-admin.sh StartBroker \
    -zkAddress localhost:2191
export JAVA_OPTS="-Xms4G -Xmx16G"
./bin/pinot-admin.sh StartServer \
    -zkAddress localhost:2191
export JAVA_OPTS="-Xms4G -Xmx4G"
./bin/pinot-admin.sh StartMinion \
    -zkAddress localhost:2191
./bin/pinot-admin.sh  StartKafka \ 
  -zkAddress=localhost:2191/kafka \
  -port 19092
./bin/pinot-admin.sh StartController -config conf/pinot-controller.conf
./bin/pinot-admin.sh StartBroker -config conf/pinot-broker.conf
./bin/pinot-admin.sh StartServer -config conf/pinot-server.conf
./bin/pinot-admin.sh StartMinion -config conf/pinot-minion.conf
<component name="ProjectRunConfigurationManager">
  <configuration default="false" name="HelixServerStarter" type="Application" factoryName="Application" nameIsGenerated="true">
    <classpathModifications>
      <entry path="$PROJECT_DIR$/pinot-plugins/pinot-metrics/pinot-yammer/target/classes" />
      <entry path="$MAVEN_REPOSITORY$/com/yammer/metrics/metrics-core/2.2.0/metrics-core-2.2.0.jar" />
    </classpathModifications>
    <option name="MAIN_CLASS_NAME" value="org.apache.pinot.server.starter.helix.HelixServerStarter" />
    <module name="pinot-server" />
    <extension name="coverage">
      <pattern>
        <option name="PATTERN" value="org.apache.pinot.server.starter.helix.*" />
        <option name="ENABLED" value="true" />
      </pattern>
    </extension>
    <method v="2">
      <option name="Make" enabled="true" />
    </method>
  </configuration>
</component>
Download Apache Pinot
Set up a cluster
Start a Pinot component in debug mode with IntelliJ
Apache Pinot
tar -zxvf apache-pinot-$PINOT_VERSION-bin.tar.gz
cd apache-pinot-$PINOT_VERSION-bin
https://archive.apache.org/dist/pinot/
OLDER_VERSION="0.10.0"
wget https://archive.apache.org/dist/pinot/apache-pinot-$OLDER_VERSION/apache-pinot-$OLDER_VERSION-bin.tar.gz
Github
Apache Maven
Brew documentation
Quick Start Examples
this Github repository
Zooinspector
Exploring Pinot
steps described above
commit
PINOT_VERSION=1.1.0 #set to the Pinot version you decide to use

wget https://downloads.apache.org/pinot/apache-pinot-$PINOT_VERSION/apache-pinot-$PINOT_VERSION-bin.tar.gz
git clone https://github.com/apache/pinot.git
cd pinot
mvn install package -DskipTests -Pbin-dist
cd build

Complex Type (Array, Map) Handling

Complex type handling in Apache Pinot.

Commonly, ingested data has a complex structure. For example, Avro schemas have records and arrays while JSON supports objects and arrays.

Apache Pinot's data model supports primitive data types (including int, long, float, double, BigDecimal, string, bytes), and limited multi-value types, such as an array of primitive types. Simple data types allow Pinot to build fast indexing structures for good query performance, but does require some handling of the complex structures.

There are two options for complex type handling:

  • Convert the complex-type data into a JSON string and then build a JSON index.

  • Use the built-in complex-type handling rules in the ingestion configuration.

On this page, we'll show how to handle these complex-type structures with each of these two approaches. We will process some example data, consisting of the field group from the Meetup events Quickstart example.

This object has two child fields and the child group is a nested array with elements of object type.

Example JSON data

JSON indexing

Apache Pinot provides a powerful JSON index to accelerate the value lookup and filtering for the column. To convert an object group with complex type to JSON, add the following to your table configuration.

json_meetupRsvp_realtime_table_config.json
{
    "ingestionConfig":{
      "transformConfigs": [
        {
          "columnName": "group_json",
          "transformFunction": "jsonFormat(\"group\")"
        }
      ],
    },
    ...
    "tableIndexConfig": {
    "loadMode": "MMAP",
    "noDictionaryColumns": [
      "group_json"
    ],
    "jsonIndexColumns": [
      "group_json"
    ]
  },

}

The config transformConfigs transforms the object group to a JSON string group_json, which then creates the JSON indexing with configuration jsonIndexColumns. To read the full spec, see meetupRsvpJson_realtime_table_config.json.

Also, note that group is a reserved keyword in SQL and therefore needs to be quoted in transformFunction.

The columnName can't use the same name as any of the fields in the source JSON data, for example, if our source data contains the field group and we want to transform the data in that field before persisting it, the destination column name would need to be something different, like group_json.

Note that you do not need to worry about the maxLength of the field group_json on the schema, because "JSON" data type does not have a maxLength and will not be truncated. This is true even though "JSON" is stored as a string internally.

The schema will look like this:

json_meetupRsvp_realtime_table_schema.json
{
  {
      "name": "group_json",
      "dataType": "JSON",
    }
    ...
}

For the full specification, see json_meetupRsvp_schema.json.

With this, you can start to query the nested fields under group. For more details about the supported JSON function, see guide).

Ingestion configurations

Though JSON indexing is a handy way to process the complex types, there are some limitations:

  • It’s not performant to group by or order by a JSON field, because JSON_EXTRACT_SCALAR is needed to extract the values in the GROUP BY and ORDER BY clauses, which invokes the function evaluation.

  • It does not work with Pinot's multi-column functions such as DISTINCTCOUNTMV.

Alternatively, from Pinot 0.8, you can use the complex-type handling in ingestion configurations to flatten and unnest the complex structure and convert them into primitive types. Then you can reduce the complex-type data into a flattened Pinot table, and query it via SQL. With the built-in processing rules, you do not need to write ETL jobs in another compute framework such as Flink or Spark.

To process this complex type, you can add the configuration complexTypeConfig to the ingestionConfig. For example:

complexTypeHandling_meetupRsvp_realtime_table_config.json
{
  "ingestionConfig": {    
    "complexTypeConfig": {
      "delimiter": ".",
      "fieldsToUnnest": ["group.group_topics"],
      "collectionNotUnnestedToJson": "NON_PRIMITIVE"
    }
  }
}

With the complexTypeConfig , all the map objects will be flattened to direct fields automatically. And with unnestFields , a record with the nested collection will unnest into multiple records. For instance, the example at the beginning will transform into two rows with this configuration example.

Flattened/unnested data

Note that:

  • The nested field group_id under group is flattened to group.group_id. The default value of the delimiter is . You can choose another delimiter by specifying the configuration delimiter under complexTypeConfig. This flattening rule also applies to maps in the collections to be unnested.

  • The nested array group_topics under group is unnested into the top-level, and converts the output to a collection of two rows. Note the handling of the nested field within group_topics, and the eventual top-level field of group.group_topics.urlkey. All the collections to unnest shall be included in the configuration fieldsToUnnest.

  • Collections not specified in fieldsToUnnestwill be serialized into JSON string, except for the array of primitive values, which will be ingested as a multi-value column by default. The behavior is defined by the collectionNotUnnestedToJson config, which takes the following values:

    • NON_PRIMITIVE- Converts the array to a multi-value column. (default)

    • ALL- Converts the array of primitive values to JSON string.

    • NONE- Does not do any conversion.

You can find the full specifications of the table config here and the table schema here.

You can then query the table with primitive values using the following SQL query:

SELECT "group.group_topics.urlkey", 
       "group.group_topics.topic_name", 
       "group.group_id" 
FROM meetupRsvp
LIMIT 10

. is a reserved character in SQL, so you need to quote the flattened columns in the query.

Infer the Pinot schema from the Avro schema and JSON data

When there are complex structures, it can be challenging and tedious to figure out the Pinot schema manually. To help with schema inference, Pinot provides utility tools to take the Avro schema or JSON data as input and output the inferred Pinot schema.

To infer the Pinot schema from Avro schema, you can use a command like this:

bin/pinot-admin.sh AvroSchemaToPinotSchema \
  -timeColumnName fields.hoursSinceEpoch \
  -avroSchemaFile /tmp/test.avsc \
  -pinotSchemaName myTable \
  -outputDir /tmp/test \
  -fieldsToUnnest entries

Note you can input configurations like fieldsToUnnest similar to the ones in complexTypeConfig. And this will simulate the complex-type handling rules on the Avro schema and output the Pinot schema in the file specified in outputDir.

Similarly, you can use the command like the following to infer the Pinot schema from a file of JSON objects.

bin/pinot-admin.sh JsonToPinotSchema \
  -timeColumnName hoursSinceEpoch \
  -jsonFile /tmp/test.json \
  -pinotSchemaName myTable \
  -outputDir /tmp/test \
  -fieldsToUnnest payload.commits

You can check out an example of this run in this PR.

Ingest streaming data from Apache Pulsar

This guide shows you how to ingest a stream of records from an Apache Pulsar topic into a Pinot table.

Pinot supports consuming data from via the pinot-pulsar plugin. You need to enable this plugin so that Pulsar specific libraries are present in the classpath.

Enable the Pulsar plugin with the following config at the time of Pinot setup: -Dplugins.include=pinot-pulsar

The pinot-pulsar plugin is not part of official 0.10.0 binary. You can download the plugin from and add it to the libs or plugins directory in pinot.

Set up Pulsar table

Here is a sample Pulsar stream config. You can use the streamConfigs section from this sample and make changes for your corresponding table.

Pulsar configuration options

You can change the following Pulsar specifc configurations for your tables

Property
Description

Authentication

The Pinot-Pulsar connector supports authentication using security tokens. To generate a token, follow the instructions in . Once generated, add the following property to streamConfigs to add an authentication token for each request:

OAuth2 Authentication

The Pinot-Pulsar connector supports authentication using OAuth2, for example, if connecting to a StreamNative Pulsar cluster. For more information, see how to . Once configured, you can add the following properties to streamConfigs:

TLS support

The Pinot-pulsar connector also supports TLS for encrypted connections. You can follow to enable TLS on your pulsar cluster. Once done, you can enable TLS in pulsar connector by providing the trust certificate file location generated in the previous step.

Also, make sure to change the brokers url from pulsar://localhost:6650 to pulsar+ssl://localhost:6650 so that secure connections are used.

For other table and stream configurations, you can headover to

Supported Pulsar versions

Pinot currently relies on Pulsar client version 2.7.2. Make sure the Pulsar broker is compatible with the this client version.

Extract record headers as Pinot table columns

Pinot's Pulsar connector supports automatically extracting record headers and metadata into the Pinot table columns. Pulsar supports a large amount of per-record metadata. Reference the for the meaning of the metadata fields.

The following table shows the mapping for record header/metadata to Pinot table column names:

Pulsar Message
Pinot table Column
Comments
Available By Default

In order to enable the metadata extraction in a Pulsar table, set the stream config metadata.populate to true. The fields eventTime, publishTime, brokerPublishTime, and key are populated by default. If you would like to extract additional fields from the Pulsar Message, populate the metadataFields config with a comma separated list of fields to populate. The fields are referenced by the field name in the Pulsar Message. For example, setting:

Will make the __metadata$messageId, __metadata$messageBytes, __metadata$eventTime, and __metadata$topicName, fields available for mapping to columns in the Pinot schema.

In addition to this, if you want to use any of these columns in your table, you have to list them explicitly in your table's schema.

For example, if you want to add only the offset and key as dimension columns in your Pinot table, it can listed in the schema as follows:

Once the schema is updated, these columns are similar to any other pinot column. You can apply ingestion transforms and / or define indexes on them.

Remember to follow the when updating schema of an existing table!

Ingest records with dynamic schemas

Storing records with dynamic schemas in a table with a fixed schema.

Some domains (e.g., logging) generate records where each record can have a different set of keys, whereas Pinot tables have a relatively static schema. For records with varying keys, it's impractical to store each field in its own table column. However, most (if not all) fields may be important, so fields should not be dropped unnecessarily.

Additionally, searching patterns on such table could also be complex and change frequently. Exact match, range query, prefix/suffix match, wildcard search and aggregation functions could be used on any old or newly created keys or values.

SchemaConformingTransformer

The is a that can transform records with dynamic schemas such that they can be ingested in a table with a static schema. The transformer takes record fields that don't exist in the schema and stores them in a type of catchall field. Moreover, it builds a __mergedTextIndex field and takes advantage of Lucene to fulfill text search.

For example, consider this record:

Let's say the table's schema contains the following fields:

  • arrayField

  • mapField

  • nestedFields

  • nestedFields.stringField

  • json_data

  • json_data_no_idx

  • __mergedTextIndex

Without this transformer, stringField field and fields ends with _noIdx would be dropped. mapField and nestedFields fields' storage needs to rely on the global setup in complexTransformers without granular customizations. However, with this transformer, the record would be transformed into the following:

Notice that there are 3 reserved (and configurable) fields json_data, json_data_no_idx and __mergedTextIndex. And the transformer does the following:

  • Flattens nested fields all the way to the leaf node and:

    • Conducts special treatments if necessary according to the config

    • If the key path matches the schema, put the data into the dedicated field

    • Otherwise, put them into json_data or json_data_no_idx depending on its key suffix

  • For keys in dedicated columns or json_data, puts them into __mergedTextIndex in the form of "Begin Anchor + value + Separator + key + End Anchor" to power the text matches.

  • Additional functionalities by configurations

    • Drop fields fieldPathsToDrop

    • Preserve the subtree without flattening fieldPathsToPreserveInput and fieldPathsToPreserveInputWithIndex

    • Skip storaging the fields but still indexing it (message in the example) fieldPathsToSkipStorage

    • Skip indexing the fields unindexableFieldSuffix

    • Optimize case insensitive search optimizeCaseInsensitiveSearch

    • Map input key path to a schema name with customizations columnNameToJsonKeyPathMap

    • Support anonymous dot, {'a.b': 'c'} vs {'a': {'b': 'c}} useAnonymousDotInFieldNames

    • Truncate value by length mergedTextIndexDocumentMaxLength

    • Double ingestion to support schema evolution fieldsToDoubleIngest

Table Configurations

SchemaConformingTransformer Configuration

To use the transformer, add the schemaConformingTransformerConfig option in the ingestionConfig section of your table configuration, as shown in the following example.

For example:

Available configuration options are listed in .

Configuration of reserved fields

Other index config of 3 reserved columns could be set like:

Specifically, customizable json index could be set according to .

Power the text search

Schema Design

With the help of SchemaConformingTransformer, all data could be kept even without specifying special dedicated columns in table schema. However, to optimize the storage and various query patterns, dedicated columns should be created based on the usage:

  • Fields with frequent exact match query, e.g. region, log_level, runtime_env

  • Fields with range query, e.g. timestamp

  • High frequency fields from messages

    • Reduce json index size

    • Optimize group by queries

Text Search

After putting each key/value pairs into the __mergedTextIndex field, there will neeed to be luceneAnalyzerClass to tokenize the document and luceneQueryParserClass to query by tokens. Some example common searching patterns and their queries are:

  • Exact key/value match TEXT_MATCH(__mergedTextIndex, '"valuer:key"')

  • Wildcard value search in a key TEXT_MATCH(__mergedTextIndex, '/.* value .*:key/')

  • Key exists check TEXT_MATCH(__mergedTextIndex, '/.*:key/')

  • Global value exact match TEXT_MATCH(__mergedTextIndex, '/"value"/')

  • Global value wildcard match TEXT_MATCH(__mergedTextIndex, '/.* value .*/')

The luceneAnalyzerClass and luceneQueryParserClass usually need to have similar delimiter set. It also needs to consider the values below.

With given example, each key/value pair would be stored as "\u0002value\u001ekey\u0003". The prefix and suffix match on key or value need to be adjusted accordingly in the luceneQueryParserClass.

Dictionary index

When dealing with extensive datasets, it's common for values to be repeated multiple times. To enhance storage efficiency and reduce query latencies, we strongly recommend employing a dictionary index for repetitive data. This is the reason Pinot enables dictionary encoding by default, even though it is advisable to disable it for columns with high cardinality.

Influence on other indexes

In Pinot, dictionaries serve as both an index and actual encoding. Consequently, when dictionaries are enabled, the behavior or layout of certain other indexes undergoes modification. The relationship between dictionaries and other indexes is outlined in the following table:

Index
Conditional
Description

Configuration

Deterministically enable or disable dictionaries

Unlike many other indexes, dictionary indexes are enabled by default, under the assumption that the count of unique values will be significantly lower than the number of rows.

If this assumption does not hold true, you can deactivate the dictionary for a specific column by setting the disabled property to true within indexes.dictionary:

Alternatively, the encodingType property can be changed. For example:

You may choose the option you prefer, but it's essential to maintain consistency, as Pinot will reject table configurations where the same column and index are defined in different locations.

Heuristically enable dictionaries

Most of the time the domain expert that creates the table knows whether a dictionary will be useful or not. For example, a column with random values or public IPs will probably have a large cardinality, so they can be immediately be targeted as raw encoded while columns like employee ids will have a small cardinality and therefore can be easily be recognized as good dictionary candidates. But sometimes the decision may not be clear. To help in these situations, Pinot can be configured to heuristically create the dictionary depending on the actual values and a relation factor.

When this heuristic is enabled, Pinot calculates a saving factor for each candidate column. This factor is the ratio between the forward index size encoded as raw and the same index encoded as a dictionary. If the saving factor for a candidate column is less than a saving ratio, the dictionary is not created.

In order to be considered as a candidate for the heuristic, a column must:

  • Be marked as dictionary encoded (columns marked as raw are always encoded as raw).

  • Be single valued (multi-valued columns are never considered by the heuristic).

  • Be of a fixed size type such as int, long, double, timestamp, etc. Variable size types like json, strings or bytes are never considered by the heuristic.

  • Not indexed by or (as they are only useful when cardinality is very large).

Optionally this feature can be applied only to metric columns, skipping dimension columns.

This functionality can be enabled within the indexingConfig object within the table configuration. The parameters that govern these heuristics are:

Parameter
Default
Description

It's important to emphasize that:

  • These parameters are configured for all columns within the table.

  • optimizeDictionary takes precedence over optimizeDictionaryForMetrics.

Parameters

Dictionaries can be configured with the following options

Parameter
Default
Description

Variable length dictionaries

The useVarLengthDictionary parameter only impacts columns with values that vary in the number of bytes they occupy. This includes column types that require a variable number of bytes, such as strings, bytes, or big decimals, and scenarios where not all values within a segment occupy the same number of bytes. For example, even strings in general require a variable number of bytes to be stored, if a segment contains only the values "a", "b", and "c" Pinot will identify that all values in the segment can be represented with the same number of bytes.

By default, useVarLengthDictionary is set to false, which means Pinot will calculate the length of the largest value contained within the segment. This length will then be used for all values. This approach ensures that all values can be stored efficiently, resulting in faster access and a more compressed layout when the lengths of values are similar.

If your dataset includes a few very large values and a multitude of very small ones, it is advisable to instruct Pinot to utilize variable-length encoding by setting useVarLengthDictionary to true. When variable encoding is employed, Pinot is required to store the length of each entry. Consequently, the cost of storing an entry becomes its actual size plus an additional 4 bytes for the offset.

On-heap dictionaries

Dictionary data is always stored off-heap. In general, it is recommended to keep dictionaries that way. However, in cases where the cardinality is small, and the on-heap memory usage is acceptable, you can copy them into memory by setting the onHeap parameter to true.

Remember: On-heap dictionaries are not recommended.

On-heap dictionaries can slightly reduce latency but will significantly increase the heap memory used by Pinot and increase garbage collection times, which may result in out of memory issues.

When off-heap dictionaries are used, data is deserialized each time it is accessed. This isn't a problem with primitive types (such as int or long), but with complex types (like strings or bytes), this means that the data is deserialized each time it is accessed. On-heap dictionaries solve this problem by keeping the data in memory in deserialized format so no allocations are needed at query time.

However, on-heap dictionaries have a cost in terms of memory usage and that cost is proportional to the number of segments that are accessed concurrently. It is important to note that, as with all other indexes, the dictionary scope is limited to segments. This means that if we have a table with 1,000 segments and a dictionary for a column, we may have 1,000 dictionaries in memory. This can be a waste of memory in cases where unique values are repeated across segments. To solve this problem, Pinot can retain a cache of the dictionary values and reuse them across segments. This cache is not shared between different tables or columns and its maximum size is controlled by the dictionary.intern.capacity option.

Only string and byte columns can be interned. Pinot ignores the intern configuration when used on columns with a different data type.

Here's an example of configuring a dictionary to use on-heap dictionaries with intern mode enabled:

0.5.0

This release includes many new features on Pinot ingestion and connectors, query capability and a revamped controller UI.

Summary

This release includes many new features on Pinot ingestion and connectors (e.g., support for filtering during ingestion which is configurable in table config; support for json during ingestion; proto buf input format support and a new Pinot JDBC client), query capability (e.g., a new GROOVY transform function UDF) and admin functions (a revamped Cluster Manager UI & Query Console UI). It also contains many key bug fixes. See details below.

The release was cut from the following commit: and the following cherry-picks:

Notable New Features

  • Allowing update on an existing instance config: PUT /instances/{instanceName} with Instance object as the pay-load ()

  • Add PinotServiceManager to start Pinot components ()

  • Support for protocol buffers input format. ()

  • Add GenericTransformFunction wrapper for simple ScalarFunctions () — Adding support to invoke any scalar function via GenericTransformFunction

  • Add Support for SQL CASE Statement ()

  • Support distinctCountRawThetaSketch aggregation that returns serialized sketch. ()

  • Add multi-value support to SegmentDumpTool () — add segment dump tool as part of the pinot-tool.sh script

  • Add json_format function to convert json object to string during ingestion. () — Can be used to store complex objects as a json string (which can later be queries using jsonExtractScalar)

  • Support escaping single quote for SQL literal () — This is especially useful for DistinctCountThetaSketch because it stores expression as literal E.g. DistinctCountThetaSketch(..., 'foo=''bar''', ...)

  • Support expression as the left-hand side for BETWEEN and IN clause ()

  • Add a new field IngestionConfig in TableConfig — FilterConfig: ingestion level filtering of records, based on filter function. () — TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release ().

  • Allow star-tree creation during segment load () — Introduced a new boolean config enableDynamicStarTreeCreation in IndexingConfig to enable/disable star-tree creation during segment load.

  • Support for Pinot clients using JDBC connection ()

  • Support customized accuracy for distinctCountHLL, distinctCountHLLMV functions by adding log2m value as the second parameter in the function. () —Adding cluster config: default.hyperloglog.log2m to allow user set default log2m value.

  • Add segment encryption on Controller based on table config ()

  • Add a constraint to the message queue for all instances in Helix, with a large default value of 100000. ()

  • Support order-by aggregations not present in SELECT () — Example: "select subject from transcript group by subject order by count() desc" This is equivalent to the following query but the return response should not contain count(). "select subject, count() from transcript group by subject order by count() desc"

  • Add geo support for Pinot queries () — Added geo-spatial data model and geospatial functions

  • Cluster Manager UI & Query Console UI revamp ( and ) — updated cluster manage UI and added table details page and segment details page

  • Add Controller API to explore Zookeeper ()

  • Support BYTES type for dictinctCount and group-by ( and ) —Add BYTES type support to DistinctCountAggregationFunction —Correctly handle BYTES type in DictionaryBasedAggregationOperator for DistinctCount

  • Support for ingestion job spec in JSON format ()

  • Improvements to RealtimeProvisioningHelper command () — Improved docs related to ingestion and plugins

  • Added GROOVY transform function UDF () — Ability to run a groovy script in the query as a UDF. e.g. string concatenation: SELECT GROOVY('{"returnType": "INT", "isSingleValue": true}', 'arg0 + " " + arg1', columnA, columnB) FROM myTable

Special notes

  • Changed the stream and metadata interface () — This PR concludes the work for the issue to extend offset support for other streams

  • TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release ().

  • Config key enable.case.insensitive.pql in Helix cluster config is deprecated, and replaced with enable.case.insensitive. ()

  • Change default segment load mode to MMAP. () —The load mode for segments currently defaults to heap.

Major Bug fixes

  • Fix bug in distinctCountRawHLL on SQL path ()

  • Fix backward incompatibility for existing stream implementations ()

  • Fix backward incompatibility in StreamFactoryConsumerProvider ()

  • Fix logic in isLiteralOnlyExpression. ()

  • Fix double memory allocation during operator setup ()

  • Allow segment download url in Zookeeper to be deep store uri instead of hardcoded controller uri ()

  • Fix a backward compatible issue of converting BrokerRequest to QueryContext when querying from Presto segment splits ()

  • Fix the issue that PinotSegmentToAvroConverter does not handle BYTES data type. ()

Backward Incompatible Changes

  • PQL queries with HAVING clause will no longer be accepted for the following reasons: () — HAVING clause does not apply to PQL GROUP-BY semantic where each aggregation column is ordered individually — The current behavior can produce inaccurate results without any notice — HAVING support will be added for SQL queries in the next release

  • Because of the standardization of the DistinctCountThetaSketch predicate strings, upgrade Broker before Server. The new Broker can handle both standard and non-standard predicate strings for backward-compatibility. ()

{
  "tableName": "pulsarTable",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "pulsar",
      "stream.pulsar.topic.name": "<your pulsar topic name>",
      "stream.pulsar.bootstrap.servers": "pulsar://localhost:6650,pulsar://localhost:6651",
      "stream.pulsar.consumer.prop.auto.offset.reset" : "smallest",
      "stream.pulsar.consumer.type": "lowlevel",
      "stream.pulsar.fetch.timeout.millis": "30000",
      "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json.JSONMessageDecoder",
      "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory",
      "realtime.segment.flush.threshold.rows": "1000000",
      "realtime.segment.flush.threshold.time": "6h"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}

streamType

This should be set to "pulsar"

stream.pulsar.topic.name

Your pulsar topic name

stream.pulsar.bootstrap.servers

Comma-separated broker list for Apache Pulsar

stream.pulsar.metadata.populate

set to true to populate metadata

stream.pulsar.metadata.fields

set to comma separated list of metadata fields

"stream.pulsar.authenticationToken":"your-auth-token"
"stream.pulsar.issuerUrl": "https://auth.streamnative.cloud"
"stream.pulsar.credsFilePath": "file:///path/to/private_creds_file
"stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster"
"stream.pulsar.tlsTrustCertsFilePath": "/path/to/ca.cert.pem"

key : String

__key : String

Yes

properties : Map<String, String>

Each header key is listed as a separate column: __header$HeaderKeyName : String

Yes

publishTime : Long

__metadata$publishTime : String

publish time as determined by the producer

Yes

brokerPublishTime: Optional

__metadata$brokerPublishTime : String

publish time as determined by the broker

Yes

eventTime : Long

__metadata$eventTime : String

Yes

messageId : MessageId -> String

__metadata$messageId : String

String representation of the MessagId field. The format is ledgerId:entryId:partitionIndex

messageId : MessageId -> bytes

__metadata$messageBytes : String

Base64 encoded version of the bytes returned from calling MessageId.toByteArray()

producerName : String

__metadata$producerName : String

schemaVersion : byte[]

__metadata$schemaVersion : String

Base64 encoded value

sequenceId : Long

__metadata$sequenceId : String

orderingKey : byte[]

__metadata$orderingKey : String

Base64 encoded value

size : Integer

__metadata$size : String

topicName : String

__metadata$topicName : String

index : String

__metadata$index : String

redeliveryCount : Integer

__metadata$redeliveryCount : String


"streamConfigs": {
  ...
        "stream.pulsar.metadata.populate": "true",
        "stream.pulsar.metadata.fields": "messageId,messageIdBytes,eventTime,topicName",
  ...
}
  "dimensionFieldSpecs": [
    {
      "name": "__key",
      "dataType": "STRING"
    },
    {
      "name": "__metadata$messageId",
      "dataType": "STRING"
    },
    ...
  ],
Apache Pulsar
our external repository
Pulsar documentation
Configure OAuth2 authentication in Pulsar clients
the official pulsar documentation
Table configuration Reference
official Pulsar documentation
schema evolution guidelines
{
  "arrayField":[0, 1, 2, 3],
  "stringField":"a",
  "intField_noIndex":9,
  "string_noIndex":"z",
  "message": "a",
  "mapField":{
    "arrayField":[0, 1, 2, 3],
    "stringField":"a",
    "intField_noIndex":9,
    "string_noIndex":"z"
  },
  "mapField_noIndex":{
    "arrayField":[0, 1, 2, 3],
    "stringField":"a",
  },
  "nestedFields":{
    "arrayField":[0, 1, 2, 3],
    "stringField":"a",
    "intField_noIndex":9,
    "string_noIndex":"z",
    "mapField":{
      "arrayField":[0, 1, 2, 3],
      "stringField":"a",
      "intField_noIndex":9,
      "string_noIndex":"z"
    }
  }
}
{
  "arrayField":[0, 1, 2, 3],
  "nestedFields.stringField":"a",
  "json_data":{
    "stringField":"a",
    "mapField":{
      "arrayField":[0, 1, 2, 3],
      "stringField":"a",
      "stringField":"aA_123"
    },
    "nestedFields":{
      "arrayField":[0, 1, 2, 3],
      "mapField":{
        "arrayField":[0, 1, 2, 3],
        "stringField":"a"
      }
    }
  },
  "json_data_no_idx":{
    "intField_noIndex":9,
    "string_noIndex":"z",
    "mapField":{
      "intField_noIndex":9,
      "string_noIndex":"z"
    },
    "mapField_noIndex":{
      "arrayField":[0, 1, 2, 3],
      "stringField":"a",
    },
    "nestedFields":{
      "intField_noIndex":9,
      "string_noIndex":"z",
      "mapField":{
        "intField_noIndex":9,
        "string_noIndex":"z"
      }
    }
  },
  "__mergedTextIndex": [
    // To be explained in following sections
  ]
}
"schemaConformingTransformerConfig": {
  "enableIndexableExtras": true,
  "indexableExtrasField": "json_data",
  "enableUnindexableExtras": true,
  "unindexableExtrasField": "json_data_no_idx",
  "unindexableFieldSuffix": "_noindex",
  "fieldPathsToDrop": [],
  "fieldPathsToSkipStorage": [
    "message"
  ],
  "columnNameToJsonKeyPathMap": {},
  "mergedTextIndexField": "__mergedTextIndex",
  "useAnonymousDotInFieldNames": true,
  "optimizeCaseInsensitiveSearch": false,
  "reverseTextIndexKeyValueOrder": true,
  "mergedTextIndexDocumentMaxLength": 32766,
  "mergedTextIndexBinaryDocumentDetectionMinLength": 512,
  "mergedTextIndexPathToExclude": [
    "_timestampMillisNegative",
    "__mergedTextIndex",
    "_timestampMillis"
  ],
  "fieldsToDoubleIngest": [],
  "jsonKeyValueSeparator": "\u001e",
  "mergedTextIndexBeginOfDocAnchor": "\u0002",
  "mergedTextIndexEndOfDocAnchor": "\u0003",
  "fieldPathsToPreserveInput": [],
  "fieldPathsToPreserveInputWithIndex": []
}
"fieldConfigList": [
  {
    "name": "json_data",
    "encodingType": "RAW",
    "indexTypes": [],
    "compressionCodec": "LZ4",
    "indexes": null,
    "properties": {
      "rawIndexWriterVersion": "4"
    },
    "tierOverwrites": null
  },
  {
    "name": "json_data_no_idx",
    "encodingType": "RAW",
    "indexTypes": [],
    "compressionCodec": "ZSTANDARD",
    "indexes": null,
    "properties": {
      "rawIndexWriterVersion": "4"
    },
    "tierOverwrites": null
  },
  {
    "name": "__mergedTextIndex",
    "encodingType": "RAW",
    "indexType": "TEXT",
    "indexTypes": [
      "TEXT"
    ],
    "compressionCodec": "LZ4",
    "indexes": null,
    "properties": {
      "enableQueryCacheForTextIndex": "false",
      "luceneAnalyzerClass": <analyzerClass>,
      "luceneAnalyzerClassArgTypes": <>,
      "luceneAnalyzerClassArgs": <>,
      "luceneMaxBufferSizeMB": "50",
      "luceneQueryParserClass": <parserClass>,
      "luceneUseCompoundFile": "true",
      "noRawDataForTextIndex": "true",
      "rawIndexWriterVersion": "4"
    },
    "tierOverwrites": null
  }
]

"jsonIndexConfigs": {
  "json_data": {
    "disabled": false,
    "maxLevels": 3,
    "excludeArray": true,
    "disableCrossArrayUnnest": true,
    "maxValueLength": 1000,
    "skipInvalidJson": true
  }
}
"jsonKeyValueSeparator": "\u001e",
"mergedTextIndexBeginOfDocAnchor": "\u0002",
"mergedTextIndexEndOfDocAnchor": "\u0003",
SchemaConformingTransformer
RecordTransformer
SchemaConformingTransformerConfig
json index indexPaths

forward

Implementation depends on whether the dictionary is enabled or not.

range

Implementation depends on whether the dictionary is enabled or not.

inverted

Requires the dictionary index to be enabled.

json

when optimizeDictionary

Disables dictionary.

text

when optimizeDictionary

Disables dictionary.

FST

Requires dictionary.

H3 (or geospatial)

Incompatible with dictionary.

Configured in tableConfig fieldConfigList
{
  "fieldConfigList": [
    {
      "name": "col1",
      "indexes": {
        "dictionary": {
          "disabled": true
        }
      }
    },
    ...
  ],
...
}
{
  "fieldConfigList": [
    {
      "name": "col1",
      "encodingType": "RAW"
    },
    ...
  ],
...
}

optimizeDictionary

false

Enables the heuristic for all columns and activates some extra rules.

optimizeDictionaryForMetrics

false

Enables the heuristic for metric columns.

noDictionarySizeRatioThreshold

0.85

The saving ratio used in the heuristics.

onHeap

false

Specifies whether the index should be loaded on heap or off heap.

useVarLengthDictionary

false

Determines how to store variable-length values.

intern

empty object

Configuration for interning. Only for on-heap dictionaries. Read about that below.

intern.capacity

null

how many values should be interning

{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name": "strColumn",
      "encodingType": "DICTIONARY",
      "indexes": {
        "dictionary": {
          "onHeap": true,
          "intern": {
            "capacity":32000
          }
        }
      }
    }
  ]
}
text index
JSON index
d1b4586
63a4fd4
a7f7f46
dafbef1
ced3a70
d902c1a
#PR4952
#PR5266
#PR5293
PR#5440
PR#5461
PR#5465
PR#5487
PR#5492
PR#5501
PR#5502
PR#5597
PR#5681
#PR5641
#PR5602
#PR5564
PR#5617
PR#5631
PR#5637
PR#5654
PR#5684
PR#5732
PR#5687
PR#5701
PR#5708
#PR5729
#PR5737
#PR5748
PR#5542
#5359
PR#5681
#PR5546
PR#5539
#5494
#5549
#5557
#5611
#5619
#5639
#5676
#5789
#PR5570
#PR5613

Segment

Discover the segment component in Apache Pinot for efficient data storage and querying within Pinot clusters, enabling optimized data processing and analysis.

Pinot tables are stored in one or more independent shards called segments. A small table may be contained by a single segment, but Pinot lets tables grow to an unlimited number of segments. There are different processes for creating segments (see ingestion). Segments have time-based partitions of table data, and are stored on Pinot servers that scale horizontally as needed for both storage and computation.

Pinot achieves this by breaking the data into smaller chunks known as segments (similar to shards/partitions in relational databases). Segments can be seen as time-based partitions.

A segment is a horizontal shard representing a chunk of table data with some number of rows. The segment stores data for all columns of the table. Each segment packs the data in a columnar fashion, along with the dictionaries and indices for the columns. The segment is laid out in a columnar format so that it can be directly mapped into memory for serving queries.

Columns can be single or multi-valued and the following types are supported: STRING, BOOLEAN, INT, LONG, FLOAT, DOUBLE, TIMESTAMP or BYTES. Only single-valued BIG_DECIMAL data type is supported.

Columns may be declared to be metric or dimension (or specifically as a time dimension) in the schema. Columns can have default null values. For example, the default null value of a integer column can be 0. The default value for bytes columns must be hex-encoded before it's added to the schema.

Pinot uses dictionary encoding to store values as a dictionary ID. Columns may be configured to be “no-dictionary” column in which case raw values are stored. Dictionary IDs are encoded using minimum number of bits for efficient storage (e.g. a column with a cardinality of 3 will use only 2 bits for each dictionary ID).

A forward index is built for each column and compressed for efficient memory use. In addition, you can optionally configure inverted indices for any set of columns. Inverted indices take up more storage, but improve query performance. Specialized indexes like Star-Tree index are also supported. For more details, see Indexing.

Creating a segment

Once the table is configured, we can load some data. Loading data involves generating pinot segments from raw data and pushing them to the pinot cluster. Data can be loaded in batch mode or streaming mode. For more details, see the ingestion overview page.

Load data in batch

Prerequisites

  1. Set up a cluster

  2. Create broker and server tenants

  3. Create an offline table

Below are instructions to generate and push segments to Pinot via standalone scripts. For a production setup, you should use frameworks such as Hadoop or Spark. For more details on setting up data ingestion jobs, see Import Data.

Job Spec YAML

To generate a segment, we need to first create a job spec YAML file. This file contains all the information regarding data format, input data location, and pinot cluster coordinates. Note that this assumes that the controller is RUNNING to fetch the table config and schema. If not, you will have to configure the spec to point at their location. For full configurations, see Ingestion Job Spec.

job-spec.yml
executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'

jobType: SegmentCreationAndTarPush
inputDirURI: 'examples/batch/baseballStats/rawdata'
includeFileNamePattern: 'glob:**/*.csv'
excludeFileNamePattern: 'glob:**/*.tmp'
outputDirURI: 'examples/batch/baseballStats/segments'
overwriteOutput: true

pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS

recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
  configs:

tableSpec:
  tableName: 'baseballStats'
  schemaURI: 'http://localhost:9000/tables/baseballStats/schema'
  tableConfigURI: 'http://localhost:9000/tables/baseballStats'
  
segmentNameGeneratorSpec:

pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'

pushJobSpec:
  pushParallelism: 2
  pushAttempts: 2
  pushRetryIntervalMillis: 1000

Create and push segment

To create and push the segment in one go, use the following:

docker run \
    --network=pinot-demo \
    --name pinot-data-ingestion-job \
    ${PINOT_IMAGE} LaunchDataIngestionJob \
    -jobSpecFile examples/docker/ingestion-job-specs/airlineStats.yaml

Sample Console Output

SegmentGenerationJobSpec:
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**/*.avro
inputDirURI: examples/batch/airlineStats/rawdata
jobType: SegmentCreationAndTarPush
outputDirURI: examples/batch/airlineStats/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://pinot-controller:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: {pushAttempts: 2, pushParallelism: 1, pushRetryIntervalMillis: 1000,
  segmentUriPrefix: null, segmentUriSuffix: null}
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.avro.AvroRecordReader,
  configClassName: null, configs: null, dataFormat: avro}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://pinot-controller:9000/tables/airlineStats/schema',
  tableConfigURI: 'http://pinot-controller:9000/tables/airlineStats', tableName: airlineStats}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 403 documents
Created dictionary for INT column: FlightNum with cardinality: 386, range: 14 to 7389
Using fixed bytes value dictionary for column: Origin, size: 294
Created dictionary for STRING column: Origin with cardinality: 98, max length in bytes: 3, range: ABQ to VPS
Created dictionary for INT column: Quarter with cardinality: 1, range: 1 to 1
Created dictionary for INT column: LateAircraftDelay with cardinality: 50, range: -2147483648 to 303
......
......
Pushing segment: airlineStats_OFFLINE_16085_16085_29 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16085_16085_29 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16085_16085_29 of table: airlineStats"}
Pushing segment: airlineStats_OFFLINE_16084_16084_30 to location: http://pinot-controller:9000 for table airlineStats
Sending request: http://pinot-controller:9000/v2/segments?tableName=airlineStats to controller: a413b0013806, version: Unknown
Response for pushing table airlineStats segment airlineStats_OFFLINE_16084_16084_30 to location http://pinot-controller:9000 - 200: {"status":"Successfully uploaded segment: airlineStats_OFFLINE_16084_16084_30 of table: airlineStats"}
bin/pinot-admin.sh LaunchDataIngestionJob \
    -jobSpecFile examples/batch/airlineStats/ingestionJobSpec.yaml

Alternately, you can separately create and then push, by changing the jobType to SegmentCreation or SegmenTarPush.

Templating Ingestion Job Spec

The Ingestion job spec supports templating with Groovy Syntax.

This is convenient if you want to generate one ingestion job template file and schedule it on a daily basis with extra parameters updated daily.

e.g. you could set inputDirURI with parameters to indicate the date, so that the ingestion job only processes the data for a particular date. Below is an example that templates the date for input and output directories.

inputDirURI: 'examples/batch/airlineStats/rawdata/${year}/${month}/${day}'
outputDirURI: 'examples/batch/airlineStats/segments/${year}/${month}/${day}'

You can pass in arguments containing values for ${year}, ${month}, ${day} when kicking off the ingestion job: -values $param=value1 $param2=value2...

docker run \
    --network=pinot-demo \
    --name pinot-data-ingestion-job \
    ${PINOT_IMAGE} LaunchDataIngestionJob \
    -jobSpecFile examples/docker/ingestion-job-specs/airlineStats.yaml
    -values year=2014 month=01 day=03

This ingestion job only generates segments for date 2014-01-03

Load data in streaming

Prerequisites

  1. Set up a cluster

  2. Create broker and server tenants

  3. Create a real-time table and set up a real-time stream

Below is an example of how to publish sample data to your stream. As soon as data is available to the real-time stream, it starts getting consumed by the real-time servers.

Kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

docker run \
  --network pinot-demo \
  --name=loading-airlineStats-data-to-kafka \
  ${PINOT_IMAGE} StreamAvroIntoKafka \
  -avroFile examples/stream/airlineStats/sample_data/airlineStats_data.avro \
  -kafkaTopic flights-realtime -kafkaBrokerList kafka:9092 -zkAddress pinot-zookeeper:2181/kafka

Run below command to stream JSON data into Kafka topic: flights-realtime

bin/pinot-admin.sh StreamAvroIntoKafka \
  -avroFile examples/stream/airlineStats/sample_data/airlineStats_data.avro \
  -kafkaTopic flights-realtime -kafkaBrokerList localhost:19092 -zkAddress localhost:2191/kafka

Ingestion FAQ

This page has a collection of frequently asked questions about ingestion with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, make a pull request.

Data processing

What is a good segment size?

While Apache Pinot can work with segments of various sizes, for optimal use of Pinot, you want to get your segments sized in the 100MB to 500MB (un-tarred/uncompressed) range. Having too many (thousands or more) tiny segments for a single table creates overhead in terms of the metadata storage in Zookeeper as well as in the Pinot servers' heap. At the same time, having too few really large (GBs) segments reduces parallelism of query execution, as on the server side, the thread parallelism of query execution is at segment level.

Can multiple Pinot tables consume from the same Kafka topic?

Yes. Each table can be independently configured to consume from any given Kafka topic, regardless of whether there are other tables that are also consuming from the same Kafka topic.

If I add a partition to a Kafka topic, will Pinot automatically ingest data from this partition?

Pinot automatically detects new partitions in Kafka topics. It checks for new partitions whenever RealtimeSegmentValidationManager periodic job runs and starts consumers for new partitions.

You can configure the interval for this job using thecontroller.realtime.segment.validation.frequencyPeriod property in the controller configuration.

Does Pinot support partition pruning on multiple partition columns?

Pinot supports multi-column partitioning for offline tables. Map multiple columns under tableIndexConfig.segmentPartitionConfig.columnPartitionMap. Pinot assigns the input data to each partition according to the partition configuration individually for each column.

The following example partitions the segment based on two columns, memberID and caseNumber. Note that each partition column is handled separately, so in this case the segment is partitioned on memberID (partition ID 1) and also partiitoned on caseNumber (partition ID 2).

"tableIndexConfig": {
      ..
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "memberId": {
            "functionName": "Modulo",
            "numPartitions": 3 
          },
          "caseNumber": {
            "functionName": "Murmur",
            "numPartitions": 12 
          }
        }
      }

For multi-column partitioning to work, you must also set routing.segementPrunerTypes as follows:

"routing": {
      "segmentPrunerTypes": ["partition"]
    }

How do I enable partitioning in Pinot when using Kafka stream?

Set up partitioner in the Kafka producer: https://docs.confluent.io/current/clients/producer.html

The partitioning logic in the stream should match the partitioning config in Pinot. Kafka uses murmur2, and the equivalent in Pinot is the Murmur function.

Set the partitioning configuration as below using same column used in Kafka:

"tableIndexConfig": {
      ..
      "segmentPartitionConfig": {
        "columnPartitionMap": {
          "column_foo": {
            "functionName": "Murmur",
            "numPartitions": 12 // same as number of kafka partitions
          }
        }
      }

and also set:

"routing": {
      "segmentPrunerTypes": ["partition"]
    }

To learn how partition works, see routing tuning.

How do I store BYTES column in JSON data?

For JSON, you can use a hex encoded string to ingest BYTES.

How do I flatten my JSON Kafka stream?

See the json_format(field) function which can store a top level json field as a STRING in Pinot.

Then you can use these json functions during query time, to extract fields from the json string.

NOTE This works well if some of your fields are nested json, but most of your fields are top level json keys. If all of your fields are within a nested JSON key, you will have to store the entire payload as 1 column, which is not ideal.

How do I escape Unicode in my Job Spec YAML file?

To use explicit code points, you must double-quote (not single-quote) the string, and escape the code point via "\uHHHH", where HHHH is the four digit hex code for the character. See https://yaml.org/spec/spec.html#escaping/in%20double-quoted%20scalars/ for more details.

Is there a limit on the maximum length of a string column in Pinot?

By default, Pinot limits the length of a String column to 512 bytes. If you want to overwrite this value, you can set the maxLength attribute in the schema as follows:

    {
      "dataType": "STRING",
      "maxLength": 1000,
      "name": "textDim1"
    },

When are new events queryable when getting ingested into a real-time table?

Events are available to queries as soon as they are ingested. This is because events are instantly indexed in memory upon ingestion.

The ingestion of events into the real-time table is not transactional, so replicas of the open segment are not immediately consistent. Pinot trades consistency for availability upon network partitioning (CAP theorem) to provide ultra-low ingestion latencies at high throughput.

However, when the open segment is closed and its in-memory indexes are flushed to persistent storage, all its replicas are guaranteed to be consistent, with the commit protocol.

How to reset a CONSUMING segment stuck on an offset which has expired from the stream?

This typically happens if:

  1. The consumer is lagging a lot.

  2. The consumer was down (server down, cluster down), and the stream moved on, resulting in offset not found when consumer comes back up.

In case of Kafka, to recover, set property "auto.offset.reset":"earliest" in the streamConfigs section and reset the CONSUMING segment. See Real-time table configs for more details about the configuration.

You can also also use the "Resume Consumption" endpoint with "resumeFrom" parameter set to "smallest" (or "largest" if you want). See Pause Stream Ingestion for more details.

Indexing

How to set inverted indexes?

Inverted indexes are set in the tableConfig's tableIndexConfig -> invertedIndexColumns list. For more info on table configuration, see Table Config Reference. For an example showing how to configure an inverted index, see Inverted Index.

Applying inverted indexes to a table configuration will generate an inverted index for all new segments. To apply the inverted indexes to all existing segments, see How to apply an inverted index to existing segments?

How to apply an inverted index to existing segments?

  1. Add the columns you want to index to the tableIndexConfig-> invertedIndexColumns list. To update the table configuration use the Pinot Swagger API: http://localhost:9000/help#!/Table/updateTableConfig.

  2. Invoke the reload API: http://localhost:9000/help#!/Segment/reloadAllSegments.

Once you've done that, you can check whether the index has been applied by querying the segment metadata API at http://localhost:9000/help#/Segment/getServerMetadata. Don't forget to include the names of the column on which you have applied the index.

The output from this API should look something like the following:

{
  "<segment-name>": {
    "segmentName": "<segment-name>",
    "indexes": {
      "<columnName>": {
        "bloom-filter": "NO",
        "dictionary": "YES",
        "forward-index": "YES",
        "inverted-index": "YES",
        "null-value-vector-reader": "NO",
        "range-index": "NO",
        "json-index": "NO"
      }
    }
  }
}

Can I retrospectively add an index to any segment?

Not all indexes can be retrospectively applied to existing segments.

If you want to add or change the sorted index column or adjust the dictionary encoding of the default forward index you will need to manually re-load any existing segments.

How to create star-tree indexes?

Star-tree indexes are configured in the table config under the tableIndexConfig -> starTreeIndexConfigs (list) and enableDefaultStarTree (boolean). See here for more about how to configure star-tree indexes: https://docs.pinot.apache.org/basics/indexing/star-tree-index#index-generation

The new segments will have star-tree indexes generated after applying the star-tree index configurations to the table configuration.

Handling time in Pinot

How does Pinot’s real-time ingestion handle out-of-order events?

Pinot does not require ordering of event time stamps. Out of order events are still consumed and indexed into the "currently consuming" segment. In a pathological case, if you have a 2 day old event come in "now", it will still be stored in the segment that is open for consumption "now". There is no strict time-based partitioning for segments, but star-indexes and hybrid tables will handle this as appropriate.

See the Components > Broker for more details about how hybrid tables handle this. Specifically, the time-boundary is computed as max(OfflineTIme) - 1 unit of granularity. Pinot does store the min-max time for each segment and uses it for pruning segments, so segments with multiple time intervals may not be perfectly pruned.

When generating star-indexes, the time column will be part of the star-tree so the tree can still be efficiently queried for segments with multiple time intervals.

What is the purpose of a hybrid table not using max(OfflineTime) to determine the time-boundary, and instead using an offset?

This lets you have an old event up come in without building complex offline pipelines that perfectly partition your events by event timestamps. With this offset, even if your offline data pipeline produces segments with a maximum timestamp, Pinot will not use the offline dataset for that last chunk of segments. The expectation is if you process offline the next time-range of data, your data pipeline will include any late events.

Why are segments not strictly time-partitioned?

It might seem odd that segments are not strictly time-partitioned, unlike similar systems such as Apache Druid. This allows real-time ingestion to consume out-of-order events. Even though segments are not strictly time-partitioned, Pinot will still index, prune, and query segments intelligently by time intervals for the performance of hybrid tables and time-filtered data.

When generating offline segments, the segments generated such that segments only contain one time interval and are well partitioned by the time column.

Geospatial

This page talks about geospatial support in Pinot.

Pinot supports SQL/MM geospatial data and is compliant with the Open Geospatial Consortium’s (OGC) OpenGIS Specifications. This includes:

  • Geospatial data types, such as point, line and polygon;

  • Geospatial functions, for querying of spatial properties and relationships.

  • Geospatial indexing, used for efficient processing of spatial operations

Geospatial data types

Geospatial data types abstract and encapsulate spatial structures such as boundary and dimension. In many respects, spatial data types can be understood simply as shapes. Pinot supports the Well-Known Text (WKT) and Well-Known Binary (WKB) forms of geospatial objects, for example:

  • POINT (0, 0)

  • LINESTRING (0 0, 1 1, 2 1, 2 2)

  • POLYGON (0 0, 10 0, 10 10, 0 10, 0 0),(1 1, 1 2, 2 2, 2 1, 1 1)

  • MULTIPOINT (0 0, 1 2)

  • MULTILINESTRING ((0 0, 1 1, 1 2), (2 3, 3 2, 5 4))

  • MULTIPOLYGON (((0 0, 4 0, 4 4, 0 4, 0 0), (1 1, 2 1, 2 2, 1 2, 1 1)), ((-1 -1, -1 -2, -2 -2, -2 -1, -1 -1)))

  • GEOMETRYCOLLECTION(POINT(2 0),POLYGON((0 0, 1 0, 1 1, 0 1, 0 0)))

Geometry vs geography

It is common to have data in which the coordinates are geographics or latitude/longitude. Unlike coordinates in Mercator or UTM, geographic coordinates are not Cartesian coordinates.

  • Geographic coordinates do not represent a linear distance from an origin as plotted on a plane. Rather, these spherical coordinates describe angular coordinates on a globe.

  • Spherical coordinates specify a point by the angle of rotation from a reference meridian (longitude), and the angle from the equator (latitude).

You can treat geographic coordinates as approximate Cartesian coordinates and continue to do spatial calculations. However, measurements of distance, length and area will be nonsensical. Since spherical coordinates measure angular distance, the units are in degrees.

Pinot supports both geometry and geography types, which can be constructed by the corresponding functions as shown in section. And for the geography types, the measurement functions such as ST_Distance and ST_Area calculate the spherical distance and area on earth respectively.

Geospatial functions

For manipulating geospatial data, Pinot provides a set of functions for analyzing geometric components, determining spatial relationships, and manipulating geometries. In particular, geospatial functions that begin with the ST_ prefix support the SQL/MM specification.

Following geospatial functions are available out of the box in Pinot:

Aggregations

ST_Union(geometry[] g1_array) → Geometry This aggregate function returns a MULTI geometry or NON-MULTI geometry from a set of geometries. it ignores NULL geometries.

Constructors

  • ST_GeomFromText(String wkt) → Geometry Returns a geometry type object from WKT representation, with the optional spatial system reference.

  • ST_GeomFromWKB(bytes wkb) → Geometry Returns a geometry type object from WKB representation.

  • ST_Point(double x, double y) → Point Returns a geometry type point object with the given coordinate values.

  • ST_Polygon(String wkt) → Polygon Returns a geometry type polygon object from WKT representation.

  • ST_GeogFromWKB(bytes wkb) → Geography Creates a geography instance from a Well-Known Binary geometry representation (WKB)

  • ST_GeogFromText(String wkt) → Geography Returns a specified geography value from Well-Known Text representation or extended (WKT).

Measurements

  • ST_Area(Geometry/Geography g) → double For geometry type, it returns the 2D Euclidean area of a geometry. For geography, returns the area of a polygon or multi-polygon in square meters using a spherical model for Earth.

  • ST_Distance(Geometry/Geography g1, Geometry/Geography g2) → double For geometry type, returns the 2-dimensional cartesian minimum distance (based on spatial ref) between two geometries in projected units. For geography, returns the great-circle distance in meters between two SphericalGeography points. Note that g1, g2 shall have the same type.

  • ST_GeometryType(Geometry g) → String Returns the type of the geometry as a string. e.g.: ST_Linestring, ST_Polygon,ST_MultiPolygon etc.

Outputs

  • ST_AsBinary(Geometry/Geography g) → bytes Returns the WKB representation of the geometry.

  • ST_AsText(Geometry/Geography g) → string Returns the WKT representation of the geometry/geography.

Conversion

  • toSphericalGeography(Geometry g) → Geography Converts a Geometry object to a spherical geography object.

  • toGeometry(Geography g) → Geometry Converts a spherical geographical object to a Geometry object.

Relationship

  • ST_Contains(Geometry/Geography, Geometry/Geography) → boolean Returns true if and only if no points of the second geometry/geography lie in the exterior of the first geometry/geography, and at least one point of the interior of the first geometry lies in the interior of the second geometry. Warning: ST_Contains on Geography only give close approximation

  • ST_Equals(Geometry, Geometry) → boolean Returns true if the given geometries represent the same geometry/geography.

  • ST_Within(Geometry, Geometry) → boolean Returns true if first geometry is completely inside second geometry.

Geospatial index

Geospatial functions are typically expensive to evaluate, and using geoindex can greatly accelerate the query evaluation. Geoindexing in Pinot is based on Uber’s H3, a hexagon-based hierarchical gridding.

A given geospatial location (longitude, latitude) can map to one hexagon (represented as H3Index). And its neighbors in H3 can be approximated by a ring of hexagons. To quickly identify the distance between any given two geospatial locations, we can convert the two locations in the H3Index, and then check the H3 distance between them. H3 distance is measured as the number of hexagons.

For example, in the diagram below, the red hexagons are within the 1 distance of the central hexagon. The size of the hexagon is determined by the resolution of the indexing. Check this table for the level of resolutions and the corresponding precision (measured in km).

Hexagonal grid in H3

How to use geoindex

To use the geoindex, first declare the geolocation field as bytes in the schema, as in the example of the QuickStart example.

geoindex schema
{
  "dataType": "BYTES",
  "name": "location_st_point",
  "transformFunction": "toSphericalGeography(stPoint(lon,lat))"
}

Note the use of transformFunction that converts the created point into SphericalGeography format, which is needed by the ST_Distance function.

Next, declare the geospatial index in the table configuration you need to

  • Verify the dictionary is disabled (see how to disable the dictionary index).

  • Enable the H3 index.

It is recommended to do the latter by using the indexes section:

geoindex tableConfig
{
  "fieldConfigList": [
    {
      "name": "location_st_point",
      "encodingType":"RAW", // this actually disables the dictionary
      "indexes": {
        "h3": {
          "resolutions": [13, 5, 6]
        }
      }
    }
  ],
  ...
}

Alternative the older way to configure H3 indexes is still supported:

geoindex tableConfig
{
  "fieldConfigList": [{
    "name": "location_st_point",
    "encodingType":"RAW", // this actually disables the dictionary
    "indexTypes":["H3"],
    "properties": {
      "resolutions": "13, 5, 6" // Here resolutions must be a string with ints separated by commas 
    }
  }],
  ...
}

The query below will use the geoindex to filter the Starbucks stores within 5km of the given point in the bay area.

SELECT address, ST_DISTANCE(location_st_point, ST_Point(-122, 37, 1))
FROM starbucksStores
WHERE ST_DISTANCE(location_st_point, ST_Point(-122, 37, 1)) < 5000
limit 1000

How geoindex works

The Pinot geoindex accelerates query evaluation while maintaining accuracy. Currently, geoindex supports the ST_Distance function in the WHERE clause.

At the high level, geoindex is used for retrieving the records within the nearby hexagons of the given location, and then use ST_Distance to accurately filter the matched results.

Geoindex example

As in the example diagram above, if we want to find all relevant points within a given distance around San Francisco (area within the red circle), then the algorithm with geoindex will:

  • First find the H3 distance x that contains the range (for example, within a red circle).

  • Then, for the points within the H3 distance (those covered by the hexagons completely within kRing(x)), directly accept those points without filtering.

  • Finally, for the points contained in the hexagons of kRing(x) at the outer edge of the red circle H3 distance, the algorithm will filter them by evaluating the condition ST_Distance(loc1, loc2) < x to find only those that are within the circle.

Quick Start Examples

This section describes quick start commands that launch all Pinot components in a single process.

Pinot ships with QuickStart commands that launch Pinot components in a single process and import pre-built datasets. These quick start examples are a good place if you're just getting started with Pinot. The examples begin with the example, after the following notes:

  • Prerequisites

    You must have either or . The examples are available in each option and work the same. The decision of which to choose depends on your installation preference and how you generally like to work. If you don't know which to choose, using Docker will make your cleanup easier after you are done with the examples.

  • Pinot versions in examples

    The Docker-based examples on this page use pinot:latest, which instructs Docker to pull and use the most recent release of Apache Pinot. If you prefer to use a specific release instead, you can designate it by replacing latest with the release number, like this: pinot:0.12.1.

    The local install-based examples that are run using the launcher scripts will use the Apache Pinot version you installed.

  • Stopping a running example

    To stop a running example, enter Ctrl+C in the same terminal where you ran the docker run command to start the example.

macOS Monterey Users

By default the Airplay receiver server runs on port 7000, which is also the port used by the Pinot Server in the Quick Start. You may see the following error when running these examples:

If you disable the Airplay receiver server and try again, you shouldn't see this error message anymore.

Batch Processing

This example demonstrates how to do batch processing with Pinot. The command:

  • Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates the baseballStats table

  • Launches a standalone data ingestion job that builds one segment for a given CSV data file for the baseballStats table and pushes the segment to the Pinot Controller.

  • Issues sample queries to Pinot

Batch JSON

This example demonstrates how to import and query JSON documents in Pinot. The command:

  • Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates the githubEvents table

  • Launches a standalone data ingestion job that builds one segment for a given JSON data file for the githubEvents table and pushes the segment to the Pinot Controller.

  • Issues sample queries to Pinot

Batch with complex data types

This example demonstrates how to do batch processing in Pinot where the the data items have complex fields that need to be unnested. The command:

  • Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates the githubEvents table

  • Launches a standalone data ingestion job that builds one segment for a given JSON data file for the githubEvents table and pushes the segment to the Pinot Controller.

  • Issues sample queries to Pinot

Streaming

This example demonstrates how to do stream processing with Pinot. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates meetupRsvp table

  • Launches a meetup stream

  • Publishes data to a Kafka topic meetupRSVPEvents that is subscribed to by Pinot.

  • Issues sample queries to Pinot

Streaming JSON

This example demonstrates how to do stream processing with JSON documents in Pinot. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates meetupRsvp table

  • Launches a meetup stream

  • Publishes data to a Kafka topic meetupRSVPEvents that is subscribed to by Pinot

  • Issues sample queries to Pinot

Streaming with minion cleanup

This example demonstrates how to do stream processing in Pinot with RealtimeToOfflineSegmentsTask and MergeRollupTask minion tasks continuously optimizing segments as data gets ingested. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, Pinot Minion, and Pinot Server.

  • Creates githubEvents table

  • Launches a GitHub events stream

  • Publishes data to a Kafka topic githubEvents that is subscribed to by Pinot.

  • Issues sample queries to Pinot

Streaming with complex data types

This example demonstrates how to do stream processing in Pinot where the stream contains items that have complex fields that need to be unnested. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, Pinot Minion, and Pinot Server.

  • Creates meetupRsvp table

  • Launches a meetup stream

  • Publishes data to a Kafka topic meetupRSVPEvents that is subscribed to by Pinot.

  • Issues sample queries to Pinot

Upsert

This example demonstrates how to do with Pinot. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates meetupRsvp table

  • Launches a meetup stream

  • Publishes data to a Kafka topic meetupRSVPEvents that is subscribed to by Pinot

  • Issues sample queries to Pinot

Upsert JSON

This example demonstrates how to do with JSON documents in Pinot. The command:

  • Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  • Creates meetupRsvp table

  • Launches a meetup stream

  • Publishes data to a Kafka topic meetupRSVPEvents that is subscribed to by Pinot

  • Issues sample queries to Pinot

Hybrid

This example demonstrates how to do hybrid stream and batch processing with Pinot. The command:

  1. Starts Apache Kafka, Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server.

  2. Creates airlineStats table

  3. Launches a standalone data ingestion job that builds segments under a given directory of Avro files for the airlineStats table and pushes the segments to the Pinot Controller.

  4. Launches a stream of flights stats

  5. Publishes data to a Kafka topic airlineStatsEvents that is subscribed to by Pinot.

  6. Issues sample queries to Pinot

Join

This example demonstrates how to do joins in Pinot using the . The command:

  • Starts Apache Zookeeper, Pinot Controller, Pinot Broker, and Pinot Server in the same container.

  • Creates the baseballStats table

  • Launches a data ingestion job that builds one segment for a given CSV data file for the baseballStats table and pushes the segment to the Pinot Controller.

  • Creates the dimBaseballTeams table

  • Launches a data ingestion job that builds one segment for a given CSV data file for the dimBaseballStats table and pushes the segment to the Pinot Controller.

  • Issues sample queries to Pinot

Failed to start a Pinot [SERVER]
java.lang.RuntimeException: java.net.BindException: Address already in use
	at org.apache.pinot.core.transport.QueryServer.start(QueryServer.java:103) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.pinot.server.starter.ServerInstance.start(ServerInstance.java:158) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da21132906]
	at org.apache.helix.manager.zk.ParticipantManager.handleNewSession(ParticipantManager.java:110) ~[pinot-all-0.9.0-jar-with-dependencies.jar:0.9.0-cf8b84e8b0d6ab62374048de586ce7da2113
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type batch
./bin/pinot-admin.sh QuickStart -type batch
pinot-admin QuickStart -type batch
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type batch_json_index
./bin/pinot-admin.sh QuickStart -type batch_json_index
pinot-admin QuickStart -type batch_json_index
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type batch_complex_type
./bin/pinot-admin.sh QuickStart -type batch_complex_type
pinot-admin QuickStart -type batch_complex_type
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type stream
./bin/pinot-admin.sh QuickStart -type stream
pinot-admin QuickStart -type stream
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type stream_json_index
./bin/pinot-admin.sh QuickStart -type stream_json_index
pinot-admin QuickStart -type stream_json_index
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type realtime_minion
./bin/pinot-admin.sh QuickStart -type realtime_minion
pinot-admin QuickStart -type realtime_minion
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type stream_complex_type
./bin/pinot-admin.sh QuickStart -type stream_complex_type
pinot-admin QuickStart -type stream_complex_type
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type upsert
./bin/pinot-admin.sh QuickStart -type upsert
pinot-admin QuickStart -type upsert
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type upsert_json_index
./bin/pinot-admin.sh QuickStart -type upsert_json_index
pinot-admin QuickStart -type upsert_json_index
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type hybrid
./bin/pinot-admin.sh QuickStart -type hybrid
pinot-admin QuickStart -type hybrid
docker run \
    -p 9000:9000 \
    apachepinot/pinot:latest QuickStart \
    -type join
./bin/pinot-admin.sh QuickStart -type join
pinot-admin QuickStart -type join
Batch Processing
installed Pinot locally
have Docker installed if you want to use the Pinot Docker image
stream processing with upsert
stream processing with upsert
Lookup UDF

Running Pinot in Docker

This guide will show you to run a Pinot cluster using Docker.

Get started setting up a Pinot cluster with Docker using the guide below.

Prerequisites:

  • Install Docker

  • Configure Docker memory with the following minimum resources:

    • CPUs: 8

    • Memory: 16.00 GB

    • Swap: 4 GB

    • Disk Image size: 60 GB

The latest Pinot Docker image is published at apachepinot/pinot:latest. View a list of all published tags on Docker Hub.

Pull the latest Docker image onto your machine by running the following command:

docker pull apachepinot/pinot:latest

To pull a specific version, modify the command like below:

docker pull apachepinot/pinot:1.2.0

Set up a cluster

Once you've downloaded the Pinot Docker image, it's time to set up a cluster. There are two ways to do this.

Quick start

Pinot comes with quick start commands that launch instances of Pinot components in the same process and import pre-built datasets.

For example, the following quick start command launches Pinot with a baseball dataset pre-loaded:

docker run \
    -p 2123:2123 \
    -p 9000:9000 \
    -p 8000:8000 \
    -p 7050:7050 \
    -p 6000:6000 \
    apachepinot/pinot:1.2.0 QuickStart \
    -type batch

For a list of all available quick start commands, see Quick Start Examples.

Below are the usages of different ports:

2123: Zookeeper Port

9000: Pinot Controller Port

8000: Pinot Broker Port

7050: Pinot Server Port

6000: Pinot Minion Port

Manual cluster

The quick start scripts launch Pinot with minimal resources. If you want to play with bigger datasets (more than a few MB), you can launch each of the Pinot components individually.

Note that these are sample configurations to be used as references. You will likely want to customize them to meet your needs for production use.

Docker

Create a Network

Create an isolated bridge network in docker

docker network create -d bridge pinot-demo

Export Docker Image tags

Export the necessary docker image tags for Pinot, Zookeeper, and Kafka.

export PINOT_IMAGE=apachepinot/pinot:1.2.0
export ZK_IMAGE=zookeeper:3.9.2
export KAFKA_IMAGE= bitnami/kafka:3.6

Start Zookeeper

Start Zookeeper in daemon mode. This is a single node zookeeper setup. Zookeeper is the central metadata store for Pinot and should be set up with replication for production use. For more information, see Running Replicated Zookeeper.

docker run \
    --network=pinot-demo \
    --name pinot-zookeeper \
    --restart always \
    -p 2181:2181 \
    -d ${ZK_IMAGE}

Start Pinot Controller

Start Pinot Controller in daemon and connect to Zookeeper.

The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.

docker run --rm -ti \
    --network=pinot-demo \
    --name pinot-controller \
    -p 9000:9000 \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log" \
    -d ${PINOT_IMAGE} StartController \
    -zkAddress pinot-zookeeper:2181

Start Pinot Broker

Start Pinot Broker in daemon and connect to Zookeeper.

The command below expects a 4GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.

docker run --rm -ti \
    --network=pinot-demo \
    --name pinot-broker \
    -p 8099:8099 \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log" \
    -d ${PINOT_IMAGE} StartBroker \
    -zkAddress pinot-zookeeper:2181

Start Pinot Server

Start Pinot Server in daemon and connect to Zookeeper.

The command below expects a 16GB memory container. Tune-Xms and-Xmx if your machine doesn't have enough resources.

docker run --rm -ti \
    --network=pinot-demo \
    --name pinot-server \
    -p 8098:8098 \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
    -d ${PINOT_IMAGE} StartServer \
    -zkAddress pinot-zookeeper:2181

Start Kafka

Optionally, you can also start Kafka for setting up real-time streams. This brings up the Kafka broker on port 9092.

docker run --rm -ti \
    --network pinot-demo --name=kafka \
    -e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
    -e KAFKA_BROKER_ID=0 \
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
    -p 9092:9092 \
    -d ${KAFKA_IMAGE}

Now all Pinot related components are started as an empty cluster.

Run the below command to check container status:

docker container ls -a

Sample Console Output

CONTAINER ID   IMAGE                     COMMAND                  CREATED              STATUS              PORTS                                                       NAMES
accc70bc7f07   bitnami/kafka:3.6         "/opt/bitnami/script…"   About a minute ago   Up About a minute   0.0.0.0:9092->9092/tcp                                      kafka
1b8b80395959   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp   pinot-server
134a67eec957   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp             pinot-broker
4fcc72cb7302   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   About a minute ago   Up About a minute   8096-8099/tcp, 0.0.0.0:9000->9000/tcp                       pinot-controller
144304524f6c   zookeeper:3.9.2           "/docker-entrypoint.…"   About a minute ago   Up About a minute   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp        pinot-zookeeper

Docker Compose

Export Docker Image tags

Optionally, export the necessary docker image tags for Pinot, Zookeeper, and Kafka.

export PINOT_IMAGE=apachepinot/pinot:1.2.0
export ZK_IMAGE=zookeeper:3.9.2
export KAFKA_IMAGE=bitnami/kafka:3.6

Create docker-compose.yml file

Create a file called docker-compose.yml that contains the following:

docker-compose.yml
version: '3.7'

services:
  pinot-zookeeper:
    image: ${ZK_IMAGE:-zookeeper:3.9.2}
    container_name: "pinot-zookeeper"
    restart: unless-stopped
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - pinot-demo
    healthcheck:
      test: ["CMD", "zkServer.sh", "status"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s

  pinot-kafka:
    image: ${KAFKA_IMAGE:-bitnami/kafka:3.6}
    container_name: "kafka"
    restart: unless-stopped
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: pinot-zookeeper:2181/kafka
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_HOST_NAME: kafka
    depends_on:
      pinot-zookeeper:
        condition: service_healthy
    networks:
      - pinot-demo
    healthcheck:
      test: [ "CMD-SHELL", "kafka-broker-api-versions.sh -bootstrap-server kafka:9092" ]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s
    deploy:
      replicas: ${KAFKA_REPLICAS:-0}  # Default to 0, meaning Kafka won't start unless KAFKA_REPLICAS is set

  pinot-controller:
    image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
    command: "StartController -zkAddress pinot-zookeeper:2181"
    container_name: "pinot-controller"
    restart: unless-stopped
    ports:
      - "9000:9000"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms1G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log"
    depends_on:
      pinot-zookeeper:
        condition: service_healthy
    networks:
      - pinot-demo
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s

  pinot-broker:
    image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
    command: "StartBroker -zkAddress pinot-zookeeper:2181"
    container_name: "pinot-broker"
    restart: unless-stopped
    ports:
      - "8099:8099"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log"
    depends_on:
      pinot-controller:
        condition: service_healthy
    networks:
      - pinot-demo
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8099/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s

  pinot-server:
    image: ${PINOT_IMAGE:-apachepinot/pinot:1.2.0}
    command: "StartServer -zkAddress pinot-zookeeper:2181"
    container_name: "pinot-server"
    restart: unless-stopped
    ports:
      - "8098:8098"
    environment:
      JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log"
    depends_on:
      pinot-broker:
        condition: service_healthy
    networks:
      - pinot-demo
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8097/health/readiness || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 10s

networks:
  pinot-demo:
    name: pinot-demo
    driver: bridge

Launch the components

Run the following command to launch all the required components:

docker compose --project-name pinot-demo up

OR, optionally, run the following command to launch all the components, including kafka:

export KAFKA_REPLICAS=1
docker compose --project-name pinot-demo up

Run the below command to check the container status:

docker container ls -a

Sample Console Output

CONTAINER ID   IMAGE                     COMMAND                  CREATED          STATUS                        PORTS                                                       NAMES
f34a046ac69f   bitnami/kafka:3.6         "/opt/bitnami/script…"   9 minutes ago    Up About a minute (healthy)   0.0.0.0:9092->9092/tcp                                      kafka
f28021bd5b1d   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   18 minutes ago   Up About a minute (healthy)   8096-8097/tcp, 8099/tcp, 9000/tcp, 0.0.0.0:8098->8098/tcp   pinot-server
e938453054b0   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   18 minutes ago   Up About a minute (healthy)   8096-8098/tcp, 9000/tcp, 0.0.0.0:8099->8099/tcp             pinot-broker
e0d0c71303a8   apachepinot/pinot:1.2.0   "./bin/pinot-admin.s…"   18 minutes ago   Up About a minute (healthy)   8096-8099/tcp, 0.0.0.0:9000->9000/tcp                       pinot-controller
4be5f168f252   zookeeper:3.9.2           "/docker-entrypoint.…"   18 minutes ago   Up About a minute (healthy)   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp        pinot-zookeeper

Once your cluster is up and running, see Exploring Pinot to learn how to run queries against the data.

If you have minikube or Docker Kubernetes installed, you can also try running the Kubernetes quick start.

Neha Pawar from the Apache Pinot team shows you how to set up a Pinot cluster

Batch import example

Step-by-step guide for pushing your own data into the Pinot cluster

This example assumes you have set up your cluster using .

Preparing your data

Let's gather our data files and put them in pinot-quick-start/rawdata.

Supported file formats are CSV, JSON, AVRO, PARQUET, THRIFT, ORC. If you don't have sample data, you can use this sample CSV.

Creating a schema

Schema is used to define the columns and data types of the Pinot table. A detailed overview of the schema can be found in .

Columns are categorized into 3 types:

Column Type
Description

In our example transcript-schema, the studentID,firstName,lastName,gender,subject columns are the dimensions, the score column is the metric and timestampInEpoch is the time column.

Once you have identified the dimensions, metrics and time columns, create a schema for your data, using the following reference.

Creating a table configuration

A table configuration is used to define the configuration related to the Pinot table. A detailed overview of the table can be found in .

Here's the table configuration for the sample CSV file. You can use this as a reference to build your own table configuration. Edit the tableName and schemaName.

Uploading your table configuration and schema

Review the directory structure so far.

Upload the table configuration using the following command.

Use the that is running on your Pinot instance to review the table configuration and schema and make sure it was successfully uploaded. This link uses localhost as an example.

Creating a segment

Pinot table data is stored as Pinot segments. A detailed overview of segments can be found in .

  1. To generate a segment, first create a job specification (JobSpec) yaml file. A JobSpec yaml file contains all the information regarding data format, input data location, and pinot cluster coordinates. Copy the following job specification file (example from Pinot quickstart file). If you're using your own data, be sure to do the following:

    • Replace transcript with your table name

    • Set the correct recordReaderSpec

  2. Depending if you're using Docker or a launcher script, choose one of the following commands to generate a segment to upload to Pinot:

Here is some sample output.

Querying your data

If everything worked, find your table in the to run queries against it.

Complex Type Examples

Additional examples that demonstrate handling of complex types.

Additional examples that demonstrate handling of complex types.

Unnest Root Level Collection

In this example, we would look at un-nesting json records that are batched together as part of a single key at the root level. We will make use of the configs to persist the individual student records as separate rows in Pinot.

Sample JSON record

Pinot Schema

The Pinot schema for this example would look as follows.

Pinot Table Configuration

The Pinot table configuration for this schema would look as follows.

Data in Pinot

Post ingestion, the student records would appear as separate records in Pinot. Note that the nested field scores is captured as a JSON field.

Unnest sibling collections

In this example, we would look at un-nesting the sibling collections "student" and "teacher".

Sample JSON Record

Pinot Schema

Pinot Table configuration

Data in Pinot

Unnest nested collection

In this example, we would look at un-nesting the nested collection "students.grades".

Sample JSON Record

Pinot Schema

Pinot Table configuration

Data in Pinot

Unnest Multi Level Array

In this example, we would look at un-nesting the array "finalExam" which is located within the array "students".

Sample JSON Record

Pinot Schema

Pinot Table configuration

Data in Pinot

Convert inner collections

In this example, the inner collection "grades" is converted into a multi value string column.

Sample JSON Record

Pinot Schema

Pinot Table configuration

Data in Pinot

Primitive Array Converted to JSON String

In this example, the array of primitives "extra_curricular" is converted to a Json string.

Sample JSON Record

Pinot Schema

Pinot Table configuration

Data in Pinot

0.6.0

This release introduced some excellent new features, including upsert, tiered storage, pinot-spark-connector, support of having clause, more validations on table config and schema, support of ordinals

Summary

This release introduced some excellent new features, including upsert, tiered storage, pinot-spark-connector, support of having clause, more validations on table config and schema, support of ordinals in GROUP BY and ORDER BY clause, array transform functions, adding push job type of segment metadata only mode, and some new APIs like updating instance tags, new health check endpoint. It also contains many key bug fixes. See details below.

The release was cut from the following commit: e5c9bec and the following cherry-picks:

  • d033a11

Notable New Features

  • Tiered storage (#5793)

  • Upsert feature (#6096, #6113, #6141, #6149, #6167)

  • Pre-generate aggregation functions in QueryContext (#5805)

  • Adding controller healthcheck endpoint: /health (#5846)

  • Add pinot-spark-connector (#5787)

  • Support multi-value non-dictionary group by (#5851)

  • Support type conversion for all scalar functions (#5849)

  • Add additional datetime functionality (#5438)

  • Support post-aggregation in ORDER-BY (#5856)

  • Support post-aggregation in SELECT (#5867)

  • Add RANGE FilterKind to support merging ranges for SQL (#5898)

  • Add HAVING support (#58895889)

  • Support for exact distinct count for non int data types (#5872)

  • Add max qps bucket count (#5922)

  • Add Range Indexing support for raw values (#5853)

  • Add IdSet and IdSetAggregationFunction (#5926)

  • [Deepstore by-pass]Add a Deepstore bypass integration test with minor bug fixes. (#5857)

  • Add Hadoop counters for detecting schema mismatch (#5873)

  • Add RawThetaSketchAggregationFunction (#5970)

  • Instance API to directly updateTags (#5902)

  • Add streaming query handler (#5717)

  • Add InIdSetTransformFunction (#5973)

  • Add ingestion descriptor in the header (#5995)

  • Zookeeper put api (#5949)

  • Feature/#5390 segment indexing reload status api (#5718)

  • Segment processing framework (#5934)

  • Support streaming query in QueryExecutor (#6027)

  • Add list of allowed tables for emitting table level metrics (#6037)

  • Add FilterOptimizer which supports optimizing both PQL and SQL query filter (#6056)

  • Adding push job type of segment metadata only mode (#5967)

  • Minion taskExecutor for RealtimeToOfflineSegments task (#6050, #6124)

  • Adding array transform functions: array_average, array_max, array_min, array_sum (#6084)

  • Allow modifying/removing existing star-trees during segment reload (#6100)

  • Implement off-heap bloom filter reader (#6118)

  • Support for multi-threaded Group By reducer for SQL. (#6044)

  • Add OnHeapGuavaBloomFilterReader (#6147)

  • Support using ordinals in GROUP BY and ORDER BY clause (#6152)

  • Merge common APIs for Dictionary (#6176)

  • Add table level lock for segment upload ([#6165])

  • Added recursive functions validation check for group by (#6186)

  • Add StrictReplicaGroupInstanceSelector (#6208)

  • Add IN_SUBQUERY support (#6022)

  • Add IN_PARTITIONED_SUBQUERY support (#6043)

  • Some UI features (#5810, #5981, #6117, #6215)

Special notes

  • Brokers should be upgraded before servers in order to keep backward-compatible:

    • Change group key delimiter from '\t' to '\0' (#5858)

    • Support for exact distinct count for non int data types (#5872)

  • Pinot Components have to be deployed in the following order:

    (PinotServiceManager -> Bootstrap services in role ServiceRole.CONTROLLER -> All remaining bootstrap services in parallel)

    • Starts Broker and Server in parallel when using ServiceManager (#5917)

    • New settings introduced and old ones deprecated:

    • Make real-time threshold property names less ambiguous (#5953)

    • Change Signature of Broker API in Controller (#6119)

  • This aggregation function is still in beta version. This PR involves change on the format of data sent from server to broker, so it works only when both broker and server are upgraded to the new version:

    • Enhance DistinctCountThetaSketchAggregationFunction (#6004)

Major Bug fixes

  • Improve performance of DistinctCountThetaSketch by eliminating empty sketches and unions. (#5798)

  • Enhance VarByteChunkSVForwardIndexReader to directly read from data buffer for uncompressed data (#5816)

  • Fixing backward-compatible issue of schema fetch call (#5885)

  • Fix race condition in MetricsHelper (#5887)

  • Fixing the race condition that segment finished before ControllerLeaderLocator created. (#5864)

  • Fix CSV and JSON converter on BYTES column (#5931)

  • Fixing the issue that transform UDFs are parsed as function name 'OTHER', not the real function names (#5940)

  • Incorporating embedded exception while trying to fetch stream offset (#5956)

  • Use query timeout for planning phase (#5990)

  • Add null check while fetching the schema (#5994)

  • Validate timeColumnName when adding/updating schema/tableConfig (#5966)

  • Handle the partitioning mismatch between table config and stream (#6031)

  • Fix built-in virtual columns for immutable segment (#6042)

  • Refresh the routing when real-time segment is committed (#6078)

  • Add support for Decimal with Precision Sum aggregation (#6053)

  • Fixing the calls to Helix to throw exception if zk connection is broken (#6069)

  • Allow modifying/removing existing star-trees during segment reload (#6100)

  • Add max length support in schema builder (#6112)

  • Enhance star-tree to skip matching-all predicate on non-star-tree dimension (#6109)

Backward Incompatible Changes

  • Make real-time threshold property names less ambiguous (#5953)

  • Enhance DistinctCountThetaSketchAggregationFunction (#6004)

  • Deep Extraction Support for ORC, Thrift, and ProtoBuf Records (#6046)

mkdir -p /tmp/pinot-quick-start/rawdata
/tmp/pinot-quick-start/rawdata/transcript.csv
studentID,firstName,lastName,gender,subject,score,timestampInEpoch
200,Lucy,Smith,Female,Maths,3.8,1570863600000
200,Lucy,Smith,Female,English,3.5,1571036400000
201,Bob,King,Male,Maths,3.2,1571900400000
202,Nick,Young,Male,Physics,3.6,1572418800000

Dimensions

Typically used in filters and group by, for slicing and dicing into data

Metrics

Typically used in aggregations, represents the quantitative data

Time

Optional column, represents the timestamp associated with each row

/tmp/pinot-quick-start/transcript-schema.json
{
  "schemaName": "transcript",
  "dimensionFieldSpecs": [
    {
      "name": "studentID",
      "dataType": "INT"
    },
    {
      "name": "firstName",
      "dataType": "STRING"
    },
    {
      "name": "lastName",
      "dataType": "STRING"
    },
    {
      "name": "gender",
      "dataType": "STRING"
    },
    {
      "name": "subject",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "score",
      "dataType": "FLOAT"
    }
  ],
  "dateTimeFieldSpecs": [{
    "name": "timestampInEpoch",
    "dataType": "LONG",
    "format" : "1:MILLISECONDS:EPOCH",
    "granularity": "1:MILLISECONDS"
  }]
}
/tmp/pinot-quick-start/transcript-table-offline.json
{
  "tableName": "transcript",
  "segmentsConfig" : {
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "replication" : "1",
    "schemaName" : "transcript"
  },
  "tableIndexConfig" : {
    "invertedIndexColumns" : [],
    "loadMode"  : "MMAP"
  },
  "tenants" : {
    "broker":"DefaultTenant",
    "server":"DefaultTenant"
  },
  "tableType":"OFFLINE",
  "metadata": {}
}
$ ls /tmp/pinot-quick-start
rawdata			transcript-schema.json	transcript-table-offline.json

$ ls /tmp/pinot-quick-start/rawdata 
transcript.csv
docker run --rm -ti \
    --network=pinot-demo \
    -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \
    --name pinot-batch-table-creation \
    apachepinot/pinot:latest AddTable \
    -schemaFile /tmp/pinot-quick-start/transcript-schema.json \
    -tableConfigFile /tmp/pinot-quick-start/transcript-table-offline.json \
    -controllerHost manual-pinot-controller \
    -controllerPort 9000 -exec
bin/pinot-admin.sh AddTable \
  -tableConfigFile /tmp/pinot-quick-start/transcript-table-offline.json \
  -schemaFile /tmp/pinot-quick-start/transcript-schema.json -exec
// /tmp/pinot-quick-start/docker-job-spec.yml or /tmp/pinot-quick-start/batch-job-spec.yml

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/tmp/pinot-quick-start/rawdata/'
includeFileNamePattern: 'glob:**/*.csv'
outputDirURI: '/tmp/pinot-quick-start/segments/'
overwriteOutput: true
pushJobSpec:
  pushFileNamePattern: 'glob:**/*.tar.gz'
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
  tableName: 'transcript'
  schemaURI: 'http://localhost:9000/tables/transcript/schema'
  tableConfigURI: 'http://localhost:9000/tables/transcript'
pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'
docker run --rm -ti \
    --network=pinot-demo \
    -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \
    --name pinot-data-ingestion-job \
    apachepinot/pinot:latest LaunchDataIngestionJob \
    -jobSpecFile /tmp/pinot-quick-start/docker-job-spec.yml
bin/pinot-admin.sh LaunchDataIngestionJob \
    -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yml
SegmentGenerationJobSpec: 
!!org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec
excludeFileNamePattern: null
executionFrameworkSpec: {extraConfigs: null, name: standalone, segmentGenerationJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner,
  segmentTarPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner,
  segmentUriPushJobRunnerClassName: org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner}
includeFileNamePattern: glob:**\/*.csv
inputDirURI: /tmp/pinot-quick-start/rawdata/
jobType: SegmentCreationAndTarPush
outputDirURI: /tmp/pinot-quick-start/segments
overwriteOutput: true
pinotClusterSpecs:
- {controllerURI: 'http://localhost:9000'}
pinotFSSpecs:
- {className: org.apache.pinot.spi.filesystem.LocalPinotFS, configs: null, scheme: file}
pushJobSpec: null
recordReaderSpec: {className: org.apache.pinot.plugin.inputformat.csv.CSVRecordReader,
  configClassName: org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig,
  configs: null, dataFormat: csv}
segmentNameGeneratorSpec: null
tableSpec: {schemaURI: 'http://localhost:9000/tables/transcript/schema', tableConfigURI: 'http://localhost:9000/tables/transcript',
  tableName: transcript}

Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Finished building StatsCollector!
Collected stats for 4 documents
Using fixed bytes value dictionary for column: studentID, size: 9
Created dictionary for STRING column: studentID with cardinality: 3, max length in bytes: 3, range: 200 to 202
Using fixed bytes value dictionary for column: firstName, size: 12
Created dictionary for STRING column: firstName with cardinality: 3, max length in bytes: 4, range: Bob to Nick
Using fixed bytes value dictionary for column: lastName, size: 15
Created dictionary for STRING column: lastName with cardinality: 3, max length in bytes: 5, range: King to Young
Created dictionary for FLOAT column: score with cardinality: 4, range: 3.2 to 3.8
Using fixed bytes value dictionary for column: gender, size: 12
Created dictionary for STRING column: gender with cardinality: 2, max length in bytes: 6, range: Female to Male
Using fixed bytes value dictionary for column: subject, size: 21
Created dictionary for STRING column: subject with cardinality: 3, max length in bytes: 7, range: English to Physics
Created dictionary for LONG column: timestampInEpoch with cardinality: 4, range: 1570863600000 to 1572418800000
Start building IndexCreator!
Finished records indexing in IndexCreator!
Finished segment seal!
Converting segment: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0 to v3 format
v3 segment location for segment: transcript_OFFLINE_1570863600000_1572418800000_0 is /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3
Deleting files in v1 segment directory: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0
Starting building 1 star-trees with configs: [StarTreeV2BuilderConfig[splitOrder=[studentID, firstName],skipStarNodeCreation=[],functionColumnPairs=[org.apache.pinot.core.startree.v2.AggregationFunctionColumnPair@3a48efdc],maxLeafRecords=1]] using OFF_HEAP builder
Starting building star-tree with config: StarTreeV2BuilderConfig[splitOrder=[studentID, firstName],skipStarNodeCreation=[],functionColumnPairs=[org.apache.pinot.core.startree.v2.AggregationFunctionColumnPair@3a48efdc],maxLeafRecords=1]
Generated 3 star-tree records from 4 segment records
Finished constructing star-tree, got 9 tree nodes and 4 records under star-node
Finished creating aggregated documents, got 6 aggregated records
Finished building star-tree in 10ms
Finished building 1 star-trees in 27ms
Computed crc = 3454627653, based on files [/var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/columns.psf, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/index_map, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/metadata.properties, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/star_tree_index, /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0/v3/star_tree_index_map]
Driver, record read time : 0
Driver, stats collector time : 0
Driver, indexing time : 0
Tarring segment from: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0 to: /var/folders/3z/qn6k60qs6ps1bb6s2c26gx040000gn/T/pinot-1583443148720/output/transcript_OFFLINE_1570863600000_1572418800000_0.tar.gz
Size for segment: transcript_OFFLINE_1570863600000_1572418800000_0, uncompressed: 6.73KB, compressed: 1.89KB
Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner
Initializing PinotFS for scheme file, classname org.apache.pinot.spi.filesystem.LocalPinotFS
Start pushing segments: [/tmp/pinot-quick-start/segments/transcript_OFFLINE_1570863600000_1572418800000_0.tar.gz]... to locations: [org.apache.pinot.spi.ingestion.batch.spec.PinotClusterSpec@243c4f91] for table transcript
Pushing segment: transcript_OFFLINE_1570863600000_1572418800000_0 to location: http://localhost:9000 for table transcript
Sending request: http://localhost:9000/v2/segments?tableName=transcript to controller: nehas-mbp.hsd1.ca.comcast.net, version: Unknown
Response for pushing table transcript segment transcript_OFFLINE_1570863600000_1572418800000_0 to location http://localhost:9000 - 200: {"status":"Successfully uploaded segment: transcript_OFFLINE_1570863600000_1572418800000_0 of table: transcript"}
Pinot in Docker
Schema
Table
Rest API
Segment
Query Console
{
  "students": [
    {
      "firstName": "Jane",
      "id": "100",
      "scores": {
        "physics": 91,
        "chemistry": 93,
        "maths": 99
      }
    },
    {
      "firstName": "John",
      "id": "101",
      "scores": {
        "physics": 97,
        "chemistry": 98,
        "maths": 99
      }
    },
    {
      "firstName": "Jen",
      "id": "102",
      "scores": {
        "physics": 96,
        "chemistry": 95,
        "maths": 100
      }
    }
  ]
}
{
  "schemaName": "students001",
  "enableColumnBasedNullHandling": false,
  "dimensionFieldSpecs": [
    {
      "name": "students.firstName",
      "dataType": "STRING",
      "notNull": false,
      "fieldType": "DIMENSION"
    },
    {
      "name": "students.id",
      "dataType": "STRING",
      "notNull": false,
      "fieldType": "DIMENSION"
    },
    {
      "name": "students.scores",
      "dataType": "JSON",
      "notNull": false,
      "fieldType": "DIMENSION"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "ts",
      "fieldType": "DATE_TIME",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ],
  "metricFieldSpecs": []
}
{
    "ingestionConfig": {
      "complexTypeConfig": {
        "fieldsToUnnest": [
          "students"
        ]
      }
  }
}
{
  "student": [
    {
      "name": "John"
    },
    {
      "name": "Jane"
    }
  ],
  "teacher": [
    {
      "physics": "Kim"
    },
    {
      "chemistry": "Lu"
    },
    {
      "maths": "Walsh"
    }
  ]
}
{
  "schemaName": "students002",
  "enableColumnBasedNullHandling": false,
  "dimensionFieldSpecs": [
    {
      "name": "student.name",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    },
    {
      "name": "teacher.physics",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    },
    {
      "name": "teacher.chemistry",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    },
    {
      "name": "teacher.maths",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    }
  ]
}
  "complexTypeConfig": {
    "fieldsToUnnest": [
      "student",
      "teacher"
    ]
  }
{
  "students": [
    {
      "name": "Jane",
      "grades": [
        {
          "physics": "A+"
        },
        {
          "maths": "A-"
        }
      ]
    },
    {
      "name": "John",
      "grades": [
        {
          "physics": "B+"
        },
        {
          "maths": "B-"
        }
      ]
    }
  ]
}
{
  "schemaName": "students003",
  "enableColumnBasedNullHandling": false,
  "dimensionFieldSpecs": [
    {
      "name": "students.name",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    },
    {
      "name": "students.grades.physics",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    },
    {
      "name": "students.grades.maths",
      "dataType": "STRING",
      "fieldType": "DIMENSION",
      "notNull": false
    }
  ]
}
  "complexTypeConfig": {
    "fieldsToUnnest": [
      "students",
      "students.grades"
    ]
  }
{
  "students": [
    {
      "name": "John",
      "grades": {
        "finalExam": [
          {
            "physics": "A+"
          },
          {
            "maths": "A-"
          }
        ]
      }
    },
    {
      "name": "Jane",
      "grades": {
        "finalExam": [
          {
            "physics": "B+"
          },
          {
            "maths": "B-"
          }
        ]
      }
    }
  ]
}
{
    "schemaName": "students004",
    "enableColumnBasedNullHandling": false,
    "dimensionFieldSpecs": [
      {
        "name": "students.name",
        "dataType": "STRING",
        "notNull": false,
        "fieldType": "DIMENSION"
      },
      {
        "name": "students.grades.finalExam.physics",
        "dataType": "STRING",
        "notNull": false,
        "fieldType": "DIMENSION"
      },
      {
        "name": "students.grades.finalExam.maths",
        "dataType": "STRING",
        "notNull": false,
        "fieldType": "DIMENSION"
      }
    ]
  }
  "complexTypeConfig": {
    "fieldsToUnnest": [
      "students",
      "students.grades.finalExam"
    ]
  }
{
  "students": [
    {
      "name": "John",
      "grades": [
        {
          "physics": "A+"
        },
        {
          "maths": "A"
        }
      ]
    },
    {
      "name": "Jane",
      "grades": [
        {
          "physics": "B+"
        },
        {
          "maths": "B-"
        }
      ]
    }
  ]
}
{
    "schemaName": "students005",
    "enableColumnBasedNullHandling": false,
    "dimensionFieldSpecs": [
      {
        "name": "students.name",
        "dataType": "STRING",
        "notNull": false,
        "fieldType": "DIMENSION"
      },
      {
        "name": "students.grades",
        "dataType": "STRING",
        "notNull": false,
        "isSingleValue": false,
        "fieldType": "DIMENSION"
      }
    ]
  }
  "complexTypeConfig": {
    "fieldsToUnnest": [
      "students"
    ]
  }
{
  "students": [
    {
      "name": "John",
      "extra_curricular": [
        "piano", "soccer"
      ]
    },
    {
      "name": "Jane",
      "extra_curricular": [
        "violin", "music"
      ]
    }
  ]
}
{
    "schemaName": "students006",
    "enableColumnBasedNullHandling": false,
    "dimensionFieldSpecs": [
      {
        "name": "students.name",
        "dataType": "STRING",
        "notNull": false,
        "fieldType": "DIMENSION"
      },
      {
        "name": "students.extra_curricular",
        "dataType": "JSON",
        "notNull": false,
        "fieldType": "DIMENSION"
      }
    ]
  }
    "complexTypeConfig": {
      "fieldsToUnnest": [
        "students"
      ], 
      "collectionNotUnnestedToJson": "ALL"
    }
ComplexType
Unnested Student Records
Unnested student records
Unnest Nested Collection
Unnested Multi Level Array
Converted Inner Collection
Primitives Converted to JSON

Table

Explore the table component in Apache Pinot, a fundamental building block for organizing and managing data in Pinot clusters, enabling effective data processing and analysis.

Pinot stores data in tables. A Pinot table is conceptually identical to a relational database table with rows and columns. Columns have the same name and data type, known as the table's schema.

Pinot schemas are defined in a JSON file. Because that schema definition is in its own file, multiple tables can share a single schema. Each table can have a unique name, indexing strategy, partitioning, data sources, and other metadata.

Pinot table types include:

  • real-time: Ingests data from a streaming source like Apache Kafka®

  • offline: Loads data from a batch source

  • hybrid: Loads data from both a batch source and a streaming source

Pinot breaks a table into multiple segments and stores these segments in a deep-store such as Hadoop Distributed File System (HDFS) as well as Pinot servers.

In the Pinot cluster, a table is modeled as a Helix resource and each segment of a table is modeled as a Helix Partition.

Table naming in Pinot follows typical naming conventions, such as starting names with a letter, not ending with an underscore, and using only alphanumeric characters.

Pinot supports the following types of tables:

Type
Description

Offline

Offline tables ingest pre-built Pinot segments from external data stores and are generally used for batch ingestion.

Real-time

Real-time tables ingest data from streams (such as Kafka) and build segments from the consumed data.

Hybrid

Hybrid Pinot tables have both real-time as well as offline tables under the hood. By default, all tables in Pinot are hybrid.

The user querying the database does not need to know the type of the table. They only need to specify the table name in the query.

e.g. regardless of whether we have an offline table myTable_OFFLINE, a real-time table myTable_REALTIME, or a hybrid table containing both of these, the query will be:

select count(*)
from myTable

Table configuration is used to define the table properties, such as name, type, indexing, routing, and retention. It is written in JSON format and is stored in Zookeeper, along with the table schema.

Use the following properties to make your tables faster or leaner:

  • Segment

  • Indexing

  • Tenants

Segments

A table is comprised of small chunks of data known as segments. Learn more about how Pinot creates and manages segments here.

For offline tables, segments are built outside of Pinot and uploaded using a distributed executor such as Spark or Hadoop. For details, see Batch Ingestion.

For real-time tables, segments are built in a specific interval inside Pinot. You can tune the following for the real-time segments.

Flush

The Pinot real-time consumer ingests the data, creates the segment, and then flushes the in-memory segment to disk. Pinot allows you to configure when to flush the segment in the following ways:

  • Number of consumed rows: After consuming the specified number of rows from the stream, Pinot will persist the segment to disk.

  • Number of rows per segment: Pinot learns and then estimates the number of rows that need to be consumed. The learning phase starts by setting the number of rows to 100,000 (this value can be changed) and adjusts it to reach the appropriate segment size. Because Pinot corrects the estimate as it goes along, the segment size might go significantly over the correct size during the learning phase. You should set this value to optimize the performance of queries.

  • Max time duration to wait: Pinot consumers wait for the configured time duration after which segments are persisted to the disk.

Replicas A segment can have multiple replicas to provide higher availability. You can configure the number of replicas for a table segment using the CLI.

Completion Mode By default, if the in-memory segment in the non-winner server is equivalent to the committed segment, then the non-winner server builds and replaces the segment. If the available segment is not equivalent to the committed segment, the server just downloads the committed segment from the controller.

However, in certain scenarios, the segment build can get very memory-intensive. In these cases, you might want to enforce the non-committer servers to just download the segment from the controller instead of building it again. You can do this by setting completionMode: "DOWNLOAD" in the table configuration.

For details, see Completion Config.

Download Scheme

A Pinot server might fail to download segments from the deep store, such as HDFS, after its completion. However, you can configure servers to download these segments from peer servers instead of the deep store. Currently, only HTTP and HTTPS download schemes are supported. More methods, such as gRPC/Thrift, are planned be added in the future.

For more details about peer segment download during real-time ingestion, refer to this design doc on bypass deep store for segment completion.

Indexing

You can create multiple indices on a table to increase the performance of the queries. The following types of indices are supported:

  • Forward Index

    • Dictionary-encoded forward index with bit compression

    • Raw value forward index

    • Sorted forward index with run-length encoding

  • Inverted Index

    • Bitmap inverted index

    • Sorted inverted index

  • Star-tree Index

  • Range Index

  • Text Index

  • Geospatial

For more details on each indexing mechanism and corresponding configurations, see Indexing.

Set up Bloomfilters on columns to make queries faster. You can also keep segments in off-heap instead of on-heap memory for faster queries.

Pre-aggregation

Aggregate the real-time stream data as it is consumed to reduce segment sizes. We add the metric column values of all rows that have the same values for all dimension and time columns and create a single row in the segment. This feature is only available on REALTIME tables.

The only supported aggregation is SUM. The columns to pre-aggregate need to satisfy the following requirements:

  • All metrics should be listed in noDictionaryColumns.

  • No multi-value dimensions

  • All dimension columns are treated to have a dictionary, even if they appear as noDictionaryColumns in the config.

The following table config snippet shows an example of enabling pre-aggregation during real-time ingestion:

pinot-table-realtime.json
    "tableIndexConfig": { 
      "noDictionaryColumns": ["metric1", "metric2"],
      "aggregateMetrics": true,
      ...
    }

Tenants

Each table is associated with a tenant. A segment resides on the server, which has the same tenant as itself. For details, see Tenant.

Optionally, override if a table should move to a server with different tenant based on segment status. The example below adds a tagOverrideConfig under the tenants section for real-time tables to override tags for consuming and completed segments.

  "broker": "brokerTenantName",
  "server": "serverTenantName",
  "tagOverrideConfig" : {
    "realtimeConsuming" : "serverTenantName_REALTIME"
    "realtimeCompleted" : "serverTenantName_OFFLINE"
  }
}

In the above example, the consuming segments will still be assigned to serverTenantName_REALTIME hosts, but once they are completed, the segments will be moved to serverTeantnName_OFFLINE.

You can specify the full name of any tag in this section. For example, you could decide that completed segments for this table should be in Pinot servers tagged as allTables_COMPLETED). To learn more about, see the Moving Completed Segments section.

Hybrid table

A hybrid table is a table composed of two tables, one offline and one real-time, that share the same name. In a hybrid table, offline segments can be pushed periodically. The retention on the offline table can be set to a high value because segments are coming in on a periodic basis, whereas the retention on the real-time part can be small.

Once an offline segment is pushed to cover a recent time period, the brokers automatically switch to using the offline table for segments for that time period and use the real-time table only for data not available in the offline table.

To learn how time boundaries work for hybrid tables, see Broker.

A typical use case for hybrid tables is pushing deduplicated, cleaned-up data into an offline table every day while consuming real-time data as it arrives. Data can remain in offline tables for as long as a few years, while the real-time data would be cleaned every few days.

Examples

Create a table config for your data, or see examples for all possible batch/streaming tables.

Prerequisites

  • Set up the cluster

  • Create broker and server tenants

Offline table creation

docker run \
    --network=pinot-demo \
    --name pinot-batch-table-creation \
    ${PINOT_IMAGE} AddTable \
    -schemaFile examples/batch/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json \
    -controllerHost pinot-controller \
    -controllerPort 9000 \
    -exec

Sample console output

Executing command: AddTable -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json -schemaFile examples/batch/airlineStats/airlineStats_schema.json -controllerHost pinot-controller -controllerPort 9000 -exec
Sending request: http://pinot-controller:9000/schemas to controller: a413b0013806, version: Unknown
{"status":"Table airlineStats_OFFLINE succesfully added"}
bin/pinot-admin.sh AddTable \
    -schemaFile examples/batch/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/batch/airlineStats/airlineStats_offline_table_config.json \
    -exec
# add schema
curl -F schemaName=@airlineStats_schema.json  localhost:9000/schemas

# add table
curl -i -X POST -H 'Content-Type: application/json' \
    -d @airlineStats_offline_table_config.json localhost:9000/tables

Check out the table config in the Rest API to make sure it was successfully uploaded.

Streaming table creation

Start Kafka

docker run \
    --network pinot-demo --name=kafka \
    -e KAFKA_ZOOKEEPER_CONNECT=pinot-zookeeper:2181/kafka \
    -e KAFKA_BROKER_ID=0 \
    -e KAFKA_ADVERTISED_HOST_NAME=kafka \
    -d wurstmeister/kafka:latest

Create a Kafka topic

docker exec \
  -t kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --zookeeper pinot-zookeeper:2181/kafka \
  --partitions=1 --replication-factor=1 \
  --create --topic flights-realtime

Create a streaming table

docker run \
    --network=pinot-demo \
    --name pinot-streaming-table-creation \
    ${PINOT_IMAGE} AddTable \
    -schemaFile examples/stream/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/docker/table-configs/airlineStats_realtime_table_config.json \
    -controllerHost pinot-controller \
    -controllerPort 9000 \
    -exec

Sample output

Executing command: AddTable -tableConfigFile examples/docker/table-configs/airlineStats_realtime_table_config.json -schemaFile examples/stream/airlineStats/airlineStats_schema.json -controllerHost pinot-controller -controllerPort 9000 -exec
Sending request: http://pinot-controller:9000/schemas to controller: 8fbe601012f3, version: Unknown
{"status":"Table airlineStats_REALTIME succesfully added"}

Start Kafka-Zookeeper

bin/pinot-admin.sh StartZookeeper -zkPort 2191

Start Kafka

bin/pinot-admin.sh  StartKafka -zkAddress=localhost:2191/kafka -port 19092

Create stream table

bin/pinot-admin.sh AddTable \
    -schemaFile examples/stream/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/stream/airlineStats/airlineStats_realtime_table_config.json \
    -exec

Check out the table config in the Rest API to make sure it was successfully uploaded.

Hybrid table creation

To create a hybrid table, you have to create the offline and real-time tables individually. You don't need to create a separate hybrid table.

"OFFLINE": {
    "tableName": "pinotTable", 
    "tableType": "OFFLINE", 
    "segmentsConfig": {
      ... 
    }, 
    "tableIndexConfig": { 
      ... 
    },  
    "tenants": {
      "broker": "myBrokerTenant", 
      "server": "myServerTenant"
    },
    "metadata": {
      ...
    }
  },
  "REALTIME": { 
    "tableName": "pinotTable", 
    "tableType": "REALTIME", 
    "segmentsConfig": {
      ...
    }, 
    "tableIndexConfig": { 
      ... 
      "streamConfigs": {
        ...
      },  
    },  
    "tenants": {
      "broker": "myBrokerTenant", 
      "server": "myServerTenant"
    },
    "metadata": {
    ...
    }
  }
}

0.7.1

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations.

Summary

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations and improvements.

It also adds several new APIs to better manage the segments and upload data to the offline table. It also contains many key bug fixes. See details below.

The release was cut from the following commit: 78152cd

and the following cherry-picks:

  • b527af3

  • 84d59e3

  • a18dc60

  • 4ec38f7

  • b48dac0

  • 5d2bc0c

  • 913492e

  • 50a4531

  • 1f21403

  • 8dbb70b

Notable New Features

  • Add a server metric: queriesDisabled to check if queries disabled or not. (#6586)

  • Optimization on GroupKey to save the overhead of ser/de the group keys (#6593) (#6559)

  • Support validation for jsonExtractKey and jsonExtractScalar functions (#6246) (#6594)

  • Real Time Provisioning Helper tool improvement to take data characteristics as input instead of an actual segment (#6546)

  • Add the isolation level config isolation.level to Kafka consumer (2.0) to ingest transactionally committed messages only (#6580)

  • Enhance StarTreeIndexViewer to support multiple trees (#6569)

  • Improves ADLSGen2PinotFS with service principal based auth, auto create container on initial run. It's backwards compatible with key based auth. (#6531)

  • Add metrics for minion tasks status (#6549)

  • Use minion data directory as tmp directory for SegmentGenerationAndPushTask to ensure directory is always cleaned up (#6560)

  • Add optional HTTP basic auth to pinot broker, which enables user- and table-level authentication of incoming queries. (#6552)

  • Add Access Control for REST endpoints of Controller (#6507)

  • Add date_trunc to scalar functions to support date_trunc during ingestion (#6538)

  • Allow tar gz with > 8gb size (#6533)

  • Add Lookup UDF Join support (#6530), (#6465), (#6383) (#6286)

  • Add cron scheduler metrics reporting (#6502)

  • Support generating derived column during segment load, so that derived columns can be added on-the-fly (#6494)

  • Support chained transform functions (#6495)

  • Add scalar function JsonPathArray to extract arrays from json (#6490)

  • Add a guard against multiple consuming segments for same partition (#6483)

  • Remove the usage of deprecated range delimiter (#6475)

  • Handle scheduler calls with proper response when it's disabled. (#6474)

  • Simplify SegmentGenerationAndPushTask handling getting schema and table config (#6469)

  • Add a cluster config to config number of concurrent tasks per instance for minion task: SegmentGenerationAndPushTaskGenerator (#6468)

  • Replace BrokerRequestOptimizer with QueryOptimizer to also optimize the PinotQuery (#6423)

  • Add additional string scalar functions (#6458)

  • Add additional scalar functions for array type (#6446)

  • Add CRON scheduler for Pinot tasks (#6451)

  • Set default Data Type while setting type in Add Schema UI dialog (#6452)

  • Add ImportData sub command in pinot admin (#6396)

  • H3-based geospatial index (#6409) (#6306)

  • Add JSON index support (#6408) (#6216) (#6346)

  • Make minion tasks pluggable via reflection (#6395)

  • Add compatibility test for segment operations upload and delete (#6382)

  • Add segment reset API that disables and then enables the segment (#6336)

  • Add Pinot minion segment generation and push task. (#6340)

  • Add a version option to pinot admin to show all the component versions (#6380)

  • Add FST index using lucene lib to speedup REGEXP_LIKE operator on text (#6120)

  • Add APIs for uploading data to an offline table. (#6354)

  • Allow the use of environment variables in stream configs (#6373)

  • Enhance task schedule api for single type/table support (#6352)

  • Add broker time range based pruner for routing. Query operators supported: RANGE, =, <, <=, >, >=, AND, OR(#6259)

  • Add json path functions to extract values from json object (#6347)

  • Create a pluggable interface for Table config tuner (#6255)

  • Add a Controller endpoint to return table creation time (#6331)

  • Add tooltips, ability to enable-disable table state to the UI (#6327)

  • Add Pinot Minion client (#6339)

  • Add more efficient use of RoaringBitmap in OnHeapBitmapInvertedIndexCreator and OffHeapBitmapInvertedIndexCreator (#6320)

  • Add decimal percentile support. (#6323)

  • Add API to get status of consumption of a table (#6322)

  • Add support to add offline and real-time tables, individually able to add schema and schema listing in UI (#6296)

  • Improve performance for distinct queries (#6285)

  • Allow adding custom configs during the segment creation phase (#6299)

  • Use sorted index based filtering only for dictionary encoded column (#6288)

  • Enhance forward index reader for better performance (#6262)

  • Support for text index without raw (#6284)

  • Add api for cluster manager to get table state (#6211)

  • Perf optimization for SQL GROUP BY ORDER BY (#6225)

  • Add support using environment variables in the format of ${VAR_NAME:DEFAULT_VALUE} in Pinot table configs. (#6271)

Special notes

  • Pinot controller metrics prefix is fixed to add a missing dot (#6499). This is a backward-incompatible change that JMX query on controller metrics must be updated

  • Legacy group key delimiter (\t) was removed to be backward-compatible with release 0.5.0 (#6589)

  • Upgrade zookeeper version to 3.5.8 to fix ZOOKEEPER-2184: Zookeeper Client should re-resolve hosts when connection attempts fail. (#6558)

  • Add TLS-support for client-pinot and pinot-internode connections (#6418) Upgrades to a TLS-enabled cluster can be performed safely and without downtime. To achieve a live-upgrade, go through the following steps:

    • First, configure alternate ingress ports for https/netty-tls on brokers, controllers, and servers. Restart the components with a rolling strategy to avoid cluster downtime.

    • Second, verify manually that https access to controllers and brokers is live. Then, configure all components to prefer TLS-enabled connections (while still allowing unsecured access). Restart the individual components.

    • Third, disable insecure connections via configuration. You may also have to set controller.vip.protocol and controller.vip.port and update the configuration files of any ingestion jobs. Restart components a final time and verify that insecure ingress via http is not available anymore.

  • PQL endpoint on Broker is deprecated (#6607)

    • Apache Pinot has adopted SQL syntax and semantics. Legacy PQL (Pinot Query Language) is deprecated and no longer supported. Use SQL syntax to query Pinot on broker endpoint /query/sql and controller endpoint /sql

Major Bug fixes

  • Fix the SIGSEGV for large index (#6577)

  • Handle creation of segments with 0 rows so segment creation does not fail if data source has 0 rows. (#6466)

  • Fix QueryRunner tool for multiple runs (#6582)

  • Use URL encoding for the generated segment tar name to handle characters that cannot be parsed to URI. (#6571)

  • Fix a bug of miscounting the top nodes in StarTreeIndexViewer (#6569)

  • Fix the raw bytes column in real-time segment (#6574)

  • Fixes a bug to allow using JSON_MATCH predicate in SQL queries (#6535)

  • Fix the overflow issue when loading the large dictionary into the buffer (#6476)

  • Fix empty data table for distinct query (#6363)

  • Fix the default map return value in DictionaryBasedGroupKeyGenerator (#6712)

  • Fix log message in ControllerPeriodicTask (#6709)

  • Fix bug #6671: RealtimeTableDataManager shuts down SegmentBuildTimeLeaseExtender for all tables in the host (#6682)

  • Fix license headers and plugin checks

Architecture

Understand how the components of Apache Pinot™ work together to create a scalable OLAP database that can deliver low-latency, high-concurrency queries at scale.

Apache Pinot™ is a distributed OLAP database designed to serve real-time, user-facing use cases, which means handling large volumes of data and many concurrent queries with very low query latencies. Pinot supports the following requirements:

  • Ultra low-latency queries (as low as 10ms P95)

  • High query concurrency (as many as 100,000 queries per second)

  • High data freshness (streaming data available for query immediately upon ingestion)

  • Large data volume (up to petabytes)

Distributed design principles

To accommodate large data volumes with stringent latency and concurrency requirements, Pinot is designed as a distributed database that supports the following requirements:

  • Highly available: Pinot has no single point of failure. When tables are configured for replication, and a node goes down, the cluster is able to continue processing queries.

  • Horizontally scalable: Operators can scale a Pinot cluster by adding new nodes when the workload increases. There are even two node types (servers and brokers) to scale query volume, query complexity, and data size independently.

  • Immutable data: Pinot assumes all stored data is immutable, which helps simplify the parts of the system that handle data storage and replication. However, Pinot still supports upserts on streaming entity data and background purges of data to comply with data privacy regulations.

  • Dynamic configuration changes: Operations like adding new tables, expanding a cluster, ingesting data, modifying an existing table, and adding indexes do not impact query availability or performance.

Core components

As described in the Pinot Components, Pinot has four node types:

  • Controller

  • Broker

  • Server

  • Minion

Apache Helix and ZooKeeper

Distributed systems do not maintain themselves, and in fact require sophisticated scheduling and resource management to function. Pinot uses Apache Helix for this purpose. Helix exists as an independent project, but it was designed by the original creators of Pinot for Pinot's own cluster management purposes, so the architectures of the two systems are well-aligned. Helix takes the form of a process on the controller, plus embedded agents on the brokers and servers. It uses Apache ZooKeeper as a fault-tolerant, strongly consistent, durable state store.

Helix maintains a picture of the intended state of the cluster, including the number of servers and brokers, the configuration and schema of all tables, connections to streaming ingest sources, currently executing batch ingestion jobs, the assignment of table segments to the servers in the cluster, and more. All of these configuration items are potentially mutable quantities, since operators routinely change table schemas, add or remove streaming ingest sources, begin new batch ingestion jobs, and so on. Additionally, physical cluster state may change as servers and brokers fail or suffer network partition. Helix works constantly to drive the actual state of the cluster to match the intended state, pushing configuration changes to brokers and servers as needed.

There are three physical node types in a Helix cluster:

  • Participant: These nodes do things, like store data or perform computation. Participants host resources, which are Helix's fundamental storage abstraction. Because Pinot servers store segment data, they are participants.

  • Spectator: These nodes see things, observing the evolving state of the participants through events pushed to the spectator. Because Pinot brokers need to know which servers host which segments, they are spectators.

  • Controller: This node observes and manages the state of participant nodes. The controller is responsible for coordinating all state transitions in the cluster and ensures that state constraints are satisfied while maintaining cluster stability.

In addition, Helix defines two logical components to express its storage abstraction:

  • Partition. A unit of data storage that lives on at least one participant. Partitions may be replicated across multiple participants. A Pinot segment is a partition.

  • Resource. A logical collection of partitions, providing a single view over a potentially large set of data stored across a distributed system. A Pinot table is a resource.

In summary, the Pinot architecture maps onto Helix components as follows:

Pinot Component
Helix Component

Segment

Helix Partition

Table

Helix Resource

Controller

Helix Controller or Helix agent that drives the overall state of the cluster

Server

Helix Participant

Broker

A Helix Spectator that observes the cluster for changes in the state of segments and servers. To support multi-tenancy, brokers are also modeled as Helix Participants.

Minion

Helix Participant that performs computation rather than storing data

Helix uses ZooKeeper to maintain cluster state. ZooKeeper sends Helix spectators notifications of changes in cluster state (which correspond to changes in ZNodes). Zookeeper stores the following information about the cluster:

Resource
Stored Properties

Controller

  • Controller that is assigned as the current leader

Servers and Brokers

  • List of servers and brokers

  • Configuration of all current servers and brokers

  • Health status of all current servers and brokers

Tables

  • List of tables

  • Table configurations

  • Table schema

  • List of the table's segments

Segment

  • Exact server locations of a segment

  • State of each segment (online/offline/error/consuming)

  • Metadata about each segment

Zookeeper, as a first-class citizen of a Pinot cluster, may use the well-known ZNode structure for operations and troubleshooting purposes. Be advised that this structure can change in future Pinot releases.

Pinot's Zookeeper Browser UI

Controller

The Pinot controller schedules and re-schedules resources in a Pinot cluster when metadata changes or a node fails. As an Apache Helix Controller, it schedules the resources that comprise the cluster and orchestrates connections between certain external processes and cluster components (e.g., ingest of real-time tables and offline tables). It can be deployed as a single process on its own server or as a group of redundant servers in an active/passive configuration.

Fault tolerance

Only one controller can be active at a time, so when multiple controllers are present in a cluster, they elect a leader. When that controller instance becomes unavailable, the remaining instances automatically elect a new leader. Leader election is achieved using Apache Helix. A Pinot cluster can serve queries without an active controller, but it can't perform any metadata-modifying operations, like adding a table or consuming a new segment.

Controller REST interface

The controller provides a REST interface that allows read and write access to all logical storage resources (e.g., servers, brokers, tables, and segments). See Pinot Data Explorer for more information on the web-based admin tool.

Broker

The broker's responsibility is to route queries to the appropriate server instances, or in the case of multi-stage queries, to compute a complete query plan and distribute it to the servers required to execute it. The broker collects and merges the responses from all servers into a final result, then sends the result back to the requesting client. The broker exposes an HTTP endpoint that accepts SQL queries in JSON format and returns the response in JSON.

Each broker maintains a query routing table. The routing table maps segments to the servers that store them. (When replication is configured on a table, each segment is stored on more than one server.) The broker computes multiple routing tables depending on the configured routing strategy for a table. The default strategy is to balance the query load across all available servers.

Advanced routing strategies are available, such as replica-aware routing, partition-based routing, and minimal server selection routing.

//This is an example ZNode config for EXTERNAL VIEW in Helix
{
  "id" : "baseballStats_OFFLINE",
  "simpleFields" : {
    ...
  },
  "mapFields" : {
    "baseballStats_OFFLINE_0" : {
      "Server_10.1.10.82_7000" : "ONLINE"
    }
  },
  ...
}

Query processing

Every query processed by a broker uses the single-stage engine or the multi-stage engine. For single-stage queries, the broker does the following:

  • Computes query routes based on the routing strategy defined in the table configuration.

  • Computes the list of segments to query on each server. (See routing for further details on this process.)

  • Sends the query to each of those servers for local execution against their segments.

  • Receives the results from each server and merges them.

  • Sends the query result to the client.

// Query: select count(*) from baseballStats limit 10

// RESPONSE
// ========
{
    "resultTable": {
        "dataSchema": {
            "columnDataTypes": ["LONG"],
            "columnNames": ["count(*)"]
        },
        "rows": [
            [97889]
        ]
    },
    "exceptions": [],
    "numServersQueried": 1,
    "numServersResponded": 1,
    "numSegmentsQueried": 1,
    "numSegmentsProcessed": 1,
    "numSegmentsMatched": 1,
    "numConsumingSegmentsQueried": 0,
    "numDocsScanned": 97889,
    "numEntriesScannedInFilter": 0,
    "numEntriesScannedPostFilter": 0,
    "numGroupsLimitReached": false,
    "totalDocs": 97889,
    "timeUsedMs": 5,
    "segmentStatistics": [],
    "traceInfo": {},
    "minConsumingFreshnessTimeMs": 0
}

For multi-stage queries, the broker performs the following:

  • Computes a query plan that runs on multiple sets of servers. The servers selected for the first stage are selected based on the segments required to execute the query, which are determined in a process similar to single-stage queries.

  • Sends the relevant portions of the query plan to one or more servers in the cluster for each stage of the query plan.

  • The servers that received query plans each execute their part of the query. For more details on this process, read about the multi-stage engine.

  • The broker receives a complete result set from the final stage of the query, which is always a single server.

  • The broker sends the query result to the client.

Server

Servers host segments on locally attached storage and process queries on those segments. By convention, operators speak of "real-time" and "offline" servers, although there is no difference in the server process itself or even its configuration that distinguishes between the two. This is merely a convention reflected in the table assignment strategy to confine the two different kinds of workloads to two groups of physical instances, since the performance-limiting factors differ between the two kinds of workloads. For example, offline servers might optimize for larger storage capacity, whereas real-time servers might optimize for memory and CPU cores.

Offline servers

Offline servers host segments created by ingesting batch data. The controller writes these segments to the offline server according to the table's replication factor and segment assignment strategy. Typically, the controller writes new segments to the deep store, and affected servers download the segment from deep store. The controller then notifies brokers that a new segment exists, and is available to participate in queries.

Because offline tables tend to have long retention periods, offline servers tend to scale based on the size of the data they store.

Real-time servers

Real-time servers ingest data from streaming sources, like Apache Kafka®, Apache Pulsar®, or AWS Kinesis. Streaming data ends up in conventional segment files just like batch data, but is first accumulated in an in-memory data structure known as a consuming segment. Each message consumed from a streaming source is written immediately to the relevant consuming segment, and is available for query processing from the consuming segment immediately, since consuming segments participate in query processing as first-class citizens. Consuming segments get flushed to disk periodically based on a completion threshold, which can be calculated by row count, ingestion time, or segment size. A flushed segment on a real-time table is called a completed segment, and is functionally equivalent to a segment created during offline ingest.

Real-time servers tend to be scaled based on the rate at which they ingest streaming data.

Minion

A Pinot minion is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing minion tasks as directed by the controller. Examples of minion tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically.

Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function without minions, they are typically present to support routine tasks like ingesting batch data.

Data ingestion overview

Pinot tables exist in two varieties: offline (or batch) and real-time. Offline tables contain data from batch sources like CSV, Avro, or Parquet files, and real-time tables contain data from streaming sources like like Apache Kafka®, Apache Pulsar®, or AWS Kinesis.

Offline (batch) ingest

Pinot ingests batch data using an ingestion job, which follows a process like this:

  1. The job transforms a raw data source (such as a CSV file) into segments. This is a potentially complex process resulting in a file that is typically several hundred megabytes in size.

  2. The job then transfers the file to the cluster's deep store and notifies the controller that a new segment exists.

  3. The controller (in its capacity as a Helix controller) updates the ideal state of the cluster in its cluster metadata map.

  4. The controller then assigns the segment to one or more "offline" servers (depending on replication factor) and notifies them that new segments are available.

  5. The servers then download the newly created segments directly from the deep store.

  6. The cluster's brokers, which watch for state changes as Helix spectators, detect the new segments and update their segment routing tables accordingly. The cluster is now able to query the new offline segments.

Real-time ingest

Ingestion is established at the time a real-time table is created, and continues as long as the table exists. When the controller receives the metadata update to create a new real-time table, the table configuration specifies the source of the streaming input data—often a topic in a Kafka cluster. This kicks off a process like this:

  1. The controller picks one or more servers to act as direct consumers of the streaming input source.

  2. The controller creates consuming segments for the new table. It does this by creating an entry in the global metadata map for a new consuming segment for each of the real-time servers selected in step 1.

  3. Through Helix functionality on the controller and the relevant servers, the servers proceed to create consuming segments in memory and establish a connection to the streaming input source. When this input source is Kafka, each server acts as a Kafka consumer directly, with no other components involved in the integration.

  4. Through Helix functionality on the controller and all of the cluster's brokers, the brokers become aware of the consuming segments, and begin including them in query routing immediately.

  5. The consuming servers simultaneously begin consuming messages from the streaming input source, storing them in the consuming segment.

  6. When a server decides its consuming segment is complete, it commits the in-memory consuming segment to a conventional segment file, uploads it to the deep store, and notifies the controller.

  7. The controller and the server create a new consuming segment to continue real-time ingestion.

  8. The controller marks the newly committed segment as online. Brokers then discover the new segment through the Helix notification mechanism, allowing them to route queries to it in the usual fashion.

Running in Kubernetes

Pinot quick start in Kubernetes

Get started running Pinot in Kubernetes.

Note: The examples in this guide are sample configurations to be used as reference. For production setup, you may want to customize it to your needs.

Prerequisites

Kubernetes

This guide assumes that you already have a running Kubernetes cluster.

If you haven't yet set up a Kubernetes cluster, see the links below for instructions:

    • Make sure to run with enough resources: minikube start --vm=true --cpus=4 --memory=8g --disk-size=50g

Pinot

Make sure that you've downloaded Apache Pinot. The scripts for the setup in this guide can be found in our.

Set up a Pinot cluster in Kubernetes

Start Pinot with Helm

The Pinot repository has pre-packaged Helm charts for Pinot and Presto. The Helm repository index file is .

Note: Specify StorageClass based on your cloud vendor. Don't mount a blob store (such as AzureFile, GoogleCloudStorage, or S3) as the data serving file system. Use only Amazon EBS/GCP Persistent Disk/Azure Disk-style disks.

  • For AWS: "gp2"

  • For GCP: "pd-ssd" or "standard"

  • For Azure: "AzureDisk"

  • For Docker-Desktop: "hostpath"

1.1.1 Update Helm dependency

1.1.2 Start Pinot with Helm

Check Pinot deployment status

Load data into Pinot using Kafka

Bring up a Kafka cluster for real-time data ingestion

Check Kafka deployment status

Ensure the Kafka deployment is ready before executing the scripts in the following steps. Run the following command:

Below is an example output showing the deployment is ready:

Create Kafka topics

Run the scripts below to create two Kafka topics for data ingestion:

Load data into Kafka and create Pinot schema/tables

The script below does the following:

  • Ingests 19492 JSON messages to Kafka topic flights-realtime at a speed of 1 msg/sec

  • Ingests 19492 Avro messages to Kafka topic flights-realtime-avro at a speed of 1 msg/sec

  • Uploads Pinot schema airlineStats

  • Creates Pinot table airlineStats to ingest data from JSON encoded Kafka topic flights-realtime

  • Creates Pinot table airlineStatsAvro to ingest data from Avro encoded Kafka topic flights-realtime-avro

Query with the Pinot Data Explorer

Pinot Data Explorer

The following script (located at ./pinot/helm/pinot) performs local port forwarding, and opens the Pinot query console in your default web browser.

Query Pinot with Superset

Bring up Superset using Helm

  1. Install the SuperSet Helm repository:

  1. Get the Helm values configuration file:

  1. For Superset to install Pinot dependencies, edit /tmp/superset-values.yaml file to add apinotdb pip dependency into bootstrapScript field.

  2. You can also build your own image with this dependency or use the image apachepinot/pinot-superset:latest instead.

  1. Replace the default admin credentials inside the init section with a meaningful user profile and stronger password.

  2. Install Superset using Helm:

  1. Ensure your cluster is up by running:

Access the Superset UI

  1. Run the below command to port forward Superset to your localhost:18088.

  1. Navigate to Superset in your browser with the admin credentials you set in the previous section.

  2. Create a new database connection with the following URI: pinot+http://pinot-broker.pinot-quickstart:8099/query?controller=http://pinot-controller.pinot-quickstart:9000/

  3. Once the database is added, you can add more data sets and explore the dashboard options.

Access Pinot with Trino

Deploy Trino

  1. Deploy Trino with the Pinot plugin installed:

  1. See the charts in the Trino Helm chart repository:

  1. In order to connect Trino to Pinot, you'll need to add the Pinot catalog, which requires extra configurations. Run the below command to get all the configurable values.

  1. To add the Pinot catalog, edit the additionalCatalogs section by adding:

Pinot is deployed at namespace pinot-quickstart, so the controller serviceURL is pinot-controller.pinot-quickstart:9000

  1. After modifying the /tmp/trino-values.yaml file, deploy Trino with:

  1. Once you've deployed Trino, check the deployment status:

Query Pinot with the Trino CLI

Once Trino is deployed, run the below command to get a runnable Trino CLI.

  1. Download the Trino CLI:

  1. Port forward Trino service to your local if it's not already exposed:

  1. Use the Trino console client to connect to the Trino service:

  1. Query Pinot data using the Trino CLI, like in the sample queries below.

Sample queries to execute

List all catalogs

List all tables

Show schema

Count total documents

Access Pinot with Presto

Deploy Presto with the Pinot plugin

  1. First, deploy Presto with default configurations:

  1. To customize your deployment, run the below command to get all the configurable values.

  1. After modifying the /tmp/presto-values.yaml file, deploy Presto:

  1. Once you've deployed the Presto instance, check the deployment status:

Query Presto using the Presto CLI

Once Presto is deployed, you can run the below command from , or follow the steps below.

  1. Download the Presto CLI:

  1. Port forward presto-coordinator port 8080 to localhost port 18080:

  1. Start the Presto CLI with the Pinot catalog:

  1. Query Pinot data with the Presto CLI, like in the sample queries below.

Sample queries to execute

List all catalogs

List all tables

Show schema

Count total documents

Delete a Pinot cluster in Kubernetes

To delete your Pinot cluster in Kubernetes, run the following command:

Operations FAQ

This page has a collection of frequently asked questions about operations with answers from the community.

This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, .

Memory

How much heap should I allocate for my Pinot instances?

Typically, Apache Pinot components try to use as much off-heap (MMAP/DirectMemory) wherever possible. For example, Pinot servers load segments in memory-mapped files in MMAP mode (recommended), or direct memory in HEAP mode. Heap memory is used mostly for query execution and storing some metadata. We have seen production deployments with high throughput and low-latency work well with just 16 GB of heap for Pinot servers and brokers. The Pinot controller may also cache some metadata (table configurations etc) in heap, so if there are just a few tables in the Pinot cluster, a few GB of heap should suffice.

DR

Does Pinot provide any backup/restore mechanism?

Pinot relies on deep-storage for storing a backup copy of segments (offline as well as real-time). It relies on Zookeeper to store metadata (table configurations, schema, cluster state, and so on). It does not explicitly provide tools to take backups or restore these data, but relies on the deep-storage (ADLS/S3/GCP/etc), and ZK to persist these data/metadata.

Alter Table

Can I change a column name in my table, without losing data?

Changing a column name or data type is considered backward incompatible change. While Pinot does support schema evolution for backward compatible changes, it does not support backward incompatible changes like changing name/data-type of a column.

How to change number of replicas of a table?

You can change the number of replicas by updating the table configuration's section. Make sure you have at least as many servers as the replication.

For offline tables, update :

For real-time tables, update :

After changing the replication, run a .

Note that if you are using replica groups, it's expected these configurations equal numReplicaGroups. If they do not match, Pinot will use numReplicaGroups.

How to set or change table retention?

By default there is no retention set for a table in Apache Pinot. You may however, set retention by setting the following properties in the section inside table configs:

  • retentionTimeUnit

  • retentionTimeValue

Updating the retention value in the table config should be good enough, there is no need to rebalance the table or reload its segments.

Rebalance

How to run a rebalance on a table?

See .

Why does my real-time table not use the new nodes I added to the cluster?

Likely explanation: num partitions * num replicas < num servers.

In real-time tables, segments of the same partition always remain on the same node. This sticky assignment is needed for replica groups and is critical if using upserts. For instance, if you have 3 partitions, 1 replica, and 4 nodes, only ¾ nodes will be used, and all of p0 segments will be on 1 node, p1 on 1 node, and p2 on 1 node. One server will be unused, and will remain unused through rebalances.

There’s nothing we can do about CONSUMING segments, they will continue to use only 3 nodes if you have 3 partitions. But we can rebalance such that completed segments use all nodes. If you want to force the completed segments of the table to use the new server use this config:

Segments

How to control the number of segments generated?

The number of segments generated depends on the number of input files. If you provide only 1 input file, you will get 1 segment. If you break up the input file into multiple files, you will get as many segments as the input files.

What are the common reasons my segment is in a BAD state ?

This typically happens when the server is unable to load the segment. Possible causes: out-of-memory, no disk space, unable to download segment from deep-store, and similar other errors. Check server logs for more information.

How to reset a segment when it runs into a BAD state?

Use the segment reset controller REST API to reset the segment:

How do I pause real-time ingestion?

Refer to .

What's the difference between Reset, Refresh, and Reload?

  • Reset: Gets a segment in ERROR state back to ONLINE or CONSUMING state. Behind the scenes, the Pinot controller takes the segment to the OFFLINE state, waits for External View to stabilize, and then moves it back to ONLINE or CONSUMING state, thus effectively resetting segments or consumers in error states.

  • Refresh: Replaces the segment with a new one, with the same name but often different data. Under the hood, the Pinot controller sets new segment metadata in Zookeeper, and notifies brokers and servers to check their local states about this segment and update accordingly. Servers also download the new segment to replace the old one, when both have different checksums. There is no separate rest API for refreshing, and it is done as part of the SegmentUpload API.

  • Reload: Loads the segment again, often to generate a new index as updated in the table configuration. Underlying, the Pinot server gets the new table configuration from Zookeeper, and uses it to guide the segment reloading. In fact, the last step of REFRESH as explained above is to load the segment into memory to serve queries. There is a dedicated rest API for reloading. By default, it doesn't download segments, but the option is provided to force the server to download the segment to replace the local one cleanly.

In addition, RESET brings the segment OFFLINE temporarily; while REFRESH and RELOAD swap the segment on server atomically without bringing down the segment or affecting ongoing queries.

Tenants

How can I make brokers/servers join the cluster without the DefaultTenant tag?

Set this property in your controller.conf file:

Now your brokers and servers should join the cluster as broker_untagged and server_untagged. You can then directly use the POST /tenants API to create the tenants you want, as in the following:

Minion

How do I tune minion task timeout and parallelism on each worker?

There are two task configurations, but they are set as part of cluster configurations, like in the following example. One controls the task's overall timeout (1hr by default) and one sets how many tasks to run on a single minion worker (1 by default). The <taskType> is the task to tune, such as MergeRollupTask or RealtimeToOfflineSegmentsTask etc.

How to I manually run a Periodic Task?

See .

Tuning and Optimizations

Do replica groups work for real-time?

Yes, replica groups work for real-time. There's 2 parts to enabling replica groups:

  1. Replica groups segment assignment.

  2. Replica group query routing.

Replica group segment assignment

Replica group segment assignment is achieved in real-time, if number of servers is a multiple of number of replicas. The partitions get uniformly sprayed across the servers, creating replica groups. For example, consider we have 6 partitions, 2 replicas, and 4 servers.

r1
r2

As you can see, the set (S0, S2) contains r1 of every partition, and (s1, S3) contains r2 of every partition. The query will only be routed to one of the sets, and not span every server. If you are are adding/removing servers from an existing table setup, you have to run for segment assignment changes to take effect.

Replica group query routing

Once replica group segment assignment is in effect, the query routing can take advantage of it. For replica group based query routing, set the following in the table config's section, and then restart brokers

Overwrite index configs at tier level

When using , user may want to have different encoding and indexing types for a column in different tiers to balance query latency and cost saving more flexibly. For example, segments in the hot tier can use dict-encoding, bloom filter and all kinds of relevant index types for very fast query execution. But for segments in the cold tier, where cost saving matters more than low query latency, one may want to use raw values and bloom filters only.

The following two examples show how to overwrite encoding type and index configs for tiers. Similar changes are also demonstrated in the .

  1. Overwriting single-column index configs using fieldConfigList. All top level fields in can be overwritten, and fields not overwritten are kept intact.

  1. Overwriting star-tree index configurations using tableIndexConfig. The StarTreeIndexConfigs is overwritten as a whole. In fact, all top level fields defined in can be overwritten, so single-column index configs defined in tableIndexConfig can also be overwritten but it's less clear than using fieldConfigList.

Credential

How do I update credentials for real-time upstream without downtime?

  1. .

  2. Wait for the pause status to change to success.

  3. Update the credential in the table config.

  4. Resume the consumption.

Ingest streaming data from Apache Kafka

This guide shows you how to ingest a stream of records from an Apache Kafka topic into a Pinot table.

Learn how to ingest data from Kafka, a stream processing platform. You should have a local cluster up and running, following the instructions in .

Install and Launch Kafka

Let's start by downloading Kafka to our local machine.

To pull down the latest Docker image, run the following command:

Download Kafka from and then extract it:

Next we'll spin up a Kafka broker:

Note: The --network pinot-demo flag is optional and assumes that you have a Docker network named pinot-demo that you want to connect the Kafka container to.

On one terminal window run this command:

Start Zookeeper

And on another window, run this command:

Start Kafka Broker

Data Source

We're going to generate some JSON messages from the terminal using the following script:

datagen.py

If you run this script (python datagen.py), you'll see the following output:

Ingesting Data into Kafka

Let's now pipe that stream of messages into Kafka, by running the following command:

We can check how many messages have been ingested by running the following command:

Output

And we can print out the messages themselves by running the following command

Output

Schema

A schema defines what fields are present in the table along with their data types in JSON format.

Create a file called /tmp/pinot/schema-stream.json and add the following content to it.

Table Config

A table is a logical abstraction that represents a collection of related data. It is composed of columns and rows (known as documents in Pinot). The table config defines the table's properties in JSON format.

Create a file called /tmp/pinot/table-config-stream.json and add the following content to it.

Create schema and table

Create the table and schema by running the appropriate command below:

Querying

Navigate to and click on the events table to run a query that shows the first 10 rows in this table.

Querying the events table

Kafka ingestion guidelines

Kafka versions in Pinot

Pinot supports two versions of the Kafka library: kafka-0.9 and kafka-2.x for low level consumers.

Post release 0.10.0, we have started shading kafka packages inside Pinot. If you are using our latest tagged docker images or master build, you should replace org.apache.kafka with shaded.org.apache.kafka in your table config.

Upgrade from Kafka 0.9 connector to Kafka 2.x connector

  • Update table config for low level consumer: stream.kafka.consumer.factory.class.name from org.apache.pinot.core.realtime.impl.kafka.KafkaConsumerFactory to org.apache.pinot.core.realtime.impl.kafka2.KafkaConsumerFactory.

Pinot does not support using high-level Kafka consumers (HLC). Pinot uses low-level consumers to ensure accurate results, supports operational complexity and scalability, and minimizes storage overhead.

How to consume from a Kafka version > 2.0.0

This connector is also suitable for Kafka lib version higher than 2.0.0. In , change the kafka.lib.version from 2.0.0 to 2.1.1 will make this Connector working with Kafka 2.1.1.

Kafka configurations in Pinot

Use Kafka partition (low) level consumer with SSL

Here is an example config which uses SSL based authentication to talk with kafka and schema-registry. Notice there are two sets of SSL options, ones starting with ssl. are for kafka consumer and ones with stream.kafka.decoder.prop.schema.registry. are for SchemaRegistryClient used by KafkaConfluentSchemaRegistryAvroMessageDecoder.

Consume transactionally-committed messages

The connector with Kafka library 2.0+ supports Kafka transactions. The transaction support is controlled by config kafka.isolation.level in Kafka stream config, which can be read_committed or read_uncommitted (default). Setting it to read_committed will ingest transactionally committed messages in Kafka stream only.

For example,

Note that the default value of this config read_uncommitted to read all messages. Also, this config supports low-level consumer only.

Use Kafka partition (low) level consumer with SASL_SSL

Here is an example config which uses SASL_SSL based authentication to talk with kafka and schema-registry. Notice there are two sets of SSL options, some for kafka consumer and ones with stream.kafka.decoder.prop.schema.registry. are for SchemaRegistryClient used by KafkaConfluentSchemaRegistryAvroMessageDecoder.

Extract record headers as Pinot table columns

Pinot's Kafka connector supports automatically extracting record headers and metadata into the Pinot table columns. The following table shows the mapping for record header/metadata to Pinot table column names:

Kafka Record
Pinot Table Column
Description

In order to enable the metadata extraction in a Kafka table, you can set the stream config metadata.populate to true.

In addition to this, if you want to use any of these columns in your table, you have to list them explicitly in your table's schema.

For example, if you want to add only the offset and key as dimension columns in your Pinot table, it can listed in the schema as follows:

Once the schema is updated, these columns are similar to any other pinot column. You can apply ingestion transforms and / or define indexes on them.

Remember to follow the when updating schema of an existing table!

Tell Pinot where to find an Avro schema

There is a standalone utility to generate the schema from an Avro file. See [infer the pinot schema from the avro schema and JSON data]() for details.

To avoid errors like The Avro schema must be provided, designate the location of the schema in your streamConfigs section. For example, if your current section contains the following:

Then add this key: "stream.kafka.decoder.prop.schema"followed by a value that denotes the location of your schema.

{ 
    "tableName": "pinotTable", 
    "tableType": "OFFLINE", 
    "segmentsConfig": {
      "replication": "3", 
      ... 
    }
    ..
{ 
    "tableName": "pinotTable", 
    "tableType": "REALTIME", 
    "segmentsConfig": {
      "replicasPerPartition": "3", 
      ... 
    }
    ..
"instanceAssignmentConfigMap": {
      "COMPLETED": {
        "tagPoolConfig": {
          "tag": "DefaultTenant_OFFLINE"
        },
        "replicaGroupPartitionConfig": {
        }
      }
    },
curl -X POST "{host}/segments/{tableNameWithType}/{segmentName}/reset"
cluster.tenant.isolation.enable=false
curl -X POST "http://localhost:9000/tenants" 
-H "accept: application/json" 
-H "Content-Type: application/json" 
-d "{\"tenantRole\":\"BROKER\",\"tenantName\":\"foo\",\"numberOfInstances\":1}"
Using "POST /cluster/configs API" on CLUSTER tab in Swagger, with this payload:
{
	"<taskType>.timeoutMs": "600000",
	"<taskType>.numConcurrentTasksPerInstance": "4"
}

p1

S0

S1

p2

S2

S3

p3

S0

S1

p4

S2

S3

p5

S0

S1

p6

S2

S3

{
    "tableName": "pinotTable", 
    "tableType": "REALTIME",
    "routing": {
        "instanceSelectorType": "replicaGroup"
    }
    ..
}
{
  ...
  "fieldConfigList": [    
    {
      "name": "ArrTimeBlk",
      "encodingType": "DICTIONARY",
      "indexes": {
        "inverted": {
          "enabled": "true"
        }
      },
      "tierOverwrites": {
        "hotTier": {
          "encodingType": "DICTIONARY",
          "indexes": { // change index types for this tier
            "bloom": {
              "enabled": "true"
            }
          }
        },
        "coldTier": {
          "encodingType": "RAW", // change encoding type for this tier
          "indexes": { } // remove all indexes
        }
      }
    }
  ],
  "tableIndexConfig": {
    "starTreeIndexConfigs": [
      {
        "dimensionsSplitOrder": [
          "AirlineID",
          "Origin",
          "Dest"
        ],
        "skipStarNodeCreationForDimensions": [],
        "functionColumnPairs": [
          "COUNT__*",
          "MAX__ArrDelay"
        ],
        "maxLeafRecords": 10
      }
    ],
...
    "tierOverwrites": {
      "hotTier": {
        "starTreeIndexConfigs": [ // create different STrTree index on this tier
          {
            "dimensionsSplitOrder": [
              "Carrier",
              "CancellationCode",
              "Origin",
              "Dest"
            ],
            "skipStarNodeCreationForDimensions": [],
            "functionColumnPairs": [
              "MAX__CarrierDelay",
              "AVG__CarrierDelay"
            ],
            "maxLeafRecords": 10
          }
        ]
      },
      "coldTier": {
        "starTreeIndexConfigs": [] // removes ST index for this tier
      }
    }
  },
 ...
make a pull request
segmentsConfig
replication
replicasPerPartition
table rebalance
segmentsConfig
Rebalance
Pause Stream Ingestion
Running a Periodic Task Manually
rebalance
routing
tiered storage
MultiDirQuickStart example
FieldConfig class
IndexingConfig class
Pause the stream ingestion

Batch Ingestion

Batch ingestion of data into Apache Pinot.

With batch ingestion you create a table using data already present in a file system such as S3. This is particularly useful when you want to use Pinot to query across large data with minimal latency or to test out new features using a simple data file.

To ingest data from a filesystem, perform the following steps, which are described in more detail in this page:

  1. Create schema configuration

  2. Create table configuration

  3. Upload schema and table configs

  4. Upload data

Batch ingestion currently supports the following mechanisms to upload the data:

  • Standalone

  • Hadoop

  • Spark

  • Flink

Here's an example using standalone local processing.

First, create a table using the following CSV data.

studentID,firstName,lastName,gender,subject,score,timestampInEpoch
200,Lucy,Smith,Female,Maths,3.8,1570863600000
200,Lucy,Smith,Female,English,3.5,1571036400000
201,Bob,King,Male,Maths,3.2,1571900400000
202,Nick,Young,Male,Physics,3.6,1572418800000

Create schema configuration

In our data, the only column on which aggregations can be performed is score. Secondly, timestampInEpoch is the only timestamp column. So, on our schema, we keep score as metric and timestampInEpoch as timestamp column.

{
  "schemaName": "transcript",
  "dimensionFieldSpecs": [
    {
      "name": "studentID",
      "dataType": "INT"
    },
    {
      "name": "firstName",
      "dataType": "STRING"
    },
    {
      "name": "lastName",
      "dataType": "STRING"
    },
    {
      "name": "gender",
      "dataType": "STRING"
    },
    {
      "name": "subject",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "score",
      "dataType": "FLOAT"
    }
  ],
  "dateTimeFieldSpecs": [{
    "name": "timestampInEpoch",
    "dataType": "LONG",
    "format" : "1:MILLISECONDS:EPOCH",
    "granularity": "1:MILLISECONDS"
  }]
}

Here, we have also defined two extra fields: format and granularity. The format specifies the formatting of our timestamp column in the data source. Currently, it's in milliseconds, so we've specified 1:MILLISECONDS:EPOCH.

Create table configuration

We define a table transcript and map the schema created in the previous step to the table. For batch data, we keep the tableType as OFFLINE.

{
  "tableName": "transcript",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "replication": 1,
    "timeColumnName": "timestampInEpoch",
    "timeType": "MILLISECONDS",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": 365
  },
  "tenants": {
    "broker":"DefaultTenant",
    "server":"DefaultTenant"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY"
    },
    "continueOnError": true,
    "rowTimeValueCheck": true,
    "segmentTimeValueCheck": false

  },
  "metadata": {}
}

Upload schema and table configs

Now that we have both the configs, upload them and create a table by running the following command:

bin/pinot-admin.sh AddTable \\
  -tableConfigFile /path/to/table-config.json \\
  -schemaFile /path/to/table-schema.json -exec

Check out the table config and schema in the \[Rest API] to make sure it was successfully uploaded.

Upload data

We now have an empty table in Pinot. Next, upload the CSV file to this empty table.

A table is composed of multiple segments. The segments can be created in the following three ways:

  • Minion based ingestion\

  • Upload API\

  • Ingestion jobs

Minion-based ingestion

Refer to SegmentGenerationAndPushTask

Upload API

There are 2 controller APIs that can be used for a quick ingestion test using a small file.

When these APIs are invoked, the controller has to download the file and build the segment locally.

Hence, these APIs are NOT meant for production environments and for large input files.

/ingestFromFile

This API creates a segment using the given file and pushes it to Pinot. All steps happen on the controller.

Example usage:

To upload a JSON file data.json to a table called foo_OFFLINE, use below command

Note that query params need to be URLEncoded. For example, {"inputFormat":"json"} in the command below needs to be converted to %7B%22inputFormat%22%3A%22json%22%7D.

curl -X POST -F [email protected] \
  -H "Content-Type: multipart/form-data" \
  "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
  batchConfigMapStr={"inputFormat":"json"}"

The batchConfigMapStr can be used to pass in additional properties needed for decoding the file. For example, in case of csv, you may need to provide the delimiter

curl -X POST -F [email protected] \
  -H "Content-Type: multipart/form-data" \
  "http://localhost:9000/ingestFromFile?tableNameWithType=foo_OFFLINE&
batchConfigMapStr={
  "inputFormat":"csv",
  "recordReader.prop.delimiter":"|"
}"

/ingestFromURI

This API creates a segment using file at the given URI and pushes it to Pinot. Properties to access the FS need to be provided in the batchConfigMap. All steps happen on the controller. Example usage:

curl -X POST "http://localhost:9000/ingestFromURI?tableNameWithType=foo_OFFLINE
&batchConfigMapStr={
  "inputFormat":"json",
  "input.fs.className":"org.apache.pinot.plugin.filesystem.S3PinotFS",
  "input.fs.prop.region":"us-central",
  "input.fs.prop.accessKey":"foo",
  "input.fs.prop.secretKey":"bar"
}
&sourceURIStr=s3://test.bucket/path/to/json/data/data.json"

Ingestion jobs

Segments can be created and uploaded using tasks known as DataIngestionJobs. A job also needs a config of its own. We call this config the JobSpec.

For our CSV file and table, the JobSpec should look like this:

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'

# Recommended to set jobType to SegmentCreationAndMetadataPush for production environment where Pinot Deep Store is configured  
jobType: SegmentCreationAndTarPush

inputDirURI: '/tmp/pinot-quick-start/rawdata/'
includeFileNamePattern: 'glob:**/*.csv'
outputDirURI: '/tmp/pinot-quick-start/segments/'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
  tableName: 'transcript'
pinotClusterSpecs:
  - controllerURI: 'http://localhost:9000'
pushJobSpec:
  pushAttempts: 2
  pushRetryIntervalMillis: 1000

For more detail, refer to Ingestion job spec.

Now that we have the job spec for our table transcript, we can trigger the job using the following command:

bin/pinot-admin.sh LaunchDataIngestionJob \\
    -jobSpecFile /tmp/pinot-quick-start/batch-job-spec.yaml

Once the job successfully finishes, head over to the \[query console] and start playing with the data.

Segment push job type

There are 3 ways to upload a Pinot segment:

  • Segment tar push

  • Segment URI push

  • Segment metadata push

Segment tar push

This is the original and default push mechanism.

Tar push requires the segment to be stored locally or can be opened as an InputStream on PinotFS. So we can stream the entire segment tar file to the controller.

The push job will:

  1. Upload the entire segment tar file to the Pinot controller.

Pinot controller will:

  1. Save the segment into the controller segment directory(Local or any PinotFS).

  2. Extract segment metadata.

  3. Add the segment to the table.

Segment URI push

This push mechanism requires the segment tar file stored on a deep store with a globally accessible segment tar URI.

URI push is light-weight on the client-side, and the controller side requires equivalent work as the tar push.

The push job will:

  1. POST this segment tar URI to the Pinot controller.

Pinot controller will:

  1. Download segment from the URI and save it to controller segment directory (local or any PinotFS).

  2. Extract segment metadata.

  3. Add the segment to the table.

Segment metadata push

This push mechanism also requires the segment tar file stored on a deep store with a globally accessible segment tar URI.

Metadata push is light-weight on the controller side, there is no deep store download involves from the controller side.

The push job will:

  1. Download the segment based on URI.

  2. Extract metadata.

  3. Upload metadata to the Pinot Controller.

Pinot Controller will:

  1. Add the segment to the table based on the metadata.

4. Segment Metadata Push with copyToDeepStore

This extends the original Segment Metadata Push for cases, where the segments are pushed to a location not used as deep store. The ingestion job can still do metadata push but ask Pinot Controller to copy the segments into deep store. Those use cases usually happen when the ingestion jobs don't have direct access to deep store but still want to use metadata push for its efficiency, thus using a staging location to keep the segments temporarily.

NOTE: the staging location and deep store have to use same storage scheme, like both on s3. This is because the copy is done via PinotFS.copyDir interface that assumes so; but also because this does copy at storage system side, so segments don't need to go through Pinot Controller at all.

To make this work, grant Pinot controllers access to the staging location. For example on AWS, this may require adding an access policy like this example for the controller EC2 instances:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::metadata-push-staging",
                "arn:aws:s3:::metadata-push-staging/*"
            ]
        }
    ]
}

Then use metadata push to add one extra config like this one:

...
jobType: SegmentCreationAndMetadataPush
...
outputDirURI: 's3://metadata-push-staging/stagingDir/'
...
pushJobSpec:
  copyToDeepStoreForMetadataPush: true
...

Consistent data push and rollback

Pinot supports atomic update on segment level, which means that when data consisting of multiple segments are pushed to a table, as segments are replaced one at a time, queries to the broker during this upload phase may produce inconsistent results due to interleaving of old and new data.

See Consistent Push and Rollback for how to enable this feature.

Segment fetchers

When Pinot segment files are created in external systems (Hadoop/spark/etc), there are several ways to push those data to the Pinot controller and server:

  1. Push segment to shared NFS and let pinot pull segment files from the location of that NFS. See Segment URI Push.

  2. Push segment to a Web server and let pinot pull segment files from the Web server with HTTP/HTTPS link. See Segment URI Push.

  3. Push segment to PinotFS(HDFS/S3/GCS/ADLS) and let pinot pull segment files from PinotFS URI. See Segment URI Push and Segment Metadata Push.

  4. Push segment to other systems and implement your own segment fetcher to pull data from those systems.

The first three options are supported out of the box within the Pinot package. As long your remote jobs send Pinot controller with the corresponding URI to the files, it will pick up the file and allocate it to proper Pinot servers and brokers. To enable Pinot support for PinotFS, you'll need to provide PinotFS configuration and proper Hadoop dependencies.

Persistence

By default, Pinot does not come with a storage layer, so all the data sent, won't be stored in case of a system crash. In order to persistently store the generated segments, you will need to change controller and server configs to add deep storage. Checkout File systems for all the info and related configs.

Tuning

Standalone

Since pinot is written in Java, you can set the following basic Java configurations to tune the segment runner job -

  • Log4j2 file location with -Dlog4j2.configurationFile

  • Plugin directory location with -Dplugins.dir=/opt/pinot/plugins

  • JVM props, like -Xmx8g -Xms4G

If you are using the docker, you can set the following under JAVA_OPTS variable.

Hadoop

You can set -D mapreduce.map.memory.mb=8192 to set the mapper memory size when submitting the Hadoop job.

Spark

You can add config spark.executor.memory to tune the memory usage for segment creation when submitting the Spark job.

Input formats

This section contains a collection of guides that will show you how to import data from a Pinot-supported input format.

Pinot offers support for various popular input formats during ingestion. By changing the input format, you can reduce the time spent doing serialization-deserialization and speed up the ingestion.

Configuring input formats

To change the input format, adjust the recordReaderSpec config in the ingestion job specification.

recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
  configs: 
			key1 : 'value1'
			key2 : 'value2'

The configuration consists of the following keys:

  • dataFormat: Name of the data format to consume.

  • className: Name of the class that implements the RecordReader interface. This class is used for parsing the data.

  • configClassName: Name of the class that implements the RecordReaderConfig interface. This class is used the parse the values mentioned in configs

  • configs: Key-value pair for format-specific configurations. This field is optional.

Supported input formats

Pinot supports multiple input formats out of the box. Specify the corresponding readers and the associated custom configurations to switch between formats.

CSV

dataFormat: 'csv'
className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
configs:
	fileFormat: 'default' #should be one of default, rfc4180, excel, tdf, mysql
	header: 'columnName separated by delimiter'
  delimiter: ','
  multiValueDelimiter: '-'

CSV Record Reader supports the following configs:

  • fileFormat: default, rfc4180, excel, tdf, mysql

  • header: Header of the file. The columnNames should be separated by the delimiter mentioned in the configuration.

  • delimiter: The character seperating the columns.

  • multiValueDelimiter: The character separating multiple values in a single column. This can be used to split a column into a list.

  • skipHeader: Skip header record in the file. Boolean.

  • ignoreEmptyLines: Ignore empty lines (instead of filling them with default values). Boolean.

  • ignoreSurroundingSpaces: ignore spaces around column names and values. Boolean

  • quoteCharacter: Single character used for quotes in CSV files.

  • recordSeparator: Character used to separate records in the input file. Default is or \r depending on the platform.

  • nullStringValue: String value that represents null in CSV files. Default is empty string.

  • skipUnParseableLines : Skip lines that cannot be parsed. Note that this would result in data loss. Boolean.

Your CSV file may have raw text fields that cannot be reliably delimited using any character. In this case, explicitly set the multiValueDelimeter field to empty in the ingestion config. multiValueDelimiter: ''

Avro

dataFormat: 'avro'
className: 'org.apache.pinot.plugin.inputformat.avro.AvroRecordReader'
configs:
    enableLogicalTypes: true

The Avro record reader converts the data in file to a GenericRecord. A Java class or .avro file is not required. By default, the Avro record reader only supports primitive types. To enable support for rest of the Avro data types, set enableLogicalTypes to true .

We use the following conversion table to translate between Avro and Pinot data types. The conversions are done using the offical Avro methods present in org.apache.avro.Conversions.

Avro Data Type
Pinot Data Type
Comment

INT

INT

LONG

LONG

FLOAT

FLOAT

DOUBLE

DOUBLE

BOOLEAN

BOOLEAN

STRING

STRING

ENUM

STRING

BYTES

BYTES

FIXED

BYTES

MAP

JSON

ARRAY

JSON

RECORD

JSON

UNION

JSON

DECIMAL

BYTES

UUID

STRING

DATE

STRING

yyyy-MM-dd format

TIME_MILLIS

STRING

HH:mm:ss.SSS format

TIME_MICROS

STRING

HH:mm:ss.SSSSSS format

TIMESTAMP_MILLIS

TIMESTAMP

TIMESTAMP_MICROS

TIMESTAMP

JSON

dataFormat: 'json'
className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'

Thrift

dataFormat: 'thrift'
className: 'org.apache.pinot.plugin.inputformat.thrift.ThriftRecordReader'
configs:
	thriftClass: 'ParserClassName'

Thrift requires the generated class using .thrift file to parse the data. The .class file should be available in the Pinot's classpath. You can put the files in the lib/ folder of Pinot distribution directory.

Parquet

dataFormat: 'parquet'
className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'

Since 0.11.0 release, the Parquet record reader determines whether to use ParquetAvroRecordReader or ParquetNativeRecordReader to read records. The reader looks for the parquet.avro.schema or avro.schema key in the parquet file footer, and if present, uses the Avro reader.

You can change the record reader manually in case of a misconfiguration.

dataFormat: 'parquet'
className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetNativeRecordReader'

For the support of DECIMAL and other parquet native data types, always use ParquetNativeRecordReader.

INT96

LONG

ParquetINT96 type converts nanoseconds

to Pinot INT64 type of milliseconds

INT64

LONG

INT32

INT

FLOAT

FLOAT

DOUBLE

DOUBLE

BINARY

BYTES

FIXED-LEN-BYTE-ARRAY

BYTES

DECIMAL

DOUBLE

ENUM

STRING

UTF8

STRING

REPEATED

MULTIVALUE/MAP (represented as MV

if parquet original type is LIST, then it is converted to MULTIVALUE column otherwise a MAP column.

For ParquetAvroRecordReader , you can refer to the Avro section above for the type conversions.

ORC

dataFormat: 'orc'
className: 'org.apache.pinot.plugin.inputformat.orc.ORCRecordReader'

ORC record reader supports the following data types -

ORC Data Type
Java Data Type

BOOLEAN

String

SHORT

Integer

INT

Integer

LONG

Integer

FLOAT

Float

DOUBLE

Double

STRING

String

VARCHAR

String

CHAR

String

LIST

Object[]

MAP

Map<Object, Object>

DATE

Long

TIMESTAMP

Long

BINARY

byte[]

BYTE

Integer

In LIST and MAP types, the object should only belong to one of the data types supported by Pinot.

Protocol Buffers

dataFormat: 'proto'
className: 'org.apache.pinot.plugin.inputformat.protobuf.ProtoBufRecordReader'
configs:
	descriptorFile: 'file:///path/to/sample.desc'

The reader requires a descriptor file to deserialize the data present in the files. You can generate the descriptor file (.desc) from the .proto file using the command -

protoc --include_imports --descriptor_set_out=/absolute/path/to/output.desc /absolute/path/to/input.proto

Indexing

This page describes the indexing techniques available in Apache Pinot

Apache Pinot™ supports the following indexing techniques:

  • Bloom filter

  • Forward index

    • Dictionary-encoded forward index with bit compression

    • Raw value forward index

    • Sorted forward index with run-length encoding

  • FST index

  • Geospatial

  • Inverted index

    • Bitmap inverted index

    • Sorted inverted index

  • JSON index

  • Range index

  • Star-tree index

  • Text search support

  • Timestamp index

By default, Pinot creates a dictionary-encoded forward index for each column.

Enabling indexes

There are two ways to enable indexes for a Pinot table.

As part of ingestion, during Pinot segment generation

Indexing is enabled by specifying the column names in the table configuration. More details about how to configure each type of index can be found in the respective index's section linked above or in the table configuration reference.

Dynamically added or removed

Indexes can also be dynamically added to or removed from segments at any point. Update your table configuration with the latest set of indexes you want to have.

For example, if you have an inverted index on the foo field and now want to also include the bar field, you would update your table configuration from this:

"tableIndexConfig": {
        "invertedIndexColumns": ["foo"],
        ...
    }

To this:

"tableIndexConfig": {
        "invertedIndexColumns": ["foo", "bar"],
        ...
    }

The updated index configuration won't be picked up unless you invoke the reload API. This API sends reload messages via Helix to all servers, as part of which indexes are added or removed from the local segments. This happens without any downtime and is completely transparent to the queries.

When adding an index, only the new index is created and appended to the existing segment. When removing an index, its related states are cleaned up from Pinot servers. You can find this API under the Segments tab on Swagger:

curl -X POST \
  "http://localhost:9000/segments/myTable/reload" \
  -H "accept: application/json"

You can also find this action on the Cluster Manager in the Pinot UI, on the specific table's page.

Not all indexes can be retrospectively applied to existing segments. For more detailed documentation on applying indexes, see the Indexing FAQ.

Tuning Index

The inverted index provides good performance for most use cases, especially if your use case doesn't have a strict low latency requirement.

You should start by using this, and if your queries aren't fast enough, switch to advanced indices like the sorted or star-tree index.

Data and Index types compatibility matrix

Matrix below show which combinations of data types, cardinality and encoding are compatible with various index types:

data type
bloom
fst
geo
inverted
json
native
text
range
startree
timestamp
vector

boolean

❌

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

int

🆗

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

long

🆗

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

float

🆗

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

🆗 (5)

double

🆗

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

big decimal

❌

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

timestamp

❌

❌

❌

🆗

❌

❌

❌

🆗

🆗 (2)

🆗

❌

string

🆗

🆗 (1)

❌

🆗

🆗 (2) (4)

🆗

🆗

🆗

🆗 (2)

🆗 (3)

❌

json

❌

❌

❌

🆗

🆗 (2)

🆗

🆗

🆗

🆗 (2)

❌

❌

bytes

🆗

❌

🆗 (2)

🆗

❌

❌

❌

🆗

🆗 (2)

❌

❌

map

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

❌

(1) Supports only dictionary-encoded columns.

(2) Supports only single value columns.

(3) Supported only if values can be parsed as long.

(4) Supported only if values can be parsed as json.

(5) Supports only multi value columns.

0.8.0

This release introduced several new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins.

Summary

This release introduced several awesome new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins (AWS Kinesis, Apache Pulsar). It contains a lot of query enhancements such as new timestamp and boolean type support and flexible numerical column comparison. It also includes many key bug fixes. See details below.

The release was cut from the following commit: fe83e95aa9124ee59787c580846793ff7456eaa5

and the following cherry-picks:

  • 668b5e0

  • ee887b9

  • c2f7fcc

  • c1ac8a1

  • 4da1dae

  • 573651b

  • c6c407d

  • 0d96c7f

  • c2637d1

Notable New Features

  • Extract time handling for SegmentProcessorFramework (#7158)

  • Add Apache Pulsar low level and high level connector (#7026)

  • Enable parallel builds for compat checker (#7149)

  • Add controller/server API to fetch aggregated segment metadata (#7102)

  • Support Dictionary Based Plan For DISTINCT (#7141)

  • Provide HTTP client to kinesis builder (#7148)

  • Add datetime function with 2 arguments (#7116)

  • Adding ability to check ingestion status for Offline Pinot table (#7070)

  • Add timestamp datatype support in JDBC (#7117)

  • Allow updating controller and broker helix hostname (#7064)

  • Cancel running Kinesis consumer tasks when timeout occurs (#7109)

  • Implement Append merger for partial upsert (#7087)

    `* SegmentProcessorFramework Enhancement (#7092)

  • Added TaskMetricsEmitted periodic controler job (#7091)

  • Support json path expressions in query. (#6998)

  • Support data preprocessing for AVRO and ORC formats (#7062)

  • Add partial upsert config and mergers (#6899)

  • Add support for range index rule recommendation(#7034) (#7063)

  • Allow reloading consuming segment by default (#7078)

  • Add LZ4 Compression Codec (#6804) ([#7035](https://github.com/apache/pinot/pull/7035

    ))

  • Make Pinot JDK 11 Compilable (#6424\

  • Introduce in-Segment Trim for GroupBy OrderBy Query (#6991)

  • Produce GenericRow file in segment processing mapper (#7013)

  • Add ago() scalar transform function (#6820)

  • Add Bloom Filter support for IN predicate(#7005) (#7007)

  • Add genericRow file reader and writer (#6997)

  • Normalize LHS and RHS numerical types for >, >=, <, and <= operators. (#6927)

  • Add Kinesis Stream Ingestion Plugin (#6661)

  • feature/#6766 JSON and Startree index information in API (#6873)

  • Support null value fields in generic row ser/de (#6968)

  • Implement PassThroughTransformOperator to optimize select queries(#6972) (#6973)

  • Optimize TIME_CONVERT/DATE_TIME_CONVERT predicates (#6957)

  • Prefetch call to fetch buffers of columns seen in the query (#6967)

  • Enabling compatibility tests in the script (#6959)

  • Add collectionToJsonMode to schema inference (#6946)

  • Add the complex-type support to decoder/reader (#6945)

  • Adding a new Controller API to retrieve ingestion status for real-time… (#6890)

  • Add support for Long in Modulo partition function. (#6929)

  • Enhance PinotSegmentRecordReader to preserve null values (#6922)

  • add complex-type support to avro-to-pinot schema inference (#6928)

  • Add correct yaml files for real-time data(#6787) (#6916)

  • Add complex-type transformation to offline segment creation (#6914)

  • Add config File support(#6787) (#6901)

  • Enhance JSON index to support nested array (#6877)

  • Add debug endpoint for tables. (#6897)

  • JSON column datatype support. (#6878)

  • Allow empty string in MV column (#6879)

  • Add Zstandard compression support with JMH benchmarking(#6804) (#6876)

  • Normalize LHS and RHS numerical types for = and != operator. (#6811)

  • Change ConcatCollector implementation to use off-heap (#6847)

  • [PQL Deprecation] Clean up the old BrokerRequestOptimizer (#6859)

  • [PQL Deprecation] Do not compile PQL broker request for SQL query (#6855)

  • Add TIMESTAMP and BOOLEAN data type support (#6719)

  • Add admin endpoint for Pinot Minon. (#6822)

  • Remove the usage of PQL compiler (#6808)

  • Add endpoints in Pinot Controller, Broker and Server to get system and application configs. (#6817)

  • Support IN predicate in ColumnValue SegmentPruner(#6756) (#6776)

  • Enable adding new segments to a upsert-enabled real-time table (#6567)

  • Interface changes for Kinesis connector (#6667)

  • Pinot Minion SegmentGenerationAndPush task: PinotFS configs inside taskSpec is always temporary and has higher priority than default PinotFS created by the minion server configs (#6744)

  • DataTable V3 implementation and measure data table serialization cost on server (#6710)

  • add uploadLLCSegment endpoint in TableResource (#6653)

  • File-based SegmentWriter implementation (#6718)

  • Basic Auth for pinot-controller (#6613)

  • UI integration with Authentication API and added login page (#6686)

  • Support data ingestion for offline segment in one pass (#6479)

  • SumPrecision: support all data types and star-tree (#6668)

  • complete compatibility regression testing (#6650)

  • Kinesis implementation Part 1: Rename partitionId to partitionGroupId (#6655)

  • Make Pinot metrics pluggable (#6640)

  • Recover the segment from controller when LLC table cannot load it (#6647)

  • Adding a new API for validating specified TableConfig and Schema (#6620)

  • Introduce a metric for query/response size on broker. (#6590)

  • Adding a controller periodic task to clean up dead minion instances (#6543)

  • Adding new validation for Json, TEXT indexing (#6541)

  • Always return a response from query execution. (#6596)

Special notes

  • After the 0.8.0 release, we will officially support jdk 11, and can now safely start to use jdk 11 features. Code is still compilable with jdk 8 (#6424)

  • RealtimeToOfflineSegmentsTask config has some backward incompatible changes (#7158)

    — timeColumnTransformFunction is removed (backward-incompatible, but rollup is not supported anyway)

    — Deprecate collectorType and replace it with mergeType

    — Add roundBucketTimePeriod and partitionBucketTimePeriod to config the time bucket for round and partition

  • Regex path for pluggable MinionEventObserverFactory is changed from org.apache.pinot.*.event.* to org.apache.pinot.*.plugin.minion.tasks.* (#6980)

  • Moved all pinot built-in minion tasks to the pinot-minion-builtin-tasks module and package them into a shaded jar (#6618)

  • Reloading consuming segment flag pinot.server.instance.reload.consumingSegment will be true by default (#7078)

  • Move JSON decoder from pinot-kafka to pinot-json package. (#7021)

  • Backward incompatible schema change through controller rest API PUT /schemas/{schemaName} will be blocked. (#6737)

  • Deprecated /tables/validateTableAndSchema in favor of the new configs/validate API and introduced new APIs for /tableConfigs to operate on the real-time table config, offline table config and schema in one shot. (#6840)

Major Bug fixes

  • Fix race condition in MinionInstancesCleanupTask (#7122)

  • Fix custom instance id for controller/broker/minion (#7127)

  • Fix UpsertConfig JSON deserialization. (#7125)

  • Fix the memory issue for selection query with large limit (#7112)

  • Fix the deleted segments directory not exist warning (#7097)

  • Fixing docker build scripts by providing JDK_VERSION as parameter (#7095)

  • Misc fixes for json data type (#7057)

  • Fix handling of date time columns in query recommender(#7018) (#7031)

  • fixing pinot-hadoop and pinot-spark test (#7030)

  • Fixing HadoopPinotFS listFiles method to always contain scheme (#7027)

  • fixed GenericRow compare for different _fieldToValueMap size (#6964)

  • Fix NPE in NumericalFilterOptimizer due to IS NULL and IS NOT NULL operator. (#7001)

  • Fix the race condition in real-time text index refresh thread (#6858) (#6990)

  • Fix deep store directory structure (#6976)

  • Fix NPE issue when consumed kafka message is null or the record value is null. (#6950)

  • Mitigate calcite NPE bug. (#6908)

  • Fix the exception thrown in the case that a specified table name does not exist (#6328) (#6765)

  • Fix CAST transform function for chained transforms (#6941)

  • Fixed failing pinot-controller npm build (#6795)

helm repo add pinot https://raw.githubusercontent.com/apache/pinot/master/helm
kubectl create ns pinot-quickstart
helm install pinot pinot/pinot \
    -n pinot-quickstart \
    --set cluster.name=pinot \
    --set server.replicaCount=2
helm dependency update
kubectl create ns pinot-quickstart
helm install -n pinot-quickstart pinot ./pinot
kubectl get all -n pinot-quickstart
helm repo add kafka https://charts.bitnami.com/bitnami
helm install -n pinot-quickstart kafka kafka/kafka --set replicas=1,zookeeper.image.tag=latest,listeners.client.protocol=PLAINTEXT
kubectl get all -n pinot-quickstart | grep kafka
pod/kafka-controller-0                   1/1     Running     0          2m
pod/kafka-controller-1                   1/1     Running     0          2m
pod/kafka-controller-2                   1/1     Running     0          2m
kubectl -n pinot-quickstart exec kafka-controller-0 -- kafka-topics.sh --bootstrap-server kafka:9092 --topic flights-realtime --create --partitions 1 --replication-factor 1
kubectl -n pinot-quickstart exec kafka-controller-0 -- kafka-topics.sh --bootstrap-server kafka:9092 --topic flights-realtime-avro --create --partitions 1 --replication-factor 1
kubectl apply -f pinot/helm/pinot/pinot-realtime-quickstart.yml
./query-pinot-data.sh
helm repo add superset https://apache.github.io/superset
helm inspect values superset/superset > /tmp/superset-values.yaml
kubectl create ns superset
helm upgrade --install --values /tmp/superset-values.yaml superset superset/superset -n superset
kubectl get all -n superset
kubectl port-forward service/superset 18088:8088 -n superset
helm repo add trino https://trinodb.github.io/charts/
helm search repo trino
helm inspect values trino/trino > /tmp/trino-values.yaml
additionalCatalogs:
  pinot: |
    connector.name=pinot
    pinot.controller-urls=pinot-controller.pinot-quickstart:9000
kubectl create ns trino-quickstart
helm install my-trino trino/trino --version 0.2.0 -n trino-quickstart --values /tmp/trino-values.yaml
kubectl get pods -n trino-quickstart
curl -L https://repo1.maven.org/maven2/io/trino/trino-cli/363/trino-cli-363-executable.jar -o /tmp/trino && chmod +x /tmp/trino
echo "Visit http://127.0.0.1:18080 to use your application"
kubectl port-forward service/my-trino 18080:8080 -n trino-quickstart
/tmp/trino --server localhost:18080 --catalog pinot --schema default
trino:default> show catalogs;
  Catalog
---------
 pinot
 system
 tpcds
 tpch
(4 rows)

Query 20211025_010256_00002_mxcvx, FINISHED, 2 nodes
Splits: 36 total, 36 done (100.00%)
0.70 [0 rows, 0B] [0 rows/s, 0B/s]
trino:default> show tables;
    Table
--------------
 airlinestats
(1 row)

Query 20211025_010326_00003_mxcvx, FINISHED, 3 nodes
Splits: 36 total, 36 done (100.00%)
0.28 [1 rows, 29B] [3 rows/s, 104B/s]
trino:default> DESCRIBE airlinestats;
        Column        |      Type      | Extra | Comment
----------------------+----------------+-------+---------
 flightnum            | integer        |       |
 origin               | varchar        |       |
 quarter              | integer        |       |
 lateaircraftdelay    | integer        |       |
 divactualelapsedtime | integer        |       |
 divwheelsons         | array(integer) |       |
 divwheelsoffs        | array(integer) |       |
......

Query 20211025_010414_00006_mxcvx, FINISHED, 3 nodes
Splits: 36 total, 36 done (100.00%)
0.37 [79 rows, 5.96KB] [212 rows/s, 16KB/s]
trino:default> select count(*) as cnt from airlinestats limit 10;
 cnt
------
 9746
(1 row)

Query 20211025_015607_00009_mxcvx, FINISHED, 2 nodes
Splits: 17 total, 17 done (100.00%)
0.24 [1 rows, 9B] [4 rows/s, 38B/s]
helm install presto pinot/presto -n pinot-quickstart
kubectl apply -f presto-coordinator.yaml
helm inspect values pinot/presto > /tmp/presto-values.yaml
helm install presto pinot/presto -n pinot-quickstart --values /tmp/presto-values.yaml
kubectl get pods -n pinot-quickstart
./pinot-presto-cli.sh
curl -L https://repo1.maven.org/maven2/com/facebook/presto/presto-cli/0.246/presto-cli-0.246-executable.jar -o /tmp/presto-cli && chmod +x /tmp/presto-cli
kubectl port-forward service/presto-coordinator 18080:8080 -n pinot-quickstart> /dev/null &
/tmp/presto-cli --server localhost:18080 --catalog pinot --schema default
presto:default> show catalogs;
 Catalog
---------
 pinot
 system
(2 rows)

Query 20191112_050827_00003_xkm4g, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0:01 [0 rows, 0B] [0 rows/s, 0B/s]
presto:default> show tables;
    Table
--------------
 airlinestats
(1 row)

Query 20191112_050907_00004_xkm4g, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0:01 [1 rows, 29B] [1 rows/s, 41B/s]
presto:default> DESCRIBE pinot.dontcare.airlinestats;
        Column        |  Type   | Extra | Comment
----------------------+---------+-------+---------
 flightnum            | integer |       |
 origin               | varchar |       |
 quarter              | integer |       |
 lateaircraftdelay    | integer |       |
 divactualelapsedtime | integer |       |
......

Query 20191112_051021_00005_xkm4g, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0:02 [80 rows, 6.06KB] [35 rows/s, 2.66KB/s]
presto:default> select count(*) as cnt from pinot.dontcare.airlinestats limit 10;
 cnt
------
 9745
(1 row)

Query 20191112_051114_00006_xkm4g, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [1 rows, 8B] [2 rows/s, 19B/s]
kubectl delete ns pinot-quickstart
Enable Kubernetes on Docker-Desktop
Install Minikube for local setup
Set up a Kubernetes Cluster using Amazon Elastic Kubernetes Service (Amazon EKS)
Set up a Kubernetes Cluster using Google Kubernetes Engine (GKE)
Set up a Kubernetes Cluster using Azure Kubernetes Service (AKS)
open source project on GitHub
here
here
Sample Output of K8s Deployment Status
# checkout pinot
git clone https://github.com/apache/pinot.git
cd pinot/helm/pinot
import datetime
import uuid
import random
import json

while True:
    ts = int(datetime.datetime.now().timestamp()* 1000)
    id = str(uuid.uuid4())
    count = random.randint(0, 1000)
    print(
        json.dumps({"ts": ts, "uuid": id, "count": count})
    )
{"ts": 1644586485807, "uuid": "93633f7c01d54453a144", "count": 807}
{"ts": 1644586485836, "uuid": "87ebf97feead4e848a2e", "count": 41}
{"ts": 1644586485866, "uuid": "960d4ffa201a4425bb18", "count": 146}
python datagen.py | docker exec -i kafka /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic events;
python datagen.py | bin/kafka-console-producer.sh --bootstrap-server localhost:9092  --topic events;
docker exec -i kafka kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic events
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic events
events:0:11940
docker exec -i kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic events
...
{"ts": 1644586485807, "uuid": "93633f7c01d54453a144", "count": 807}
{"ts": 1644586485836, "uuid": "87ebf97feead4e848a2e", "count": 41}
{"ts": 1644586485866, "uuid": "960d4ffa201a4425bb18", "count": 146}
...
{
  "schemaName": "events",
  "dimensionFieldSpecs": [
    {
      "name": "uuid",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "count",
      "dataType": "INT"
    }
  ],
  "dateTimeFieldSpecs": [{
    "name": "ts",
    "dataType": "TIMESTAMP",
    "format" : "1:MILLISECONDS:EPOCH",
    "granularity": "1:MILLISECONDS"
  }]
}
{
  "tableName": "events",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "ts",
    "schemaName": "events",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "events",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "kafka:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
docker run --rm -ti  --network=pinot-demo  -v /tmp/pinot:/tmp/pinot  apachepinot/pinot:1.0.0 AddTable  -schemaFile /tmp/pinot/schema-stream.json  -tableConfigFile /tmp/pinot/table-config-stream.json  -controllerHost pinot-controller  -controllerPort 9000 -exec
bin/pinot-admin.sh AddTable -schemaFile /tmp/pinot/schema-stream.json -tableConfigFile /tmp/pinot/table-config-stream.json
  {
    "tableName": "transcript",
    "tableType": "REALTIME",
    "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
    },
    "tenants": {},
    "tableIndexConfig": {
      "loadMode": "MMAP",
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "LowLevel",
        "stream.kafka.topic.name": "transcript-topic",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.zk.broker.url": "pinot-zookeeper:2191/kafka",
        "stream.kafka.broker.list": "localhost:9092",
        "schema.registry.url": "",
        "security.protocol": "SSL",
        "ssl.truststore.location": "",
        "ssl.keystore.location": "",
        "ssl.truststore.password": "",
        "ssl.keystore.password": "",
        "ssl.key.password": "",
        "stream.kafka.decoder.prop.schema.registry.rest.url": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.truststore.location": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.keystore.location": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.truststore.password": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.keystore.password": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.keystore.type": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.truststore.type": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.key.password": "",
        "stream.kafka.decoder.prop.schema.registry.ssl.protocol": ""
      }
    },
    "metadata": {
      "customConfigs": {}
    }
  }
  {
    "tableName": "transcript",
    "tableType": "REALTIME",
    "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
    },
    "tenants": {},
    "tableIndexConfig": {
      "loadMode": "MMAP",
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "LowLevel",
        "stream.kafka.topic.name": "transcript-topic",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.zk.broker.url": "pinot-zookeeper:2191/kafka",
        "stream.kafka.broker.list": "kafka:9092",
        "stream.kafka.isolation.level": "read_committed"
      }
    },
    "metadata": {
      "customConfigs": {}
    }
  }
"streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.topic.name": "mytopic",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "kafka:9092",
        "stream.kafka.schema.registry.url": "https://xxx",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
        "stream.kafka.decoder.prop.schema.registry.rest.url": "https://xxx",
        "stream.kafka.decoder.prop.basic.auth.credentials.source": "USER_INFO",
        "stream.kafka.decoder.prop.schema.registry.basic.auth.user.info": "schema_registry_username:schema_registry_password",
        "sasl.mechanism": "PLAIN" ,
        "security.protocol": "SASL_SSL" ,
        "sasl.jaas.config":"org.apache.kafka.common.security.scram.ScramLoginModule required username=\"kafkausername\" password=\"kafkapassword\";",
        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "24h",
        "realtime.segment.flush.autotune.initialRows": "3000000",
        "realtime.segment.flush.threshold.segment.size": "500M"
      },

Record key: any type <K>

__key : String

For simplicity of design, we assume that the record key is always a UTF-8 encoded String

Record Headers: Map<String, String>

Each header key is listed as a separate column: __header$HeaderKeyName : String

For simplicity of design, we directly map the string headers from kafka record to pinot table column

Record metadata - offset : long

__metadata$offset : String

Record metadata - partition : int

__metadata$partition : String

Record metadata - recordTimestamp : long

__metadata$recordTimestamp : String

  "dimensionFieldSpecs": [
    {
      "name": "__key",
      "dataType": "STRING"
    },
    {
      "name": "__metadata$offset",
      "dataType": "STRING"
    },
    {
      "name": "__metadata$partition",
      "dataType": "STRING"
    },
    ...
  ],
...
"streamConfigs": {
  "streamType": "kafka",
  "stream.kafka.consumer.type": "lowlevel",
  "stream.kafka.topic.name": "",
  "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder",
  "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
  "stream.kafka.broker.list": "",
  "stream.kafka.consumer.prop.auto.offset.reset": "largest"
  ...
}
Set up a cluster
kafka.apache.org/quickstart#quickstart_download
localhost:9000/#/query
Kafka 2.0 connector pom.xml
schema evolution guidelines
https://docs.pinot.apache.org/basics/data-import/complex-type#infer-the-pinot-schema-from-the-avro-schema-and-json-data
docker pull wurstmeister/kafka:latest
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
docker run --network pinot-demo --name=kafka -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181/kafka -e KAFKA_BROKER_ID=0 -e KAFKA_ADVERTISED_HOST_NAME=kafka wurstmeister/kafka:latest
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Forward index

The forward index is the mechanism Pinot employs to store the values of each column. At a conceptual level, the forward index can be thought of as a mapping from document IDs (also known as row indices) to the actual column values of each row.

Forward indexes are enabled by default, meaning that columns will have a forward index unless explicitly disabled. Disabling the forward index can save storage space when other indexes sufficiently cover the required data patterns. For information on how to disable the forward index and its implications, refer to .

Dictionary encoded vs raw value

How forward indexes are implemented depends on the index encoding and whether the column is sorted.

When the encoding is set to RAW, the forward index is implemented as an array, where the indices correspond to document IDs and the values represent the actual row values. For more details, refer to the section.

In the case of DICTIONARY encoding, the forward index doesn't store the actual row values but instead stores dictionary IDs. This introduces an additional level of indirection when reading values, but it allows for more efficient physical layouts when unique number of values in the column is significantly smaller than the number of rows.

The DICTIONARY encoding can be even more efficient if the segment is sorted by the indexed column. You can learn more about the and the in their respective sections.

When working out whether a column should use dictionary encoded or raw value encoding, the following comparison table may help:

Dictionary
Raw Value

Dictionary-encoded forward index with bit compression (default)

In this approach, each unique value in a column is assigned an ID, and a dictionary is constructed to map these IDs back to their corresponding values. Instead of storing the actual values, the default forward index stores these bit-compressed IDs. This method is particularly effective when dealing with columns containing few unique values, as it significantly improves space efficiency.

The below diagram shows the dictionary encoding for two columns with integer and string types. ForcolA, dictionary encoding saved a significant amount of space for duplicated values.

The diagram below illustrates dictionary encoding for two columns with different data types (integer and string). For colA, dictionary encoding leads to significant space savings due to duplicated values. However, for colB, which contains mostly unique values, the compression effect is limited, and padding overhead may be high.

To know more about dictionary encoding, see .

When using the dictionary-encoded forward index for multi-value column, to further compress the forward index for repeated multi-value entires, enable the MV_ENTRY_DICT compression type which adds another level of dictionary encoding on the multi-value entries. This may be useful, for example, in cases where you pre-join a fact table with dimension table, where the multi-value entries in the dimension table are repeated after joining with the fact table.

It can be enabled with parameter:

Parameter
Default
Description

Sorted forward index with run-length encoding

When a column is physically sorted, Pinot employs a sorted forward index with run-length encoding, which builds upon dictionary encoding. Instead of storing dictionary IDs for each document ID, this approach stores pairs of start and end document IDs for each unique value.

(For simplicity, this diagram does not include the dictionary encoding layer.)

Sorted forward indexes offer the benefits of efficient compression and data locality and can also serve as an inverted index. They are active when two conditions are met: the segment is sorted by the column, and the dictionary is enabled for that column. Refer to the for details on enabling the dictionary.

When dealing with multiple segments, it's crucial to ensure that data is sorted within each segment. Sorting across segments is not necessary.

To guarantee that a segment is sorted by a particular column, follow these steps:

  • For real-time tables, use the tableIndexConfig.sortedColumn property. If there is exactly one column specified in that array, Pinot will sort the segment by that column upon committing.

  • For offline tables, you must pre-sort the data by the specified column before ingesting it into Pinot.

It's crucial to note that for offline tables, the tableIndexConfig.sortedColumn property is indeed ignored.

Additionally, for online tables, even though this property is specified as a JSON array, at most one column should be included. Using an array with more than one column is incorrect and will not result in segments being sorted by all the columns listed in the array.

When a real-time segment is committed, rows will be sorted by the sorting column and it will be transformed into an offline segment.

During the creation of an offline segment, which also applies when a real-time segment is committed, Pinot scans the data in each column. If it detects that all values within a column are sorted in ascending order, Pinot concludes that the segment is sorted based on that particular column. In case this happens on more than one column, all of them are considered as sorting columns. Consequently, whether a segment is sorted by a column or not solely depends on the actual data distribution within the segment and entirely disregards the value of the sortedColumn property. This approach also implies that two segments belonging to the same table may have a different number of sorting columns. In the extreme scenario where a segment contains only one row, Pinot will consider all columns within that segment as sorting columns.

Here is an example of a table configuration that illustrates these concepts:

Checking sort status

You can check the sorted status of a column in a segment by running the following:

Alternatively, for offline tables and for committed segments in real-time tables, you can retrieve the sorted status from the getServerMetadata endpoint. The following example is based on the :

Raw value forward index

The raw value forward index stores actual values instead of IDs. This means that it eliminates the need for dictionary lookups when fetching values, which can result in improved query performance. Raw forward index is particularly effective for columns with a large number of unique values, where dictionary encoding doesn't provide significant compression benefits.

As shown in the diagram below, dictionary encoding can lead to numerous random memory accesses for dictionary lookups. In contrast, the raw value forward index allows for sequential value scanning, which can enhance query performance when applied appropriately.

Note: Raw value forward index currently does not support inverted index (all others JSON/TEXT/Range/etc are supported). Also, since reading a value from this index requires reading the entire chunk in memory and decompressing, it is not suitable for heavy random reads.

The raw format is used in two scenarios:

  1. When the dictionary is disabled for a column, as specified in the .

  2. When the encoding is set to RAW in the .

When using the raw format, you can configure the following parameters:

Parameter
Default
Description

The compressionCodec parameter has the following valid values:

  • PASS_THROUGH

  • SNAPPY

  • ZSTANDARD

  • LZ4

  • GZIP (Introduced in release 1.2.0)

  • null (the JSON null value, not "null"), which is the default. In this case, PASS_THROUGH will be used for metrics and LZ4 for other columns.

deriveNumDocsPerChunk is only used when the datatype may have a variable length, such as with string, big decimal, bytes, etc. By default, Pinot uses a fixed number of elements that was chosen empirically. If changed to true, Pinot will use a heuristic value that depends on the column data.

rawIndexWriterVersion changes the algorithm used to create the index. This changes the actual data layout, but modern versions of Pinot can read indexes written in older versions. The latest version right now is 4.

targetDocsPerChunk changes the target number of docs to store in a chunk. For rawIndexWriterVersion versions 2 and 3, this will store exactly targetDocsPerChunk per chunk. For rawIndexWriterVersion version 4, this config is used in conjunction with targetMaxChunkSize and chunk size is determined with the formula min(lengthOfLongestDocumentInSegment * targetDocsPerChunk, targetMaxChunkSize). A negative value will disable dynamic chunk sizing and use the static targetMaxChunkSize.

targetMaxChunkSize changes the target max chunk size. For rawIndexWriterVersion versions 2 and 3, this can only be used with deriveNumDocsPerChunk. For rawIndexWriterVersion version 4, this sets the upper bound for a dynamically calculated chunk size. Documents larger than the targetMaxChunkSize will be given their own 'huge' chunk, therefore, it is recommended to size this such that huge chunks are avoided.

Raw forward index configuration

The recommended way to configure the forward index using raw format is by including the parameters explained above in the indexes.forward object. For example:

Deprecated

An alternative method to configure the raw format parameters is available. This older approach can still be used, although it is not recommended. Here are the details of this older method:

  • chunkCompressionType: This parameter can be defined as a sibling of name and encodingType in the fieldConfigList section.

  • deriveNumDocsPerChunk: You can configure this parameter with the property deriveNumDocsPerChunkForRawIndex. Note that in properties, all values must be strings, so valid values for this property are "true" and "false".

  • rawIndexWriterVersion: This parameter can be configured using the property rawIndexWriterVersion. Again, in properties, all values must be strings, so valid values for this property are "2", "3", and so on.

For example:

While this older method is still supported, it is not the recommended way to configure these parameters. There are no plans to remove support for this older method, but keep in mind that any new parameters added in the future may only be configurable in the forward JSON object.

Disabling the forward index

Traditionally the forward index has been a mandatory index for all columns in the on-disk segment file format.

However, certain columns may only be used as a filter in the WHERE clause for all queries. In such scenarios the forward index is not necessary as essentially other indexes and structures in the segments can provide the required SQL query functionality. Forward index just takes up extra storage space for such scenarios and can ideally be freed up.

Thus, to provide users an option to save storage space, a knob to disable the forward index is now available.

Forward index on one or more columns(s) in your Pinot table can be disabled with the following limitations:

  • Only supported for immutable (offline) segments.

  • If the column has a then the column must be of single-value type and use range index version 2.

  • MV columns with duplicates within a row will lose the duplicated entries on forward index regeneration. The ordering of data with an MV row may also change on regeneration. A backfill is required in such scenarios (to preserve duplicates or ordering).

  • If forward index regeneration support on reload (i.e. re-enabling the forward index for a forward index disabled column) is required then the dictionary and inverted index must be enabled on that particular column.

Sorted columns will allow the forward index to be disabled, but this operation will be treated as a no-op and the index (which acts as both a forward index and inverted index) will be created.

To disable the forward index, in under fieldConfigList, set the disabled property to true as shown below:

The older way to do so is still supported, but not recommended.

A table reload operation must be performed for the above config to take effect. Enabling / disabling other indexes on the column can be done via the usual options.

The forward index can also be regenerated for a column where it is disabled by enabling the index and reloading the segment. The forward index can only be regenerated if the dictionary and inverted index have been enabled for the column. If either have been disabled then the only way to get the forward index back is to regenerate the segments via the offline jobs and re-push / refresh the data.

Warning:

For multi-value (MV) columns the following invariants cannot be maintained after regenerating the forward index for a forward index disabled column:

  • Ordering guarantees of the MV values within a row

  • If entries within an MV row are duplicated, the duplicates will be lost. Regenerate the segments via your offline jobs and re-push / refresh the data to get back the original MV data with duplicates.

We will work on removing the second invariant in the future.

Examples of queries which will fail after disabling the forward index for an example column, columnA, can be found below:

Select

Forward index disabled columns cannot be present in the SELECT clause even if filters are added on it.

Group By Order By

Forward index disabled columns cannot be present in the GROUP BY and ORDER BY clauses. They also cannot be part of the HAVING clause.

Aggregation Queries

A subset of the aggregation functions do work when the forward index is disabled such as MIN, MAX, DISTINCTCOUNT, DISTINCTCOUNTHLL and more. Some of the other aggregation functions will not work such as the below:

Distinct

Forward index disabled columns cannot be present in the SELECT DISTINCT clause.

Range Queries

To run queries on single-value columns where the filter clause contains operators such as >, <, >=, <= a version 2 range index must be present. Without the range index such queries will fail as shown below:

Provides compression when low to medium cardinality.

Eliminates padding overhead

Allows for indexing (esp inv index).

No inv index (only JSON/Text/FST index)

Adds one level of dereferencing, so can increase disk seeks

Eliminates additional dereferencing, so good when all docs of interest are contiguous

For Strings, adds padding to make all values equal length in the dictionary

Chunk de-compression overhead with docs selected don't have spatial locality

dictIdCompressionType

null

The compression that will be used for dictionary-encoded forward index

Part of a tableConfig
{
    "tableIndexConfig": {
        "sortedColumn": [
            "column_name"
        ],
        ...
    }
}
$ grep memberId <segment_name>/v3/metadata.properties | grep isSorted
column.memberId.isSorted = true
curl -X GET \
  "http://localhost:9000/segments/baseballStats/metadata?columns=playerID&columns=teamID" \
  -H "accept: application/json" 2>/dev/null | \
  jq -c  '.[] | . as $parent |  
          .columns[] | 
          [$parent .segmentName, .columnName, .sorted]'
["baseballStats_OFFLINE_0","teamID",false]
["baseballStats_OFFLINE_0","playerID",false]

chunkCompressionType

null

The compression that will be used. Replaced by compressionCodec since release 1.2.0

compressionCodec

null

The compression that will be used. Introduced in release 1.2.0

deriveNumDocsPerChunk

false

Modifies the behavior when storing variable length values (like string or bytes)

rawIndexWriterVersion

2

The version initially used

targetDocsPerChunk

1000

The target number of docs per chunk

targetMaxChunkSize

1MB

The target max chunk size

Configured in tableConfig fieldConfigList
{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name": "playerID",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "compressionCodec": "PASS_THROUGH", // or "SNAPPY", "ZSTANDARD", "LZ4" or "GZIP"
          "deriveNumDocsPerChunk": false,
          "rawIndexWriterVersion": 2
        }
      }
    },
    ...
  ],
...
}
Configured in tableConfig fieldConfigList
{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name": "playerID",
      "encodingType": "RAW",
      "chunkCompressionType": "PASS_THROUGH", // it can also be defined here
      "properties": {
        "deriveNumDocsPerChunkForRawIndex": "false", // here the string value has to be used
        "rawIndexWriterVersion": "2" // here the string value has to be used
      }
    },
    ...
  ],
...
}
Configured in tableConfig fieldConfigList
{
  "tableName": "somePinotTable",
  "fieldConfigList": [
    {
      "name":"columnA",
      "indexes": {
        "forward": {
          "disabled": true
        }
      }
    },
    ...
  ],
  ...
}
Configured in tableConfig fieldConfigList
"fieldConfigList":[
  {
     "name":"columnA",
     "properties": {
        "forwardIndexDisabled": "true"
      }
  }
]
SELECT columnA
FROM myTable
    WHERE columnA = 10
SELECT *
FROM myTable
SELECT SUM(columnB)
FROM myTable
GROUP BY columnA
SELECT SUM(columnB), columnA
FROM myTable
GROUP BY columnA
ORDER BY columnA
SELECT MIN(columnA)
FROM myTable
GROUP BY columnB
HAVING MIN(columnA) > 100
ORDER BY columnB
SELECT SUM(columnA), AVG(columnA)
FROM myTable
SELECT MAX(ADD(columnA, columnB))
FROM myTable
SELECT DISTINCT columnA
FROM myTable
SELECT columnB
FROM myTable
    WHERE columnA > 1000
Disabling the Forward Index
raw value forward index
dictionary encoded forward index
sorted forward index
Dictionary index
dictionary documentation
Batch Quick Start
dictionary documentation
field config list
range index
table config
table config
Sorted forward index

Stream ingestion

This guide shows you how to ingest a stream of records into a Pinot table.

Apache Pinot lets users consume data from streams and push it directly into the database. This process is called stream ingestion. Stream ingestion makes it possible to query data within seconds of publication.

Stream ingestion provides support for checkpoints for preventing data loss.

To set up Stream ingestion, perform the following steps, which are described in more detail in this page:

  1. Create schema configuration

  2. Create table configuration

  3. Create ingestion configuration

  4. Upload table and schema spec

Here's an example where we assume the data to be ingested is in the following format:

{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"Maths","score":3.8,"timestamp":1571900400000}
{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"History","score":3.5,"timestamp":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Maths","score":3.2,"timestamp":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Chemistry","score":3.6,"timestamp":1572418800000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Geography","score":3.8,"timestamp":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"English","score":3.5,"timestamp":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Maths","score":3.2,"timestamp":1572678000000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Physics","score":3.6,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"Maths","score":3.8,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"English","score":3.5,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"History","score":3.2,"timestamp":1572854400000}
{"studentID":212,"firstName":"Nick","lastName":"Young","gender":"Male","subject":"History","score":3.6,"timestamp":1572854400000}

Create schema configuration

The schema defines the fields along with their data types. The schema also defines whether fields serve as dimensions , metrics, or timestamp. For more details on schema configuration, see creating a schema.

For our sample data, the schema configuration looks like this:

/tmp/pinot-quick-start/transcript-schema.json
{
  "schemaName": "transcript",
  "dimensionFieldSpecs": [
    {
      "name": "studentID",
      "dataType": "INT"
    },
    {
      "name": "firstName",
      "dataType": "STRING"
    },
    {
      "name": "lastName",
      "dataType": "STRING"
    },
    {
      "name": "gender",
      "dataType": "STRING"
    },
    {
      "name": "subject",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "score",
      "dataType": "FLOAT"
    }
  ],
  "dateTimeFieldSpecs": [{
    "name": "timestamp",
    "dataType": "LONG",
    "format" : "1:MILLISECONDS:EPOCH",
    "granularity": "1:MILLISECONDS"
  }]
}

Create table configuration with ingestion configuration

The next step is to create a table where all the ingested data will flow and can be queried. For details about each table component, see the table reference.

The table configuration contains an ingestion configuration (ingestionConfig), which specifies how to ingest streaming data into Pinot. For details, see the ingestion configuration reference.

Example table config with ingestionConfig

For our sample data and schema, the table config will look like this:

{
  "tableName": "transcript",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
  },
  "metadata": {
    "customConfigs": {}
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
        "streamConfigMaps": [
          {
            "realtime.segment.flush.threshold.rows": "0",
            "stream.kafka.decoder.prop.format": "JSON",
            "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "streamType": "kafka",
            "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.consumer.type": "LOWLEVEL",
            "realtime.segment.flush.threshold.segment.rows": "50000",
            "stream.kafka.broker.list": "localhost:9876",
            "realtime.segment.flush.threshold.time": "3600000",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
            "stream.kafka.topic.name": "transcript-topic"
          }
        ]
      },
      "transformConfigs": [],
      "continueOnError": true,
      "rowTimeValueCheck": true,
      "segmentTimeValueCheck": false
    },
    "isDimTable": false
  }
}

Example ingestionConfig for multi-topics ingestion

From this PR, Pinot starts to support ingesting data from multiple stream partitions. (It is currently in Beta mode, and only supports multiple Kafka topics. Other stream types would be supported in the near future.) For our sample data and schema, assume that we duplicate it to 2 topics, transcript-topic1 and transcript-topic2. If we want to ingest from both topics, then the table config will look like this:

{
  "tableName": "transcript",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
  },
  "metadata": {
    "customConfigs": {}
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
        "streamConfigMaps": [
          {
            "realtime.segment.flush.threshold.rows": "0",
            "stream.kafka.decoder.prop.format": "JSON",
            "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "streamType": "kafka",
            "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.consumer.type": "LOWLEVEL",
            "realtime.segment.flush.threshold.segment.rows": "50000",
            "stream.kafka.broker.list": "localhost:9876",
            "realtime.segment.flush.threshold.time": "3600000",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
            "stream.kafka.topic.name": "transcript-topic1"
          },
          {
            "realtime.segment.flush.threshold.rows": "0",
            "stream.kafka.decoder.prop.format": "JSON",
            "key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "streamType": "kafka",
            "value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
            "stream.kafka.consumer.type": "LOWLEVEL",
            "realtime.segment.flush.threshold.segment.rows": "50000",
            "stream.kafka.broker.list": "localhost:9876",
            "realtime.segment.flush.threshold.time": "3600000",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
            "stream.kafka.topic.name": "transcript-topic2"
          }
        ]
      },
      "transformConfigs": [],
      "continueOnError": true,
      "rowTimeValueCheck": true,
      "segmentTimeValueCheck": false
    },
    "isDimTable": false
  }
}

With multi-topics ingestion: (details please refer to the design doc)

  • All transform functions would apply to both topics' ingestions.

  • Existing instance assignment strategy would all work as usual.

  • Partition changes would still be handled in the same way.

  • Underlying ingestion still works as LOWLEVEL mode, where

    • transcript-topic1 segments would be named like transcript__0__0__20250101T0000Z

    • transcript-topic2 segments would be named like transcript__10000__0__20250101T0000Z

Upload schema and table config

Now that we have our table and schema configurations, let's upload them to the Pinot cluster. As soon as the configs are uploaded, Pinot will start ingesting available records from the topic.

docker run \
    --network=pinot-demo \
    -v /tmp/pinot-quick-start:/tmp/pinot-quick-start \
    --name pinot-streaming-table-creation \
    apachepinot/pinot:latest AddTable \
    -schemaFile /tmp/pinot-quick-start/transcript-schema.json \
    -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \
    -controllerHost pinot-quickstart \
    -controllerPort 9000 \
    -exec
bin/pinot-admin.sh AddTable \
    -schemaFile /path/to/transcript-schema.json \
    -tableConfigFile /path/to/transcript-table-realtime.json \
    -exec

Tune the stream config

Throttle stream consumption

There are some scenarios where the message rate in the input stream can come in bursts which can lead to long GC pauses on the Pinot servers or affect the ingestion rate of other real-time tables on the same server. If this happens to you, throttle the consumption rate during stream ingestion to better manage overall performance.

Stream consumption throttling can be tuned using the stream config topic.consumption.rate.limit which indicates the upper bound on the message rate for the entire topic.

Here is the sample configuration on how to configure the consumption throttling:

{
  "tableName": "transcript",
  "tableType": "REALTIME",
  ...
  "ingestionConfig": {
    "streamIngestionConfig":,
    "streamConfigMaps": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "transcript-topic",
      ...
      "topic.consumption.rate.limit": 1000
    }
  },
  ...

Some things to keep in mind while tuning this config are:

  • Since this configuration applied to the entire topic, internally, this rate is divided by the number of partitions in the topic and applied to each partition's consumer.

  • In case of multi-tenant deployment (where you have more than 1 table in the same server instance), you need to make sure that the rate limit on one table doesn't step on/starve the rate limiting of another table. So, when there is more than 1 table on the same server (which is most likely to happen), you may need to re-tune the throttling threshold for all the streaming tables.

Once throttling is enabled for a table, you can verify by searching for a log that looks similar to:

A consumption rate limiter is set up for topic <topic_name> in table <tableName> with rate limit: <rate_limit> (topic rate limit: <topic_rate_limit>, partition count: <partition_count>)

In addition, you can monitor the consumption rate utilization with the metric COSUMPTION_QUOTA_UTILIZATION.

Note that any configuration change for topic.consumption.rate.limit in the stream config will NOT take effect immediately. The new configuration will be picked up from the next consuming segment. In order to enforce the new configuration, you need to trigger forceCommit APIs. Refer to Pause Stream Ingestion for more details.

$ curl -X POST {controllerHost}/tables/{tableName}/forceCommit

Custom ingestion support

You can also write an ingestion plugin if the platform you are using is not supported out of the box. For a walkthrough, see Stream Ingestion Plugin.

Pause stream ingestion

There are some scenarios in which you may want to pause the real-time ingestion while your table is available for queries. For example, if there is a problem with the stream ingestion and, while you are troubleshooting the issue, you still want the queries to be executed on the already ingested data. For these scenarios, you can first issue a Pause request to a Controller host. After troubleshooting with the stream is done, you can issue another request to Controller to resume the consumption.

$ curl -X POST {controllerHost}/tables/{tableName}/pauseConsumption
$ curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption

When a Pause request is issued, the controller instructs the real-time servers hosting your table to commit their consuming segments immediately. However, the commit process may take some time to complete. Note that Pause and Resume requests are async. An OK response means that instructions for pausing or resuming has been successfully sent to the real-time server. If you want to know if the consumption has actually stopped or resumed, issue a pause status request.

$ curl -X POST {controllerHost}/tables/{tableName}/pauseStatus

It's worth noting that consuming segments on real-time servers are stored in volatile memory, and their resources are allocated when the consuming segments are first created. These resources cannot be altered if consumption parameters are changed midway through consumption. It may take hours before these changes take effect. Furthermore, if the parameters are changed in an incompatible way (for example, changing the underlying stream with a completely new set of offsets, or changing the stream endpoint from which to consume messages), it will result in the table getting into an error state.

The pause and resume feature is helpful in these instances. When a pause request is issued by the operator, consuming segments are committed without starting new mutable segments. Instead, new mutable segments are started only when the resume request is issued. This mechanism provides the operators as well as developers with more flexibility. It also enables Pinot to be more resilient to the operational and functional constraints imposed by underlying streams.

There is another feature called Force Commit which utilizes the primitives of the pause and resume feature. When the operator issues a force commit request, the current mutable segments will be committed and new ones started right away. Operators can now use this feature for all compatible table config parameter changes to take effect immediately.

$ curl -X POST {controllerHost}/tables/{tableName}/forceCommit

(v 0.12.0+) Once submitted, the forceCommit API returns a jobId that can be used to get the current progress of the forceCommit operation. A sample response and status API call:

$ curl -X POST {controllerHost}/tables/{tableName}/forceCommit
{
  "forceCommitJobId": "6757284f-b75b-45ce-91d8-a277bdbc06ae",
  "forceCommitStatus": "SUCCESS",
  "jobMetaZKWriteStatus": "SUCCESS"
}

$ curl -X GET {controllerHost}/tables/forceCommitStatus/6757284f-b75b-45ce-91d8-a277bdbc06ae
{
  "jobId": "6757284f-b75b-45ce-91d8-a277bdbc06ae",
  "segmentsForceCommitted": "[\"airlineStats__0__0__20230119T0700Z\",\"airlineStats__1__0__20230119T0700Z\",\"airlineStats__2__0__20230119T0700Z\"]",
  "submissionTimeMs": "1674111682977",
  "numberOfSegmentsYetToBeCommitted": 0,
  "jobType": "FORCE_COMMIT",
  "segmentsYetToBeCommitted": [],
  "tableName": "airlineStats_REALTIME"
}

The forceCommit request just triggers a regular commit before the consuming segments reaching the end criteria, so it follows the same mechanism as regular commit. It is one-time shot request, and not retried automatically upon failure. But it is idempotent so one may keep issuing it till success if needed.

This API is async, as it doesn't wait for the segment commit to complete. But a status entry is put in ZK to track when the request is issued and the consuming segments included. The consuming segments tracked in the status entry are compared with the latest IdealState to indicate the progress of forceCommit. However, this status is not updated or deleted upon commit success or failure, so that it could become stale. Currently, the most recent 100 status entries are kept in ZK, and the oldest ones only get deleted when the total number is about to exceed 100.

For incompatible parameter changes, an option is added to the resume request to handle the case of a completely new set of offsets. Operators can now follow a three-step process: First, issue a pause request. Second, change the consumption parameters. Finally, issue the resume request with the appropriate option. These steps will preserve the old data and allow the new data to be consumed immediately. All through the operation, queries will continue to be served.

$ curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption?resumeFrom=smallest
$ curl -X POST {controllerHost}/tables/{tableName}/resumeConsumption?resumeFrom=largest

Handle partition changes in streams

If a Pinot table is configured to consume using a Low Level (partition-based) stream type, then it is possible that the partitions of the table change over time. In Kafka, for example, the number of partitions may increase. In Kinesis, the number of partitions may increase or decrease -- some partitions could be merged to create a new one, or existing partitions split to create new ones.

Pinot runs a periodic task called RealtimeSegmentValidationManager that monitors such changes and starts consumption on new partitions (or stops consumptions from old ones) as necessary. Since this is a periodic task that is run on the controller, it may take some time for Pinot to recognize new partitions and start consuming from them. This may delay the data in new partitions appearing in the results that pinot returns.

If you want to recognize the new partitions sooner, then manually trigger the periodic task so as to recognize such data immediately.

Infer ingestion status of real-time tables

Often, it is important to understand the rate of ingestion of data into your real-time table. This is commonly done by looking at the consumption lag of the consumer. The lag itself can be observed in many dimensions. Pinot supports observing consumption lag along the offset dimension and time dimension, whenever applicable (as it depends on the specifics of the connector).

The ingestion status of a connector can be observed by querying either the /consumingSegmentsInfo API or the table's /debug API, as shown below:

# GET /tables/{tableName}/consumingSegmentsInfo
curl -X GET "http://<controller_url:controller_admin_port>/tables/meetupRsvp/consumingSegmentsInfo" -H "accept: application/json"

# GET /debug/tables/{tableName}
curl -X GET "http://localhost:9000/debug/tables/meetupRsvp?type=REALTIME&verbosity=1" -H "accept: application/json"

A sample response from a Kafka-based real-time table is shown below. The ingestion status is displayed for each of the CONSUMING segments in the table.

{
  "_segmentToConsumingInfoMap": {
    "meetupRsvp__0__0__20221019T0639Z": [
      {
        "serverName": "Server_192.168.0.103_7000",
        "consumerState": "CONSUMING",
        "lastConsumedTimestamp": 1666161593904,
        "partitionToOffsetMap": { // <<-- Deprecated. See currentOffsetsMap for same info
          "0": "6"
        },
        "partitionOffsetInfo": {
          "currentOffsetsMap": {
            "0": "6" // <-- Current consumer position
          },
          "latestUpstreamOffsetMap": {
            "0": "6"  // <-- Upstream latest position
          },
          "recordsLagMap": {
            "0": "0"  // <-- Lag, in terms of #records behind latest
          },
          "recordsAvailabilityLagMap": {
            "0": "2"  // <-- Lag, in terms of time
          }
        }
      }
    ],
Term
Description

currentOffsetsMap

Current consuming offset position per partition

latestUpstreamOffsetMap

(Wherever applicable) Latest offset found in the upstream topic partition

recordsLagMap

(Whenever applicable) Defines how far behind the current record's offset / pointer is from upstream latest record. This is calculated as the difference between the latestUpstreamOffset and currentOffset for the partition when the lag computation request is made.

recordsAvailabilityLagMap

(Whenever applicable) Defines how soon after record ingestion was the record consumed by Pinot. This is calculated as the difference between the time the record was consumed and the time at which the record was ingested upstream.

Monitor real-time ingestion

Real-time ingestion includes 3 stages of message processing: Decode, Transform, and Index.

In each of these stages, a failure can happen which may or may not result in an ingestion failure. The following metrics are available to investigate ingestion issues:

  1. Decode stage -> an error here is recorded as INVALID_REALTIME_ROWS_DROPPED

  2. Transform stage -> possible errors here are:

    1. When a message gets dropped due to the FILTER transform, it is recorded as REALTIME_ROWS_FILTERED

    2. When the transform pipeline sets the $INCOMPLETE_RECORD_KEY$ key in the message, it is recorded as INCOMPLETE_REALTIME_ROWS_CONSUMED , only when continueOnError configuration is enabled. If the continueOnError is not enabled, the ingestion fails.

  3. Index stage -> When there is failure at this stage, the ingestion typically stops and marks the partition as ERROR.

There is yet another metric called ROWS_WITH_ERROR which is the sum of all error counts in the 3 stages above.

Furthermore, the metric REALTIME_CONSUMPTION_EXCEPTIONS gets incremented whenever there is a transient/permanent stream exception seen during consumption.

These metrics can be used to understand why ingestion failed for a particular table partition before diving into the server logs.

Minion

Explore the minion component in Apache Pinot, empowering efficient data movement and segment generation within Pinot clusters.

A Pinot minion is an optional cluster component that executes background tasks on table data apart from the query processes performed by brokers and servers. Minions run on independent hardware resources, and are responsible for executing minion tasks as directed by the controller. Examples of minon tasks include converting batch data from a standard format like Avro or JSON into segment files to be loaded into an offline table, and rewriting existing segment files to purge records as required by data privacy laws like GDPR. Minion tasks can run once or be scheduled to run periodically.

Minions isolate the computational burden of out-of-band data processing from the servers. Although a Pinot cluster can function with or without minions, they are typically present to support routine tasks like batch data ingest.

Starting a minion

Make sure you've set up Zookeeper. If you're using Docker, make sure to pull the Pinot Docker image. To start a minion:

Usage: StartMinion
    -help                                                   : Print this message. (required=false)
    -minionHost               <String>                      : Host name for minion. (required=false)
    -minionPort               <int>                         : Port number to start the minion at. (required=false)
    -zkAddress                <http>                        : HTTP address of Zookeeper. (required=false)
    -clusterName              <String>                      : Pinot cluster name. (required=false)
    -configFileName           <Config File Name>            : Minion Starter Config file. (required=false)
docker run \
    --network=pinot-demo \
    --name pinot-minion \
    -d ${PINOT_IMAGE} StartMinion \
    -zkAddress pinot-zookeeper:2181
bin/pinot-admin.sh StartMinion \
    -zkAddress localhost:2181

Interfaces

Pinot task generator

The Pinot task generator interface defines the APIs for the controller to generate tasks for minions to execute.

public interface PinotTaskGenerator {

  /**
   * Initializes the task generator.
   */
  void init(ClusterInfoAccessor clusterInfoAccessor);

  /**
   * Returns the task type of the generator.
   */
  String getTaskType();

  /**
   * Generates a list of tasks to schedule based on the given table configs.
   */
  List<PinotTaskConfig> generateTasks(List<TableConfig> tableConfigs);

  /**
   * Returns the timeout in milliseconds for each task, 3600000 (1 hour) by default.
   */
  default long getTaskTimeoutMs() {
    return JobConfig.DEFAULT_TIMEOUT_PER_TASK;
  }

  /**
   * Returns the maximum number of concurrent tasks allowed per instance, 1 by default.
   */
  default int getNumConcurrentTasksPerInstance() {
    return JobConfig.DEFAULT_NUM_CONCURRENT_TASKS_PER_INSTANCE;
  }

  /**
   * Performs necessary cleanups (e.g. remove metrics) when the controller leadership changes.
   */
  default void nonLeaderCleanUp() {
  }
}

PinotTaskExecutorFactory

Factory for PinotTaskExecutor which defines the APIs for Minion to execute the tasks.

public interface PinotTaskExecutorFactory {

  /**
   * Initializes the task executor factory.
   */
  void init(MinionTaskZkMetadataManager zkMetadataManager);

  /**
   * Returns the task type of the executor.
   */
  String getTaskType();

  /**
   * Creates a new task executor.
   */
  PinotTaskExecutor create();
}
public interface PinotTaskExecutor {

  /**
   * Executes the task based on the given task config and returns the execution result.
   */
  Object executeTask(PinotTaskConfig pinotTaskConfig)
      throws Exception;

  /**
   * Tries to cancel the task.
   */
  void cancel();
}

MinionEventObserverFactory

Factory for MinionEventObserver which defines the APIs for task event callbacks on minion.

public interface MinionEventObserverFactory {

  /**
   * Initializes the task executor factory.
   */
  void init(MinionTaskZkMetadataManager zkMetadataManager);

  /**
   * Returns the task type of the event observer.
   */
  String getTaskType();

  /**
   * Creates a new task event observer.
   */
  MinionEventObserver create();
}
public interface MinionEventObserver {

  /**
   * Invoked when a minion task starts.
   *
   * @param pinotTaskConfig Pinot task config
   */
  void notifyTaskStart(PinotTaskConfig pinotTaskConfig);

  /**
   * Invoked when a minion task succeeds.
   *
   * @param pinotTaskConfig Pinot task config
   * @param executionResult Execution result
   */
  void notifyTaskSuccess(PinotTaskConfig pinotTaskConfig, @Nullable Object executionResult);

  /**
   * Invoked when a minion task gets cancelled.
   *
   * @param pinotTaskConfig Pinot task config
   */
  void notifyTaskCancelled(PinotTaskConfig pinotTaskConfig);

  /**
   * Invoked when a minion task encounters exception.
   *
   * @param pinotTaskConfig Pinot task config
   * @param exception Exception encountered during execution
   */
  void notifyTaskError(PinotTaskConfig pinotTaskConfig, Exception exception);
}

Built-in tasks

SegmentGenerationAndPushTask

The PushTask can fetch files from an input folder e.g. from a S3 bucket and converts them into segments. The PushTask converts one file into one segment and keeps file name in segment metadata to avoid duplicate ingestion. Below is an example task config to put in TableConfig to enable this task. The task is scheduled every 10min to keep ingesting remaining files, with 10 parallel task at max and 1 file per task.

NOTE: You may want to simply omit "tableMaxNumTasks" due to this caveat: the task generates one segment per file, and derives segment name based on the time column of the file. If two files happen to have same time range and are ingested by tasks from different schedules, there might be segment name conflict. To overcome this issue for now, you can omit “tableMaxNumTasks” and by default it’s Integer.MAX_VALUE, meaning to schedule as many tasks as possible to ingest all input files in a single batch. Within one batch, a sequence number suffix is used to ensure no segment name conflict. Because the sequence number suffix is scoped within one batch, tasks from different batches might encounter segment name conflict issue said above.

When performing ingestion at scale remember that Pinot will list all of the files contained in the `inputDirURI` every time a `SegmentGenerationAndPushTask` job gets scheduled. This could become a bottleneck when fetching files from a cloud bucket like GCS. To prevent this make `inputDirURI` point to the least number of files possible.

  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "APPEND",
      "segmentIngestionFrequency": "DAILY",
      "batchConfigMaps": [
        {
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "us-west-2",
          "input.fs.prop.secretKey": "....",
          "input.fs.prop.accessKey": "....",
          "inputDirURI": "s3://my.s3.bucket/batch/airlineStats/rawdata/",
          "includeFileNamePattern": "glob:**/*.avro",
          "excludeFileNamePattern": "glob:**/*.tmp",
          "inputFormat": "avro"
        }
      ]
    }
  },
  "task": {
    "taskTypeConfigsMap": {
      "SegmentGenerationAndPushTask": {
        "schedule": "0 */10 * * * ?",
        "tableMaxNumTasks": "10"
      }
    }
  }

RealtimeToOfflineSegmentsTask

See Pinot managed Offline flows for details.

MergeRollupTask

See Minion merge rollup task for details.

Enable tasks

Tasks are enabled on a per-table basis. To enable a certain task type (e.g. myTask) on a table, update the table config to include the task type:

{
  ...
  "task": {
    "taskTypeConfigsMap": {
      "myTask": {
        "myProperty1": "value1",
        "myProperty2": "value2"
      }
    }
  }
}

Under each enable task type, custom properties can be configured for the task type.

There are also two task configs to be set as part of cluster configs like below. One controls task's overall timeout (1hr by default) and one for how many tasks to run on a single minion worker (1 by default).

Using "POST /cluster/configs" API on CLUSTER tab in Swagger, with this payload
{
	"RealtimeToOfflineSegmentsTask.timeoutMs": "600000",
	"RealtimeToOfflineSegmentsTask.numConcurrentTasksPerInstance": "4"
}

Schedule tasks

Auto-schedule

There are 2 ways to enable task scheduling:

Controller level schedule for all minion tasks

Tasks can be scheduled periodically for all task types on all enabled tables. Enable auto task scheduling by configuring the schedule frequency in the controller config with the key controller.task.frequencyPeriod. This takes period strings as values, e.g. 2h, 30m, 1d.

Per table and task level schedule

Tasks can also be scheduled based on cron expressions. The cron expression is set in the schedule config for each task type separately. This config in the controller config, controller.task.scheduler.enabled should be set to true to enable cron scheduling.

As shown below, the RealtimeToOfflineSegmentsTask will be scheduled at the first second of every minute (following the syntax defined here).

  "task": {
    "taskTypeConfigsMap": {
      "RealtimeToOfflineSegmentsTask": {
        "bucketTimePeriod": "1h",
        "bufferTimePeriod": "1h",
        "schedule": "0 * * * * ?"
      }
    }
  },

Manual schedule

Tasks can be manually scheduled using the following controller rest APIs:

Rest API
Description

POST /tasks/schedule

Schedule tasks for all task types on all enabled tables

POST /tasks/schedule?taskType=myTask

Schedule tasks for the given task type on all enabled tables

POST /tasks/schedule?tableName=myTable_OFFLINE

Schedule tasks for all task types on the given table

POST /tasks/schedule?taskType=myTask&tableName=myTable_OFFLINE

Schedule tasks for the given task type on the given table

Schedule task on specific instances

Tasks can be scheduled on specific instances using the following config at task level:

  "task": {
    "taskTypeConfigsMap": {
      "RealtimeToOfflineSegmentsTask": {
        "bucketTimePeriod": "1h",
        "bufferTimePeriod": "1h",
        "schedule": "0 * * * * ?",
        "minionInstanceTag": "tag1_MINION"
      }
    }
  },

By default, the value is minion_untagged to have backward-compatibility. This will allow users to schedule tasks on specific nodes and isolate tasks among tables / task-types.

Rest API
Description

POST /tasks/schedule?taskType=myTask&tableName=myTable_OFFLINE&minionInstanceTag=tag1_MINION

Schedule tasks for the given task type of the given table on the minion nodes tagged as tag1_MINION.

Task level advanced configs

allowDownloadFromServer

When a task is executed on a segment, the minion node fetches the segment from deepstore. If the deepstore is not accessible, the minion node can download the segment from the server node. This is controlled by the allowDownloadFromServer config in the task config. By default, this is set to false.

We can also set this config at a minion instance level pinot.minion.task.allow.download.from.server (default is false). This instance level config helps in enforcing this behaviour if the number of tables / tasks is pretty high and we want to enable for all. Note: task-level config will override instance-level config value.

Plug-in custom tasks

To plug in a custom task, implement PinotTaskGenerator, PinotTaskExecutorFactory and MinionEventObserverFactory (optional) for the task type (all of them should return the same string for getTaskType()), and annotate them with the following annotations:

Implementation
Annotation

PinotTaskGenerator

@TaskGenerator

PinotTaskExecutorFactory

@TaskExecutorFactory

MinionEventObserverFactory

@EventObserverFactory

After annotating the classes, put them under the package of name org.apache.pinot.*.plugin.minion.tasks.*, then they will be auto-registered by the controller and minion.

Example

See SimpleMinionClusterIntegrationTest where the TestTask is plugged-in.

Task Manager UI

In the Pinot UI, there is Minion Task Manager tab under Cluster Manager page. From that minion task manager tab, one can find a lot of task related info for troubleshooting. Those info are mainly collected from the Pinot controller that schedules tasks or Helix that tracks task runtime status. There are also buttons to schedule tasks in an ad hoc way. Below are some brief introductions to some pages under the minion task manager tab.

This one shows which types of Minion Task have been used. Essentially which task types have created their task queues in Helix.

Clicking into a task type, one can see the tables using that task. And a few buttons to stop the task queue, cleaning up ended tasks etc.

Then clicking into any table in this list, one can see how the task is configured for that table. And the task metadata if there is one in ZK. For example, MergeRollupTask tracks a watermark in ZK. If the task is cron scheduled, the current and next schedules are also shown in this page like below.

At the bottom of this page is a list of tasks generated for this table for this specific task type. Like here, one MergeRollup task has been generated and completed.

Clicking into a task from that list, we can see start/end time for it, and the subtasks generated for that task (as context, one minion task can have multiple subtasks to process data in parallel). In this example, it happened to have one sub-task here, and it shows when it starts and stops and which minion worker it's running.

Clicking into this subtask, one can see more details about it like the input task configs and error info if the task failed.

Task-related metrics

There is a controller job that runs every 5 minutes by default and emits metrics about Minion tasks scheduled in Pinot. The following metrics are emitted for each task type:

  • NumMinionTasksInProgress: Number of running tasks

  • NumMinionSubtasksRunning: Number of running sub-tasks

  • NumMinionSubtasksWaiting: Number of waiting sub-tasks (unassigned to a minion as yet)

  • NumMinionSubtasksError: Number of error sub-tasks (completed with an error/exception)

  • PercentMinionSubtasksInQueue: Percent of sub-tasks in waiting or running states

  • PercentMinionSubtasksInError: Percent of sub-tasks in error

The controller also emits metrics about how tasks are cron scheduled:

  • cronSchedulerJobScheduled: Number of current cron schedules registered to be triggered regularly according their cron expressions. It's a Gauge.

  • cronSchedulerJobTrigger: Number of cron scheduled triggered, as a Meter.

  • cronSchedulerJobSkipped: Number of late cron scheduled skipped, as a Meter.

  • cronSchedulerJobExecutionTimeMs: Time used to complete task generation, as a Timer.

For each task, the minion will emit these metrics:

  • TASK_QUEUEING: Task queueing time (task_dequeue_time - task_inqueue_time), assuming the time drift between helix controller and pinot minion is minor, otherwise the value may be negative

  • TASK_EXECUTION: Task execution time, which is the time spent on executing the task

  • NUMBER_OF_TASKS: number of tasks in progress on that minion. Whenever a Minion starts a task, increase the Gauge by 1, whenever a Minion completes (either succeeded or failed) a task, decrease it by 1

  • NUMBER_TASKS_EXECUTED: Number of tasks executed, as a Meter.

  • NUMBER_TASKS_COMPLETED: Number of tasks completed, as a Meter.

  • NUMBER_TASKS_CANCELLED: Number of tasks cancelled, as a Meter.

  • NUMBER_TASKS_FAILED: Number of tasks failed, as a Meter. Different from fatal failure, the task encountered an error which can not be recovered from this run, but it may still succeed by retrying the task.

  • NUMBER_TASKS_FATAL_FAILED: Number of tasks fatal failed, as a Meter. Different from failure, the task encountered an error, which will not be recoverable even with retrying the task.

Text search support

This page talks about support for text search in Pinot.

This text index method is recommended over the experimental native text index.

Click to skip the background info and go straight to the procedure to enable this text index.

Why do we need text search?

Pinot supports super-fast query processing through its indexes on non-BLOB like columns. Queries with exact match filters are run efficiently through a combination of dictionary encoding, inverted index, and sorted index.

This is useful for a query like the following, which looks for exact matches on two columns of type STRING and INT respectively:

SELECT COUNT(*) 
FROM Foo 
WHERE STRING_COL = 'ABCDCD' 
AND INT_COL > 2000

For arbitrary text data that falls into the BLOB/CLOB territory, we need more than exact matches. This often involves using regex, phrase, fuzzy queries on BLOB like data. Text indexes can efficiently perform arbitrary search on STRING columns where each column value is a large BLOB of text using the TEXT_MATCH function, like this:

SELECT COUNT(*) 
FROM Foo 
WHERE TEXT_MATCH (<column_name>, '<search_expression>')

where <column_name> is the column text index is created on and <search_expression> conforms to one of the following:

Search Expression Type

Example

Phrase query

TEXT_MATCH (<column_name>, '"distributed system"')

Term Query

TEXT_MATCH (<column_name>, 'Java')

Boolean Query

TEXT_MATCH (<column_name>, 'Java AND c++')

Prefix Query

TEXT_MATCH (<column_name>, 'stream*')

Regex Query

TEXT_MATCH (<column_name>, '/Exception.*/')

Not Query

TEXT_MATCH (<column_name>, '*:* NOT c%')

NOT TEXT_MATCH (<column_name>, 'c%')

Current restrictions

Pinot supports text search with the following requirements:

  • The column type should be STRING.

  • The column should be single-valued.

  • Using a text index in coexistence with other Pinot indexes is not supported.

Sample Datasets

Text search should ideally be used on STRING columns where doing standard filter operations (EQUALITY, RANGE, BETWEEN) doesn't fit the bill because each column value is a reasonably large blob of text.

Apache Access Log

Consider the following snippet from an Apache access log. Each line in the log consists of arbitrary data (IP addresses, URLs, timestamps, symbols etc) and represents a column value. Data like this is a good candidate for doing text search.

Let's say the following snippet of data is stored in the ACCESS_LOG_COL column in a Pinot table.

109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://almhuette-raith.at/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
46.72.177.4 - - [12/Dec/2015:18:31:08 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
46.72.177.4 - - [12/Dec/2015:18:31:08 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://almhuette-raith.at/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
83.167.113.100 - - [12/Dec/2015:18:31:25 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
83.167.113.100 - - [12/Dec/2015:18:31:25 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://almhuette-raith.at/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
95.29.198.15 - - [12/Dec/2015:18:32:10 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
95.29.198.15 - - [12/Dec/2015:18:32:11 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://almhuette-raith.at/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
109.184.11.34 - - [12/Dec/2015:18:32:56 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
109.184.11.34 - - [12/Dec/2015:18:32:56 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://almhuette-raith.at/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
91.227.29.79 - - [12/Dec/2015:18:33:51 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"

Here are some examples of search queries on this data:

Count the number of GET requests.

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(ACCESS_LOG_COL, 'GET')

Count the number of POST requests that have administrator in the URL (administrator/index)

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(ACCESS_LOG_COL, 'post AND administrator AND index')

Count the number of POST requests that have a particular URL and handled by Firefox browser

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(ACCESS_LOG_COL, 'post AND administrator AND index AND firefox')

Resume text

Let's consider another example using text from job candidate resumes. Each line in this file represents skill-data from resumes of different candidates.

This data is stored in the SKILLS_COL column in a Pinot table. Each line in the input text represents a column value.

Distributed systems, Java, C++, Go, distributed query engines for analytics and data warehouses, Machine learning, spark, Kubernetes, transaction processing
Java, Python, C++, Machine learning, building and deploying large scale production systems, concurrency, multi-threading, CPU processing
C++, Python, Tensor flow, database kernel, storage, indexing and transaction processing, building large scale systems, Machine learning
Amazon EC2, AWS, hadoop, big data, spark, building high performance scalable systems, building and deploying large scale production systems, concurrency, multi-threading, Java, C++, CPU processing
Distributed systems, database development, columnar query engine, database kernel, storage, indexing and transaction processing, building large scale systems
Distributed systems, Java, realtime streaming systems, Machine learning, spark, Kubernetes, distributed storage, concurrency, multi-threading
CUDA, GPU, Python, Machine learning, database kernel, storage, indexing and transaction processing, building large scale systems
Distributed systems, Java, database engine, cluster management, docker image building and distribution
Kubernetes, cluster management, operating systems, concurrency, multi-threading, apache airflow, Apache Spark,
Apache spark, Java, C++, query processing, transaction processing, distributed storage, concurrency, multi-threading, apache airflow
Big data stream processing, Apache Flink, Apache Beam, database kernel, distributed query engines for analytics and data warehouses
CUDA, GPU processing, Tensor flow, Pandas, Python, Jupyter notebook, spark, Machine learning, building high performance scalable systems
Distributed systems, Apache Kafka, publish-subscribe, building and deploying large scale production systems, concurrency, multi-threading, C++, CPU processing, Java
Realtime stream processing, publish subscribe, columnar processing for data warehouses, concurrency, Java, multi-threading, C++,

Here are some examples of search queries on this data:

Count the number of candidates that have "machine learning" and "gpu processing": This is a phrase search (more on this further in the document) where we are looking for exact match of phrases "machine learning" and "gpu processing", not necessarily in the same order in the original data.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"Machine learning" AND "gpu processing"')

Count the number of candidates that have "distributed systems" and either 'Java' or 'C++': This is a combination of searching for exact phrase "distributed systems" along with other terms.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"distributed systems" AND (Java C++)')

Query Log

Next, consider a snippet from a log file containing SQL queries handled by a database. Each line (query) in the file represents a column value in the QUERY_LOG_COL column in a Pinot table.

SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560988800000 AND 1568764800000 GROUP BY dimensionCol3 TOP 2500
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560988800000 AND 1568764800000 GROUP BY dimensionCol3 TOP 2500
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1545436800000 AND 1553212800000 GROUP BY dimensionCol3 TOP 2500
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1537228800000 AND 1537660800000 GROUP BY dimensionCol3 TOP 2500
SELECT dimensionCol2, dimensionCol4, timestamp, dimensionCol5, dimensionCol6 FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1561366800000 AND 1561370399999 AND dimensionCol3 = 2019062409 LIMIT 10000
SELECT dimensionCol2, dimensionCol4, timestamp, dimensionCol5, dimensionCol6 FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1563807600000 AND 1563811199999 AND dimensionCol3 = 2019072215 LIMIT 10000
SELECT dimensionCol2, dimensionCol4, timestamp, dimensionCol5, dimensionCol6 FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1563811200000 AND 1563814799999 AND dimensionCol3 = 2019072216 LIMIT 10000
SELECT dimensionCol2, dimensionCol4, timestamp, dimensionCol5, dimensionCol6 FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1566327600000 AND 1566329400000 AND dimensionCol3 = 2019082019 LIMIT 10000
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560834000000 AND 1560837599999 AND dimensionCol3 = 2019061805 LIMIT 0
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560870000000 AND 1560871800000 AND dimensionCol3 = 2019061815 LIMIT 0
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560871800001 AND 1560873599999 AND dimensionCol3 = 2019061815 LIMIT 0
SELECT count(dimensionCol2) FROM FOO WHERE dimensionCol1 = 18616904 AND timestamp BETWEEN 1560873600000 AND 1560877199999 AND dimensionCol3 = 2019061816 LIMIT 0

Here are some examples of search queries on this data:

Count the number of queries that have GROUP BY

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(QUERY_LOG_COL, '"group by"')

Count the number of queries that have the SELECT count... pattern

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(QUERY_LOG_COL, '"select count"')

Count the number of queries that use BETWEEN filter on timestamp column along with GROUP BY

SELECT COUNT(*) 
FROM MyTable 
WHERE TEXT_MATCH(QUERY_LOG_COL, '"timestamp between" AND "group by"')

Read on for concrete examples on each kind of query and step-by-step guides covering how to write text search queries in Pinot.

A column in Pinot can be dictionary-encoded or stored RAW. In addition, we can create an inverted index and/or a sorted index on a dictionary-encoded column.

The text index is an addition to the type of per-column indexes users can create in Pinot. However, it only supports text index on a RAW column, not a dictionary-encoded column.

Enable a text index

Enable a text index on a column in the table configuration by adding a new section with the name "fieldConfigList".

"fieldConfigList":[
  {
     "name":"text_col_1",
     "encodingType":"RAW",
     "indexTypes":["TEXT"]
  },
  {
     "name":"text_col_2",
     "encodingType":"RAW",
     "indexTypes":["TEXT"]
  }
]

Each column that has a text index should also be specified as noDictionaryColumns in tableIndexConfig:

"tableIndexConfig": {
   "noDictionaryColumns": [
     "text_col_1",
     "text_col_2"
 ]}

You can configure text indexes in the following scenarios:

  • Adding a new table with text index enabled on one or more columns.

  • Adding a new column with text index enabled to an existing table.

  • Enabling a text index on an existing column.

When you're using a text index, add the indexed column to the noDictionaryColumns columns list to reduce unnecessary storage overhead.

For instructions on that configuration property, see the Raw value forward index documentation.

Text index creation

Once the text index is enabled on one or more columns through a table configuration, segment generation code will automatically create the text index (per column).

Text index is supported for both offline and real-time segments.

Text parsing and tokenization

The original text document (denoted by a value in the column that has text index enabled) is parsed, tokenized and individual "indexable" terms are extracted. These terms are inserted into the index.

Pinot's text index is built on top of Lucene. Lucene's standard english text tokenizer generally works well for most classes of text. To build a custom text parser and tokenizer to suit particular user requirements, this can be made configurable for the user to specify on a per-column text-index basis.

There is a default set of "stop words" built in Pinot's text index. This is a set of high frequency words in English that are excluded for search efficiency and index size, including:

"a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it",
"no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "than", "there", "these", 
"they", "this", "to", "was", "will", "with", "those"

Any occurrence of these words will be ignored by the tokenizer during index creation and search.

In some cases, users might want to customize the set. A good example would be when IT (Information Technology) appears in the text that collides with "it", or some context-specific words that are not informative in the search. To do this, one can config the words in fieldConfig to include/exclude from the default stop words:

"fieldConfigList":[
  {
     "name":"text_col_1",
     "encodingType":"RAW",
     "indexType":"TEXT",
     "properties": {
        "stopWordInclude": "incl1, incl2, incl3",
        "stopWordExclude": "it"
     }
  }
]

The words should be comma separated and in lowercase. Words appearing in both lists will be excluded as expected.

Writing text search queries

The TEXT_MATCH function enables using text search in SQL/PQL.

TEXT_MATCH(text_column_name, search_expression)

  • text_column_name - name of the column to do text search on.

  • search_expression - search query

You can use TEXT_MATCH function as part of queries in the WHERE clause, like this:

SELECT COUNT(*) FROM Foo WHERE TEXT_MATCH(...)
SELECT * FROM Foo WHERE TEXT_MATCH(...)

You can also use the TEXT_MATCH filter clause with other filter operators. For example:

SELECT COUNT(*) FROM Foo WHERE TEXT_MATCH(...) AND some_other_column_1 > 20000
SELECT COUNT(*) FROM Foo WHERE TEXT_MATCH(...) AND some_other_column_1 > 20000 AND some_other_column_2 < 100000

You can combine multiple TEXT_MATCH filter clauses:

SELECT COUNT(*) FROM Foo WHERE TEXT_MATCH(text_col_1, ....) AND TEXT_MATCH(text_col_2, ...)

TEXT_MATCH can be used in WHERE clause of all kinds of queries supported by Pinot.

  • Selection query which projects one or more columns

    • User can also include the text column name in select list

  • Aggregation query

  • Aggregation GROUP BY query

The search expression (the second argument to TEXT_MATCH function) is the query string that Pinot will use to perform text search on the column's text index.

Phrase query

This query is used to seek out an exact match of a given phrase, where terms in the user-specified phrase appear in the same order in the original text document.

The following example reuses the earlier example of resume text data containing 14 documents to walk through queries. In this sentence, "document" means the column value. The data is stored in the SKILLS_COL column and we have created a text index on this column.

Java, C++, worked on open source projects, coursera machine learning
Machine learning, Tensor flow, Java, Stanford university,
Distributed systems, Java, C++, Go, distributed query engines for analytics and data warehouses, Machine learning, spark, Kubernetes, transaction processing
Java, Python, C++, Machine learning, building and deploying large scale production systems, concurrency, multi-threading, CPU processing
C++, Python, Tensor flow, database kernel, storage, indexing and transaction processing, building large scale systems, Machine learning
Amazon EC2, AWS, hadoop, big data, spark, building high performance scalable systems, building and deploying large scale production systems, concurrency, multi-threading, Java, C++, CPU processing
Distributed systems, database development, columnar query engine, database kernel, storage, indexing and transaction processing, building large scale systems
Distributed systems, Java, realtime streaming systems, Machine learning, spark, Kubernetes, distributed storage, concurrency, multi-threading
CUDA, GPU, Python, Machine learning, database kernel, storage, indexing and transaction processing, building large scale systems
Distributed systems, Java, database engine, cluster management, docker image building and distribution
Kubernetes, cluster management, operating systems, concurrency, multi-threading, apache airflow, Apache Spark,
Apache spark, Java, C++, query processing, transaction processing, distributed storage, concurrency, multi-threading, apache airflow
Big data stream processing, Apache Flink, Apache Beam, database kernel, distributed query engines for analytics and data warehouses
CUDA, GPU processing, Tensor flow, Pandas, Python, Jupyter notebook, spark, Machine learning, building high performance scalable systems
Distributed systems, Apache Kafka, publish-subscribe, building and deploying large scale production systems, concurrency, multi-threading, C++, CPU processing, Java
Realtime stream processing, publish subscribe, columnar processing for data warehouses, concurrency, Java, multi-threading, C++,
C++, Java, Python, realtime streaming systems, Machine learning, spark, Kubernetes, transaction processing, distributed storage, concurrency, multi-threading, apache airflow
Databases, columnar query processing, Apache Arrow, distributed systems, Machine learning, cluster management, docker image building and distribution
Database engine, OLAP systems, OLTP transaction processing at large scale, concurrency, multi-threading, GO, building large scale systems

This example queries the SKILLS_COL column to look for documents where each matching document MUST contain phrase "Distributed systems":

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"Distributed systems"')

The search expression is '\"Distributed systems\"'

  • The search expression is always specified within single quotes '<your expression>'

  • Since we are doing a phrase search, the phrase should be specified within double quotes inside the single quotes and the double quotes should be escaped

    • '\"<your phrase>\"'

The above query will match the following documents:

Distributed systems, Java, C++, Go, distributed query engines for analytics and data warehouses, Machine learning, spark, Kubernetes, transaction processing
Distributed systems, database development, columnar query engine, database kernel, storage, indexing and transaction processing, building large scale systems
Distributed systems, Java, realtime streaming systems, Machine learning, spark, Kubernetes, distributed storage, concurrency, multi-threading
Distributed systems, Java, database engine, cluster management, docker image building and distribution
Distributed systems, Apache Kafka, publish-subscribe, building and deploying large scale production systems, concurrency, multi-threading, C++, CPU processing, Java
Databases, columnar query processing, Apache Arrow, distributed systems, Machine learning, cluster management, docker image building and distribution

But it won't match the following document:

Distributed data processing, systems design experience

This is because the phrase query looks for the phrase occurring in the original document "as is". The terms as specified by the user in phrase should be in the exact same order in the original document for the document to be considered as a match.

NOTE: Matching is always done in a case-insensitive manner.

The next example queries the SKILLS_COL column to look for documents where each matching document MUST contain phrase "query processing":

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"query processing"')

The above query will match the following documents:

Apache spark, Java, C++, query processing, transaction processing, distributed storage, concurrency, multi-threading, apache airflow
Databases, columnar query processing, Apache Arrow, distributed systems, Machine learning, cluster management, docker image building and distribution"

Term query

Term queries are used to search for individual terms.

This example will query the SKILLS_COL column to look for documents where each matching document MUST contain the term 'Java'.

As mentioned earlier, the search expression is always within single quotes. However, since this is a term query, we don't have to use double quotes within single quotes.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, 'Java')

Composite query using Boolean operators

The Boolean operators AND and OR are supported and we can use them to build a composite query. Boolean operators can be used to combine phrase and term queries in any arbitrary manner

This example queries the SKILLS_COL column to look for documents where each matching document MUST contain the phrases "machine learning" and "tensor flow". This combines two phrases using the AND Boolean operator.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"Machine learning" AND "Tensor Flow"')

The above query will match the following documents:

Machine learning, Tensor flow, Java, Stanford university,
C++, Python, Tensor flow, database kernel, storage, indexing and transaction processing, building large scale systems, Machine learning
CUDA, GPU processing, Tensor flow, Pandas, Python, Jupyter notebook, spark, Machine learning, building high performance scalable systems

This example queries the SKILLS_COL column to look for documents where each document MUST contain the phrase "machine learning" and the terms 'gpu' and 'python'. This combines a phrase and two terms using Boolean operators.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"Machine learning" AND gpu AND python')

The above query will match the following documents:

CUDA, GPU, Python, Machine learning, database kernel, storage, indexing and transaction processing, building large scale systems
CUDA, GPU processing, Tensor flow, Pandas, Python, Jupyter notebook, spark, Machine learning, building high performance scalable systems

When using Boolean operators to combine term(s) and phrase(s) or both, note that:

  • The matching document can contain the terms and phrases in any order.

  • The matching document may not have the terms adjacent to each other (if this is needed, use appropriate phrase query).

Use of the OR operator is implicit. In other words, if phrase(s) and term(s) are not combined using AND operator in the search expression, the OR operator is used by default:

This example queries the SKILLS_COL column to look for documents where each document MUST contain ANY one of:

  • phrase "distributed systems" OR

  • term 'java' OR

  • term 'C++'.

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"distributed systems" Java C++')

Grouping using parentheses is supported:

This example queries the SKILLS_COL column to look for documents where each document MUST contain

  • phrase "distributed systems" AND

  • at least one of the terms Java or C++

Here the terms Java and C++ are grouped without any operator, which implies the use of OR. The root operator AND is used to combine this with phrase "distributed systems"

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, '"distributed systems" AND (Java C++)')

Prefix query

Prefix queries can be done in the context of a single term. We can't use prefix matches for phrases.

This example queries the SKILLS_COL column to look for documents where each document MUST contain text like stream, streaming, streams etc

SELECT SKILLS_COL 
FROM MyTable 
WHERE TEXT_MATCH(SKILLS_COL, 'stream*')

The above query will match the following documents:

Distributed systems, Java, realtime streaming systems, Machine learning, spark, Kubernetes, distributed storage, concurrency, multi-threading
Big data stream processing, Apache Flink, Apache Beam, database kernel, distributed query engines for analytics and data warehouses
Realtime stream processing, publish subscribe, columnar processing for data warehouses, concurrency, Java, multi-threading, C++,
C++, Java, Python, realtime streaming systems, Machine learning, spark, Kubernetes, transaction processing, distributed storage, concurrency, multi-threading, apache airflow

Regular Expression Query

Phrase and term queries work on the fundamental logic of looking up the terms in the text index. The original text document (a value in the column with text index enabled) is parsed, tokenized, and individual "indexable" terms are extracted. These terms are inserted into the index.

Based on the nature of the original text and how the text is segmented into tokens, it is possible that some terms don't get indexed individually. In such cases, it is better to use regular expression queries on the text index.

Consider a server log as an example where we want to look for exceptions. A regex query is suitable here as it is unlikely that 'exception' is present as an individual indexed token.

Syntax of a regex query is slightly different from queries mentioned earlier. The regular expression is written between a pair of forward slashes (/).

SELECT SKILLS_COL 
FROM MyTable 
WHERE text_match(SKILLS_COL, '/.*Exception/')

The above query will match any text document containing "exception".

Phrase search with wildcard term matching

Phrase search with wildcard and prefix term matching can match patterns like "pache pino" to the text "Apache Pinot" directly. The kind of queries is very common in use case like log search where user needs to search substrings across term boundary in long text. To enable such search (which can be more costly because Lucene by default does not allow * to start a pattern to avoid costly term matching), one can add a new config key to the column text index config:

"fieldConfigList":[
  {
     "name":"text_col_1",
     "encodingType":"RAW",
     "indexType":"TEXT",
     "properties": {
        "enablePrefixSuffixMatchingInPhraseQueries": "true"
     }
  }
]

With this config enabled, one can now perform the pharse wildcard search using the following syntax like

SELECT SKILLS_COL 
FROM MyTable 
WHERE text_match(SKILLS_COL, '*pache pino*')

to match the string "Apache pinot" in the SIKLLS_COL. Boolean expressions like 'pache pino AND apche luce' are are supported.

Deciding Query Types

Combining phrase and term queries using Boolean operators and grouping lets you build a complex text search query expression.

The key thing to remember is that phrases should be used when the order of terms in the document is important and when separating the phrase into individual terms doesn't make sense from end user's perspective.

An example would be phrase "machine learning".

TEXT_MATCH(column, '"machine learning"')

However, if we are searching for documents matching Java and C++ terms, using phrase query "Java C++" will actually result in in partial results (could be empty too) since now we are relying the on the user specifying these skills in the exact same order (adjacent to each other) in the resume text.

TEXT_MATCH(column, '"Java C++"')

Term query using Boolean AND operator is more appropriate for such cases

TEXT_MATCH(column, 'Java AND C++')

Text Index Tuning

To improve Lucene index creation time, some configs have been provided. Field Config properties luceneUseCompoundFile and luceneMaxBufferSizeMB can provide faster index writing at but may increase file descriptors and/or memory pressure.

JSON index

This page describes configuring the JSON index for Apache Pinot.

The JSON index can be applied to JSON string columns to accelerate value lookups and filtering for the column.

When to use JSON index

JSON strings can be used to represent arrays, maps, and nested fields without forcing a fixed schema. While JSON strings are flexible, filtering on JSON string columns is expensive, so consider the use case.

Suppose we have some JSON records similar to the following sample record stored in the person column:

{
  "name": "adam",
  "age": 30,
  "country": "us",
  "addresses":
  [
    {
      "number" : 112,
      "street" : "main st",
      "country" : "us"
    },
    {
      "number" : 2,
      "street" : "second st",
      "country" : "us"
    },
    {
      "number" : 3,
      "street" : "third st",
      "country" : "ca"
    }
  ]
}

Without an index, to look up the key and filter records based on the value, Pinot must scan and reconstruct the JSON object from the JSON string for every record, look up the key and then compare the value.

For example, in order to find all persons whose name is "adam", the query will look like:

SELECT *
FROM mytable
WHERE JSON_EXTRACT_SCALAR(person, '$.name', 'STRING') = 'adam'

The JSON index is designed to accelerate the filtering on JSON string columns without scanning and reconstructing all the JSON objects.

Enable and configure a JSON index

To enable the JSON index, you can configure the following options in the table configuration:

Config Key
Description
Type
Default

maxLevels

Max levels to flatten the json object (array is also counted as one level)

int

-1 (unlimited)

excludeArray

Whether to exclude array when flattening the object

boolean

false (include array)

disableCrossArrayUnnest

Whether to not unnest multiple arrays (unique combination of all elements in those arrays). If document contains two arrays holding, respectively M and N elements, then flattening produces M*N documents. If number of such combinations reaches 100k, error with "Got too many combinations" message is thrown.

boolean

false (calculate unique combination of all elements)

includePaths

Only include the given paths, e.g. "$.a.b", "$.a.c[*]" (mutual exclusive with excludePaths). Paths under the included paths will be included, e.g. "$.a.b.c" will be included when "$.a.b" is configured to be included.

Set<String>

null (include all paths)

excludePaths

Exclude the given paths, e.g. "$.a.b", "$.a.c[*]" (mutual exclusive with includePaths). Paths under the excluded paths will also be excluded, e.g. "$.a.b.c" will be excluded when "$.a.b" is configured to be excluded.

Set<String>

null (include all paths)

excludeFields

Exclude the given fields, e.g. "b", "c", even if it is under the included paths.

Set<String>

null (include all fields)

indexPaths

Index the given paths, e.g. *.*, a.**. Paths matches the indexed paths will be indexed, e.g. a.** will index everything whose first layer is "a", *.* will index everything with maxLevels=2. This config could work together with other configs, e.g. includePaths, excludePaths, maxLevels but usually does not have to because it should be flexible enough to catch any scenarios.

Set<String>

null that is equivalent to ** (include all fields)

maxValueLength

If the value of a json node (not the whole document) is longer than given value then replace it with $SKIPPED$ before indexing.

int

0 (disabled)

skipInvalidJson

If set, while adding json to index, instead of throwing exception, replace ill-formed json with empty key/path and $SKIPPED$ value .

boolean

false (disabled)

Recommended way to configure

The recommended way to configure a JSON index is in the fieldConfigList.indexes object, within the json key.

json index defined in tableConfig
{
  "fieldConfigList": [
    {
      "name": "person",
      "indexes": {
        "json": {
          "maxLevels": 2,
          "excludeArray": false,
          "disableCrossArrayUnnest": true,
          "includePaths": null,
          "excludePaths": null,
          "excludeFields": null,
          "indexPaths": null
        }
      }
    }
  ],
  ...
}

All options are optional, so the following is a valid configuration that use the default parameter values:

json index defined in tableConfig
{
  "fieldConfigList": [
    {
      "name": "person",
      "indexes": {
        "json": {}
      }
    }
  ],
  ...
}

Deprecated ways to configure JSON indexes

There are two older ways to configure the indexes that can be configured in the tableIndexConfig section inside table config.

The first one uses the same JSON explained above, but it is defined inside tableIndexConfig.jsonIndexConfigs.<column name>:

older way to configure json indexes in table config
{
  "tableIndexConfig": {
    "jsonIndexConfigs": {
      "person": {
        "maxLevels": 2,
        "excludeArray": false,
        "disableCrossArrayUnnest": true,
        "includePaths": null,
        "excludePaths": null,
        "excludeFields": null,
        "indexPaths": null
      },
      ...
    },
    ...
  }
}

Like in the previous case, all parameters are optional, so the following is also valid:

json index with default config
{
  "tableIndexConfig": {
    "jsonIndexConfigs": {
      "person": {},
      ...
    },
    ...
  }
}

The last option does not support to configure any parameter. In order to use this option, add the name of the column in tableIndexConfig.jsonIndexColumns like in this example:

json index with default config
{
  "tableIndexConfig": {
    "jsonIndexColumns": [
      "person",
      ...
    ],
    ...
  }
}

Example:

With the following JSON document:

{
  "name": "adam",
  "age": 20,
  "addresses": [
    {
      "country": "us",
      "street": "main st",
      "number": 1
    },
    {
      "country": "ca",
      "street": "second st",
      "number": 2
    }
  ],
  "skills": [
    "english",
    "programming"
  ]
}

Using the default setting, we will flatten the document into the following records:

{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1,
  "skills[1]": "programming"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "addresses[1].number": 2,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "addresses[1].number": 2,
  "skills[1]": "programming"
}

With maxValueLength set to 9:

{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1,
  "skills[1]": "$SKIPPED$"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "addresses[1].number": 2,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "addresses[1].number": 2,
  "skills[1]": "$SKIPPED$"
}

With maxLevels set to 1:

{
  "name": "adam",
  "age": 20
}

With maxLevels set to 2:

{
  "name": "adam",
  "age": 20,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "skills[1]": "programming"
}

With excludeArray set to true:

{
  "name": "adam",
  "age": 20
}

With disableCrossArrayUnnest set to true:

{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1
},
{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "addresses[0].number": 1
},
{
  "name": "adam",
  "age": 20,
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "skills[1]": "programming"
}

When cross array un-nesting is disabled, then number of documents produced during JSON flattening is the sum of all array sizes, e.g. 2+2 = 4 in the example above.

With disableCrossArrayUnnest set to false:

{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].number": 1,
  "addresses[0].street": "main st",
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
  "addresses[0].number": 1,
  "addresses[0].street": "main st",
  "skills[1]": "programming"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].number": 2,
  "addresses[1].street": "second st",
  "skills[0]": "english"
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
  "addresses[1].number": 2,
  "addresses[1].street": "second st",
  "skills[1]": "programming"
}

When cross array un-nesting is enabled, then number of documents produced during JSON flattening is the product of all array sizes, e.g. 2*2 = 4 in the example above. If JSON contains multiple large nested arrays, it might be necessary to disable cross array un-nesting (disableCrossArrayUnnest=true) to avoid hitting the 100k flattened documents limit and triggering 'Got to many combinations' error.

With includePaths set to ["$.name", "$.addresses[*].country"]:

{
  "name": "adam",
  "addresses[0].country": "us"
},
{
  "name": "adam",
  "addresses[1].country": "ca"
}

With excludePaths set to ["$.age", "$.addresses[*].number"]:

{
  "name": "adam",
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "skills[0]": "english"
},
{
  "name": "adam",
  "addresses[0].country": "us",
  "addresses[0].street": "main st",
  "skills[1]": "programming"
},
{
  "name": "adam",
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "skills[0]": "english"
},
{
  "name": "adam",
  "addresses[1].country": "ca",
  "addresses[1].street": "second st",
  "skills[1]": "programming"
}

With excludeFields set to ["age", "street"]:

{
  "name": "adam",
  "addresses[0].country": "us",
  "addresses[0].number": 1,
  "skills[0]": "english"
},
{
  "name": "adam",
  "addresses[0].country": "us",
  "addresses[0].number": 1,
  "skills[1]": "programming"
},
{
  "name": "adam",
  "addresses[1].country": "ca",
  "addresses[1].number": 2,
  "skills[0]": "english"
},
{
  "name": "adam",
  "addresses[1].country": "ca",
  "addresses[1].number": 2,
  "skills[1]": "programming"
}

With indexPaths set to ["*", "address..country"]:

{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
},
{
  "name": "adam",
  "age": 20,
  "addresses[0].country": "us",
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
},
{
  "name": "adam",
  "age": 20,
  "addresses[1].country": "ca",
}

With skipInvalidJson set to true, if we corrupt the original JSON, e.g. to

{ _invalid_json_
  "name": "adam",
  "age": 20,
  "addresses": [...]
  "skills": [...]
}

then flattening will be produce:

{ "": "$SKIPPED$" }

Note that the JSON index can only be applied to STRING/JSON columns whose values are JSON strings.

To reduce unnecessary storage overhead when using a JSON index, we recommend that you add the indexed column to the noDictionaryColumns columns list.

For instructions on that configuration property, see the Raw value forward index documentation.

How to use the JSON index

The JSON index can be used via the JSON_MATCH predicate for filtering: JSON_MATCH(<column>, '<filterExpression>'). For example, to find every entry with the name "adam":

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.name"=''adam''')

Note that the quotes within the filter expression need to be escaped.

The JSON index can also be used via the JSON_EXTRACT_INDEX predicate for value extraction (optionally with filtering): JSON_EXTRACT_INDEX(<column>, '<jsonPath>', ['resultsType'], ['filter']). For example, to extract every value for path $.name when the path $.id is less than 10:

SELECT jsonextractindex(repo, '$.name', 'STRING', 'dummyValue', '"$.id" < 10')
FROM mytable

More in-depth examples can be found in the JSON_EXTRACT_INDEX function documentation.

Supported filter expressions

Simple key lookup

Find all persons whose name is "adam":

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.name"=''adam''')

Chained key lookup

Find all persons who have an address (one of the addresses) with number 112:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].number"=112')

Find all persons who have at least one address that is not in the US:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].country" != ''us''')

Regex based lookup

Find all persons who have an address (one of the addresses) where the street contains the term 'st':

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, 'REGEXP_LIKE("$.addresses[*].street", ''.*st.*'')')

Range lookup

Find all persons whose age is greater than 18:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.age" > 18')

Nested filter expression

Find all persons whose name is "adam" and also have an address (one of the addresses) with number 112:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.name"=''adam'' AND "$.addresses[*].number"=112')

NOT IN and != can't be used in nested filter expressions in Pinot versions older than 1.2.0. Note that IS NULL cannot be used in nested filter expressions currently.

Array access

Find all persons whose first address has number 112:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[0].number"=112')

Since JSON index works based on flattened JSON documents, if cross array un-nesting is disabled ( disableCrossArrayUnnest = true ), then querying more than one array in a single JSON_MATCH function call returns empty result, e.g.

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].country"=''us'' AND "$.skills[*]"=''english''')

In such cases expression should be split into multiple JSON_MATCH calls, e.g.

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].country"=''us''')
AND   JSON_MATCH(person, '"$.skills[*]"=''english''')

Existence check

Find all persons who have a phone field within the JSON:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.phone" IS NOT NULL')

Find all persons whose first address does not contain floor field within the JSON:

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[0].floor" IS NULL')

JSON context is maintained

The JSON context is maintained for object elements within an array, meaning the filter won't cross-match different objects in the array.

To find all persons who live on "main st" in "ca":

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].street"=''main st'' AND "$.addresses[*].country"=''ca''')

This query won't match "adam" because none of his addresses matches both the street and the country.

If you don't want JSON context, use multiple separate JSON_MATCH predicates. For example, to find all persons who have addresses on "main st" and have addresses in "ca" (matches need not have the same address):

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[*].street"=''main st''') 
  AND JSON_MATCH(person, '"$.addresses[*].country"=''ca''')

This query will match "adam" because one of his addresses matches the street and another one matches the country.

The array index is maintained as a separate entry within the element, so in order to query different elements within an array, multiple JSON_MATCH predicates are required. For example, to find all persons who have first address on "main st" and second address on "second st":

SELECT ...
FROM mytable
WHERE JSON_MATCH(person, '"$.addresses[0].street"=''main st''') 
  AND JSON_MATCH(person, '"$.addresses[1].street"=''second st''')

Supported JSON values

Object

See examples above.

Array

["item1", "item2", "item3"]

To find the records with array element "item1" in "arrayCol":

SELECT ...
FROM mytable
WHERE JSON_MATCH(arrayCol, '"$[*]"=''item1''')

To find the records with second array element "item2" in "arrayCol":

SELECT ...
FROM mytable
WHERE JSON_MATCH(arrayCol, '"$[1]"=''item2''')

Value

123
1.23
"Hello World"

To find the records with value 123 in "valueCol":

SELECT ...
FROM mytable
WHERE JSON_MATCH(valueCol, '"$"=123')

Null

null

To find the records with null in "nullableCol":

SELECT ...
FROM mytable
WHERE JSON_MATCH(nullableCol, '"$" IS NULL')

Limitations

  1. The key (left-hand side) of the filter expression must be the leaf level of the JSON object, for example, "$.addresses[*]"='main st' won't work.

0.9.0

Summary

This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.

The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9

Support Segment Merge and Roll-up

LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.

To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.

At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.

Major Changes:

  • Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)

  • Merge/Rollup task scheduler for offline tables. (#7178)

  • Fix MergeRollupTask uploading segments not updating their metadata (#7289)

  • MergeRollupTask integration tests (#7283)

  • Add mergeRollupTask delay metrics (#7368)

  • MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)

  • Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)

UI Improvement

This release also sees improvements to Pinot’s query console UI.

  • Cmd+Enter shortcut to run query in query console (#7359)

  • Showing tooltip in SQL Editor (#7387)

  • Make the SQL Editor box expandable (#7381)

  • Fix tables ordering by number of segments (#7564)

SQL Improvements

There have also been improvements and additions to Pinot’s SQL implementation.

New functions:

  • IN (#7542)

  • LASTWITHTIME (#7584)

  • ID_SET on MV columns (#7355)

  • Raw results for Percentile TDigest and Est (#7226),

  • Add timezone as argument in function toDateTime (#7552)

New predicates are supported:

  • LIKE(#7214)

  • REGEXP_EXTRACT(#7114)

  • FILTER(#7566)

Query compatibility improvements:

  • Infer data type for Literal (#7332)

  • Support logical identifier in predicate (#7347)

  • Support JSON queries with top-level array path expression. (#7511)

  • Support configurable group by trim size to improve results accuracy (#7241)

Performance Improvements

This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:

  • Reduce the disk usage for segment conversion task (#7193)

  • Simplify association between Java Class and PinotDataType for faster mapping (#7402)

  • Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)

  • Replace MINUS with STRCMP (#7394)

  • Bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)

  • Use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)

  • Add option to limit thread usage per query (#7492)

  • Improved range queries (#7513)

  • Faster bitmap scans (#7530)

  • Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)

  • Map bitmaps through a bounded window to avoid excessive disk pressure (#7535)

  • Allow RLE compression of bitmaps for smaller file sizes (#7582)

  • Support raw index properties for columns with JSON and RANGE indexes (#7615)

  • Enhance BloomFilter rule to include IN predicate(#7444) (#7624)

  • Introduce LZ4_WITH_LENGTH chunk compression type (#7655)

  • Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)

  • Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)

  • During segment pruning, release the bloom filter after each segment is processed (#7668)

  • Fix JSONPath cache inefficient issue (#7409)

  • Optimize getUnpaddedString with SWAR padding search (#7708)

  • Lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)

  • Inline binary comparison ops to prevent function call overhead (#7709)

  • Memoize literals in query context in order to deduplicate them (#7720)

Other Notable New Features and Changes

  • Human Readable Controller Configs (#7173)

  • Add the support of geoToH3 function (#7182)

  • Add Apache Pulsar as Pinot Plugin (#7223) (#7247)

  • Add dropwizard metrics plugin (#7263)

  • Introduce OR Predicate Execution On Star Tree Index (#7184)

  • Allow to extract values from array of objects with jsonPathArray (#7208)

  • Add Realtime table metadata and indexes API. (#7169)

  • Support array with mixing data types (#7234)

  • Support force download segment in reload API (#7249)

  • Show uncompressed znRecord from zk api (#7304)

  • Add debug endpoint to get minion task status. (#7300)

  • Validate CSV Header For Configured Delimiter (#7237)

  • Add auth tokens and user/password support to ingestion job command (#7233)

  • Add option to store the hash of the upsert primary key (#7246)

  • Add null support for time column (#7269)

  • Add mode aggregation function (#7318)

  • Support disable swagger in Pinot servers (#7341)

  • Delete metadata properly on table deletion (#7329)

  • Add basic Obfuscator Support (#7407)

  • Add AWS sts dependency to enable auth using web identity token. (#7017)(#7445)

  • Mask credentials in debug endpoint /appconfigs (#7452)

  • Fix /sql query endpoint now compatible with auth (#7230)

  • Fix case sensitive issue in BasicAuthPrincipal permission check (#7354)

  • Fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)

  • Add segmentNameGeneratorType config to IndexingConfig (#7346)

  • Support trigger PeriodicTask manually (#7174)

  • Add endpoint to check minion task status for a single task. (#7353)

  • Showing partial status of segment and counting CONSUMING state as good segment status (#7327)

  • Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)

  • Check schema backward-compatibility when updating schema through addSchema with override (#7374)

  • Optimize IndexedTable (#7373)

  • Support indices remove in V3 segment format (#7301)

  • Optimize TableResizer (#7392)

  • Introduce resultSize in IndexedTable (#7420)

  • Offset based real-time consumption status checker (#7267)

  • Add causes to stack trace return (#7460)

  • Create controller resource packages config key (#7488)

  • Enhance TableCache to support schema name different from table name (#7525)

  • Add validation for realtimeToOffline task (#7523)

  • Unify CombineOperator multi-threading logic (#7450)

  • Support no downtime rebalance for table with 1 replica in TableRebalancer (#7532)

  • Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)

  • Adjust tuner api (#7553)

  • Adding config for metrics library (#7551)

  • Add geo type conversion scalar functions (#7573)

  • Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)

  • Add MV raw forward index and MV BYTES data type (#7595)

  • Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)

  • Improve get tenant API to differentiate offline and real-time tenants (#7548)

  • Refactor query rewriter to interfaces and implementations to allow customization (#7576)

  • In ServiceStartable, apply global cluster config in ZK to instance config (#7593)

  • Make dimension tables creation bypass tenant validation (#7559)

  • Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)

  • Reject query with identifiers not in schema (#7590)

  • Round Robin IP addresses when retry uploading/downloading segments (#7585)

  • Support multi-value derived column in offline table reload (#7632)

  • Support segmentNamePostfix in segment name (#7646)

  • Add select segments API (#7651)

  • Controller getTableInstance() call now returns the list of live brokers of a table. (#7556)

  • Allow MV Field Support For Raw Columns in Text Indices (#7638)

  • Allow override distinctCount to segmentPartitionedDistinctCount (#7664)

  • Add a quick start with both UPSERT and JSON index (#7669)

  • Add revertSegmentReplacement API (#7662)

  • Smooth segment reloading with non blocking semantic (#7675)

  • Clear the reused record in PartitionUpsertMetadataManager (#7676)

  • Replace args4j with picocli (#7665)

  • Handle datetime column consistently (#7645)(#7705)

  • Allow to carry headers with query requests (#7696) (#7712)

  • Allow adding JSON data type for dimension column types (#7718)

  • Separate SegmentDirectoryLoader and tierBackend concepts (#7737)

  • Implement size balanced V4 raw chunk format (#7661)

  • Add presto-pinot-driver lib (#7384)

Major Bug fixes

  • Fix null pointer exception for non-existed metric columns in schema for JDBC driver (#7175)

  • Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)

  • Fixed pinot java client to add zkClient close (#7196)

  • Ignore query json parse errors (#7165)

  • Fix shutdown hook for PinotServiceManager (#7251) (#7253)

  • Make STRING to BOOLEAN data type change as backward compatible schema change (#7259)

  • Replace gcp hardcoded values with generic annotations (#6985)

  • Fix segment conversion executor for in-place conversion (#7265)

  • Fix reporting consuming rate when the Kafka partition level consumer isn't stopped (#7322)

  • Fix the issue with concurrent modification for segment lineage (#7343)

  • Fix TableNotFound error message in PinotHelixResourceManager (#7340)

  • Fix upload LLC segment endpoint truncated download URL (#7361)

  • Fix task scheduling on table update (#7362)

  • Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)

  • Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema (#7366)

  • Fix currentOffset volatility in consuming segment(#7365)

  • Fix misleading error msg for missing URI (#7367)

  • Fix the correctness of getColumnIndices method (#7370)

  • Fix SegmentZKMetadta time handling (#7375)

  • Fix retention for cleaning up segment lineage (#7424)

  • Fix segment generator to not return illegal filenames (#7085)

  • Fix missing LLC segments in segment store by adding controller periodic task to upload them (#6778)

  • Fix parsing error messages returned to FileUploadDownloadClient (#7428)

  • Fix manifest scan which drives /version endpoint (#7456)

  • Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)

  • Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)

  • Fix pql double quote checker exception (#7485)

  • Fix minion metrics exporter config (#7496)

  • Fix segment unable to retry issue by catching timeout exception during segment replace (#7509)

  • Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)

  • Fix segment generation commands (#7527)

  • Return non zero from main with exception (#7482)

  • Fix parquet plugin shading error (#7570)

  • Fix the lowest partition id is not 0 for LLC (#7066)

  • Fix star-tree index map when column name contains '.' (#7623)

  • Fix cluster manager URLs encoding issue(#7639)

  • Fix fieldConfig nullable validation (#7648)

  • Fix verifyHostname issue in FileUploadDownloadClient (#7703)

  • Fix TableCache schema to include the built-in virtual columns (#7706)

  • Fix DISTINCT with AS function (#7678)

  • Fix SDF pattern in DataPreprocessingHelper (#7721)

  • Fix fields missing issue in the source in ParquetNativeRecordReader (#7742)

Star-tree index

This page describes the indexing techniques available in Apache Pinot.

In this page you will learn what a star-tree index is and gain a conceptual understanding of how one works.

Unlike other index techniques which work on a single column, the star-tree index is built on multiple columns and utilizes pre-aggregated results to significantly reduce the number of values to be processed, resulting in improved query performance.

One of the biggest challenges in real-time OLAP systems is achieving and maintaining tight SLAs on latency and throughput on large data sets. Existing techniques such as sorted index or inverted index help improve query latencies, but speed-ups are still limited by the number of documents that need to be processed to compute results. On the other hand, pre-aggregating the results ensures a constant upper bound on query latencies, but can lead to storage space explosion.

Use the star-tree index to utilize pre-aggregated documents to achieve both low query latencies and efficient use of storage space for aggregation and group-by queries.

Existing solutions

Consider the following data set, which is used here as an example to discuss these indexes:

Country
Browser
Locale
Impressions

CA

Chrome

en

400

CA

Firefox

fr

200

MX

Safari

es

300

MX

Safari

en

100

USA

Chrome

en

600

USA

Firefox

es

200

USA

Firefox

en

400

Sorted index

In this approach, data is sorted on a primary key, which is likely to appear as filter in most queries in the query set.

This reduces the time to search the documents for a given primary key value from linear scan O(n) to binary search O(logn), and also keeps good locality for the documents selected.

While this is a significant improvement over linear scan, there are still a few issues with this approach:

  • While sorting on one column does not require additional space, sorting on additional columns requires additional storage space to re-index the records for the various sort orders.

  • While search time is reduced from O(n) to O(logn), overall latency is still a function of the total number of documents that need to be processed to answer a query.

Inverted index

In this approach, for each value of a given column, we maintain a list of document id’s where this value appears.

Below are the inverted indexes for columns ‘Browser’ and ‘Locale’ for our example data set:

Browser
Doc Id

Firefox

1,5,6

Chrome

0,4

Safari

2,3

Locale
Doc Id

en

0,3,4,6

es

2,5

fr

1

For example, if we want to get all the documents where ‘Browser’ is ‘Firefox’, we can look up the inverted index for ‘Browser’ and identify that it appears in documents [1, 5, 6].

Using an inverted index, we can reduce the search time to constant time O(1). The query latency, however, is still a function of the selectivity of the query: it increases with the number of documents that need to be processed to answer the query.

Pre-aggregation

In this technique, we pre-compute the answer for a given query set upfront.

In the example below, we have pre-aggregated the total impressions for each country:

Country
Impressions

CA

600

MX

400

USA

1200

With this approach, answering queries about total impressions for a country is a value lookup, because we have eliminated the need to process a large number of documents. However, to be able to answer queries that have multiple predicates means we would need to pre-aggregate for various combinations of different dimensions, which leads to an exponential increase in storage space.

Star-tree solution

On one end of the spectrum we have indexing techniques that improve search times with a limited increase in space, but don't guarantee a hard upper bound on query latencies. On the other end of the spectrum, we have pre-aggregation techniques that offer a hard upper bound on query latencies, but suffer from exponential explosion of storage space

The star-tree data structure offers a configurable trade-off between space and time and lets us achieve a hard upper bound for query latencies for a given use case. The following sections cover the star-tree data structure, and explain how Pinot uses this structure to achieve low latencies with high throughput.

Definitions

Tree structure

The star-tree index stores data in a structure that consists of the following properties:

Star-tree index structure
  • Root node (Orange): Single root node, from which the rest of the tree can be traversed.

  • Leaf node (Blue): A leaf node can containing at most T records, where T is configurable.

  • Non-leaf node (Green): Nodes with more than T records are further split into children nodes.

  • Star node (Yellow): Non-leaf nodes can also have a special child node called the star node. This node contains the pre-aggregated records after removing the dimension on which the data was split for this level.

  • Dimensions split order ([D1, D2]): Nodes at a given level in the tree are split into children nodes on all values of a particular dimension. The dimensions split order is an ordered list of dimensions that is used to determine the dimension to split on for a given level in the tree.

Node properties

The properties stored in each node are as follows:

  • Dimension: The dimension that the node is split on

  • Start/End Document Id: The range of documents this node points to

  • Aggregated Document Id: One single document that is the aggregation result of all documents pointed by this node

Index generation

The star-tree index is generated in the following steps:

  • The data is first projected as per the dimensionsSplitOrder. Only the dimensions from the split order are reserved, others are dropped. For each unique combination of reserved dimensions, metrics are aggregated per configuration. The aggregated documents are written to a file and served as the initial star-tree documents (separate from the original documents).

  • Sort the star-tree documents based on the dimensionsSplitOrder. It is primary-sorted on the first dimension in this list, and then secondary sorted on the rest of the dimensions based on their order in the list. Each node in the tree points to a range in the sorted documents.

  • The tree structure can be created recursively (starting at root node) as follows:

    • If a node has more than T records, it is split into multiple children nodes, one for each value of the dimension in the split order corresponding to current level in the tree.

    • A star node can be created (per configuration) for the current node, by dropping the dimension being split on, and aggregating the metrics for rows containing dimensions with identical values. These aggregated documents are appended to the end of the star-tree documents.

      If there is only one value for the current dimension, a star node won’t be created because the documents under the star node are identical to the single node.

  • The above step is repeated recursively until there are no more nodes to split.

  • Multiple star-trees can be generated based on different configurations (dimensionsSplitOrder, aggregations, T)

Aggregation

Aggregation is configured as a pair of aggregation functions and the column to apply the aggregation.

All types of aggregation function that have a bounded-sized intermediate result are supported.

Supported functions

  • COUNT

  • MIN

  • MAX

  • SUM

  • SUM_PRECISION

    • The maximum precision can be optionally configured in functionParameters using the key precision. For example: {"precision": 20}.

  • AVG

  • MIN_MAX_RANGE

  • PERCENTILE_EST

  • PERCENTILE_RAW_EST

  • PERCENTILE_TDIGEST

    • The compression factor for the TDigest histogram can be optionally configured in functionParameters using the key compressionFactor. For example: {"compressionFactor": 200}. If not configured, the default value of 100 will be used.

  • PERCENTILE_RAW_TDIGEST

    • The compression factor for the TDigest histogram can be optionally configured in functionParameters using the key compressionFactor. For example: {"compressionFactor": 200}. If not configured, the default value of 100 will be used.

  • DISTINCT_COUNT_BITMAP

    • NOTE: The intermediate result RoaringBitmap is not bounded-sized, use carefully on high cardinality columns.

  • DISTINCT_COUNT_HLL

    • The log2m value for the HyperLogLog structure can be optionally configured in functionParameters , for example: {"log2m": 16}. If not configured, the default value of 8 will be used. Remember that a larger log2m value leads to better accuracy but also a larger memory footprint.

  • DISTINCT_COUNT_RAW_HLL

    • The log2m value for the HyperLogLog structure can be optionally configured in functionParameters , for example: {"log2m": 16}. If not configured, the default value of 8 will be used. Remember that a larger log2m value leads to better accuracy but also a larger memory footprint.

  • DISTINCT_COUNT_HLL_PLUS

    • The p (precision value of normal set) and sp (precision value of sparse set) values for the HyperLogLogPlus structure can be optionally configured in functionParameters, for example: {"p": 16, "sp": 32}. If not configured, p will have the default value of 14 and sp will have the default value of 0.

  • DISTINCT_COUNT_RAW_HLL_PLUS

    • The p (precision value of normal set) and sp (precision value of sparse set) values for the HyperLogLogPlus structure can be optionally configured in functionParameters, for example: {"p": 16, "sp": 32}. If not configured, p will have the default value of 14 and sp will have the default value of 0.

  • DISTINCT_COUNT_THETA_SKETCH

    • The nominalEntries value for the Theta Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with DISTINCT_COUNT_THETA_SKETCH having nominalEntries=8192 or less for any power of 2.

  • DISTINCT_COUNT_RAW_THETA_SKETCH

    • The nominalEntries value for the Theta Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with DISTINCT_COUNT_RAW_THETA_SKETCH having nominalEntries=8192 or less for any power of 2.

  • DISTINCT_COUNT_TUPLE_SKETCH

    • The nominalEntries value for the Tuple Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with DISTINCT_COUNT_TUPLE_SKETCH having nominalEntries=8192 or less for any power of 2.

  • DISTINCT_COUNT_RAW_INTEGER_SUM_TUPLE_SKETCH

    • The nominalEntries value for the Tuple Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with DISTINCT_COUNT_RAW_INTEGER_SUM_TUPLE_SKETCH having nominalEntries=8192 or less for any power of 2.

  • SUM_VALUES_INTEGER_SUM_TUPLE_SKETCH

    • The nominalEntries value for the Tuple Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with SUM_VALUES_INTEGER_SUM_TUPLE_SKETCH having nominalEntries=8192 or less for any power of 2.

  • AVG_VALUE_INTEGER_SUM_TUPLE_SKETCH

    • The nominalEntries value for the Tuple Sketch can be optionally configured in functionParameters, for example: {"nominalEntries": 4096}. If not configured, the default value of 16384 will be used. Note that the nominalEntries provided at query time should be less than or equal to the value used to construct the star-tree index. For instance, a star-tree index with {"nominalEntries": 8192} can be used with AVG_VALUE_INTEGER_SUM_TUPLE_SKETCH having nominalEntries=8192 or less for any power of 2.

  • DISTINCT_COUNT_CPC_SKETCH

    • The lgK value for the CPC Sketch can be optionally configured in functionParameters, for example: {"lgK": 13}. If not configured, the default value of 12 will be used. Note that the nominalEntries provided at query time should be 2 ^ lgK in order for a star-tree index to be used. For instance, a star-tree index with {"lgK": 13} can be used with DISTINCTCOUNTCPCSKETCH having nominalEntries=8192.

  • DISTINCT_COUNT_RAW_CPC_SKETCH

  • DISTINCT_COUNT_ULL

    • The p value (precision parameter) for the UltraLogLog structure can be optionally configured in functionParameters, for example: {"p": 20}. If not configured, the default value of 12 will be used.

  • DISTINCT_COUNT_RAW_ULL

    • The p value (precision parameter) for the UltraLogLog structure can be optionally configured in functionParameters, for example: {"p": 20}. If not configured, the default value of 12 will be used.

Unsupported functions

  • DISTINCT_COUNT

    • Intermediate result Set is unbounded.

  • SEGMENT_PARTITIONED_DISTINCT_COUNT:

    • Intermediate result Set is unbounded.

  • PERCENTILE

    • Intermediate result List is unbounded.

Functions to be supported

  • ST_UNION

Index generation configuration

Multiple index generation configurations can be provided to generate multiple star-trees. Each configuration should contain the following properties:

Property
Description

dimensionsSplitOrder

An ordered list of dimension names can be specified to configure the split order. Only the dimensions in this list are reserved in the aggregated documents. The nodes will be split based on the order of this list. For example, split at level i is performed on the values of dimension at index i in the list. - The star-tree dimension does not have to be a dimension column in the table, it can also be time column, date-time column, or metric column if necessary. - The star-tree dimension column should be dictionary encoded in order to generate the star-tree index. - All columns in the filter and group-by clause of a query should be included in this list in order to use the star-tree index.

skipStarNodeCreationForDimensions

(Optional, default empty): A list of dimension names for which to not create the Star-Node.

functionColumnPairs

A list of aggregation function and column pairs (split by double underscore “__”). E.g. SUM__Impressions (SUM of column Impressions) or COUNT__*.

aggregationConfigs

Check

maxLeafRecords

(Optional, default 10000): The threshold T to determine whether to further split each node.

`functionColumnPairs` and `aggregationConfigs` are interchangeable. Consider using `aggregationConfigs` since it supports additional parameters like compression.

AggregationConfigs

All aggregations of a query should be included in `aggregationConfigs` or in `functionColumnPairs` in order to use the star-tree index.

Property
Description

columnName

(Required) Name of the column to aggregate. The column can be either dictionary encoded or raw.

aggregationFunction

(Required) Name of the aggregation function to use.

compressionCodec

(Optional, default PASS_THROUGH, introduced in release 1.1.0) Used to configure the compression enabled on the star-tree-index. Useful when aggregating on columns that contain big values. For example, a BYTES column containing HLL counters serialisations used to calculate DISTINCTCOUNTHLL. In this case setting "compressionCodec": "LZ4" can significantly reduce the space used by the index. Equivalent to compressionCodec in

deriveNumDocsPerChunk

(Optional, introduced in release 1.2.0) Equivalent to deriveNumDocsPerChunk in

indexVersion

(Optional, introduced in release 1.2.0) Equivalent to rawIndexWriterVersion in

targetMaxChunkSize

(Optional, introduced in release 1.2.0) Equivalent to targetMaxChunkSize in

targetDocsPerChunk

(Optional, introduced in release 1.2.0) Equivalent to targetDocsPerChunk in

functionParameters

(Optional) A configuration map used to pass in additional configurations to the aggregation function. For example, on DISTINCTCOUNTHLL, this could look like {"log2m": 16} in order to build the star-tree index using DISTINCTCOUNTHLL with a non-default value for log2m. Note that the index will only be used for queries using the same value for log2m with DISTINCTCOUNTHLL.

Default index generation configuration

A default star-tree index can be added to a segment by using the boolean config enableDefaultStarTree under the tableIndexConfig.

A default star-tree will have the following configuration:

  • All dictionary-encoded single-value dimensions with cardinality smaller or equal to a threshold (10000) will be included in the dimensionsSplitOrder, sorted by their cardinality in descending order.

  • All dictionary-encoded Time/DateTime columns will be appended to the _dimensionsSplitOrder _following the dimensions, sorted by their cardinality in descending order. Here we assume that time columns will be included in most queries as the range filter column and/or the group by column, so for better performance, we always include them as the last elements in the dimensionsSplitOrder.

  • Include COUNT(*) and SUM for all numeric metrics in the functionColumnPairs.

  • Use default maxLeafRecords (10000).

Example

For our example data set, in order to solve the following query efficiently:

SELECT SUM(Impressions) 
FROM myTable 
WHERE Country = 'USA' 
AND Browser = 'Chrome' 
GROUP BY Locale

We may configure the star-tree index as follows:

"tableIndexConfig": {
  "starTreeIndexConfigs": [{
    "dimensionsSplitOrder": [
      "Country",
      "Browser",
      "Locale"
    ],
    "skipStarNodeCreationForDimensions": [
    ],
    "functionColumnPairs": [
      "SUM__Impressions"
    ],
    "maxLeafRecords": 1
  }],
  ...
}

Alternatively using aggregationConfigs instead of functionColumnPairs and enabling compression on the aggregation:

"tableIndexConfig": {
  "starTreeIndexConfigs": [{
    "dimensionsSplitOrder": [
      "Country",
      "Browser",
      "Locale"
    ],
    "skipStarNodeCreationForDimensions": [
    ],
    "aggregationConfigs": [
      {
        "columnName": "Impressions",
        "aggregationFunction": "SUM",
        "compressionCodec": "LZ4"
      }
    ],
    "maxLeafRecords": 1
  }],
  ...
}

Note: In above example configs maxLeafRecords is set to 1 so that all of the dimension combinations are pre-aggregated for clarity in visual below.

The star-tree and documents should be something like below:

Tree structure

The values in the parentheses are the aggregated sum of Impressions for all the documents under the node.

Star-tree documents

Country
Browser
Locale
SUM__Impressions

CA

Chrome

en

400

CA

Firefox

fr

200

MX

Safari

en

100

MX

Safari

es

300

USA

Chrome

en

600

USA

Firefox

en

400

USA

Firefox

es

200

CA

*

en

400

CA

*

fr

200

CA

*

*

600

MX

Safari

*

400

USA

Firefox

*

600

USA

*

en

1000

USA

*

es

200

USA

*

*

1200

*

Chrome

en

1000

*

Firefox

en

400

*

Firefox

es

200

*

Firefox

fr

200

*

Firefox

*

800

*

Safari

en

100

*

Safari

es

300

*

Safari

*

400

*

*

en

1500

*

*

es

500

*

*

fr

200

*

*

*

2200

Query execution

For query execution, the idea is to first check metadata to determine whether the query can be solved with the star-tree documents, then traverse the Star-Tree to identify documents that satisfy all the predicates. After applying any remaining predicates that were missed while traversing the star-tree to the identified documents, apply aggregation/group-by on the qualified documents.

The algorithm to traverse the tree can be described as follows:

  • Start from root node.

  • For each level, what child node(s) to select depends on whether there are any predicates/group-by on the split dimension for the level in the query.

    • If there is no predicate or group-by on the split dimension, select the Star-Node if exists, or all child nodes to traverse further.

    • If there are predicate(s) on the split dimension, select the child node(s) that satisfy the predicate(s).

    • If there is no predicate, but there is a group-by on the split dimension, select all child nodes except Star-Node.

  • Recursively repeat the previous step until all leaf nodes are reached, or all predicates are satisfied.

  • Collect all the documents pointed by the selected nodes.

    • If all predicates and group-by's are satisfied, pick the single aggregated document from each selected node.

    • Otherwise, collect all the documents in the document range from each selected node.note

Predicates

Supported Predicates

  • EQ (=)

  • NOT EQ (!=)

  • IN

  • NOT IN

  • RANGE (>, >=, <, <=, BETWEEN)

  • AND

Unsupported Predicates

  • REGEXP_LIKE: It is intentionally left unsupported because it requires scanning the entire dictionary.

  • IS NULL: Currently NULL value info is not stored in star-tree index, and the dimension will be indexed as default value. A workaround is to do col = <default> instead.

  • IS NOT NULL: Same as IS NULL. A workaround is to do col != <default>.

Limited Support Predicates

  • OR

    • It can be applied to predicates on the same dimension, e.g. WHERE d1 < 10 OR d1 > 50)

    • It CANNOT be applied to predicates on multiple dimensions because star-tree index will double counting with pre-aggregated results.

  • NOT (Added since 1.2.0)

    • It can be applied to simple predicate and NOT

    • It CANNOT be applied on top of AND/OR because star-tree index will double counting with pre-aggregated results.

In scenarios where you have a transform on a column(s) which is in the dimension split order (should include all columns that are either a predicate or a group by column in target query(ies)) AND used in a group-by, then Star-tree index will get applied automatically. If a transform is applied to a column(s) which is used in predicate (WHERE clause) then Star-tree index won't apply.

For e.g if query contains round(colA,600) as roundedValue from tableA group by roundedValue and colA is included in dimensionSplitOrder then Pinot will use the pre-aggregated records to first scan matching records and then apply transform round() to derive roundedValue.

AggregationConfigs
Raw value forward index
Raw value forward index
Raw value forward index
Raw value forward index
Raw value forward index

0.10.0

Summary

This release introduces some new great features, performance enhancements, UI improvements, and bug fixes which are described in details in the following sections. The release was cut from this commit fd9c58a.

Dependency Graph

The dependency graph for plug-and-play architecture that was introduced in release 0.3.0 has been extended and now it contains new nodes for Pinot Segment SPI.

Dependency graph after introducing pinot-segment-api.

SQL Improvements

  • Implement NOT Operator (#8148)

  • Add DistinctCountSmartHLLAggregationFunction which automatically store distinct values in Set or HyperLogLog based on cardinality (#8189)

  • Add LEAST and GREATEST functions (#8100)

  • Handle SELECT * with extra columns (#7959)

  • Add FILTER clauses for aggregates (#7916)

  • Add ST_Within function (#7990)

  • Handle semicolon in query (#7861)

  • Add EXPLAIN PLAN (#7568)

UI Enhancements

  • Show Reported Size and Estimated Size in human readable format in UI (#8199)

  • Make query console state URL based (#8194)

  • Improve query console to not show query result when multiple columns have the same name (#8131)

  • Improve Pinot dashboard tenant view to show correct amount of servers and brokers (#8115)

  • Fix issue with opening new tabs from Pinot Dashboard (#8021)

  • Fix issue with Query console going blank on syntax error (#8006)

  • Make query stats always show even there's error (#7981)

  • Implement OIDC auth workflow in UI (#7121)

  • Add tooltip and modal for table status (#7899)

  • Add option to wrap lines in custom code mirror (#7857)

  • Add ability to comment out queries with cmd + / (#7841)

  • Return exception when unavailable segments on empty broker response (#7823)

  • Properly handle the case where segments are missing in externalview (#7803)

  • Add TIMESTAMP to datetime column Type (#7746)

Performance Improvements

  • Reuse regex matcher in dictionary based LIKE queries (#8261)

  • Early terminate orderby when columns already sorted (#8228)

  • Do not do another pass of Query Automaton Minimization (#8237)

  • Improve RangeBitmap by upgrading RoaringBitmap (#8206)

  • Optimize geometry serializer usage when literal is available (#8167)

  • Improve performance of no-dictionary group by (#8195)

  • Allocation free DataBlockCache lookups (#8140)

  • Prune unselected THEN statements in CaseTransformFunction (#8138)

  • Aggregation delay conversion to double (#8139)

  • Reduce object allocation rate in ExpressionContext or FunctionContext (#8124)

  • Lock free DimensionDataTableManager (#8102)

  • Improve json path performance during ingestion by upgrading JsonPath (#7819)

  • Reduce allocations and speed up StringUtil.sanitizeString (#8013)

  • Faster metric scans - ForwardIndexReader (#7920)

  • Unpeel group by 3 ways to enable vectorization (#7949)

  • Power of 2 fixed size chunks (#7934)

  • Don't use mmap for compression except for huge chunks (#7931)

  • Exit group-by marking loop early (#7935)

  • Improve performance of base chunk forward index write (#7930)

  • Cache JsonPaths to prevent compilation per segment (#7826)

  • Use LZ4 as default compression mode (#7797)

  • Peel off special case for 1 dimensional groupby (#7777)

  • Bump roaringbitmap version to improve range queries performance (#7734)

Other Notable Features

  • Adding NoopPinotMetricFactory and corresponding changes (#8270)

  • Allow to specify fixed segment name for SegmentProcessorFramework (#8269)

  • Move all prestodb dependencies into a separated module (#8266)

  • Include docIds in Projection and Transform block (#8262)

  • Automatically update broker resource on broker changes (#8249)

  • Update ScalarFunction annotation from name to names to support function alias. (#8252)

  • Implemented BoundedColumnValue partition function (#8224)

  • Add copy recursive API to pinotFS (#8200)

  • Add Support for Getting Live Brokers for a Table (without type suffix) (#8188)

  • Pinot docker image - cache prometheus rules (#8241)

  • In BrokerRequestToQueryContextConverter, remove unused filterExpressionContext (#8238)

  • Adding retention period to segment delete REST API (#8122)

  • Pinot docker image - upgrade prometheus and scope rulesets to components (#8227)

  • Allow segment name postfix for SegmentProcessorFramework (#8230)

  • Superset docker image - update pinotdb version in superset image (#8231)

  • Add retention period to deleted segment files and allow table level overrides (#8176)

  • Remove incubator from pinot and superset (#8223)

  • Adding table config overrides for disabling groovy (#8196)

  • Optimise sorted docId iteration order in mutable segments (#8213)

  • Adding secure grpc query server support (#8207)

  • Move Tls configs and utils from pinot-core to pinot-common (#8210)

  • Reduce allocation rate in LookupTransformFunction (#8204)

  • Allow subclass to customize what happens pre/post segment uploading (#8203)

  • Enable controller service auto-discovery in Jersey framework (#8193)

  • Add support for pushFileNamePattern in pushJobSpec (#8191)

  • Add additionalMatchLabels to helm chart (#7177)

  • Simulate rsvps after meetup.com retired the feed (#8180)

  • Adding more checkstyle rules (#8197)

  • Add persistence.extraVolumeMounts and persistence.extraVolumes to Kubernetes statefulsets (#7486)

  • Adding scala profile for kafka 2.x build and remove root pom scala dependencies (#8174)

  • Allow real-time data providers to accept non-kafka producers (#8190)

  • Enhance revertReplaceSegments api (#8166)

  • Adding broker level config for disabling Pinot queries with Groovy (#8159)

  • Make presto driver query pinot server with SQL (#8186)

  • Adding controller config for disabling Groovy in ingestionConfig (#8169)

  • Adding main method for LaunchDataIngestionJobCommand for spark-submit command (#8168)

  • Add auth token for segment replace rest APIs (#8146)

  • Add allowRefresh option to UploadSegment (#8125)

  • Add Ingress to Broker and Controller helm charts (#7997)

  • Improve progress reporter in SegmentCreationMapper (#8129)

  • St_* function error messages + support literal transform functions (#8001)

  • Add schema and segment crc to SegmentDirectoryContext (#8127)

  • Extend enableParallePushProtection support in UploadSegment API (#8110)

  • Support BOOLEAN type in Config Recommendation Engine (#8055)

  • Add a broker metric to distinguish exception happens when acquire channel lock or when send request to server (#8105)

  • Add pinot.minion prefix on minion configs for consistency (#8109)

  • Enable broker service auto-discovery in Jersey framework (#8107)

  • Timeout if waiting server channel lock takes a long time (#8083)

  • Wire EmptySegmentPruner to routing config (#8067)

  • Support for TIMESTAMP data type in Config Recommendation Engine (#8087)

  • Listener TLS customization (#8082)

  • Add consumption rate limiter for LLConsumer (#6291)

  • Implement Real Time Mutable FST (#8016)

  • Allow quickstart to get table files from filesystem (#8093)

  • Add support for instant segment deletion (#8077)

  • Add a config file to override quickstart configs (#8059)

  • Add pinot server grpc metadata acl (#8030)

  • Move compatibility verifier to a separate module (#8049)

  • Move hadoop and spark ingestion libs from plugins directory to external-plugins (#8048)

  • Add global strategy for partial upsert (#7906)

  • Upgrade kafka to 2.8.1 (#7883)

  • Created EmptyQuickstart command (#8024)

  • Allow SegmentPushUtil to push real-time segment (#8032)

  • Add ignoreMerger for partial upsert (#7907)

  • Make task timeout and concurrency configurable (#8028)

  • Return 503 response from health check on shut down (#7892)

  • Pinot-druid-benchmark: set the multiValueDelimiterEnabled to false when importing TPC-H data (#8012)

  • Cleanup: Remove remaining occurrences of incubator. (#8023)

  • Refactor segment loading logic in BaseTableDataManager to decouple it with local segment directory (#7969)

  • Improving segment replacement/revert protocol (#7995)

  • PinotConfigProvider interface (#7984)

  • Enhance listSegments API to exclude the provided segments from the output (#7878)

  • Remove outdated broker metric definitions (#7962)

  • Add skip key for realtimeToOffline job validation (#7921)

  • Upgrade async-http-client (#7968)

  • Allow Reloading Segments with Multiple Threads (#7893)

  • Ignore query options in commented out queries (#7894)

  • Remove TableConfigCache which does not listen on ZK changes (#7943)

  • Switch to zookeeper of helm 3.0x (#7955)

  • Use a single react hook for table status modal (#7952)

  • Add debug logging for real-time ingestion (#7946)

  • Separate the exception for transform and indexing for consuming records (#7926)

  • Disable JsonStatementOptimizer (#7919)

  • Make index readers/loaders pluggable (#7897)

  • Make index creator provision pluggable (#7885)

  • Support loading plugins from multiple directories (#7871)

  • Update helm charts to honour readinessEnabled probes flags on the Controller, Broker, Server and Minion StatefulSets (#7891)

  • Support non-selection-only GRPC server request handler (#7839)

  • GRPC broker request handler (#7838)

  • Add validator for SDF (#7804)

  • Support large payload in zk put API (#7364)

  • Push JSON Path evaluation down to storage layer (#7820)

  • When upserting new record, index the record before updating the upsert metadata (#7860)

  • Add Post-Aggregation Gapfilling functionality. (#7781)

  • Clean up deprecated fields from segment metadata (#7853)

  • Remove deprecated method from StreamMetadataProvider (#7852)

  • Obtain replication factor from tenant configuration in case of dimension table (#7848)

  • Use valid bucket end time instead of segment end time for merge/rollup delay metrics (#7827)

  • Make pinot start components command extensible (#7847)

  • Make upsert inner segment update atomic (#7844)

  • Clean up deprecated ZK metadata keys and methods (#7846)

  • Add extraEnv, envFrom to statefulset help template (#7833)

  • Make openjdk image name configurable (#7832)

  • Add getPredicate() to PredicateEvaluator interface (#7840)

  • Make split commit the default commit protocol (#7780)

  • Pass Pinot connection properties from JDBC driver (#7822)

  • Add Pinot client connection config to allow skip fail on broker response exception (#7816)

  • Change default range index version to v2 (#7815)

  • Put thread timer measuring inside of wall clock timer measuring (#7809)

  • Add getRevertReplaceSegmentRequest method in FileUploadDownloadClient (#7796)

  • Add JAVA_OPTS env var in docker image (#7799)

  • Split thread cpu time into three metrics (#7724)

  • Add config for enabling real-time offset based consumption status checker (#7753)

  • Add timeColumn, timeUnit and totalDocs to the json segment metadata (#7765)

  • Set default Dockerfile CMD to -help (#7767)

  • Add getName() to PartitionFunction interface (#7760)

  • Support Native FST As An Index Subtype for FST Indices (#7729)

  • Add forceCleanup option for 'startReplaceSegments' API (#7744)

  • Add config for keystore types, switch tls to native implementation, and add authorization for server-broker tls channel (#7653)

  • Extend FileUploadDownloadClient to send post request with json body (#7751)

Major Bug Fixes

  • Fix string comparisons (#8253)

  • Bugfix for order-by all sorted optimization (#8263)

  • Fix dockerfile (#8239)

  • Ensure partition function never return negative partition (#8221)

  • Handle indexing failures without corrupting inverted indexes (#8211)

  • Fixed broken HashCode partitioning (#8216)

  • Fix segment replace test (#8209)

  • Fix filtered aggregation when it is mixed with regular aggregation (#8172)

  • Fix FST Like query benchmark to remove SQL parsing from the measurement (#8097)

  • Do not identify function types by throwing exceptions (#8137)

  • Fix regression bug caused by sharing TSerializer across multiple threads (#8160)

  • Fix validation before creating a table (#8103)

  • Check cron schedules from table configs after subscribing child changes (#8113)

  • Disallow duplicate segment name in tar file (#8119)

  • Fix storage quota checker NPE for Dimension Tables (#8132)

  • Fix TraceContext NPE issue (#8126)

  • Update gcloud libraries to fix underlying issue with api's with CMEK (#8121)

  • Fix error handling in jsonPathArray (#8120)

  • Fix error handling in json functions with default values (#8111)

  • Fix controller config validation failure for customized TLS listeners (#8106)

  • Validate the numbers of input and output files in HadoopSegmentCreationJob (#8098)

  • Broker Side validation for the query with aggregation and col but without group by (#7972)

  • Improve the proactive segment clean-up for REVERTED (#8071)

  • Allow JSON forward indexes (#8073)

  • Fix the PinotLLCRealtimeSegmentManager on segment name check (#8058)

  • Always use smallest offset for new partitionGroups (#8053)

  • Fix RealtimeToOfflineSegmentsTaskExecutor to handle time gap (#8054)

  • Refine segment consistency checks during segment load (#8035)

  • Fixes for various JDBC issues (#7784)

  • Delete tmp- segment directories on server startup (#7961)

  • Fix ByteArray datatype column metadata getMaxValue NPE bug and expose maxNumMultiValues (#7918)

  • Fix the issues that Pinot upsert table's uploaded segments get deleted when a server restarts. (#7979)

  • Fixed segment upload error return (#7957)

  • Fix QuerySchedulerFactory to plug in custom scheduler (#7945)

  • Fix the issue with grpc broker request handler not started correctly (#7950)

  • Fix real-time ingestion when an entire batch of messages is filtered out (#7927)

  • Move decode method before calling acquireSegment to avoid reference count leak (#7938)

  • Fix semaphore issue in consuming segments (#7886)

  • Add bootstrap mode for PinotServiceManager to avoid glitch for health check (#7880)

  • Fix the broker routing when segment is deleted (#7817)

  • Fix obfuscator not capturing secretkey and keytab (#7794)

  • Fix segment merge delay metric when there is empty bucket (#7761)

  • Fix QuickStart by adding types for invalid/missing type (#7768)

  • Use oldest offset on newly detected partitions (#7756)

  • Fix javadoc to compatible with jdk8 source (#7754)

  • Handle null segment lineage ZNRecord for getSelectedSegments API (#7752)

  • Handle fields missing in the source in ParquetNativeRecordReader (#7742)

Backward Incompatible Changes

  • Fix the issue with HashCode partitioning function (#8216)

  • Fix the issue with validation on table creation (#8103)

  • Change PinotFS API's (#8603)

Stream ingestion with Upsert

Upsert support in Apache Pinot.

Pinot provides native support of upserts during real-time ingestion. There are scenarios where records need modifications, such as correcting a ride fare or updating a delivery status.

Partial upserts are convenient as you only need to specify the columns where values change, and you ignore the rest.

Overview of upserts in Pinot

See an overview of how upserts work in Pinot 1.0.

Enable upserts in Pinot

To enable upserts on a Pinot table, do the following:

  1. Define the primary key in the schema

  2. Enable upserts in the table configurations

Define the primary key in the schema

To update a record, you need a primary key to uniquely identify the record. To define a primary key, add the field primaryKeyColumns to the schema definition. For example, the schema definition of UpsertMeetupRSVP in the quick start example has this definition.

upsert_meetupRsvp_schema.json
{
    "primaryKeyColumns": ["event_id"]
}

Note this field expects a list of columns, as the primary key can be a composite.

When two records of the same primary key are ingested, the record with the greater comparison value (timeColumn by default) is used. When records have the same primary key and event time, then the order is not determined. In most cases, the later ingested record will be used, but this may not be true in cases where the table has a column to sort by.

Partition the input stream by the primary key

An important requirement for the Pinot upsert table is to partition the input stream by the primary key. For Kafka messages, this means the producer shall set the key in the send API. If the original stream is not partitioned, then a streaming processing job (such as with Flink) is needed to shuffle and repartition the input stream into a partitioned one for Pinot's ingestion.

Additionally if using segmentPartitionConfigto leverage Broker segment pruning then it's important to ensure that the partition function used matches both on the Kafka producer side as well as Pinot. In Kafka default for Java client is 32-bit murmur2 hash and for all other languages such as Python its CRC32 (Cyclic Redundancy Check 32-bit).

Enable upsert in the table configurations

To enable upsert, make the following configurations in the table configurations.

Upsert modes

Full upsert

The upsert mode defaults to FULL . FULL upsert means that a new record will replace the older record completely if they have same primary key. Example config:

{
  "upsertConfig": {
    "mode": "FULL"
  }
}

Partial upserts

Partial upsert lets you choose to update only specific columns and ignore the rest.

To enable the partial upsert, set the mode to PARTIAL and specify partialUpsertStrategies for partial upsert columns. Since release-0.10.0, OVERWRITE is used as the default strategy for columns without a specified strategy. defaultPartialUpsertStrategy is also introduced to change the default strategy for all columns.

Note that null handling must be enabled for partial upsert to work.

For example:

release-0.8.0
{
  "upsertConfig": {
    "mode": "PARTIAL",
    "partialUpsertStrategies":{
      "rsvp_count": "INCREMENT",
      "group_name": "IGNORE",
      "venue_name": "OVERWRITE"
    }
  },
  "tableIndexConfig": {
    "nullHandlingEnabled": true
  }
}
release-0.10.0
{
  "upsertConfig": {
    "mode": "PARTIAL",
    "defaultPartialUpsertStrategy": "OVERWRITE",
    "partialUpsertStrategies":{
      "rsvp_count": "INCREMENT",
      "group_name": "IGNORE"
    }
  },
  "tableIndexConfig": {
    "nullHandlingEnabled": true
  }
}

Pinot supports the following partial upsert strategies:

Strategy
Description

OVERWRITE

Overwrite the column of the last record

INCREMENT

Add the new value to the existing values

APPEND

Add the new item to the Pinot unordered set

UNION

Add the new item to the Pinot unordered set if not exists

IGNORE

Ignore the new value, keep the existing value (v0.10.0+)

MAX

Keep the maximum value betwen the existing value and new value (v0.12.0+)

MIN

Keep the minimum value betwen the existing value and new value (v0.12.0+)

With partial upsert, if the value is null in either the existing record or the new coming record, Pinot will ignore the upsert strategy and the null value:

(null, newValue) -> newValue

(oldValue, null) -> oldValue

(null, null) -> null

None upserts

If set mode to NONE, the upsert is disabled.

Comparison column

By default, Pinot uses the value in the time column (timeColumn in tableConfig) to determine the latest record. That means, for two records with the same primary key, the record with the larger value of the time column is picked as the latest update. However, there are cases when users need to use another column to determine the order. In such case, you can use option comparisonColumn to override the column used for comparison. For example,

{
  "upsertConfig": {
    "mode": "FULL",
    "comparisonColumn": "anotherTimeColumn"
  }
}

For partial upsert table, the out-of-order events won't be consumed and indexed. For example, for two records with the same primary key, if the record with the smaller value of the comparison column came later than the other record, it will be skipped.

NOTE: Please use comparisonColumns for single comparison column instead of comparisonColumn as it is currently deprecated. You may see unrecognizedProperties when using the old config, but it's converted to comparisonColumns automatically when adding the table.

Multiple comparison columns

In some cases, especially where partial upsert might be employed, there may be multiple producers of data each writing to a mutually exclusive set of columns, sharing only the primary key. In such a case, it may be helpful to use one comparison column per producer group so that each group can manage its own specific versioning semantics without the need to coordinate versioning across other producer groups.

{
  "upsertConfig": {
    "mode": "PARTIAL",
    "defaultPartialUpsertStrategy": "OVERWRITE",
    "partialUpsertStrategies":{},
    "comparisonColumns": ["secondsSinceEpoch", "otherComparisonColumn"]
  }
}

Documents written to Pinot are expected to have exactly 1 non-null value out of the set of comparisonColumns; if more than 1 of the columns contains a value, the document will be rejected. When new documents are written, whichever comparison column is non-null will be compared against only that same comparison column seen in prior documents with the same primary key. Consider the following examples, where the documents are assumed to arrive in the order specified in the array.

[
  {
    "event_id": "aa",
    "orderReceived": 1,
    "description" : "first",
    "secondsSinceEpoch": 1567205394
  },
  {
    "event_id": "aa",
    "orderReceived": 2,
    "description" : "update",
    "secondsSinceEpoch": 1567205397
  },
  {
    "event_id": "aa",
    "orderReceived": 3,
    "description" : "update",
    "secondsSinceEpoch": 1567205396
  },
  {
    "event_id": "aa",
    "orderReceived": 4,
    "description" : "first arrival, other column",
    "otherComparisonColumn": 1567205395
  },
  {
    "event_id": "aa",
    "orderReceived": 5,
    "description" : "late arrival, other column",
    "otherComparisonColumn": 1567205392
  },
  {
    "event_id": "aa",
    "orderReceived": 6,
    "description" : "update, other column",
    "otherComparisonColumn": 1567205398
  }
]

The following would occur:

  1. orderReceived: 1

  • Result: persisted

  • Reason: first doc seen for primary key "aa"

  1. orderReceived: 2

  • Result: persisted (replacing orderReceived: 1)

  • Reason: comparison column (secondsSinceEpoch) larger than that previously seen

  1. orderReceived: 3

  • Result: rejected

  • Reason: comparison column (secondsSinceEpoch) smaller than that previously seen

  1. orderReceived: 4

  • Result: persisted (replacing orderReceived: 2)

  • Reason: comparison column (otherComparisonColumn) larger than previously seen (never seen previously), despite the value being smaller than that seen for secondsSinceEpoch

  1. orderReceived: 5

  • Result: rejected

  • Reason: comparison column (otherComparisonColumn) smaller than that previously seen

  1. orderReceived: 6

  • Result: persist (replacing orderReceived: 4)

  • Reason: comparison column (otherComparisonColumn) larger than that previously seen

Metadata time-to-live (TTL)

In Pinot, the metadata map is stored in heap memory. To decrease in-memory data and improve performance, minimize the time primary key entries are stored in the metadata map (metadata time-to-live (TTL)). Limiting the TTL is especially useful for primary keys with high cardinality and frequent updates.

Since the metadata TTL is applied on the first comparison column, the time unit of upsert TTL is the same as the first comparison column.

Configure how long primary keys are stored in metadata

To configure how long primary keys are stored in metadata, specify the length of time in upsertTTL. For example:{

  "upsertConfig": {
    "mode": "FULL",
    "enableSnapshot": true,
    "enablePreload": true,
    "metadataTTL": 86400
  }
}

In this example, Pinot will retain primary keys in metadata for 1 day.

Note that enabling upsert snapshot is required for metadata TTL for in-memory validDocsIDs recovery.

Delete column

Upsert Pinot table can support soft-deletes of primary keys. This requires the incoming record to contain a dedicated boolean single-field column that serves as a delete marker for a primary key. Once the real-time engine encounters a record with delete column set to true , the primary key will no longer be part of the queryable set of documents. This means the primary key will not be visible in the queries, unless explicitly requested via query option skipUpsert=true.

{ 
    "upsertConfig": {  
        ... 
        "deleteRecordColumn": <column_name>
    } 
}

Note that the delete column has to be a single-value boolean column.

// In the Schema
{
    ...
    {
      "name": "<delete_column_name>",
      "dataType": "BOOLEAN"
    },
    ...
}

Note that when deleteRecordColumn is added to an existing table, it will require a server restart to actually pick up the upsert config changes.

A deleted primary key can be revived by ingesting a record with the same primary, but with higher comparison column value(s).

Note that when reviving a primary key in a partial upsert table, the revived record will be treated as the source of truth for all columns. This means any previous updates to the columns will be ignored and overwritten with the new record's values.

Deleted Keys time-to-live (TTL)

The above config deleteRecordColumn only soft-deletes the primary key. To decrease in-memory data and improve performance, minimize the time deleted-primary-key entries are stored in the metadata map (deletedKeys time-to-live (TTL)). Limiting the TTL is especially useful for deleted-primary-keys where there are no future updates foreseen.

Configure how long deleted-primary-keys are stored in metadata

To configure how long primary keys are stored in metadata, specify the length of time in deletedKeysTTL For example:

  "upsertConfig": {
    "mode": "FULL",
    "deleteRecordColumn": <column_name>,
    "deletedKeysTTL": 86400
  }
}

In this example, Pinot will retain the deleted-primary-keys in metadata for 1 day.

Note that the value of this field deletedKeysTTL should be the same as the unit of comparison column. If your comparison column is having values which corresponds to seconds, this config should also have values in seconds (see above example).

Data consistency with deletes and compaction together

When using deletedKeysTTL together with UpsertCompactionTask, there can be a scenario where a segment containing deleted-record (where deleteRecordColumn = true was set for the primary key) gets compacted first and a previous old record is not yet compacted. During server restart, now the old record is added to the metadata manager map and is treated as non-deleted. To prevent data inconsistencies in this scenario, we have added a new config enableDeletedKeysCompactionConsistency which when set to true, will ensure that the deleted records are not compacted until all the previous records from all other segments are compacted for the deleted primary-key.

{
  "upsertConfig": {
    "mode": "FULL",
    "deleteRecordColumn": <column_name>,
    "deletedKeysTTL": 86400,
    "enableDeletedKeysCompactionConsistency": true
  }
}

Data consistency when queries and upserts happen concurrently

Upserts in Pinot enable real-time updates and ensure that queries always retrieve the latest version of a record, making them a powerful feature for managing mutable data efficiently. However, in applications with extremely high QPS and high ingestion rates, queries and upserts happening concurrently can sometimes lead to inconsistencies in query results.

For example, consider a table with 1 million primary keys. A distinct count query should always return 1 million, regardless of how new records are ingested and older records are invalidated. However, at high ingestion and query rates, the query may occasionally return a count slightly above or below 1 million. This happens because queries determine valid records by acquiring validDocIds bitmaps from multiple segments, which indicate which documents are currently valid. Since acquiring these bitmaps is not atomic with respect to ongoing upserts, a query may capture an inconsistent view of the data, leading to overcounting or undercounting of valid records.

This is a classic concurrency issue where reads and writes happen simultaneously, leading to temporary inconsistencies. Typically, such issues are resolved using locks or snapshots to maintain a stable view of the data during query execution. To address this, two new consistency modes - SYNC and SNAPSHOT - have been introduced for upsert enabled tables to ensure consistent query results even when queries and upserts occur concurrently and at very high throughput.

By default, the consistency mode is NONE, meaning the system operates as before. The SYNC mode ensures consistency by blocking upserts while queries execute, guaranteeing that queries always see a stable upserted data view. However, this can introduce write latency. Alternatively, the SNAPSHOT mode creates a consistent snapshot of validDocIds bitmaps for queries to use. This allows upserts to continue without blocking queries, making it more suitable for workloads with both high query and write rates. These new consistency modes provide flexibility, allowing applications to balance consistency guarantees against performance trade-offs based on their specific requirements.

{
  "upsertConfig": {
    "consistencyMode": "SYNC", // or "SNAPSHOT", "NONE"
...
  }
}

Use strictReplicaGroup for routing

The upsert Pinot table can use only the low-level consumer for the input streams. As a result, it uses the partitioned replica-group assignment implicitly for the segments. Moreover, upsert poses the additional requirement that all segments of the same partition must be served from the same server to ensure the data consistency across the segments. Accordingly, it requires to use strictReplicaGroup as the routing strategy. To use that, configure instanceSelectorType in Routing as the following:

{
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  }
}

Using implicit partitioned replica-group assignment from low-level consumer won't persist the instance assignment (mapping from partition to servers) to the ZooKeeper, and new added servers will be automatically included without explicit reassigning instances (usually through rebalance). This can cause new segments of the same partition assigned to a different server and break the requirement of upsert.

To prevent this, we recommend using explicit partitioned replica-group instance assignment to ensure the instance assignment is persisted. Note that numInstancesPerPartition should always be 1 in replicaGroupPartitionConfig.

Enable validDocIds snapshots for upsert metadata recovery

Upsert snapshot support is also added in release-0.12.0. To enable the snapshot, set the enableSnapshot to true. For example:

{
  "upsertConfig": {
    "mode": "FULL",
    "enableSnapshot": true
  }
}

Upsert maintains metadata in memory containing which docIds are valid in a particular segment (ValidDocIndexes). This metadata gets lost during server restarts and needs to be recreated again. ValidDocIndexes can not be recovered easily after out-of-TTL primary keys get removed. Enabling snapshots addresses this problem by adding functions to store and recover validDocIds snapshot for Immutable Segments

The snapshots are taken on every segment commit to ensure that they are consistent with the persisted data in case of abrupt shutdown. We recommend that you enable this feature so as to speed up server boot times during restarts.

The lifecycle for validDocIds snapshots are shows as follows,

  1. If snapshot is enabled, snapshots for existing segments are taken or refreshed when the next consuming segment gets started.

  2. The snapshot files are kept on disk until the segments get removed, e.g. due to data retention or manual deletion.

  3. If snapshot is disabled, the existing snapshot for a segment is cleaned up when the segment gets loaded by the server, e.g. when the server restarts.

Enable preload for faster server restarts

Upsert preload feature can make it faster to restore the upsert states when server restarts. To enable the preload feature, set the enablePreload to true. To enable preloading, enableSnapshot: true should also be set in the table config. For example:

{
  "upsertConfig": {
    "mode": "FULL",
    "enableSnapshot": true,
    "enablePreload": true
  }
}

Under the hood, it uses the validDocIds snapshots to identify the valid docs and restore their upsert metadata quickly instead of performing a whole upsert comparison flow. The flow is triggered before the server is marked as ready, after which the server starts to load the remaining segments without snapshots (hence the name preload).

The feature also requires you to specify pinot.server.instance.max.segment.preload.threads: N in the server config where N should be replaced with the number of threads that should be used for preload. It's 0 by default to disable the preloading feature.

A bug was introduced in v1.2.0 that when enablePreload and enableSnapshot flags are set to true but max.segment.preload.threads is left as 0, the preloading mechanism is still enabled but segments fail to get loaded as there is no threads for preloading. This was fixed in newer versions, but for v1.2.0, if enablePreload and enableSnapshot are set to true, remember to set max.segment.preload.threads to a positive value as well. Server restart is needed to get max.segment.preload.threads config change into effect.

Handle out-of-order events

There are 2 configs added related to handling out-of-order events.

dropOutOfOrderRecord

To enable dropping of out-of-order record, set the dropOutOfOrderRecord to true. For example:

{
  "upsertConfig": {
    ...,
    "dropOutOfOrderRecord": true
  }
}

This feature doesn't persist any out-of-order event to the consuming segment. If not specified, the default value is false.

  • When false, the out-of-order record gets persisted to the consuming segment, but the MetadataManager mapping is not updated thus this record is not referenced in query or in any future updates. You can still see the records when using skipUpsert query option.

  • When true, the out-of-order record doesn't get persisted at all and the MetadataManager mapping is not updated so this record is not referenced in query or in any future updates. You cannot see the records when using skipUpsert query option.

outOfOrderRecordColumn

This is to identify out-of-order events programmatically. To enable this config, add a boolean field in your table schema, say isOutOfOrder and enable via this config. For example:

{
  "upsertConfig": {
    ...,
    "outOfOrderRecordColumn": "isOutOfOrder"
  }
}

This feature persists a true / false value to the isOutOfOrder field based on the orderness of the event. You can filter out out-of-order events while using skipUpsert to avoid any confusion. For example:

select key, val from tbl1 where isOutOfOrder = false option(skipUpsert=false)

Use custom metadata manager

Pinot supports custom PartitionUpsertMetadataManager that handle records and segments updates.

{
  "upsertConfig": {
    "metadataManagerClass": org.apache.pinot.segment.local.upsert.CustomPartitionUpsertMetadataManager
  }
}

Adding custom upsert managers

You can add custom PartitionUpsertMetadataManager as follows:

  • Create a new java project. Make sure you keep the package name as org.apache.pinot.segment.local.upsert.xxx

  • In your java project include the dependency

<dependency>
  <groupId>org.apache.pinot</groupId>
  <artifactId>pinot-segment-local</artifactId>
  <version>1.0.0</version>
 </dependency>
include 'org.apache.pinot:pinot-common:1.0.0'
  • Add your custom partition manager that implements PartitionUpsertMetadataManager interface

//Example custom partition manager

class CustomPartitionUpsertMetadataManager implements PartitionUpsertMetadataManager {}
  • Add your custom TableUpsertMetadataManager that implements BaseTableUpsertMetadataManager interface

//Example custom table upsert metadata manager

public class CustomTableUpsertMetadataManager extends BaseTableUpsertMetadataManager {}
  • Place the compiled JAR in the /plugins directory in pinot. You will need to restart all Pinot instances if they are already running.

  • Now, you can use the custom upsert manager in table configs as follows:

{
  "upsertConfig": {
    "metadataManagerClass": org.apache.pinot.segment.local.upsert.CustomPartitionUpsertMetadataManager
  }
}

⚠️ The upsert manager class name is case-insensitive as well.

Upsert table limitations

There are some limitations for the upsert Pinot tables.

  • The upsert feature is supported for Real-time tables only, and not for Hybrid or Offline tables.

  • The high-level consumer is not allowed for the input stream ingestion, which means stream.[consumerName].consumer.type must always be lowLevel.

  • The star-tree index cannot be used for indexing, as the star-tree index performs pre-aggregation during the ingestion.

  • Unlike append-only tables, out-of-order events (with comparison value in incoming record less than the latest available value) won't be consumed and indexed by Pinot partial upsert table, these late events will be skipped.

Best practices

Unlike other real-time tables, Upsert table takes up more memory resources as it needs to bookkeep the record locations in memory. As a result, it's important to plan the capacity beforehand, and monitor the resource usage. Here are some recommended practices of using Upsert table.

Create the topic/stream with more partitions.

The number of partitions in input streams determines the partition numbers of the Pinot table. The more partitions you have in input topic/stream, more Pinot servers you can distribute the Pinot table to and therefore more you can scale the table horizontally. Do note that you can't increase the partitions in future for upsert enabled tables so you need to start with good enough partitions (atleast 2-3X the number of pinot servers)

Memory usage

Upsert table maintains an in-memory map from the primary key to the record location. So it's recommended to use a simple primary key type and avoid composite primary keys to save the memory cost. Beware when using JSON column as primary key, same key-values in different order would be considered as different primary keys. In addition, consider the hashFunction config in the Upsert config, which can be MD5 or MURMUR3, to store the 128-bit hashcode of the primary key instead. This is useful when your primary key takes more space. But keep in mind, this hash may introduce collisions, though the chance is very low.

Monitoring

Set up a dashboard over the metric pinot.server.upsertPrimaryKeysCount.tableName to watch the number of primary keys in a table partition. It's useful for tracking its growth which is proportional to the memory usage growth. **** The total memory usage by upsert is roughly (primaryKeysCount * (sizeOfKeyInBytes + 24))

Capacity planning

It's useful to plan the capacity beforehand to ensure you will not run into resource constraints later. A simple way is to measure the rate of the primary keys in the input stream per partition and extrapolate the data to a specific time period (based on table retention) to approximate the memory usage. A heap dump is also useful to check the memory usage so far on an upsert table instance.

Example

Putting these together, you can find the table configurations of the quick start examples as the following:

{
  "tableName": "upsertMeetupRsvp",
  "tableType": "REALTIME",
  "tenants": {},
  "segmentsConfig": {
    "timeColumnName": "mtime",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1",
    "replication": "1"
  },
  "tableIndexConfig": {
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "event_id": {
          "functionName": "Hashcode",
          "numPartitions": 2
        }
      }
    }
  },
  "instanceAssignmentConfigMap": {
    "CONSUMING": {
      "tagPoolConfig": {
        "tag": "DefaultTenant_REALTIME"
      },
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 1,
        "partitionColumn": "event_id",
        "numPartitions": 2,
        "numInstancesPerPartition": 1
      }
    }
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ],
    "instanceSelectorType": "strictReplicaGroup"
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
      "streamConfigMaps": [
        {
          "streamType": "kafka",
          "stream.kafka.topic.name": "upsertMeetupRSVPEvents",
          "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
          "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
          "stream.kafka.zk.broker.url": "localhost:2191/kafka",
          "stream.kafka.broker.list": "localhost:19092"
        }
      ]
    }
  },
  "upsertConfig": {
    "mode": "FULL",
    "enableSnapshot": true,
    "enablePreload": true
  },
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexType": "H3",
      "properties": {
        "resolutions": "5"
      }
    }
  ],
  "metadata": {
    "customConfigs": {}
  }
}
{
  "tableName": "upsertPartialMeetupRsvp",
  "tableType": "REALTIME",
  "tenants": {},
  "segmentsConfig": {
    "timeColumnName": "mtime",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1",
    "replication": "1"
  },
  "tableIndexConfig": {
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "event_id": {
          "functionName": "Hashcode",
          "numPartitions": 2
        }
      }
    },
    "nullHandlingEnabled": true
  },
  "instanceAssignmentConfigMap": {
    "CONSUMING": {
      "tagPoolConfig": {
        "tag": "DefaultTenant_REALTIME"
      },
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 1,
        "partitionColumn": "event_id",
        "numPartitions": 2,
        "numInstancesPerPartition": 1
      }
    }
  },
  "routing": {
    "segmentPrunerTypes": [
      "partition"
    ],
    "instanceSelectorType": "strictReplicaGroup"
  },
  "ingestionConfig": {
    "streamIngestionConfig": {
      "streamConfigMaps": [
        {
          "streamType": "kafka",
          "stream.kafka.topic.name": "upsertPartialMeetupRSVPEvents",
          "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
          "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
          "stream.kafka.zk.broker.url": "localhost:2191/kafka",
          "stream.kafka.broker.list": "localhost:19092"
        }
      ]
    }
  },
  "upsertConfig": {
    "mode": "PARTIAL",
    "partialUpsertStrategies": {
      "rsvp_count": "INCREMENT",
      "group_name": "UNION",
      "venue_name": "APPEND"
    }
  },
  "fieldConfigList": [
    {
      "name": "location",
      "encodingType": "RAW",
      "indexType": "H3",
      "properties": {
        "resolutions": "5"
      }
    }
  ],
  "metadata": {
    "customConfigs": {}
  }
}

Pinot server maintains a primary key to record location map across all the segments served in an upsert-enabled table. As a result, when updating the config for an existing upsert table (e.g. change the columns in the primary key, change the comparison column), servers need to be restarted in order to apply the changes and rebuild the map.

Quick Start

To illustrate how the full upsert works, the Pinot binary comes with a quick start example. Use the following command to creates a real-time upsert table meetupRSVP.

# stop previous quick start cluster, if any
bin/quick-start-upsert-streaming.sh

You can also run partial upsert demo with the following command

# stop previous quick start cluster, if any
bin/quick-start-partial-upsert-streaming.sh

As soon as data flows into the stream, the Pinot table will consume it and it will be ready for querying. Head over to the Query Console to check out the real-time data.

Query the upsert table

For partial upsert you can see only the value from configured column changed based on specified partial upsert strategy.

Query the partial upsert table

An example for partial upsert is shown below, each of the event_id kept being unique during ingestion, meanwhile the value of rsvp_count incremented.

Explain partial upsert table

To see the difference from the non-upsert table, you can use a query option skipUpsert to skip the upsert effect in the query result.

Disable the upsert during query via query option

FAQ

Can I change primary key columns in existing upsert table?

Yes, you can add or delete columns to primary keys as long as input stream is partitioned on one of the primary key columns. However, you need to restart all Pinot servers so that it can rebuild the primary key to record location map with the new columns.

0.11.0

Summary

Apache Pinot 0.11.0 has introduced many new features to extend the query abilities, e.g. the Multi-Stage query engine enables Pinot to do distributed joins, more sql syntax(DML support), query functions and indexes(Text index, Timestamp index) supported for new use cases. And as always, more integrations with other systems(E.g. Spark3, Flink).

Note: there is a major upgrade for Apache Helix to 1.0.4, so make sure you upgrade the system in the order of:

Helix Controller -> Pinot Controller -> Pinot Broker -> Pinot server

Multi-Stage Query Engine

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as JOIN, OVER window, MATCH_RECOGNIZE and eventually, make Pinot support closer to full ANSI SQL semantics. More to read: https://docs.pinot.apache.org/developers/advanced/v2-multi-stage-query-engine

Pause Stream Consumption on Apache Pinot

Pinot operators can pause real-time consumption of events while queries are being executed, and then resume consumption when ready to do so again.\

More to read: https://medium.com/apache-pinot-developer-blog/pause-stream-consumption-on-apache-pinot-772a971ef403

Gap-filling function

The gapfilling functions allow users to interpolate data and perform powerful aggregations and data processing over time series data. More to read: https://www.startree.ai/blog/gapfill-function-for-time-series-datasets-in-pinot

Add support for Spark 3.x (#8560)

Long waiting feature for segment generation on Spark 3.x.

Add Flink Pinot connector (#8233)

Similar to the Spark Pinot connector, this allows Flink users to dump data from the Flink application to Pinot.

Show running queries and cancel query by id (#9171)

This feature allows better fine-grained control on pinot queries.

Timestamp Index (#8343)

This allows users to have better query performance on the timestamp column for lower granularity. See: https://docs.pinot.apache.org/basics/indexing/timestamp-index

Native Text Indices (#8384)

Wanna search text in real time? The new text indexing engine in Pinot supports the following capabilities:

  1. New operator: LIKE

select * FROM foo where text_col LIKE 'a%'
  1. New operator: CONTAINS

select * from foo where text_col CONTAINS 'bar'
  1. Native text index, built from the ground up, focusing on Pinot’s time series use cases and utilizing existing Pinot indices and structures(inverted index, bitmap storage).

  2. Real Time Text Index

Read more: https://medium.com/@atri.jiit/text-search-time-series-style-681af37ba42e

Adding DML definition and parse SQL InsertFile (#8557)

Now you can use INSERT INTO [database.]table FROM FILE dataDirURI OPTION ( k=v ) [, OPTION (k=v)]* to load data into Pinot from a file using Minion. See: https://docs.pinot.apache.org/basics/data-import/from-query-console

Deduplication (#8708)

This feature supports enabling deduplication for real-time tables, via a top-level table config. At a high level, primaryKey (as defined in the table schema) hashes are stored into in-memory data structures, and each incoming row is validated against it. Duplicate rows are dropped.

The expectation while using this feature is for the stream to be partitioned by the primary key, strictReplicaGroup routing to be enabled, and the configured stream consumer type to be low level. These requirements are therefore mandated via table config API's input validations.

Functions support and changes:

  • Add support for functions arrayConcatLong, arrayConcatFloat, arrayConcatDouble (#9131)

  • Add support for regexpReplace scalar function (#9123)

  • Add support for Base64 Encode/Decode Scalar Functions (#9114)

  • Optimize like to regexp conversion to do not include unnecessary ^._ and ._$ (#8893)

  • Support DISTINCT on multiple MV columns (#8873)

  • Support DISTINCT on single MV column (#8857)

  • Add histogram aggregation function (#8724)

  • Optimize dateTimeConvert scalar function to only parse the format once (#8939)

  • Support conjugates for scalar functions, add more scalar functions (#8582)

  • add FIRSTWITHTIME aggregate function support #7647 (#8181)

  • Add PercentileSmartTDigestAggregationFunction (#8565)

  • Simplify the parameters for DistinctCountSmartHLLAggregationFunction (#8566)

  • add scalar function for cast so it can be calculated at compile time (#8535)

  • Scalable Gapfill Implementation for Avg/Count/Sum (#8647)

  • Add commonly used math, string and date scalar functions in Pinot (#8304)

  • Datetime transform functions (#8397)

  • Scalar function for url encoding and decoding (#8378)

  • Add support for IS NULL and NOT IS NULL in transform functions (#8264)

  • Support st_contains using H3 index (#8498)

The full list of features introduced in this release

  • add query cancel APIs on controller backed by those on brokers (#9276)

  • Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. (#9265)

  • Adding endpoint to download local log files for each component (#9259)

  • Add metrics to track controller segment download and upload requests in progress (#9258)

  • add a freshness based consumption status checker (#9244)

  • Force commit consuming segments (#9197)

  • Adding kafka offset support for period and timestamp (#9193)

  • Make upsert metadata manager pluggable (#9186)

  • Adding logger utils and allow change logger level at runtime (#9180)

  • Proper null handling in equality, inequality and membership operators for all SV column data types (#9173)

  • support to show running queries and cancel query by id (#9171)

  • Enhance upsert metadata handling (#9095)

  • Proper null handling in Aggregation functions for SV data types (#9086)

  • Add support for IAM role based credentials in Kinesis Plugin (#9071)

  • Task genrator debug api (#9058)

  • Add Segment Lineage List API #9005 (#9006)

  • [colocated-join] Adds Support for instancePartitionsMap in Table Config (#8989)

  • Support pause/resume consumption of real-time tables (#8986)

  • #8970 Minion tab in Pinot UI (#8978)

  • Add Protocol Buffer Stream Decoder (#8972)

  • Update minion task metadata ZNode path (#8959)

  • add /tasks/{taskType}/{tableNameWithType}/debug API (#8949)

  • Defined a new broker metric for total query processing time (#8941)

  • Proper null handling in SELECT, ORDER BY, DISTINCT, and GROUP BY (#8927)

  • fixing REGEX OPTION parser (#8905)

  • Enable key value byte stitching in PulsarMessageBatch (#8897)

  • Add property to skip adding hadoop jars to package (#8888)

  • Support DISTINCT on multiple MV columns (#8873)

  • Implement Mutable FST Index (#8861)

  • Support DISTINCT on single MV column (#8857)

  • Add controller API for reload segment task status (#8828)

  • Spark Connector, support for TIMESTAMP and BOOLEAN fields (#8825)

  • Allow moveToFinalLocation in METADATA push based on config (#8823) (#8815)

  • allow up to 4GB per bitmap index (#8796)

  • Deprecate debug options and always use query options (#8768)

  • Streamed segment download & untar with rate limiter to control disk usage (#8753)

  • Improve the Explain Plan accuracy (#8738)

  • allow to set https as the default scheme (#8729)

  • Add histogram aggregation function (#8724)

  • Allow table name with dots by a PinotConfiguration switch (#8713)

  • Disable Groovy function by default (#8711)

  • Deduplication (#8708)

  • Add pluggable client auth provider (#8670)

  • Adding pinot file system command (#8659)

  • Allow broker to automatically rewrite expensive function to its approximate counterpart (#8655)

  • allow to take data outside the time window by negating the window filter (#8640)

  • Support BigDecimal raw value forward index; Support BigDecimal in many transforms and operators (#8622)

  • Ingestion Aggregation Feature (#8611)

  • Enable uploading segments to real-time tables (#8584)

  • Package kafka 0.9 shaded jar to pinot-distribution (#8569)

  • Simplify the parameters for DistinctCountSmartHLLAggregationFunction (#8566)

  • Add PercentileSmartTDigestAggregationFunction (#8565)

  • Add support for Spark 3.x (#8560)

  • Adding DML definition and parse SQL InsertFile (#8557)

  • endpoints to get and delete minion task metadata (#8551)

  • Add query option to use more replica groups (#8550)

  • Only discover public methods annotated with @ScalarFunction (#8544)

  • Support single-valued BigDecimal in schema, type conversion, SQL statements and minimum set of transforms. (#8503)

  • Add connection based FailureDetector (#8491)

  • Add endpoints for some finer control on minion tasks (#8486)

  • Add adhoc minion task creation endpoint (#8465)

  • Rewrite PinotQuery based on expression hints at instance/segment level (#8451)

  • Allow disabling dict generation for High cardinality columns (#8398)

  • add segment size metric on segment push (#8387)

  • Implement Native Text Operator (#8384)

  • Change default memory allocation for consuming segments from on-heap to off-heap (#8380)

  • New Pinot storage metrics for compressed tar.gz and table size w/o replicas (#8358)

  • add a experiment API for upsert heap memory estimation (#8355)

  • Timestamp type index (#8343)

  • Upgrade Helix to 1.0.4 in Pinot (#8325)

  • Allow overriding expression in query through query config (#8319)

  • Always handle null time values (#8310)

  • Add prefixesToRename config for renaming fields upon ingestion (#8273)

  • Added multi column partitioning for offline table (#8255)

  • Automatically update broker resource on broker changes (#8249)

Vulnerability fixs

Pinot has resolved all the high-level vulnerabilities issues:

  • Add a new workflow to check vulnerabilities using trivy (#9044)

  • Disable Groovy function by default (#8711)

  • Upgrade netty due to security vulnerability (#8328)

  • Upgrade protobuf as the current version has security vulnerability (#8287)

  • Upgrade to hadoop 2.10.1 due to cves (#8478)

  • Upgrade Helix to 1.0.4 (#8325)

  • Upgrade thrift to 0.15.0 (#8427)

  • Upgrade jetty due to security issue (#8348)

  • Upgrade netty (#8346)

  • Upgrade snappy version (#8494)

Bug fixs

  • Nested arrays and map not handled correctly for complex types (#9235)

  • Fix empty data block not returning schema (#9222)

  • Allow mvn build with development webpack; fix instances default value (#9179)

  • Fix the race condition of reflection scanning classes (#9167)

  • Fix ingress manifest for controller and broker (#9135)

  • Fix jvm processors count (#9138)

  • Fix grpc query server not setting max inbound msg size (#9126)

  • Fix upsert replace (#9132)

  • Fix the race condition for partial upsert record read (#9130)

  • Fix log msg, as it missed one param value (#9124)

  • Fix authentication issue when auth annotation is not required (#9110)

  • Fix segment pruning that can break server subquery (#9090)

  • Fix the NPE for ADLSGen2PinotFS (#9088)

  • Fix cross merge (#9087)

  • Fix LaunchDataIngestionJobCommand auth header (#9070)

  • Fix catalog skipping (#9069)

  • Fix adding util for getting URL from InstanceConfig (#8856)

  • Fix string length in MutableColumnStatistics (#9059)

  • Fix instance details page loading table for tenant (#9035)

  • Fix thread safety issue with java client (#8971)

  • Fix allSegmentLoaded check (#9010)

  • Fix bug in segmentDetails table name parsing; style the new indexes table (#8958)

  • Fix pulsar close bug (#8913)

  • Fix REGEX OPTION parser (#8905)

  • Avoid reporting negative values for server latency. (#8892)

  • Fix getConfigOverrides in MinionQuickstart (#8858)

  • Fix segment generation error handling (#8812)

  • Fix multi stage engine serde (#8689)

  • Fix server discovery (#8664)

  • Fix Upsert config validation to check for metrics aggregation (#8781)

  • Fix multi value column index creation (#8848)

  • Fix grpc port assignment in multiple server quickstart (#8834)

  • Spark Connector GRPC reader fix for reading real-time tables (#8824)

  • Fix auth provider for minion (#8831)

  • Fix metadata push mode in IngestionUtils (#8802)

  • Misc fixes on segment validation for uploaded real-time segments (#8786)

  • Fix a typo in ServerInstance.startQueryServer() (#8794)

  • Fix the issue of server opening up query server prematurely (#8785)

  • Fix regression where case order was reversed, add regression test (#8748)

  • Fix dimension table load when server restart or reload table (#8721)

  • Fix when there're two index filter operator h3 inclusion index throw exception (#8707)

  • Fix the race condition of reading time boundary info (#8685)

  • Fix pruning in expressions by max/min/bloom (#8672)

  • Fix GcsPinotFs listFiles by using bucket directly (#8656)

  • Fix column data type store for data table (#8648)

  • Fix the potential NPE for timestamp index rewrite (#8633)

  • Fix on timeout string format in KinesisDataProducer (#8631)

  • Fix bug in segment rebalance with replica group segment assignment (#8598)

  • Fix the upsert metadata bug when adding segment with same comparison value (#8590)

  • Fix the deadlock in ClusterChangeMediator (#8572)

  • Fix BigDecimal ser/de on negative scale (#8553)

  • Fix table creation bug for invalid real-time consumer props (#8509)

  • Fix the bug of missing dot to extract sub props from ingestion job filesytem spec and minion segmentNameGeneratorSpec (#8511)

  • Fix to query inconsistencies under heavy upsert load (resolves #7958) (#7971)

  • Fix ChildTraceId when using multiple child threads, make them unique (#8443)

  • Fix the group-by reduce handling when query times out (#8450)

  • Fix a typo in BaseBrokerRequestHandler (#8448)

  • Fix TIMESTAMP data type usage during segment creation (#8407)

  • Fix async-profiler install (#8404)

  • Fix ingestion transform config bugs. (#8394)

  • Fix upsert inconsistency by snapshotting the validDocIds before reading the numDocs (#8392)

  • Fix bug when importing files with the same name in different directories (#8337)

  • Fix the missing NOT handling (#8366)

  • Fix setting of metrics compression type in RealtimeSegmentConverter (#8350)

  • Fix segment status checker to skip push in-progress segments (#8323)

  • Fix datetime truncate for multi-day (#8327)

  • Fix redirections for routes with access-token (#8285)

  • Fix CSV files surrounding space issue (#9028)

  • Fix suppressed exceptions in GrpcBrokerRequestHandler(#8272)

1.3.0

Release Notes for 1.3.0

This release brings significant improvements, including enhancements to the multistage query engine and the introduction of an experimental time series query engine for efficient analysis. Key features include database query quotas, cursor-based pagination for large result sets, multi-stream ingestion, and new function support for URL and GeoJson. Security vulnerabilities and several bug fixes and performance enhancements have been addressed, ensuring a more robust and versatile platform.

Multistage Engine Improvements

Reuse common expressions in a query (spool) ,

Refines query plan reuse in Apache Pinot by allowing reuse across stages instead of subtrees. Stages are natural boundaries in the query plan, divided into pull-based operators. To execute queries, Pinot introduces stages connected by MailboxSendOperator and MailboxReceiveOperator. The proposal modifies MailboxSendOperator to send data to multiple stages, transforming stage connections into a Directed Acyclic Graph (DAG) for greater efficiency and flexibility.

Segment Plan for MultiStage Queries ,

It focuses on providing comprehensive execution plans, including physical operator details. The new explain mode aligns with Calcite terminology and uses a broker-server communication flow to analyze and transform query plans into explained physical plans without executing them. A new ExplainedPlanNode is introduced to enrich query execution plans with physical details, ensuring better transparency and debugging capabilities for users.

DataBlock Serde Performance Improvements ,

Improve the performance of DataBlock building, serialization, and deserialization by reducing memory allocation and copies without altering the binary format. Benchmarks show 1x to 3x throughput gains, with significant reductions in memory allocation, minimizing GC-related latency issues in production. The improvement is achieved by changes to the buffers and the addition of a couple of stream classes.

Notable Improvements and Bug Fixes

  • Allow adding and subtracting timestamp types.

  • Remove PinotAggregateToSemiJoinRule to avoid mistakenly removing DISTINCT from the IN clause.

  • Support the use of timestamp indexes.

  • Support for polymorphic scalar comparison functions(=, !=, >, >=, <, <=).

  • Optimized MergeEqInFilterOptimizer by reducing the hash computation of expression.

  • Add support for is_enable_group_trim aggregate option.

  • Add support for is_leaf_return_final_result aggregate option.

  • Override the return type from NOW to TIMESTAMP.

  • Fix broken BIG_DECIMAL aggregations (MIN / MAX / SUM / AVG).

  • Add cluster configuration to limit the number of multi-stage queries running concurrently.

  • Allow filter for lookup JOIN.

  • Fix the bug where the query option is completely overridden when generating a leaf stage query.

  • Fix timestamp literal handling in the multi-stage query engine.

  • Add TLS support to mailboxes used in the multi-stage engine.

  • Allow configuring TLS between brokers and servers.

  • Add tablesQueried metadata to BrokerResponse for multi-stage queries.

  • Apply filter reduce expressions Calcite rule at the end to prevent literal-only filter pushdown to leaf stage.

  • Add support for all data types in return type inference from string literals for JSON extract functions.

  • Add support for the IGNORE NULLS option for the FIRST_VALUE and LAST_VALUE window functions.

  • Fix for window frame upper bound offset extraction in PinotWindowExchangeNodeInsertRule.

  • Add support for defining custom window frame bounds for window functions.

  • Improvements to allow using DISTINCTCOUNTTHETASKETCH with filter arguments.

  • Fix for ROUND scalar function in the multi-stage query engine.

  • Support for enabling LogicalProject pushdown optimizations to eliminate the exchange of unused columns.

  • Support for COALESCE as a variadic scalar function.

  • Support for lookup join.

  • Add NULLIF scalar function.

  • Compute all groups for the group by queries with only filtered aggregations.

  • Add broker API to run a query on both query engines and compare results.

  • Database handling improvement in the multi-stage engine.

  • Adds per-block row tracking for CROSS JOINs to prevent OOM while allowing large joins to function within memory limits.

  • OOM Protection Support for Multi-Stage Queries. ,

  • Refactor function registry for multi-stage engine.

  • Enforce max rows in join limit on joined rows with left input.

  • Argument type is used to look up the function for the literal-only query.

  • Ensure broker queries fail when the multi-stage engine is disabled, aligning behaviour with the controller to improve user experience.

Timeseries Engine Support in Pinot

Introduction of a Generic Time Series Query Engine in Apache Pinot, enabling native support for various time-series query languages (e.g., PromQL, M3QL) through a pluggable framework. This enhancement addresses limitations in Pinot’s current SQL-based query engines for time-series analysis, providing optimized performance and usability for observability use cases, especially those requiring high-cardinality metrics.

NOTE: Timeseries Engine support in Pinot is currently in an Experimental state.

Key Features

Pluggable Time-Series Query Language:

  • Pinot will support multiple time-series query languages, such as PromQL and Uber’s M3QL, via plugins like pinot-m3ql.

  • Example queries:

    • Plot hourly order counts for specific merchants.

    • Perform week-over-week analysis of order counts.

  • These plugins will leverage a new SPI module to enable seamless integration of custom query languages.

Pluggable Time-Series Operators:

  • Custom operators specific to each query language (e.g., nonNegativeDerivative or holt_winters) can be implemented within language-specific plugins without modifying Pinot’s core code.

  • Extensible operator abstractions will allow stakeholders to define unique time-series analysis functions.

Advantages of the New Engine:

  • Optimized for Time-Series Data: Processes data in series rather than rows, improving performance and simplifying the addition of complex analysis functions.

  • Reduced Complexity in Pinot Core: The engine reuses existing components like the Multi-Stage Engine (MSE) Query Scheduler, Query Dispatcher, and Mailbox. At the same time, language parsers and planners remain modular in plugins.

  • Improved Usability: Users can run concise and powerful time-series queries in their preferred language, avoiding the verbosity and limitations of SQL.

Impact on Observability Use Cases:

This new engine significantly enhances Pinot’s ability to handle complex time-series analyses efficiently, making it an ideal database for high-cardinality metrics and observability workloads.

The improvement is a step forward in transforming Pinot into a robust and versatile platform for time-series analytics, enabling seamless integration of diverse query languages and custom operators.

Here are some of the key PRs that have been merged as part of this feature:

  • Pinot time series engine SPI.

  • Add combine and segment level operators for time series.

  • Working E2E quickstart for time series engine.

  • Handling NULL cases in sum, min, max series builders.

  • Remove unnecessary time series materialization and minor cleanups.

  • Fix offset handling and effective time filter and enable Group-By expressions.

  • Enabling JSON column for Group-By in time series.

  • Fix bug in handling empty filters in time series.

  • Minor time series engine improvements.

  • Fix time series query correctness issue.

  • Define time series ID and broker response name tag semantics.

  • Use num docs from the value block in the time series aggregation operator.

  • Make time buckets half open on the left.

  • Fix Server Selection Bug + Enforce Timeout.

  • Response Size Limit, Metrics and Series Limit.

  • Refactor to enable Broker reduction.

  • Enable streaming response for time series.

  • Add time series exchange operator, plan node and serde.

  • Add support for partial aggregate and complex intermediate type.

  • Complete support for multi-server queries.

Database Query Quota

Introduces the ability to impose query rate limits at the database level, covering all queries made to tables within a database. A database-level rate limiter is implemented, and a new method, acquireDatabase(databaseName), is added to the QueryQuotaManager interface to check database query quotas.

Database Query Quota Configuration

  • Query and storage quotas are now provisioned similarly to table quotas but managed separately in a DatabaseConfig znode.

  • Details about the DatabaseConfig znode:

    • It does not represent a logical database entity.

    • Its absence does not prevent table creation under a database.

    • Deletion does not remove tables within the database.

Default and Override Quotas

  • A default query quota (databaseMaxQueriesPerSecond: 1000) is provided in ClusterConfig.

  • Overrides for specific databases can be configured via znodes (e.g., PROPERTYSTORE/CONFIGS/DATABASE/).

APIs for Configuration

Method
Path
Description

Dynamic Quota Updates

  • Quotas are determined by a combination of default cluster-level quotas and database-specific overrides.

  • Per-broker quotas are adjusted dynamically based on the number of live brokers.

  • Updates are handled via:

    • A custom DatabaseConfigRefreshMessage is sent to brokers upon database config changes.

    • A ClusterConfigChangeListener in ClusterChangeMediator to process updates in cluster configs.

    • Adjustments to per-broker quotas upon broker resource changes.

    • Creation of database rate limiters during the OFFLINE -> ONLINE state transition of tables in BrokerResourceOnlineOfflineStateModel.

This feature provides fine-grained control over query rate limits, ensuring scalability and efficient resource management for databases within Pinot.

Binary Workload Scheduler for Constrained Execution

Introduction of the BinaryWorkloadScheduler, which categorizes queries into two distinct workloads to ensure cluster stability and prioritize critical operations:

Workload Categories:

1. Primary Workload:

  • Default category for all production traffic.

  • Queries are executed using an unbounded FCFS (First-Come, First-Served) scheduler.

  • Designed for high-priority, critical queries to maintain consistent availability and performance.

2. Secondary Workload:

  • Reserved for ad-hoc queries, debugging tools, dashboards/notebooks, development environments, and one-off tests.

  • Imposes several constraints to minimize impact on the primary workload:

    • Limited concurrent queries: Caps the number of in-progress queries, with excess queries queued.

    • Thread restrictions: Limits the number of worker threads per query and across all queries in the secondary workload.

    • Queue pruning: Queries stuck in the queue too long are pruned based on time or queue length.

Key Benefits:

  • Prioritization: Guarantees the primary workload remains unaffected by resource-intensive or long-running secondary queries.

  • Stability: Protects cluster availability by preventing incidents caused by poorly optimized or excessive ad-hoc queries.

  • Scalability: Efficiently manages traffic in multi-tenant clusters, maintaining service reliability across workloads.

Cursors Support ,

Cursor support will allow Pinot clients to consume query results in smaller chunks. This feature allows clients to work with lesser resources esp. memory. Application logic is more straightforward with cursors. For example an app UI paginates through results in a table or a graph. Cursor support has been implemented using APIs.

API

Method
Path
Description

SPI

The feature provides two SPIs to extend the feature to support other implementations:

  • ResponseSerde: Serialize/Deserialize the response.

  • ResponseStore: Store responses in a storage system. Both SPIs use Java SPI and the default ServiceLoader to find implementation of the SPIs. All implementation should be annotated with AutoService to help generate files for discovering the implementations.

URL Functions Support

Implemented various URL functions to handle multiple aspects of URL processing, including extraction, encoding/decoding, and manipulation, making them useful for tasks involving URL parsing and modification

URL Extraction Methods

  • urlProtocol(String url): Extracts the protocol (scheme) from the URL.

  • urlDomain(String url): Extracts the domain from the URL.

  • urlDomainWithoutWWW(String url): Extracts the domain without the leading "www." if present.

  • urlTopLevelDomain(String url): Extracts the top-level domain (TLD) from the URL.

  • urlFirstSignificantSubdomain(String url): Extracts the first significant subdomain from the URL.

  • cutToFirstSignificantSubdomain(String url): Extracts the first significant subdomain and the top-level domain from the URL.

  • cutToFirstSignificantSubdomainWithWWW(String url): Returns the part of the domain that includes top-level subdomains up to the "first significant subdomain", without stripping "www.".

  • urlPort(String url): Extracts the port from the URL.

  • urlPath(String url): Extracts the path from the URL without the query string.

  • urlPathWithQuery(String url): Extracts the path from the URL with the query string.

  • urlQuery(String url): Extracts the query string without the initial question mark (?) and excludes the fragment (#) and everything after it.

  • urlFragment(String url): Extracts the fragment identifier (without the hash symbol) from the URL.

  • urlQueryStringAndFragment(String url): Extracts the query string and fragment identifier from the URL.

  • extractURLParameter(String url, String name): Extracts the value of a specific query parameter from the URL.

  • extractURLParameters(String url): Extracts all query parameters from the URL as an array of name=value pairs.

  • extractURLParameterNames(String url): Extracts all parameter names from the URL query string.

  • urlHierarchy(String url): Generates a hierarchy of URLs truncated at path and query separators.

  • urlPathHierarchy(String url): Generates a hierarchy of path elements from the URL, excluding the protocol and host.

URL Manipulation Methods

  • urlEncode(String url): Encodes a string into a URL-safe format.

  • urlDecode(String url) Decodes a URL-encoded string.

  • urlEncodeFormComponent(String url): Encodes the URL string following RFC-1866 standards, with spaces encoded as +.

  • urlDecodeFormComponent(String url): Decodes the URL string following RFC-1866 standards, with + decoded as a space.

  • urlNetloc(String url): Extracts the network locality (username:password@host:port) from the URL.

  • cutWWW(String url): Removes the leading "www." from a URL’s domain.

  • cutQueryString(String url): Removes the query string, including the question mark.

  • cutFragment(String url): Removes the fragment identifier, including the number sign.

  • cutQueryStringAndFragment(String url): Removes both the query string and fragment identifier.

  • cutURLParameter(String url, String name): Removes a specific query parameter from a URL.

  • cutURLParameters(String url, String[] names): Removes multiple specific query parameters from a URL.

Multi Stream Ingestion Support ,

  • Add support to ingest from multiple source by a single table

  • Use existing interface (TableConfig) to define multiple streams

  • Separate the partition id definition between Stream and Pinot segment

  • Compatible with existing stream partition auto expansion logics The feature does not change any existing interfaces. Users could define the table config in the same way and combine with any other transform functions or instance assignment strategies.

New Scalar Functions Support.

  • intDiv and intDivOrZero: Perform integer division, with intDivOrZero returning zero for division by zero or when dividing a minimal negative number by minus one.

  • isFinite, isInfinite, and isNaN: Check if a double value is finite, infinite, or NaN, respectively.

  • ifNotFinite: Returns a default value if the given value is not finite.

  • moduloOrZero and positiveModulo: Variants of the modulo operation, with moduloOrZero returning zero for division by zero or when dividing a minimal negative number by minus one.

  • negate: Returns the negation of a double value.

  • gcd and lcm: Calculate the greatest common divisor and least common multiple of two long values, respectively.

  • hypot: Computes the hypotenuse of a right-angled triangle given the lengths of the other two sides.

  • byteswapInt and byteswapLong: Perform byte swapping on integer and long values.

GeoJSON Support

Add support for GeoJSON Scalar functions:

Supported data types:

  • Point

  • LineString

  • Polygon

  • MultiPoint

  • MultiLineString

  • MultiPolygon

  • GeometryCollection

  • Feature

  • FeatureCollection

Improved Implementation of Distinct Operators.

Main optimizations:

  • Add per data type DistinctTable and utilize primitive type if possible

  • Specialize single-column case to reduce overhead

  • Allow processing null values with dictionary based operators

  • Specialize unlimited LIMIT case

  • Do not create priority queue before collecting LIMIT values

  • Add support for null ordering

Upsert Improvements

Features and Improvements

Track New Segments for Upsert Tables

  • Improvement for addressing a race condition where newly uploaded segments may be processed by the server before brokers add them to the routing table, potentially causing queries to miss valid documents.

  • Introduce a configurable newSegmentTrackingTimeMs (default 10s) to track new segments on the server side, allowing them to be accessed as optional segments until brokers update their routing tables.

Ensure Upsert Deletion Consistency with Compaction Flow Enabled

Enhancement addresses inconsistencies in upsert-compaction by introducing a mechanism to track the distinct segment count for primary keys. By ensuring a record exists in only one segment before compacting deleted records, it prevents older non-deleted records from being incorrectly revived during server restarts, ensuring consistent table state.

Consistent Segments Tracking for Consistent Upsert View

This improves consistent upsert view handling by addressing segment tracking and query inconsistencies. Key changes include:

  • Complete and Consistent Segment Tracking: Introduced a new Set to track segments before registration to the table manager, ensuring synchronized segment membership and validDocIds access.

  • Improved Segment Replacement: Added DuoSegmentDataManager to register both mutable and immutable segments during replacement, allowing queries to access a complete data view without blocking ingestion.

  • Query Handling Enhancements: Queries now acquire the latest consuming segments to avoid missing newly ingested data if the broker's routing table isn't updated.

  • Misc Fixes: Addressed edge cases, such as updating _numDocsIndexed before metadata updates, returning empty bitmaps instead of null, and preventing bitmap re-acquisition outside locking logic. These changes, gated by the new feature flag upsertConfig.consistencyMode, are tested with unit and stress tests in a staging environment to ensure reliability.

Other Notable Improvements and Bug Fixes

  • Config for max output segment size in UpsertCompactMerge task.

  • Add config for ignoreCrcMismatch for upsert-compaction task.

  • Upsert small segment merger task in minions.

  • Fix to acquire segmentLock before taking segment snapshot.

  • Update upsert TTL watermark in replaceSegment.

  • Fix checks on largest comparison value for upsert ttl and allow to add segments out of ttl.

  • More observability and metrics to track the upsert rate of deletion.

Lucene and Text Search Improvements

  • Store index metadata file for Lucene text indexes.

  • Runtime configurability for Lucene analyzers and query parsers, enabling dynamic text tokenization and advanced log search capabilities like case-sensitive/insensitive searches.

Security Improvements and Vulnerability Fixes

  • Force SSL cert reload daily using the scheduled thread.

  • Allow configuring TLS between brokers and servers for the multi-stage engine.

  • Strip Matrix parameter from BasePath checking.

  • Disable replacing environment variables and system properties in get table configs REST API.

  • Dependencies upgrade for vulnerabilities.

  • TLS Configuration Support for QueryServer and Dispatch Clients.

  • Returning tables names failing authorization in Exception for Multi-State Engine Queries.

  • TLS Port support for Minion.

  • Upgrade the hadoop version to 3.3.6 to fix vulnerabilities.

  • Fix vulnerabilities for msopenjdk 11 pinot-base-runtime image.

Miscellaneous Improvements

  • Allow setting ForwardIndexConfig default settings via cluster config.

  • Extend Merge Rollup Capabilities for Datasketches.

  • Skip task validation during table creation with schema.

  • Add capability to configure sketch precision / accuracy for different rollup buckets. Helpful in a space-saving for use cases where historical data does not require high accuracy.

  • Add support for application-level query quota.

  • Improvement to allow setting ForwardIndexConfig default settings via cluster config.

  • Enhanced mutable Index class to be as pluggable.

  • Improvement to allow configurable initial capacity for IndexedTable.

  • Add a new segment reload API for flexible control, allowing specific segments to be reloaded on designated servers and enabling workload management through batch processing and replica group targeting.

  • Add a server API to list segments that need to be refreshed for a table.

  • Introduced the ability to erase dimension values before rollup in merged segments, reducing cardinality and optimizing space for less critical historical data.

  • Add support for immutable CLPForwardIndex creator and related classes.

  • Add support for Minion Task to support automatic Segment Refresh.

  • Add support for S3A Connector.

  • Add support for hex decimal to long scalar functions.

  • Remove emitting null value fields during data transformation for SchemaConformingTransformer.

  • Improved CSV record reader to skip unparseable lines.

  • Add the ability to specify a target instance for segment reloading and improve API response messages when segments are not found on the target instances.

  • Add support for JSON Path Exists function.

  • Improvement for MSQ explain and stageStats when dealing with empty tables.

  • Improvement for dynamically adjusting GroupByResultHolder's initial capacity based on filter predicates to optimize resource allocation and improve performance for filtered group-by queries.

  • Add support for the isEqualSet Function.

  • Improvement to ensure consistent index configuration by constructing IndexLoadingConfig and SegmentGeneratorConfig from table config and schema, fixing inconsistencies and honouring FieldConfig.EncodingType.

  • Add usage of CLPMutableForwardIndexV2 by default to improve ingestion performance and efficiency.

  • Add support for application-level query quota.

  • Add null handling support for aggregations grouped by MV columns.

  • Add support to enable the capability to specify zstd and lz4 segment compression via config.

  • Add support for map data type on UI.

  • Add support for ComplexType in SchemaInfo to render Complex Column count in UI.

  • Introduced raw fwd index version V5 containing implicit num doc length, improving space efficiency.

  • Improvement for colocated Joins without hints.

  • Enhanced optimizeDictionary is used to optimize var-width type columns optionally.

  • Add support for BETWEEN in NumericalFilterOptimizer.

  • Add support for NULLIF scalar function.

  • Improvement for allowing usage of star-tree index with null handling enabled when no null values in segment columns.

  • Improvement Improvement for avoiding using setter in IndexLoadingConfig for consuming segment.

  • Implement consistent data push for Spark3 segment generation and metadata push jobs.

  • Improvement in addressing ingestion delays in real-time tables with many partitions by mitigating simultaneous segment commits across consumers.

  • Improve query options validation and error handling.

  • Add support an arbitrary number of WHEN THEN clauses in the scalar CASE function.

  • Add support for configuring Theta and Tuple aggregation functions.

  • Add support for Map type in complex schema.

  • Add TTL watermark storage/loading for the dedup feature to prevent stale metadata from being added to the store when loading segments.

  • Polymorphic scalar function implementation for BETWEEN.

  • Polymorphic binary arithmetic scalar functions.

  • Improvement for Adaptive Server Selection to penalize servers returning server-side exceptions.

  • Add a server-level configuration for the segment server upload to the deep store.

  • Add support to upload segments in batch mode with METADATA upload type.

  • Remove recreateDeletedConsumingSegment flag from RealtimeSegmentValidationManager.

  • Kafka3 support for realtime ingestion.

  • Allow the building of an index on the preserved field in SchemaConformingTransformer.

  • Add support to differentiate null and emptyLists for multi-value columns in avro decoder.

  • Broker config to set default query null handling behavior.

  • Moves the untarring method to BaseTaskExecutor to enable downloading and untarring from a peer server if deepstore untarring fails and allows DownloadFromServer to be enabled.

  • Optimize Adaptive Server Selection.

  • New SPI to support custom executor services, providing default implementations for cached and fixed thread pools.

  • Introduction of shared IdealStateUpdaterLock for PinotLLCRealtimeSegmentManager to prevent race conditions and timeouts during large segment updates.

  • Support for configuring aggregation function parameters in the star-tree index.

  • Write support for creating Pinot segments in the Pinot Spark connector.

  • Array flattening support in SchemaConformingTransformer.

  • Allow table names in TableConfigs with or without database name when database context is passed.

  • Improvement in null handling performance for nullable single input aggregation functions.

  • Improvement in column-based null handling by refining method naming, adding documentation and updating validation and constructor logic to support column-specific null strategies.

  • UI load time improvements.

  • Enhanced the noRawDataForTextIndex config to skip writing raw data when re-using the mutable index is enabled, fixing a global disable issue and improving ingestion performance.

  • Improvements to polymorphic scalar comparison functions for better backward compatibility.

  • Add TablePauseStatus to track the pause details.

  • Check stale dedup metadata when adding new records/segments.

  • Improve error messages with star-tree indexes creation.

  • Adds support for ZStandard and LZ4 compression in tar archives, enhancing efficiency and reducing CPU bottlenecks for large-scale data operations.

  • Support for IPv6 in Net Utils.

  • Optimize NullableSingleInputAggregationFunction when the entire block is null based on the null bitmap’s cardinality.

  • Supporting extra headers in the request to support the database in routing the requests.

  • Adds routing policy details to query error messages for unavailable segments, providing context to ease confusion and expedite issue triage.

  • Refactoring and cleanup for permissions and access. ,

  • Prevent 500 error for non-existent tasktype in /tasks/{taskType}/tasks API.

  • Changed STREAM_DATA_LOSS from a Meter to a Gauge to accurately reflect data loss detection and ensure proper cleanup.

Bug Fixes

  • Fix typo in RefreshSegmentTaskExecutor logger.

  • Fix to avoid handling JSON_ARRAY as multi-value JSON during transformation.

  • Fix for partition-enabled instance assignment with minimized movement.

  • Fix v1 query engine behaviour for aggregations without group by where the limit is zero.

  • Fix metadata fetch by increasing timeout for the Kafka client connection.

  • Fix integer overflow in GroupByUtils.

  • Fix for using PropertiesWriter to escape index_map keys properly.

  • Fix query option validation for group-by queries.

  • Fix for making RecordExtractor preserve empty array/map and map entries with empty values.

  • Fix CRC mismatch during deep store upload retry task.

  • Fix for allowing reload for UploadedRealtimeSegmentName segments.

  • Fix default value handling in REGEXP_EXTRACT transform function.

  • Fix for Spark upsert table backfill support.

  • Fix long value parsing in jsonextractscalar.

  • Fix deep store upload retry for infinite retention tables.

  • Fix to ensure deterministic index processing order across server replicas and runs to prevent inconsistent segment data file layouts and unnecessary synchronization.

  • Fix for real-time validation NPE when stream partition is no longer available.

  • Fix for handling NULL values encountered in CLPDecodeTransformFunction.

  • Fix for TextMatchFilterOptimizer grouping for the inner compound query.

  • Fix for removing redundant API calls on the home page.

  • Fix the missing precondition check for the V5 writer version in BaseChunkForwardIndexWriter.

  • Fix for computing all groups for the group by queries with only filtered aggregations.

  • Fix for race condition in IdealStateGroupCommit.

  • Fix default column handling when the forward index is disabled.

  • Fix bug with server return final aggregation result when null handling is enabled.

  • Fix Kubernetes Routing Issue in Helm chart.

  • Fix raw index conversion from v4.

  • Fix for consuming segments cleanup on server startup.

  • Fix for making S3PinotFS listFiles return directories when non-recursive.

  • Fix for rebalancer EV converge check for low disk mode.

  • Fix for copying native text index during format conversion.

  • Fix for enforcing removeSegment flow with _enableDeletedKeysCompactionConsistency.

  • Fix for Init BrokerQueryEventListener.

  • Fix for supporting ComplexFieldSpec in Schema and column metadata.

  • Fix race condition in shared literal transform functions.

  • Fix for honouring the column max length property while populating min/max values for column metadata.

  • Fix for skipping encoding the path URL for the Azure deep store.

  • Fix for handling DUAL SQL queries in Into JDBC client.

  • Fix TLS configuration for HTTP clients.

  • Fix bugs in DynamicBrokerSelection.

  • Fix literal type handling in LiteralValueExtractor.

  • Fix for handling NULL values appropriately during segment reload for newly derived columns.

  • Fix filtered aggregate with ordering.

  • Fix implementing a table-level lock to prevent parallel updates to the SegmentLineage ZK record and align real-time table ideal state updates with minion task locking for consistency.

  • Fix INT overflow issue for FixedByteSVMutableForwardIndex with large segment size.

  • Fix preload enablement checks to consider the preload executor and refine numMissedSegments logging to exclude unchanged segments, preventing incorrect missing segment reports.

  • Fix a bug in resource status evaluation during service startup, ensuring resources return GOOD when servers have no assigned segments, addressing issues with small tables and segment redistribution.

  • Fix RealtimeProvisioningHelperCommand to allow using just schemaFile along with sampleCompletedSegmentDir.

POST

/databases/{databaseName}/quotas?maxQueriesPerSecond=

Sets the database query quota

GET

/databases/{databaseName}/quotas

Get the database query quota

POST

/query/sql

New broker API parameter has been added to trigger pagination.

GET

/resultStore/{requestId}/results

Broker API that can be used to iterate over the result set of a query submitted using the above API.

GET

/resultStore/{requestId}/

Returns the BrokerResponse metadata of the query.

GET

/resultStore

Lists all the requestIds of all the query results available in the response store.

DELETE

/resultStore/{requestId}/

Delete the results of a query.

ST_GeomFromGeoJson(string) -> binary
ST_GeogFromGeoJson(string) -> binary
ST_AsGeoJson(binary) -> string
#14507
Design Doc
#13733
#14212
#13303
#13304
#14782
#14719
#14690
#13711
#14732
#14664
#14645
#14614
#14689
#14574
#14523
#14603
#14502
#14476
#14387
#14384
#14448
#14289
#14264
#14304
#14273
#14285
#14284
#14198
#14195
#13966
#14203
#14211
#13746
#14040
#13981
#13598
#13955
#13573
#13922
#13673
#13732
Design Doc
#13885
#13999
#14048
#14084
#14092
#14104
#14141
#14192
#14227
#14251
#14286
#14331
#14413
#14426
#14501
#14582
#14598
#14611
#14631
#14676
#13544
#13847
#14110
Design Doc
#14646
#13790
Design Doc
#14671
#14405
#14701
#13992
#13347
#13677
#14772
#14668
#14477
#14179
#14147
#14094
#13838
#13948
#13003
#14535
#14387
#14383
#14002
#13892
#13645
#13195
#12943
#12561)
#14030
#14773
#14625
#14683
#14373
#14226
#14773
#14609
#14620
#14544
#14544
#14355
#14288
#14300
#14474
#14435
#14351
#14396
#14393
#14376
#14374
#14001
#14313
#14258
#14241
#14226
#14071
#14008
#14245
#14254
#14105
#13943
#13994
#14163
#14203
#14177
#14190
#14139
#14170
#14158
#14125
#14167
#13906)
#14137
#14113
#14089
#14029
#14093
#13646
#14024
#13891
#13993
#13572
#13977
#13964
#13952
#13921
#13947
#13835
#13748
#13890
#13934
#13791
#13839
#13296
#13776
#13870
#13803
#13848
#13818
#13782
#13805
#13758
#13417
#13706
#13696
#13633
#13537
#13712
#14763
#14738
#14726
#13564
#14638
#14610
#12018
#14618
#14547
#14506
#14494
#14489
#14443
#14337
#14406
#14391
#14392
#14364
#14299
#14295
#14265
#14211
#14237
#14215
#14181
#13450
#14171
#14174
#14073
#14178
#14172
#13914
#13995
#13905
#13916
#13897
#13850
#13846
#13477
#13816
#13715
#13212
#13784
#13735
#13717
#13747
#13541
#13727
Apache Pinot 1.0 Upserts overview

1.0.0

This page covers the latest changes included in the Apache Pinot™ 1.0.0 release, including new features, enhancements, and bug fixes.

1.0.0 (2023-09-19)

This release includes the several new features, enhancements, and bug fixes, including the following highlights:

  • Multi-stage query engine: new features, enhancements, and bug fixes. Learn how to enable and use the multi-stage query engine or more about how the multi-stage query engine works.

Multi-stage query engine new features

  • Support for window functions

    • Initial (phase 1) Query runtime for window functions with ORDER BY within the OVER() clause (#10449)

    • Support for the ranking ROW_NUMBER() window function (#10527, #10587)

  • Set operations support:

    • Support SetOperations (UNION, INTERSECT, MINUS) compilation in query planner (#10535)

  • Timestamp and Date Operations

  • Support TIMESTAMP type and date ops functions (#11350)

  • Aggregate functions

    • Support more aggregation functions that are currently implementable (#11208)

    • Support multi-value aggregation functions (#11216)

  • Support Sketch based functions (#11153), (#11517)

  • Make Intermediate Stage Worker Assignment Tenant Aware (#10617)

  • Evaluate literal expressions during query parsing, enabling more efficient query execution (#11438)

  • Added support for partition parallelism in partitioned table scans, allowing for more efficient data retrieval (#11266)

  • [multistage]Adding more tuple sketch scalar functions and integration tests (#11517)

Multi-stage query engine enhancements

  • Turn on v2 engine by default (#10543)

  • Introduced the ability to stream leaf stage blocks for more efficient data processing (#11472).

  • Early terminate SortOperator if there is a limit (#11334)

  • Implement ordering for SortExchange (#10408)

  • Table level Access Validation, QPS Quota, Phase Metrics for multistage queries (#10534)

  • Support partition based leaf stage processing (#11234)

  • Populate queryOption down to leaf (#10626)

  • Pushdown explain plan queries from the controller to the broker (#10505)

  • Enhanced the multi-stage group-by executor to support limiting the number of groups,

  • improving query performance and resource utilization (#11424).

  • Improved resilience and reliability of the multi-stage join operator, now with added support for hash join right table protection (#11401).

Multi-stage query engine bug fixes

  • Fix Predicate Pushdown by Using Rule Collection (#10409)

  • Try fixing mailbox cancel race condition (#10432)

  • Catch Throwable to Propagate Proper Error Message (#10438)

  • Fix tenant detection issues (#10546)

  • Handle Integer.MIN_VALUE in hashCode based FieldSelectionKeySelector (#10596)

  • Improve error message in case of non-existent table queried from the controller (#10599)

  • Derive SUM return type to be PostgreSQL compatible (#11151)

Index SPI

  • Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details here

Null value support for pinot queries

  • NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.

Upsert enhancements

Delete support in upsert enabled tables (#10703)

Support added to extend upserts and allow deleting records from a realtime table. The design details can be found here.

Preload segments with upsert snapshots to speedup table loading (#11020)

Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).

TTL configs for upsert primary keys (#10915)

Adds support for specifying expiry TTL for upsert primary key metadata cleanup.

Segment compaction for upsert real-time tables (#10463)

Adds a new minion task to compact segments belonging to a real-time table with upserts.

Pinot Spark Connector for Spark3 (#10394)

  • Added spark3 support for Pinot Spark Connector (#10394)

  • Also added support to pass pinot query options to spark connector (#10443)

PinotDataBufferFactory and new PinotDataBuffer implementations (#10528)

Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.

Query functions enhancements

  • Add PercentileKLL aggregation function (#10643)

  • Support for ARG_MIN and ARG_MAX Functions (#10636)

  • refactor argmin/max to exprmin/max and make it calcite compliant (#11296)

  • Integer Tuple Sketch support (#10427)

  • Adding vector scalar functions (#11222)

  • [feature] multi-value datetime transform variants (#10841)

  • FUNNEL_COUNT Aggregation Function (#10867)

  • [multistage] Add support for RANK and DENSE_RANK ranking window functions (#10700)

  • add theta sketch scalar (#11153)

  • Register dateTimeConverter,timeConvert,dateTrunc, regexpReplace to v2 functions (#11097)

  • Add extract(quarter/dow/doy) support (#11388)

  • Funnel Count - Multiple Strategies (no partitioning requisites) (#11092)

  • Add Boolean assertion transform functions. (#11547)

JSON and CLP encoded message ingestion and querying

  • Add clpDecode transform function for decoding CLP-encoded fields. (#10885)

  • Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. (#11006)

  • Add SchemaConformingTransformer to transform records with varying keys to fit a table's schema without dropping fields. (#11210)

Tier level index config override (#10553)

  • Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.

Ingestion connectors and features

  • Kinesis stream header extraction (#9713)

  • Extract record keys, headers and metadata from Pulsar sources (#10995)

  • Realtime pre-aggregation for Distinct Count HLL & Big Decimal (#10926)

  • Added support to skip unparseable records in the csv record reader (#11487)

  • Null support for protobuf ingestion. (#11553)

UI enhancements

  • Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires (#10389)

  • AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. (#10925)

Pinot docker image improvements and enhancements

  • Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK (#10422)

  • Support multi-arch pinot docker image (#10429)

  • Update dockerfile with recent jdk distro changes (#10963)

Operational improvements

Rebalance

  • Rebalance status API (#10359)

  • Tenant level rebalance API Tenant rebalance and status tracking APIs (#11128)

Config to use customized broker query thread pool (#10614)

Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.

pinot.broker.enable.bounded.http.async.executor
pinot.broker.http.async.executor.max.pool.size
pinot.broker.http.async.executor.core.pool.size
pinot.broker.http.async.executor.queue.size

This feature allows better management of broker resources.

Drop results support (#10419)

Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.

Make column order deterministic in segment (#10468)

In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.

Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

Adds options to configure helix timeouts external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.

Enable case insensitivity by default (#10771)

This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql

Newly added APIs and client methods

  • Add Server API to get tenant pools (#11273)

  • Add new broker query point for querying multi-stage engine (#11341)

  • Add a new controller endpoint for segment deletion with a time window (#10758)

  • New API to get tenant tags (#10937)

  • Instance retag validation check api (#11077)

  • Use PUT request to enable/disable table/instance (#11109)

  • Update the pinot tenants tables api to support returning broker tagged tables (#11184)

  • Add requestId for BrokerResponse in pinot-broker and java-client (#10943)

  • Provide results in CompletableFuture for java clients and expose metrics (#10326)

Cleanup and backward incompatible changes

High level consumers are no longer supported

  • Cleanup HLC code (#11326)

  • Remove support for High level consumers in Apache Pinot (#11017)

Type information preservation of query literals

  • [feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform (#10380) String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.

Controller job status ZNode path update

  • Moving Zk updates for reload, force_commit to their own Znodes which … (#10451) The status of previously completed reload jobs will not be available after this change is deployed.

Metric names for mutable indexes to change

  • Implement mutable index using index SPI (#10687) Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.

Update in controller API to enable / disable / drop instances

  • Update getTenantInstances call for controller and separate POST operations on it (#10993)

Change in substring query function definition

  • Change substring to comply with standard sql definition (#11502)

Full list of features added

  • Allow queries on multiple tables of same tenant to be executed from controller UI #10336

  • Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig #10352

  • [Index SPI] IndexType (#10191)

  • Simplify filtered aggregate transform operator creation (#10410)

  • Introduce BaseProjectOperator and ValueBlock (#10405)

  • Add support to create realtime segment in local (#10433)

  • Refactor: Pass context instead on individual arguments to operator (#10413)

  • Add "processAll" mode for MergeRollupTask (#10387)

  • Upgrade h2 version from 1.x to 2.x (#10456)

  • Added optional force param to the table configs update API (#10441)

  • Enhance broker reduce to handle different column names from server response (#10454)

  • Adding fields to enable/disable dictionary optimization. (#10484)

  • Remove converted H2 type NUMERIC(200, 100) from BIG_DECIMAL (#10483)

  • Add JOIN support to PinotQuery (#10421)

  • Add testng on verifier (#10491)

  • Clean up temp consuming segment files during server start (#10489)

  • make pinot k8s sts and deployment start command configurable (#10509)

  • Fix Bottleneck for Server Bootstrap by Making maxConnsPerRoute Configurable (#10487)

  • Type match between resultType and function's dataType (#10472)

  • create segment zk metadata cache (#10455)

  • Allow ValueBlock length to increase in TransformFunction (#10515)

  • Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

  • Enhance error reporting (#10531)

  • Combine "GET /segments" API & "GET /segments/{tableName}/select" (#10412)

  • Exposed the CSV header map as part of CSVRecordReader (#10542)

  • Moving Zk updates for reload,force_commit to their own Znodes which will spread out Zk write load across jobTypes (#10451)

  • Enabling dictionary override optimization on the segment reload path as well. (#10557)

  • Make broker's rest resource packages configurable (#10588)

  • Check EV not exist before allowing creating the table (#10593)

  • Adding an parameter (toSegments) to the endSegmentReplacement API (#10630)

  • update target tier for segments if tierConfigs is provided (#10642)

  • Add support for custom compression factor for Percentile TDigest aggregation functions (#10649)

  • Utility to convert table config into updated format (#10623)

  • Segment lifecycle event listener support (#10536)

  • Add server metrics to capture gRPC activity (#10678)

  • Separate and parallelize BloomFilter based semgment pruner (#10660)

  • API to expose the contract/rules imposed by pinot on tableConfig #10655

  • Add description field to metrics in Pinot (#10744)

  • changing the dedup store to become pluggable #10639

  • Make the TimeUnit in the DATETRUNC function case insensitive. (#10750)

  • [feature] Consider tierConfigs when assigning new offline segment #10746

  • Compress idealstate according to estimated size #10766

  • 10689: Update for pinot helm release version 0.2.7 (#10723)

  • Fail the query if a filter's rhs contains NULL. (#11188)

  • Support Off Heap for Native Text Indices (#10842)

  • refine segment reload executor to avoid creating threads unbounded #10837

  • compress nullvector bitmap upon seal (#10852)

  • Enable case insensitivity by default (#10771)

  • Push out-of-order events metrics for full upsert (#10944)

  • [feature] add requestId for BrokerResponse in pinot-broker and java-client #10943

  • Provide results in CompletableFuture for java clients and expose metrics #10326

  • Add minion observability for segment upload/download failures (#10978)

  • Enhance early terminate for combine operator (#10988)

  • Add fromController method that accepts a PinotClientTransport (#11013)

  • Ensure min/max value generation in the segment metadata. (#10891)

  • Apply some allocation optimizations on GrpcSendingMailbox (#11015)

  • When enable case-insensitive, don't allow to add newly column name which have the same lowercase name with existed columns. (#10991)

  • Replace Long attributes with primitive values to reduce boxing (#11059)

  • retry KafkaConsumer creation in KafkaPartitionLevelConnectionHandler.java (#253) (#11040)

  • Support for new dataTime format in DateTimeGranularitySpec without explicitly setting size (#11057)

  • Returning 403 status code in case of authorization failures (#11136)

  • Simplify compatible test to avoid test against itself (#11163)

  • Updated code for setting value of segment min/max property. (#10990)

  • Add stat to track number of segments that have valid doc id snapshots (#11110)

  • Add brokerId and brokerReduceTimeMs to the broker response stats (#11142)

  • safely multiply integers to prevent overflow (#11186)

  • Move largest comparison value update logic out of map access (#11157)

  • Optimize DimensionTableDataManager to abort unnecesarry loading (#11192)

  • Refine isNullsLast and isAsc functions. (#11199)

  • Update the pinot tenants tables api to support returning broker tagged tables (#11184)

  • add multi-value support for native text index (#11204)

  • Add percentiles report in QuerySummary (#11299)

  • Add meter for broker responses with unavailable segments (#11301)

  • Enhance Minion task management (#11315)

  • add additional lucene index configs (#11354)

  • Add DECIMAL data type to orc record reader (#11377)

  • add configuration to fail server startup on non-good status checker (#11347)

  • allow passing freshness checker after an idle threshold (#11345)

  • Add broker validation for hybrid tableConfig creation (#7908)

  • Support partition parallelism for partitioned table scan (#11266)

Vulnerability fixes, bugfixes, cleanups and deprecations

  • Remove support for High level consumers in Apache Pinot (#11017)

  • Fix JDBC driver check for username (#10416)

  • [Clean up] Remove getColumnName() from AggregationFunction interface (#10431)

  • fix jersey TerminalWriterInterceptor MessageBodyWriter not found issue (#10462)

  • Bug fix: Start counting operator execution time from first NoOp block (#10450)

  • Fix unavailable instances issues for StrictReplicaGroup (#10466)

  • Change shell to bash (#10469)

  • Fix the double destroy of segment data manager during server shutdown (#10475)

  • Remove "isSorted()" precondition check in the ForwardIndexHandler (#10476)

  • Fix null handling in streaming selection operator (#10453)

  • Fix jackson dependencies (#10477)

  • Startree index build enhancement (#10905)

  • optimize queries where lhs and rhs of predicate are equal (#10444)

  • Trivial fix on a warning detected by static checker (#10492)

  • wait for full segment commit protocol on force commit (#10479)

  • Fix bug and add test for noDict -> Dict conversion for sorted column (#10497)

  • Make column order deterministic in segment (#10468)

  • Type match between resultType and function's dataType (#10472)

  • Allow empty segmentsTo for segment replacement protocol (#10511)

  • Use string as default compatible type for coalesce (#10516)

  • Use threadlocal variable for genericRow to make the MemoryOptimizedTable threadsafe (#10502)

  • Fix shading in spark2 connector pom file (#10490)

  • Fix ramping delay caused by long lasting sequence of unfiltered messa… (#10418)

  • Do not serialize metrics in each Operator (#10473)

  • Make pinot-controller apply webpack production mode when bin-dist profile is used. (#10525)

  • Fix FS props handling when using /ingestFromUri (#10480)

  • Clean up v0_deprecated batch ingestion jobs (#10532)

  • Deprecate kafka 0.9 support (#10522)

  • safely multiply integers to prevent overflow (#11186)

  • Reduce timeout for codecov and not fail the job in any case (#10547)

  • Fix DataTableV3 serde bug for empty array (#10583)

  • Do not record operator stats when tracing is enabled (#10447)

  • Forward auth token for logger APIs from controller to other controllers and brokers (#10590)

  • Bug fix: Partial upsert default strategy is null (#10610)

  • Fix flaky test caused by EV check during table creation (#10616)

  • Fix withDissabledTrue typo (#10624)

  • Cleanup unnecessary mailbox id ser/de (#10629)

  • no error metric for queries where all segments are pruned (#10589)

  • bug fix: to keep QueryParser thread safe when handling many read requests on class RealtimeLuceneTextIndex (#10620)

  • Fix static DictionaryIndexConfig.DEFAULT_OFFHEAP being actually onheap (#10632)

  • 10567: [cleanup pinot-integration-test-base], clean query generations and some other refactoring. (#10648)

  • Fixes backward incompatability with SegmentGenerationJobSpec for segment push job runners (#10645)

  • Bug fix to get the toSegments list correctly (#10659)

  • 10661: Fix for failing numeric comparison in where clause for IllegalStateException. (#10662)

  • Fixes partial upsert not reflecting multiple comparison column values (#10693)

  • Fix Bug in Reporting Timer Value for Min Consuming Freshness (#10690)

  • Fix typo of rowSize -> columnSize (#10699)

  • update segment target tier before table rebalance (#10695)

  • Fix a bug in star-tree filter operator which can incorrecly filter documents (#10707)

  • Enhance the instrumentation for a corner case where the query doesn't go through DocIdSetOp (#10729)

  • bug fix: add missing properties when edit instance config (#10741)

  • Making segmentMapper do the init and cleanup of RecordReader (#10874)

  • Fix githubEvents table for quickstart recipes (#10716)

  • Minor Realtime Segment Commit Upload Improvements (#10725)

  • Return 503 for all interrupted queries. Refactor the query killing code. (#10683)

  • Add decoder initialization error to the server's error cache (#10773)

  • bug fix: add @JsonProperty to SegmentAssignmentConfig (#10759)

  • ensure we wait the full no query timeout before shutting down (#10784)

  • Clean up KLL functions with deprecated convention (#10795)

  • Redefine the semantics of SEGMENT_STREAMED_DOWNLOAD_UNTAR_FAILURES metric to count individual segment fetch failures. (#10777)

  • fix excpetion during exchange routing causes stucked pipeline (#10802)

  • [bugfix] fix floating point and integral type backward incompatible issue (#10650)

  • [pinot-core] Start consumption after creating segment data manager (#11227)

  • Fix IndexOutOfBoundException in filtered aggregation group-by (#11231)

  • Fix null pointer exception in segment debug endpoint #11228

  • Clean up RangeIndexBasedFilterOperator. (#11219)

  • Fix the escape/unescape issue for property value in metadata (#11223)

  • Fix a bug in the order by comparator (#10818)

  • Keeps nullness attributes of merged in comparison column values (#10704)

  • Add required JSON annotation in H3IndexResolution (#10792)

  • Fix a bug in SELECT DISTINCT ORDER BY. (#10827)

  • jsonPathString should return null instead of string literal "null" (#10855)

  • Bug Fix: Segment Purger cannot purge old segments after schema evolution (#10869)

  • Fix #10713 by giving metainfo more priority than config (#10851)

  • Close PinotFS after Data Manager Shutdowns (#10888)

  • bump awssdk version for a bugfix on http conn leakage (#10898)

  • Fix MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10909)

  • Fix a bug in SELECT DISTINCT ORDER BY LIMIT. (#10887)

  • Fix an integer overflow bug. (#10940)

  • Return true when _resultSet is not null (#10899)

  • Fixing table name extraction for lateral join queries (#10933)

  • Fix casting when prefetching mmap'd segment larger than 2GB (#10936)

  • Null check before closing reader (#10954)

  • Fixes SQL wildcard escaping in LIKE queries (#10897)

  • [Clean up] Do not count DISTINCT as aggregation (#10985)

  • do not readd lucene readers to queue if segment is destroyed #10989

  • Message batch ingestion lag fix (#10983)

  • Fix a typo in snapshot lock (#11007)

  • When extracting root-level field name for complex type handling, use the whole delimiter (#11005)

  • update jersey to fix Denial of Service (DoS) (#11021)

  • Update getTenantInstances call for controller and separate POST operations on it (#10993)

  • update freemaker to fix Server-side Template Injection (#11019)

  • format double 0 properly to compare with h2 results (#11049)

  • Fix double-checked locking in ConnectionFactory (#11014)

  • Remove presto-pinot-driver and pinot-java-client-jdk8 module (#11051)

  • Make RequestUtils always return a string array when getTableNames (#11069)

  • Fix BOOL_AND and BOOL_OR result type (#11033)

  • [cleanup] Consolidate some query and controller/broker methods in integration tests (#11064)

  • Fix grpc regression on multi-stage engine (#11086)

  • Delete an obsolete TODO. (#11080)

  • Minor fix on AddTableCommand.toString() (#11082)

  • Allow using Lucene text indexes on mutable MV columns. (#11093)

  • Allow offloading multiple segments from same table in parallel (#11107)

  • Added serviceAccount to minion-stateless (#11095)

  • Bug fix: TableUpsertMetadataManager is null (#11129)

  • Fix reload bug (#11131)

  • Allow extra aggregation types in RealtimeToOfflineSegmentsTask (#10982)

  • Fix a bug when use range index to solve EQ predicate (#11146)

  • Sanitise API inputs used as file path variables (#11132)

  • Fix NPE when nested query doesn't have gapfill (#11155)

  • Fix the NPE when query response error stream is null (#11154)

  • Make interface methods non private, for java 8 compatibility (#11164)

  • Increment nextDocId even if geo indexing fails (#11158)

  • Fix the issue of consuming segment entering ERROR state due to stream connection errors (#11166)

  • In TableRebalancer, remove instance partitions only when reassigning instances (#11169)

  • Remove JDK 8 unsupported code (#11176)

  • Fix compat test by adding -am flag to build pinot-integration-tests (#11181)

  • dont duplicate register scalar function in CalciteSchema (#11190)

  • Fix the storage quota check for metadata push (#11193)

  • Delete filtering NULL support dead code paths. (#11198)

  • [bugfix] Do not move real-time segments to working dir on restart (#11226)

  • Fix a bug in ExpressionScanDocIdIterator for multi-value. (#11253)

  • Exclude NULLs when PredicateEvaluator::isAlwaysTrue is true. (#11261)

  • UI: fix sql query options seperator (#10770)

  • Fix a NullPointerException bug in ScalarTransformFunctionWrapper. (#11309)

  • [refactor] improve disk read for partial upsert handler (#10927)

  • Fix the wrong query time when the response is empty (#11349)

  • getMessageAtIndex should actually return the value in the streamMessage for compatibility (#11355)

  • Remove presto jdk8 related dependencies (#11285)

  • Remove special routing handling for multiple consuming segments (#11371)

  • Properly handle shutdown of TableDataManager (#11380)

  • Fixing the stale pinot ServerInstance in _tableTenantServersMap (#11386)

  • Fix the thread safety issue for mutable forward index (#11392)

  • Fix RawStringDistinctExecutor integer overflow (#11403)

  • [logging] fix consume rate logging bug to respect 1 minute threshold (#11421)

1.2.0

Release Notes for 1.2.0

This release comes with several Improvements and Bug Fixes for the Multistage Engine, Upserts and Compaction. There are a ton of other small features and general bug fixes.

Multistage Engine Improvements

Features

New Window Functions: LEAD, LAG, FIRST_VALUE, LAST_VALUE

  • LEAD allows you to access values after the current row in a frame.

  • LAG allows you to access values before the current row in a frame.

  • FIRST_VALUE and LAST_VALUE return the respective extremal values in the frame.

Support for Logical Database in V2 Engine

  • V2 Engine now supports a "database" construct, enabling table namespace isolation within the same Pinot cluster.

  • Improves user experience when multiple users are using the same Pinot Cluster.

  • Access control policies can be set at the database level.

  • Database can be selected in a query using a SET statement, such as SET database=my_db;.

Improved Multi-Value (MV) and Array Function Support

  • Added array sum aggregation functions for point-wise array operations .

  • Added support for valueIn MV transform function .

  • Fixed bug in numeric casts for MV columns in filters .

  • Fixed NPE in ArrayAgg when a column contains no data .

  • Fixed array literal handling .

Support for WITHIN GROUP Clause and ListAgg

  • WITHIN GROUP Clause can be used to process rows in a given order within a group.

  • One of the most common use-cases for this is the ListAgg function, which when combined with WITHIN GROUP can be used to concatenate strings in a given order.

Scalar/Transform Function and Set Operation Improvements

  • Added Geospatial Scalar Function support for use in intermediate stage in the v2 query engine .

  • Fix 'WEEK' transform function .

  • Support EXTRACT as a scalar function .

  • Added support for ALL modifier for INTERSECT and EXCEPT Set Operations .

Improved Literal Handling Support

  • Fixed bug in handling literal arguments in aggregation functions like Percentile .

  • Allow INT and FLOAT literals .

  • Fixed literal handling for all types .

  • Fixed null literal handling for null intolerant functions .

Metrics Improvements

  • Added new metrics for tracking queries executed globally and at the table level .

  • New metrics to track join counts and window function counts .

  • Multiple meters and timers to track Multistage Engine Internals .

Notable Improvements and Bug Fixes

  • Improved Window operators resiliency, with new checks to make sure the window doesn't grow too large .

  • Optimized Group Key generation .

  • Fixed SortedMailboxReceiveOperator to honor convention of pulling at most 1 EOS block .

  • Improvement in how execution stats are handled .

  • Use Protobuf instead of Reflection for Plan Serialization .

Upsert Compaction and Minion Improvements

Features and Improvements

Minion Resource Isolation

  • Minions now support resource isolation based on an instance tag.

  • Instance tag is configured at table level, and can be set for each task on a table.

  • This enables you to implement arbitrary resource isolation strategies, i.e. you can use a set of Minion Nodes for running any set of tasks across any set of tables.

Greedy Upsert Compaction Scheduling

  • Upsert compaction now schedules segments for compaction based on the number of invalid docs.

  • This helps the compaction task to handle arbitrary temporal distribution of invalid docs.

Notable Improvements

  • Minions can now download segments from servers when deepstore copy is missing. This feature is enabled via a cluster level config allowDownloadFromServer .

  • Added support for TLS Port in Minions .

  • New metrics added for Minions to track segment/record processing information .

Bug Fixes

  • Minions can now handle invalid instance tags in Task Configs gracefully. Prior to this change, Minions would be stuck in IN_PROGRESS state until task timeout .

  • Fix bug to return validDocIDsMetadata from all servers .

  • Upsert compaction doesn't retain maxLength information and trims string fields .

Upsert Improvements

Features and Improvements

Consistent Table View for Upsert Tables

  • Adds different modes of consistency guarantees for Upsert tables.

  • Adds a new UpsertConfig called consistencyMode which can be set to NONE, SYNC, SNAPSHOT.

  • SYNC is optimized for data freshness but can lead to elevated query latencies and is best for low-qps use-cases. In this mode, the ingestion threads will take a WLock when updating validDocID bitmaps.

  • SNAPSHOT mode can handle high-qps/high-ingestion use-cases by getting the list of valid docs from a snapshot of validDocID. The snapshot can be refreshed every few seconds and the tolerance can be set via a query option upsertViewFreshnessMs.

Pluggable Partial Upsert Merger

  • Partial Upsert merges the old record and the new incoming record to generate the final ingested record.

  • Pinot now allows users to customize how this merge of an old row and the new row is computed.

  • This allows a column value in the new row to be an arbitrary function of the old and the new row.

Support for Uploading Externally Partitioned Segments for Upsert Backfill

  • Segments uploaded for Upsert Backfill can now explicitly specify the Kafka partition they belong to.

  • This enables backfilling an Upsert table where the externally generated segments are partitioned using an arbitrary hash function on an arbitrary primary key.

Misc Improvements and Bug Fixes

  • Fixed a Bug in Handling Equal Comparison Column Values in Upsert, which could lead to data inconsistency ()

  • Upsert snapshot will now snapshot only those segments which have updates. .

Notable Features

JSON Support Improvements

  • JSON Index can now be used for evaluating Regex and Range Predicates.

  • jsonExtractIndex now supports contextual array filters. .

  • JSON column type now supports filter predicates like =, !=, IN and NOT IN. This is convenient for scenarios where the JSON values are very small. .

  • JSON_MATCH now supports exclusive predicates correctly. For instance, you can use predicates such as JSON_MATCH(person, '"$.addresses[*].country" != ''us''' to find all people who have at least one address that is not in the US. .

  • jsonExtractIndex supports extracting Multi-Value JSON Fields, and also supports providing any default value when the key doesn't exist. .

  • Added isJson UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. .

  • Fix ArrayIndexOutOfBoundsException in jsonExtractIndex. .

Lucene and Text Search Improvements

  • Improved Segment Build Time for Lucene Text Index by 40-60%. This improvement is realized when a consuming segment commits and changes to an ImmutableSegment. This significantly helps in lowering ingestion lag at commit time due to a large text index .

  • Phrase Search can run 3x faster when the Lucene Index Config enablePrefixSuffixMatchingInPhraseQueries is set to true. This is achieved by rewriting phrase search query to a wildcard and prefix matching query .

  • Fixed bug in TextMatchFilterOptimizer that was not applying precedence to the filter expressions properly, which could lead to incorrect results. .

  • Fixed bug in handling NOT text_match which could have returned incorrect results. .

  • Added SchemaConformingTranformerV2 to enhance text search abilities. .

  • Added metrics to track Lucene NRT Refresh Delay .

  • Switched to NRTCachingDirectory for Realtime segments and prevented duplicates in the Realtime Lucene Index to avoid IndexOutOfBounds query time exceptions. .

  • Lucene Version is upgraded to 9.11.1. .

New Funnel Functions

  • Added funnelMaxStep function which can be used to calculate max funnel steps for a given sliding window .

  • Added funnelCompleteCount to calculate the number of completed funnels, and funnelMatchStep to get the funnel match array.

Support for Interning for OnHeapByteDictionary

  • This can reduce the heap usage of a dictionary encoded byte column, for a certain distribution of duplicate values. See for details.

Column Major Builder On By Default for New Tables

  • Prior to this feature, on a segment commit, Pinot would convert all the columnar data from the Mutable Segment to row-major, and then re-build column major Immutable Segments.

  • This feature skips the row-major conversion and is expected to be both space and time efficient.

  • It can help lower ingestion lag from segment commits, especially helpful when your segments are large.

Support for SQL Formatting in Query Editor

  • You can now prettify SQL right in the Controller UI!

Hash Function for UUID Primary Keys

  • Added a new lossless hash-function for Upsert Primary Keys optimized for UUIDs.

  • The hash function can reduce Old Gen by up to 30%.

  • It maps a UUID to a 16 byte array, vs encoding it in a UTF string which would take 36 bytes.

Column Level Index Skip Query Option

  • Convenient for debugging impact of indexes on query performance or results.

  • You can add the skipIndexes option to your query to skip any number of indexes. e.g. SET skipIndexes=inverted,range;

New UDFs and Scalar Functions

  • New GeoHash functions: encodeGeoHash, decodeGeoHash, decodeGeoHashLatitude and decodeGeoHashLongitude.

  • dateBin can be used to align a timestamp to the nearest time bucket.

  • prefixes, suffixes and uniqueNgrams UDFs for generating all respective string subsequences from a string input. .

  • Added isJson UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. .

  • splitPart UDF has minor improvements. .

CLP Compression Codec in Forward Indexes

  • is a compressed log processor which has really high compression ratio for certain log types.

  • To enable this, you can set the compressionCodec in the fieldConfigList of the column you want to target.

Misc. Improvements

  • Enable segment preloading at partition level .

  • Use Temurin instead of AdoptOpenJdk

  • Adding record reader config/context param to record transformer

  • Removing legacy commons-lang dependency

  • 12508: Feature add segment rows flush config

  • ADSS Race Condition and update to client error codes

  • Add ExceptionMapper to convert Exception to Response Object for Broker REST API's

  • Add FunnelMaxStepAggregationFunction and FunnelCompleteCountAggregationFunction

  • Add GZIP Compression Codec (#11434)

  • Add PodDisruptionBudgets to the Pinot Helm chart

  • Add Postgres compliant name aliasing for String Functions.

  • Add SchemaConformingTransformerV2 to enhance text search abilities

  • Add a benchmark to measure multi-stage block serde cost

  • Add a plan version field to QueryRequest Protobuf Message

  • Add a post-validator visitor that verifies there are no cast to bytes

  • Add a safe version of CLStaticHttpHandler that disallows path traversal.

  • Add ability to track filtered messages offset

  • Add back 'numRowsResultSet' to BrokerResponse, and retain it when result table id hidden

  • Add back profile for shade

  • Add back some exclude deps from hadoop-mapreduce-client-core

  • Add backward compatibility regression test suite for multi-stage query engine

  • Add base class for custom object accumulator

  • Add clickstream example table for funnel analysis

  • Add config option for timezone

  • Add config to skip record ingestion on string column length exceeding configured max schema length

  • Add controller API to get allLiveInstances

  • Add isJson UDF

  • Add list of collaborators to asf.yaml

  • Add locking logic to get consistent table view for upsert tables

  • Add metric to track number of segments missed in upsert-snapshot

  • Add metrics for SEGMENTS_WITH_LESS_REPLICAS monitoring

  • Add mode to allow adding dummy events for non-matching steps

  • Add offset based lag metrics

  • Add protobuf codegen decoder

  • Add retry policy to wait for job id to persist during rebalancing

  • Add round-robin logic during downloadSegmentFromPeer

  • Add schema as input to the decoder.

  • Add splitPartWithLimit and splitPartFromEnd UDFs

  • Add support for creating raw derived columns during segment reload

  • Add support for raw JSON filter predicates

  • Add the possibility of configuring ForwardIndexes with compressionCodec

  • Add upsert-snapshot timer metric

  • Add validation check for forward index disabled if it's a REALTIME table

  • Added PR compatability test against release 1.1.0

  • Added kafka partition number to metadata.

  • Added pinot-error-code header in query response

  • Added tests for additional data types in SegmentPreProcessorTest.java

  • Adding a cluster config to enable instance pool and replica group configuration in table config

  • Adding batch api support for WindowFunction

  • Adding bytes string data type integration tests

  • Adding registerExtraComponents to allow registering additional components in various services

  • Adding support of insecure TLS

  • Adding support to insecure TLS when creating SSLFactory

  • Adds AGGREGATE_CASE_TO_FILTER rule

  • Adds per-column, query-time index skip option

  • Allow Aggregations in Case Expressions

  • Allow PintoHelixResourceManager subclasses to be used in the controller starter by providing an overridable PinotHelixResouceManager object creator function

  • Allow RequestContext to consider http-headers case-insensitivity

  • Allow Server throttling just before executing queries on server to allow max CPU and disk utilization

  • Allow all raw index config in star-tree index

  • Allow apply both environment variables and system properties to user and table configs, Environment variables take precedence over system properties

  • Allow configurable queryWorkerThreads in Pinot server side GrpcQueryServer

  • Allow dynamically setting the log level even for loggers that aren't already explicitly configured

  • Allow passing custom record reader to be inited/closed in SegmentProcessorFramework

  • Allow passing database context through database http header

  • Allow stop to interrupt the consumer thread and safely release the resource

  • Allow user configurable regex library for queries

  • Allow using 'serverReturnFinalResult' to optimize server partitioned table

  • Assign default value to newly added derived column upon reload

  • Avoid port conflict in integration tests

  • Better handling of null tableNames

  • CLP as a compressionCodec

  • Change helm app version to 1.0.0 for Apache Pinot latest release version

  • Clean Google Dependencies

  • Clean up BrokerRequestHandler and BrokerResponse

  • Clean up arbitrary sleep in /GrpcBrokerClusterIntegrationTest

  • Cleaning up vector index comments and exceptions

  • Cleanup HTTP components dependencies and upgrade Thrift

  • Cleanup Javax and Jakarta dependencies

  • Cleanup deprecated query options

  • Cleanup the consumer interfaces and legacy code

  • Cleanup unnecessary dependencies under pinot-s3

  • Cleanup unused aggregate internal hint

  • Consistency in API response for live broker

  • Consolidate bouncycastle libraries

  • Consolidate nimbus-jose-jwt version to 9.37.3

  • ControllerRequestClient accepts headers. Useful for authN tests

  • Custom configuration property reader for segment metadata files

  • Delete database API

  • Deprecate PinotHelixResourceManager#getAllTables() in favour of getAllTables(String databaseName)

  • Detect expired messages in Kafka. Log and set a gauge.

  • Do not hard code resource class in BaseClusterIntegrationTest

  • Do not pause ingestion when upsert snapshot flow errors out

  • Don't drop original field during flatten

  • Don't enforce -realTimeInstanceCount and -offlineInstanceCount options when creating broker tenants

  • Egalpin/skip indexes minor changes

  • Emit Metrics for Broker Adaptive Server Selector type

  • Emit table size related metrics only in lead controller

  • Enable complexType handling in SegmentProcessFramework

  • Enable more integration tests to run on the v2 multi-stage query engine

  • Enabling avroParquet to read Int96 as bytes

  • Enhance Kinesis consumer

  • Enhance Parquet Test

  • Enhance ProtoSerializationUtils to handle class move

  • Enhance Pulsar consumer

  • Enhance PulsarConsumerTest

  • Enhance commit threshold to accept size threshold without setting rows to 0

  • Enhance json index to support regexp and range predicate evaluation

  • Enhancement: Sketch value aggregator performance

  • Ensure FieldConfig.getEncodingType() is never null

  • Ensure all the lists used in PinotQuery are ArrayList

  • Ensure brokerId and requestId are always set in BrokerResponse

  • Enter segment preloading at partition level

  • Exclude dimensions from star-tree index stored type check

  • Expose more helper API in TableDataManager

  • Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness

  • Extract json individual array elements from json index for the transform function jsonExtractIndex

  • Fetch query quota capacity utilization rate metric in a callback function

  • First with time

  • GitHub Actions checkout v4

  • Gzip compression, ensure uncompressed size can be calculated from compressed buffer

  • Handle errors gracefully during multi-stage stats collection in the broker

  • Handle shaded classes in all methods of kafka factory

  • Hash Function for UUID Primary Keys

  • Ignore case when checking for Direct Memory OOM

  • Improve Retention Manager Segment Lineage Clean Up

  • Improve error message for max rows in join limit breach

  • Improve exception logging when we fail to index / transform message

  • Improve logging in range index handler for index updates

  • Improve upsert compaction threshold validations

  • Improve warn logs for requesting validDocID snapshots

  • Improved metrics for server grpc query

  • Improved null check for varargs

  • Improved segment build time for Lucene text index realtime to offline conversion

  • In ClusterTest, make start port higher to avoid potential conflict with Kafka

  • Introduce PinotLogicalAggregate and remove internal hint

  • Introduce retries while creating stream message decoder for more robustness

  • Isolate bad server configs during broker startup phase

  • Issue #12367

  • Json extract index filter support

  • Json extract index mv

  • Keep get tables API with and without database

  • Lint failure

  • Logging a warn message instead of throwing exception

  • Made the error message around dimension table size clearer

  • Make Helix state transition handling idempotent

  • Make KafkaConsumerFactory method less restrictive to avoid incompatibility

  • Make task manager APIs database aware

  • Metric for count of tables configured with various tier backends

  • Metric for upsert tables count

  • Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions

  • Minmaxrange null

  • Modify consumingSegmentsInfo endpoint to indicate how many servers failed

  • Move offset validation logic to consumer classes

  • Move package org.apache.calcite to org.apache.pinot.calcite

  • Move resolveComparisonTies from addOrReplaceSegment to base class

  • Move some mispositioned tests under pinot-core

  • Move wildfly-openssl dependency management to root pom

  • Moving deleteSegment call from POST to DELETE call

  • Optimize unnecessary extra array allocation and conversion for raw derived column during segment reload

  • Pass explicit TypeRef when evaluating MV jsonPath

  • Percentile operations supporting null

  • Prepare for next development iteration

  • Propagate Disable User Agent Config to Http Client

  • Properly handle complex type transformer in segment processor framework

  • Properly return response if SegmentCompletion is aborted

  • Publish helm 0.2.8

  • Publish helm 0.2.9

  • Pull janino dependency to root pom

  • Pull pulsar version definitaion into root POM

  • Query response opt

  • Re-enable the Spotless plugin for Java 21

  • Readme - How to setup Pinot UI for development

  • Record enricher

  • Refactor PinotTaskManager class

  • Refactored CommonsConfigurationUtils for loading properties configuration.

  • Refactored compatibility-verifier module

  • Refactoring removeSegment flow in upsert

  • Refine PeerServerSegmentFinder

  • Refine SegmentFetcherFactory

  • Replace custom fmpp plugin with fmpp-maven-plugin

  • Reposition query submission spot for adaptive server selection

  • Reset controller port when stopping the controller in ControllerTest

  • Rest Endpoint to Create ZNode

  • Return clear error message when no common broker found for multi-stage query with tables from different tenants

  • Returning tables names failing authorization in Exception of Multi State Engine Queries

  • Revert " Adding record reader config/context param to record transformer (#12520)"

  • Revert "Using local copy of segment instead of downloading from remote (#12863)"

  • Short circuit SubPlanFragmenter because we don't support multiple sub-plans yet

  • Simplify Google dependencies by importing BOM

  • Specify version for commons-validator

  • Support NOT in StarTree Index

  • Support empty strings as json nodes^

  • Supporting human-readable format when configuring broker response size

  • Use ArrayList instead of LinkedList in SortOperator

  • Use a two server setup for multi-stage query engine backward compatibility regression test suite

  • Use more efficient variants of URLEncoder::encode and URLDecoder::decode

  • Use parameterized log messages instead of string concatenation

  • Use separate action for /tasks/scheduler/jobDetails API

  • Use try-with-resources to close file walk stream in LocalPinotFS

  • Using local copy of segment instead of downloading from remote

  • [Adaptive Server Selector] Add metrics for Stats Manager Queue Size

  • [Cleanup] Move classes in pinot-common to the correct package

  • [Feature] Add Support for SQL Formatting in Query Editor

  • [HELM]: Added additional probes options and startup probe.

  • [HELM]: Added checksum config annotation in stateful set for broker, controller and server

  • [HELM]: Added namespace support in K8s deployment.

  • [HELM]: zookeeper chart upgrade to version 13.2.0

  • [Minor] Add Nullable annotation to HttpHeaders in BrokerRequestHandler

  • [Minor] Small refactor of raw index creator constructor to be more clear

  • [Multi-stage] Clean up RelNode to Operator handling

  • [null-aggr] Add null handling support in mode aggregation

  • [partial-upsert] configure early release of _partitionGroupConsumerSemaphore in RealtimeSegmentDataManager

  • [spark-connector] Add option to fail read when there are invalid segments

  • add Netty arm64 dependencies

  • add Netty unit test

  • add SegmentContext to collect validDocIds bitmaps for many segments together

  • add skipUnavailableServers query option

  • add insecure mode when Pinot uses TLS connections

  • add instrumentation to json index getMatchingFlattenedDocsMap()

  • add jmx to promethues metric exporting rule for realtimeRowsFiltered

  • add metrics for IdeaState update

  • add some metrics for upsert table preloading

  • add some tests on jsonPathString

  • add test cases in RequestUtilsTest

  • add unit test for JsonAsyncHttpPinotClientTransport

  • add unit test for QueryServer

  • add unit test for ServerChannels

  • add unit test for StringFunctions encodeUrl

  • add unit tests for pinot-jdbc-client

  • add url assertion to SegmentCompletionProtocolTest

  • adjust the llc partition consuming metric reporting logic

  • allow passing null http headers object to translateTableName

  • allow to set segment when use SegmentProcessorFramework

  • auto renew jvm default sslconext when it's loaded from files

  • avoid useless intermediate byte array allocation for VarChunkV4Reader's getStringMV

  • aws sdk 2.25.3

  • build-helper-maven-plugin 3.5.0

  • cache ssl contexts and reuse them

  • clean up jetbrain nullable annotation

  • cleanup: maven no transfer progress

  • close JDBC connections

  • do not fail on duplicate relaxed vars (#13214)z

  • dropwizard metrics 4.2.25

  • dynamic chunk sizing for v4 raw forward index

  • enable Netty leak detection

  • enable parallel Maven in pinot linter script

  • ensure inverse And/OrFilterOperator implementations match the query

  • exclude .mvn directory from source assembly

  • extend CompactedPinotSegmentRecordReader so that it can skip deleteRecord

  • get startTime outside the executor task to avoid flaky time checks

  • handle absent segments so that catchup checker doesn't get stuck on them

  • handle overflow for MutableOffHeapByteArrayStore buffer starting size

  • handle segments not tracked by partition mgr and add skipUpsertView query option

  • handle table name translation on missed api resources

  • hash4j version upgrade to 0.17.0

  • including the underlying exception in the logging output

  • int96 parity with native parquet reader

  • jsonExtractIndex support array of default values

  • log the log rate limiter rate for dropped broker logs

  • make http listener ssl config swappable

  • make reflection calls compatible with 0.9.11 [#12958](https://github.com/apache/

  • maven: no transfer progress

  • missed to delete the temp dir

  • move shouldReplaceOnComparisonTie to base class to be more reusable

  • reduce Java enum .values() usage in TimerContext

  • reduce logging for SpecialValueTransformer

  • reduce regex pattern compilation in Pinot jdbc

  • refactor TlsUtils class

  • refine when to registerSegment while doing addSegment and replaceSegment for upsert tables for better data consistency

  • reformat AdminConsoleIntegrationTest.java

  • reformat ClusterTest.java

  • release segment mgrs more reliably

  • replaced getServer with getServers

  • report rebalance job status for the early returns like noops

  • require noDictionaryColumns with aggregationConfigs

  • share the same table config object

  • track segments for snapshotting even if they lost all comparisons

  • untrack the segment out of TTL

  • update ControllerJobType from enum to string

  • update RewriterConstants so that expr min max would not collide with columns start with "parent"

  • update access control check error handling to catch throwable and log errors

Bug Fixes

  • Use gte(lte) to replace between() which has a bug

  • Fix the ConcurrentModificationException for And/Or DocIdSet

  • Upgrade RoaringBitmap to 1.0.5 to pick up the fix for RangeBitmap.between()

  • bugfix: do not move src ByteBuffer position for LZ4 length prefixed decompress

  • Bug Fix createDictionaryForColumn does not take into account inverted index

  • fix Cluster Manager error

  • fix for quick start Cluster Manager issue

  • Adding config for having suffix for client ID for realtime consumer

  • Addressed comments and fixed tests from pull request 12389. /uptime and /start-time endpoints working all components

  • Bigfix. Added missing paramName

  • Bug fix: Do not ignore scheme property

  • Bug fix: Handle missing shade config overwrites for Kafka

  • BugFix: Fix merge result from more than one server

  • Bugfix. Allow tenant rebalance with downtime as true

  • Bugfix. Avoid passing null table name input to translation util

  • Bugfix. Correct wrong method call from scheduleTask() to scheduleTaskForDatabase()

  • Bugfix. Maintain literal data type during function evaluation

  • Cleanup: Fix grammar in error message, also improve readability.

  • Fix Bug in Handling Equal Comparison Column Values in Upsert

  • Fix ColumnMinMaxValueGenerator

  • Fix JavaEE related dependencies

  • Fix Logging Location for CPU-Based Query Killing

  • Fix PulsarUtils to not share buffer

  • Fix URI construction so that AddSchema command line tool works when override flag is set to true

  • Fix [Type]ArrayList elements() method usage

  • Fix a typo when calculating query freshness

  • Fix an overflow in PinotDataBuffer.readFrom

  • Fix bug in logging in UpsertCompaction task

  • Fix bug to return validDocIDsMetadata from all servers

  • Fix connection issues if using JDBC and Hikari (#12267)

  • Fix controller host / port / protocol CLI option description for admin commands

  • Fix environment variables not applied when creating table

  • Fix error message for insufficient number of untagged brokers during tenant creation

  • Fix few metric rules which were affected by the database prefix handling

  • Fix file handle leaks in Pinot Driver (apache#12263)

  • Fix flakiness of ControllerPeriodicTasksIntegrationTest

  • Fix issue with startree index metadata loading for columns with '__' in name

  • Fix metric rule pattern regex

  • Fix pinot-parquet NoClassFound issue

  • Fix segment size check in OfflineClusterIntegrationTest

  • Fix some resource leak in tests

  • Fix the NPE from IS update metrics

  • Fix the NPE when metadataTTL is enabled without delete column

  • Fix the ServletConfig loading issue with swagger.

  • Fix the issue that map flatten shouldn't remove the map field from the record

  • Fix the race condition for H3InclusionIndexFilterOperator

  • Fix the time segment pruner on TIMESTAMP data type

  • Fix time stats in SegmentIndexCreationDriverImpl

  • Fixed infer logical type name from avro union schema

  • Fixing instance type to resolve and

  • Helm: bug fix for chart rendering issue.

  • Try to amend kafka common package with pinot shaded package prefix

  • Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes

  • Upgrade com.microsoft.azure:msal4j from 1.3.5 to 1.3.10 for CVE fixing

  • [bugfix] Handling null value for kafka client id suffix

  • bugfix: fixing jdbc client sql feature not supported exception

  • bugfix: re-add support for not text_match

  • bugfix: reduce enum array allocation in QueryLogger

  • bugfix: use consumerDir during lucene realtime segment conversion

  • cleanup: fix apache rat violation

  • fix GuavaRateLimiter acquire method

  • fix fieldsToRead class not in decoder

  • fix flakey test, avoid early finalization

  • fix merging null multi value in partial upsert

  • fix race condition in ScalingThreadPoolExecutor

  • fix shared buffer, tests

  • fix(build): update node version to 16

  • fixing CVE critical issues by resolving kerby/jline and wildfly libraries

  • fixing pinot-adls high severity CVEs

  • fixing swagger setup using localhost as host name

  • swagger-ui upgrade to 5.15.0 Fixes

  • upgrade jettison version to fix CVE

#12878
#13340
#12591
#12695
#13324
#13443
#13425
#13358
#13345
#13146
#13457
#13483
#13463
#13151
#13166
#13282
#13078
#13344
#13345
#13255
#12982
#13032
#13035
#13180
#13428
#13441
#12394
#12406
#12517
#12704
#13136
#13221
#12459
#12786
#12461
#12960
#13247
#12943
#12710
#13092
#12431
#13157
#12976
#11983
13107
#12395
#13285
#12568
#12683
#12531
#13283
#13139
#12748
#12603
#13479
#12744
#13094
#13050
#12680
#13009
#12372
#12788
#13307
#13308
#13505
#13176
#13231
#13228
#12342
#12223
#12770
#11725
#12538
#12414
#12392
#12603
#12437
#12504
CLP
#12451
#12533
#12520
#13480
#12681
#13104
#13292
#13231
#12668
#13153
#12795
#12788
#13336
#13267
#12475
#13124
#12602
#13198
#12979
#12638
#13193
#12685
#13379
#12386
#13103
#12498
#12603
#13346
#12976
#12581
#12336
#13382
#13298
#12980
#13372
#12353
#12981
#12437
#13037
#13283
#12218
#12383
#12838
#12921
#13447
#12338
#12755
#13131
#12993
#12387
#13465
#12416
#12425
#12643
#12414
#12613
#13495
#13169
#12930
#13225
#13011
#13404
#13156
#12529
#12417
#13418
#13005
#13208
#12648
#13390
#12654
#12504
#12436
#13297
#13179
#12379
#13150
#12905
#12760
#13040
#12697
#12904
#13295
#12201
#12706
#12609
#13481
#12440
#12765
#12782
#12608
#13400
#13257
#13490
#13236
#12514
#12482
#12747
#12942
#13467
#12484
#12806
#13082
#12946
#12812
#12948
#12684
#12568
#13020
#12430
#13017
#13200
#12451
#13355
#13147
#13338
#12466
#12767
#12235
#12550
#12802
#13496
#13087
#12538
#12657
#13232
#13394
#12594
#13381
#13424
#13280
#13177
#12673
#12744
#13402
#13291
#13036
#12931
#12922
#12683
#12532
#12804
#12294
#12546
#13163
#12886
#12815
#12766
#12940
#12505
#12522
#12252
#12523
#13015
#12837
#13396
#12884
#12597
#12663
#13115
#12524
#12271
#12530
#12479
#13258
#13206
#12465
#13230
#12724
#13002
#13420
#12992
#12408
#12243
#12964
#13201
#13359
#13449
#12933
#12936
#12737
#13327
#13399
#12497
#13235
#13195
#12526
#13114
#13306
#12456
#12935
#12988
#12555
#12510
#12783
#13371
#13030
#13145
#13054
#13029
#12863
#12340
#13478
#11725
#13165
#13059
#13380
#13083
#12816
#13093
#13325
#12227
#13256
#13080
#12493
#12486
#12694
#13387
#12525
#13164
#12759
#13266
#12722
#12954
#12557
#12633
#12599
#12616
#13391
#13137
#13373
#12627
#12764
#13341
#12462
#12978
#12562
#12548
#12404
#13427
#12444
#12494
#12600
#12945
#12483
#12751
#13199
#12558
#13352
#13250
#12883
#13215
#13415
#12792
#12968
#13248
#12496
#12748
#13041
#12455
#12528
#12637
#13353
#12579
#12970
#13138
#12515
#12709
#12552
#12531
#13216
#12545
#13281
#12464
#12463
#13388
#12449
#12518
#13357
#13209
#12595
#12611
#12604
#12539
#13048
#12632
#12610
#13168
#12512
#13060
#12332
#13437
#12778
#13246
#12726
#12791
#12607
#13451
#12395
#12502
#13058
#13318
#12671
#13320
#13354
#12947
#13152
#12419
#12431
#12411
#13237
#12560
#13234
#13290
#12356
#13337
#12554
#12856
#12615
#13389
#12794
#13313
#13262
#13122
#13243
#12487
#12789
#13429
#13224
#12677
#12678
#13264
#13056
#13314
#12580
#13279
#12480
#12372
#12478
#13094
#12476
#12500
#13186
#13095
#13031
#13360
#12587
#12924
#12566
#12571
#13254
#12908
#12567

0.12.0

Multi-Stage Query Engine

New join semantics support

  • Left join (#9466)

  • In-equi join (#9448)

  • Full join (#9907)

  • Right join (#9907)

  • Semi join (#9367)

  • Using keyword (#9373)

New sql semantics support:

  • Having (#9274)

  • Order by (#9279)

  • In/NotIn clause (#9374)

  • Cast (#9384)

  • LIke/Rexlike (#9654)

  • Range predicate (#9445)

Performance enhancement

  • Thread safe query planning (#9344)

  • Partial query execution and round robin scheduling (#9753)

  • Improve data table serde (#9731)

Major updates

  • Force commit consuming segments by @sajjad-moradi in #9197

  • add a freshness based consumption status checker by @jadami10 in #9244

  • Add metrics to track controller segment download and upload requests in progress by @gviedma in #9258

  • Adding endpoint to download local log files for each component by @xiangfu0 in #9259

  • [Feature] Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. by @61yao in #9265

  • add query cancel APIs on controller backed by those on brokers by @klsince in #9276

  • Add Spark Job Launcher tool by @KKcorps in #9288

  • Enable Consistent Data Push for Standalone Segment Push Job Runners by @yuanbenson in #9295

  • Allow server to directly return the final aggregation result by @Jackie-Jiang in #9304

  • TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in #9306

  • Adaptive Server Selection by @vvivekiyer in #9311

  • [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in #9312

  • Allow ingestion of errored records with incorrect datatype by @KKcorps in #9320

  • Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356

  • skip late cron job with max allowed delay by @klsince in #9372

  • Do not allow implicit cast for BOOLEAN and TIMESTAMP by @Jackie-Jiang in #9385

  • Add missing properties in CSV plugin by @KKcorps in #9399

  • set MDC so that one can route minion task logs to separate files cleanly by @klsince in #9400

  • Add a new API to fix segment date time in metadata by @KKcorps in #9413

  • Update get bytes to return raw bytes of string and support getBytesMV by @61yao in #9441

  • Exposing consumer's record lag in /consumingSegmentsInfo by @navina in #9515

  • Do not create dictionary for high-cardinality columns by @KKcorps in #9527

  • get task runtime configs tracked in Helix by @klsince in #9540

  • Add more options to json index by @Jackie-Jiang in #9543

  • add SegmentTierAssigner and refine restful APIs to get segment tier info by @klsince in #9598

  • Add segment level debug API by @saurabhd336 in #9609

  • Add record availability lag for Kafka connector by @navina in #9621

  • notify servers that need to move segments to new tiers via SegmentReloadMessage by @klsince in #9624

  • Allow to configure multi-datadirs as instance configs and a Quickstart example about them by @klsince in #9705

  • Customize stopword for Lucene Index by @jasperjiaguo in #9708

  • Add memory optimized dimension table by @KKcorps in #9802

  • ADLS file system upgrade by @xiangfu0 in #9855

  • Added Delete Schema/Table pinot admin commands by @bagipriyank in #9857

  • Adding new ADLSPinotFS auth type: DEFAULT by @xiangfu0 in #9860

  • Add rate limit to Kinesis requests by @KKcorps in #9863

  • Adding configs for zk client timeout by @xiangfu0 in #9975

Other features/changes

  • Show most recent scheduling errors by @satishwaghela in #9161

  • Do not use aggregation result for distinct query in IntermediateResultsBlock by @Jackie-Jiang in #9262

  • Emit metrics for ratio of actual consumption rate to rate limit in real-time tables by @sajjad-moradi in #9201

  • add metrics entry offlineTableCount by @walterddr in #9270

  • refine query cancel resp msg by @klsince in #9242

  • add @ManualAuthorization annotation for non-standard endpoints by @apucher in #9252

  • Optimize ser/de to avoid using output stream by @Jackie-Jiang in #9278

  • Add Support for Covariance Function by @SabrinaZhaozyf in #9236

  • Throw an exception when MV columns are present in the order-by expression list in selection order-by only queries by @somandal in #9078

  • Improve server query cancellation and timeout checking during execution by @jasperjiaguo in #9286

  • Add capabilities to ingest from another stream without disabling the real-time table by @sajjad-moradi in #9289

  • Add minMaxInvalid flag to avoid unnecessary needPreprocess by @npawar in #9238

  • Add array cardinality function by @walterddr in #9300

  • TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in #9306

  • Add support for custom null values in CSV record reader by @KKcorps in #9318

  • Infer parquet reader type based on file metadata by @saurabhd336 in #9294

  • Add Support for Cast Function on MV Columns by @SabrinaZhaozyf in #9296

  • Allow ingestion of errored records with incorrect datatype by @KKcorps in #9320

  • [Feature] Not Operator Transformation by @61yao in #9330

  • Handle null string in CSV decoder by @KKcorps in #9340

  • [Feature] Not scalar function by @61yao in #9338

  • Add support for EXTRACT syntax and converts it to appropriate Pinot expression by @tanmesh in #9184

  • Add support for Auth in controller requests in java query client by @KKcorps in #9230

  • delete all related minion task metadata when deleting a table by @zhtaoxiang in #9339

  • BloomFilterRule should only recommend for supported column type by @yuanbenson in #9364

  • Support all the types in ParquetNativeRecordReader by @xiangfu0 in #9352

  • Improve segment name check in metadata push by @zhtaoxiang in #9359

  • Allow expression transformer cotinue on error by @xiangfu0 in #9376

  • skip late cron job with max allowed delay by @klsince in #9372

  • Enhance and filter predicate evaluation efficiency by @jasperjiaguo in #9336

  • Deprecate instanceId Config For Broker/Minion Specific Configs by @ankitsultana in #9308

  • Optimize combine operator to fully utilize threads by @Jackie-Jiang in #9387

  • Terminate the query after plan generation if timeout by @jasperjiaguo in #9386

  • [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in #9312

  • [Feature] Support Coalesce for Column Names by @61yao in #9327

  • Disable logging for interrupted exceptions in kinesis by @KKcorps in #9405

  • Benchmark thread cpu time by @jasperjiaguo in #9408

  • Use ISODateTimeFormat as default for SIMPLE_DATE_FORMAT by @KKcorps in #9378

  • Extract the common logic for upsert metadata manager by @Jackie-Jiang in #9435

  • Make minion task metadata manager methods more generic by @saurabhd336 in #9436

  • Always pass clientId to kafka's consumer properties by @navina in #9444

  • Adaptive Server Selection by @vvivekiyer in #9311

  • Refine IndexHandler methods a bit to make them reentrant by @klsince in #9440

  • use MinionEventObserver to track finer grained task progress status on worker by @klsince in #9432

  • Allow spaces in input file paths by @KKcorps in #9426

  • Add support for gracefully handling the errors while transformations by @KKcorps in #9377

  • Cache Deleted Segment Names in Server to Avoid SegmentMissingError by @ankitsultana in #9423

  • Handle Invalid timestamps by @KKcorps in #9355

  • refine minion worker event observer to track finer grained progress for tasks by @klsince in #9449

  • spark-connector should use v2/brokers endpoint by @itschrispeck in #9451

  • Remove netty server query support from presto-pinot-driver to remove pinot-core and pinot-segment-local dependencies by @xiangfu0 in #9455

  • Adaptive Server Selection: Address pending review comments by @vvivekiyer in #9462

  • track progress from within segment processor framework by @klsince in #9457

  • Decouple ser/de from DataTable by @Jackie-Jiang in #9468

  • collect file info like mtime, length while listing files for free by @klsince in #9466

  • Extract record keys, headers and metadata from Stream sources by @navina in #9224

  • [pinot-spark-connector] Bump spark connector max inbound message size by @cbalci in #9475

  • refine the minion task progress api a bit by @klsince in #9482

  • add parsing for AT TIME ZONE by @agavra in #9477

  • Eliminate explosion of metrics due to gapfill queries by @elonazoulay in #9490

  • ForwardIndexHandler: Change compressionType during segmentReload by @vvivekiyer in #9454

  • Introduce Segment AssignmentStrategy Interface by @GSharayu in #9309

  • Add query interruption flag check to broker groupby reduction by @jasperjiaguo in #9499

  • adding optional client payload by @walterddr in #9465

  • [feature] distinct from scalar functions by @61yao in #9486

  • Check data table version on server only for null handling by @Jackie-Jiang in #9508

  • Add docId and column name to segment read exception by @KKcorps in #9512

  • Sort scanning based operators by cardinality in AndDocIdSet evaluation by @jasperjiaguo in #9420

  • Do not fail CI when codecov upload fails by @Jackie-Jiang in #9522

  • [Upsert] persist validDocsIndex snapshot for Pinot upsert optimization by @deemoliu in #9062

  • broker filter by @dongxiaoman in #9391

  • [feature] coalesce scalar by @61yao in #9487

  • Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356

  • [GHA] add cache timeout by @walterddr in #9524

  • Optimize PinotHelixResourceManager.hasTable() by @Jackie-Jiang in #9526

  • Include exception when upsert metadata manager cannot be created by @Jackie-Jiang in #9532

  • allow to config task expire time by @klsince in #9530

  • expose task finish time via debug API by @klsince in #9534

  • Remove the wrong warning log in KafkaPartitionLevelConsumer by @Jackie-Jiang in #9536

  • starting http server for minion worker conditionally by @klsince in #9542

  • Make StreamMessage generic and a bug fix by @vvivekiyer in #9544

  • Improve primary key serialization performance by @KKcorps in #9538

  • [Upsert] Skip removing upsert metadata when shutting down the server by @Jackie-Jiang in #9551

  • add array element at function by @walterddr in #9554

  • Handle the case when enableNullHandling is true and an aggregation function is used w/ a column that has an empty null bitmap by @nizarhejazi in #9566

  • Support segment storage format without forward index by @somandal in #9333

  • Adding SegmentNameGenerator type inference if not explicitly set in config by @timsants in #9550

  • add version information to JMX metrics & component logs by @agavra in #9578

  • remove unused RecordTransform/RecordFilter classes by @agavra in #9607

  • Support rewriting forward index upon changing compression type for existing raw MV column by @vvivekiyer in #9510

  • Support Avro's Fixed data type by @sajjad-moradi in #9642

  • [feature] [kubernetes] add loadBalancerSourceRanges to service-external.yaml for controller and broker by @jameskelleher in #9494

  • Limit up to 10 unavailable segments to be printed in the query exception by @Jackie-Jiang in #9617

  • remove more unused filter code by @agavra in #9620

  • Do not cache record reader in segment by @Jackie-Jiang in #9604

  • make first part of user agent header configurable by @rino-kadijk in #9471

  • optimize order by sorted ASC, unsorted and order by DESC cases by @gortiz in #8979

  • Enhance cluster config update API to handle non-string values properly by @Jackie-Jiang in #9635

  • Reverts recommender REST API back to PUT (reverts PR #9326) by @yuanbenson in #9638

  • Remove invalid pruner names from server config by @Jackie-Jiang in #9646

  • Using usageHelp instead of deprecated help in picocli commands by @navina in #9608

  • Handle unique query id on server by @Jackie-Jiang in #9648

  • stateless group marker missing several by @walterddr in #9673

  • Support reloading consuming segment using force commit by @Jackie-Jiang in #9640

  • Improve star-tree to use star-node when the predicate matches all the non-star nodes by @Jackie-Jiang in #9667

  • add FetchPlanner interface to decide what column index to prefetch by @klsince in #9668

  • Improve star-tree traversal using ArrayDeque by @Jackie-Jiang in #9688

  • Handle errors in combine operator by @Jackie-Jiang in #9689

  • return different error code if old version is not on master by @SabrinaZhaozyf in #9686

  • Support creating dictionary at runtime for an existing column by @vvivekiyer in #9678

  • check mutable segment explicitly instead of checking existence of indexDir by @klsince in #9718

  • Remove leftover file before downloading segmentTar by @npawar in #9719

  • add index key and size map to segment metadata by @walterddr in #9712

  • Use ideal state as source of truth for segment existence by @Jackie-Jiang in #9735

  • Close Filesystem on exit with Minion Tasks by @KKcorps in #9681

  • render the tables list even as the table sizes are loading by @jadami10 in #9741

  • Add Support for IP Address Function by @SabrinaZhaozyf in #9501

  • bubble up error messages from broker by @agavra in #9754

  • Add support to disable the forward index for existing columns by @somandal in #9740

  • show table metadata info in aggregate index size form by @walterddr in #9733

  • Preprocess immutable segments from REALTIME table conditionally when loading them by @klsince in #9772

  • revert default timeout nano change in QueryConfig by @agavra in #9790

  • AdaptiveServerSelection: Update stats for servers that have not responded by @vvivekiyer in #9801

  • Add null value index for default column by @KKcorps in #9777

  • [MergeRollupTask] include partition info into segment name by @zhtaoxiang in #9815

  • Adding a consumer lag as metric via a periodic task in controller by @navina in #9800

  • Deserialize Hyperloglog objects more optimally by @priyen in #9749

  • Download offline segments from peers by @wirybeaver in #9710

  • Thread Level Usage Accounting and Query Killing on Server by @jasperjiaguo in #9727

  • Add max merger and min mergers for partial upsert by @deemoliu in #9665

  • #9518 added pinot helm 0.2.6 with secure version pinot 0.11.0 by @bagipriyank in #9519

  • Combine the read access for replication config by @snleee in #9849

  • add v1 ingress in helm chart by @jhisse in #9862

  • Optimize AdaptiveServerSelection for replicaGroup based routing by @vvivekiyer in #9803

  • Do not sort the instances in InstancePartitions by @Jackie-Jiang in #9866

  • Merge new columns in existing record with default merge strategy by @navina in #9851

  • Support disabling dictionary at runtime for an existing column by @vvivekiyer in #9868

  • support BOOL_AND and BOOL_OR aggregate functions by @agavra in #9848

  • Use Pulsar AdminClient to delete unused subscriptions by @navina in #9859

  • add table sort function for table size by @jadami10 in #9844

  • In Kafka consumer, seek offset only when needed by @Jackie-Jiang in #9896

  • fallback if no broker found for the specified table name by @klsince in #9914

  • Allow liveness check during server shutting down by @Jackie-Jiang in #9915

  • Allow segment upload via Metadata in MergeRollup Minion task by @KKcorps in #9825

  • Add back the Helix workaround for missing IS change by @Jackie-Jiang in #9921

  • Allow uploading real-time segments via CLI by @KKcorps in #9861

  • Add capability to update and delete table config via CLI by @KKcorps in #9852

  • default to TAR if push mode is not set by @klsince in #9935

  • load startree index via segment reader interface by @klsince in #9828

  • Allow collections for MV transform functions by @saurabhd336 in #9908

  • Construct new IndexLoadingConfig when loading completed real-time segments by @vvivekiyer in #9938

  • Make GET /tableConfigs backwards compatible in case schema does not match raw table name by @timsants in #9922

  • feat: add compressed file support for ORCRecordReader by @etolbakov in #9884

  • Add Variance and Standard Deviation Aggregation Functions by @snleee in #9910

  • enable MergeRollupTask on real-time tables by @zhtaoxiang in #9890

  • Update cardinality when converting raw column to dict based by @vvivekiyer in #9875

  • Add back auth token for UploadSegmentCommand by @timsants in #9960

  • Improving gz support for avro record readers by @snleee in #9951

  • Default column handling of noForwardIndex and regeneration of forward index on reload path by @somandal in #9810

  • [Feature] Support coalesce literal by @61yao in #9958

  • Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly by @npawar in #9988

  • handle pending minion tasks properly when getting the task progress status by @klsince in #9911

  • allow gauge stored in metric registry to be updated by @zhtaoxiang in #9961

  • support case-insensitive query options in SET syntax by @agavra in #9912

  • pin versions-maven-plugin to 2.13.0 by @jadami10 in #9993

  • Pulsar Connection handler should not spin up a consumer / reader by @navina in #9893

  • Handle in-memory segment metadata for index checking by @Jackie-Jiang in #10017

  • Support the cross-account access using IAM role for S3 PinotFS by @snleee in #10009

  • report minion task metadata last update time as metric by @zhtaoxiang in #9954

  • support SKEWNESS and KURTOSIS aggregates by @agavra in #10021

  • emit minion task generation time and error metrics by @zhtaoxiang in #10026

  • Use the same default time value for all replicas by @Jackie-Jiang in #10029

  • Reduce the number of segments to wait for convergence when rebalancing by @saurabhd336 in #10028

UI Update & Improvement

  • Allow hiding query console tab based on cluster config (#9261)

  • Allow hiding pinot broker swagger UI by config (#9343)

  • Add UI to show fine-grained minion task progress (#9488)

  • Add UI to track segment reload progress (#9521)

  • Show minion task runtime config details in UI (#9652)

  • Redefine the segment status (#9699)

  • Show an option to reload the segments during edit schema (#9762)

  • Load schema UI async (#9781)

  • Fix blank screen when redirect to unknown app route (#9888)

Library version upgrade

  • Upgrade h3 lib from 3.7.2 to 4.0.0 to lower glibc requirement (#9335)

  • Upgrade ZK version to 3.6.3 (#9612)

  • Upgrade snakeyaml from 1.30 to 1.33 (#9464)

  • Upgrade RoaringBitmap from 0.9.28 to 0.9.35 (#9730)

  • Upgrade spotless-maven-plugin from 2.9.0 to 2.28.0 (#9877)

  • Upgrade decode-uri-component from 0.2.0 to 0.2.2 (#9941)

BugFixes

  • Fix bug with logging request headers by @abhs50 in #9247

  • Fix a UT that only shows up on host with more cores by @klsince in #9257

  • Fix message count by @Jackie-Jiang in #9271

  • Fix issue with auth AccessType in Schema REST endpoints by @sajjad-moradi in #9293

  • Fix PerfBenchmarkRunner to skip the tmp dir by @Jackie-Jiang in #9298

  • Fix thrift deserializer thread safety issue by @saurabhd336 in #9299

  • Fix transformation to string for BOOLEAN and TIMESTAMP by @Jackie-Jiang in #9287

  • [hotfix] Add VARBINARY column to switch case branch by @walterddr in #9313

  • Fix annotation for "/recommender" endpoint by @sajjad-moradi in #9326

  • Fix jdk8 build issue due to missing pom dependency by @somandal in #9351

  • Fix pom to use pinot-common-jdk8 for pinot-connector jkd8 java client by @somandal in #9353

  • Fix log to reflect job type by @KKcorps in #9381

  • [Bugfix] schema update bug fix by @MeihanLi in #9382

  • fix histogram null pointer exception by @jasperjiaguo in #9428

  • Fix thread safety issues with SDF (WIP) by @saurabhd336 in #9425

  • Bug fix: failure status in ingestion jobs doesn't reflect in exit code by @KKcorps in #9410

  • Fix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator by @Jackie-Jiang in #9434

  • Fix the bug of hybrid table request using the same request id by @Jackie-Jiang in #9443

  • Fix the range check for range index on raw column by @Jackie-Jiang in #9453

  • Fix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction by @ankitsultana in #9461

  • extend PinotFS impls with listFilesWithMetadata and some bugfix by @klsince in #9478

  • fix null transform bound check by @walterddr in #9495

  • Fix JsonExtractScalar when no value is extracted by @Jackie-Jiang in #9500

  • Fix AddTable for real-time tables by @npawar in #9506

  • Fix some type convert scalar functions by @Jackie-Jiang in #9509

  • fix spammy logs for ConfluentSchemaRegistryRealtimeClusterIntegrationTest [MINOR] by @agavra in #9516

  • Fix timestamp index on column of preserved key by @Jackie-Jiang in #9533

  • Fix record extractor when ByteBuffer can be reused by @Jackie-Jiang in #9549

  • Fix explain plan ALL_SEGMENTS_PRUNED_ON_SERVER node by @somandal in #9572

  • Fix time validation when data type needs to be converted by @Jackie-Jiang in #9569

  • UI: fix incorrect task finish time by @jayeshchoudhary in #9557

  • Fix the bug where uploaded segments cannot be deleted on real-time table by @Jackie-Jiang in #9579

  • [bugfix] correct the dir for building segments in FileIngestionHelper by @zhtaoxiang in #9591

  • Fix NonAggregationGroupByToDistinctQueryRewriter by @Jackie-Jiang in #9605

  • fix distinct result return by @walterddr in #9582

  • Fix GcsPinotFS by @lfernandez93 in #9556

  • fix DataSchema thread-safe issue by @walterddr in #9619

  • Bug fix: Add missing table config fetch for /tableConfigs list all by @timsants in #9603

  • Fix re-uploading segment when the previous upload failed by @Jackie-Jiang in #9631

  • Fix string split which should be on whole separator by @Jackie-Jiang in #9650

  • Fix server request sent delay to be non-negative by @Jackie-Jiang in #9656

  • bugfix: Add missing BIG_DECIMAL support for GenericRow serde by @timsants in #9661

  • Fix extra restlet resource test which should be stateless by @Jackie-Jiang in #9674

  • AdaptiveServerSelection: Fix timer by @vvivekiyer in #9697

  • fix PinotVersion to be compatible with prometheus by @agavra in #9701

  • Fix the setup for ControllerTest shared cluster by @Jackie-Jiang in #9704

  • [hotfix]groovy class cache leak by @walterddr in #9716

  • Fix TIMESTAMP index handling in SegmentMapper by @Jackie-Jiang in #9722

  • Fix the server admin endpoint cache to reflect the config changes by @Jackie-Jiang in #9734

  • [bugfix] fix case-when issue by @walterddr in #9702

  • [bugfix] Let StartControllerCommand also handle "pinot.zk.server", "pinot.cluster.name" in default conf/pinot-controller.conf by @thangnd197 in #9739

  • [hotfix] semi-join opt by @walterddr in #9779

  • Fixing the rebalance issue for real-time table with tier by @snleee in #9780

  • UI: show segment debug details when segment is in bad state by @jayeshchoudhary in #9700

  • Fix the replication in segment assignment strategy by @GSharayu in #9816

  • fix potential fd leakage for SegmentProcessorFramework by @klsince in #9797

  • Fix NPE when reading ZK address from controller config by @Jackie-Jiang in #9751

  • have query table list show search bar; fix InstancesTables filter by @jadami10 in #9742

  • [pinot-spark-connector] Fix empty data table handling in GRPC reader by @cbalci in #9837

  • [bugfix] fix mergeRollupTask metrics by @zhtaoxiang in #9864

  • Bug fix: Get correct primary key count by @KKcorps in #9876

  • Fix issues for real-time table reload by @Jackie-Jiang in #9885

  • UI: fix segment status color remains same in different table page by @jayeshchoudhary in #9891

  • Fix bloom filter creation on BYTES by @Jackie-Jiang in #9898

  • [hotfix] broker selection not using table name by @walterddr in #9902

  • Fix race condition when 2 segment upload occurred for the same segment by @jackjlli in #9905

  • fix timezone_hour/timezone_minute functions by @agavra in #9949

  • [Bugfix] Move brokerId extraction to BaseBrokerStarter by @jackjlli in #9965

  • Fix ser/de for StringLongPair by @Jackie-Jiang in #9985

  • bugfix dir check for HadoopPinotFS.copyFromLocalDir by @klsince in #9979

  • Bugfix: Use correct exception import in TableRebalancer. by @mayankshriv in #10025

  • Fix NPE in AbstractMetrics From Race Condition by @ankitsultana in #10022

1.1.0

Release Notes for 1.1.0

Summary

This release comes with several features, including SQL, UI, and performance enhancements. Also included are bug fixes across multiple features such as the V2 multi-stage query engine, ingestion, storage format, and SQL support.

Multi-stage query engine

Features

Support RelDistribution-based trait planning (#11976, #12079)

  • Adds support for RelDistribution optimization for more accurate leaf-stage direct exchange/shuffle. Also extends partition optimization beyond leaf stage to entire query plan.

  • Applies optimization based on distribution trait in the mailbox/worker assignment stage

    • Fixes previous direct exchange which was decided based on the table partition hint. Now direct exchange is decided via distribution trait: it will applied if-and-only-if the trait propagated matches the exchange requirement.

    • As a side effect, is_colocated_by_join_keys query option is reintroduced to ensure dynamic broadcast which can also benefit from direct exchange optimization

    • Allows propagation of partition distribution trait info across the tree to be used during Physical Planning phase. It can be used in the following scenarios (will follow up in separate PRs)

  • Note on backward incompatbility

    • is_colocated_by_join_keys hint is now required for making colocated joins

      • it should only affect semi-join b/c it is the only one utilizing broadcast exchange but were pulled to act as direct exchange.

      • inner/left/right/full join should automatically apply colocation thus the backward incompatibility should not affect these.

Leaf stage planning with multi-semi join support (#11937)

  • Solves the limitation of pinotQuery that supports limited amount of PlanNodes.

  • Splits the ServerRequest planning into 2 stages

    • First plan as much as possible into PinotQuery

    • for any remainder nodes that cannot be planned into PinotQuery, will be run together with the LeafStageTransferrableBlockOperator as the input locally.

Support for ArrayAgg aggregation function (#11822)

  • Usage: ArrayAgg(column, 'dataType' [, 'isDistinct'])

  • Float type column is treated as Double in the multistage engine, so FLOAT type is not supported.

  • Supports data BOOLEAN, INT, LONG, FLOAT(only in V1), DOUBLE, STRING, TIMESTAMP. E.g. ArrayAgg(intCol, 'INT') returns ARRAY<INT>

Enhancements

  • Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support

    concat as || operator (#12025)

  • Capability for constant filter in QueryContext, with support for server to handle it (#11956)

  • Tests for filter pushdown (#11994)

  • Enhancements to query plan tests (#11966)

  • Refactor PlanFragmenter to make the logic clear (#11912)

  • Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

    • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)

    • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

  • Security enhancement to add RBAC authorization checks for multi-stage query engine (#11830)

  • Enhancement to leaf-stage execution stats NPE handling (#11805)

  • Enhancement to add a framework to back-propagate metadata across opChains (#11746)

  • Use of BinaryArray to wire proto for multi-stage engine bytes literal handling (#11738)

  • Enable dynamic broadcast for SEMI joins. Adds a fallback option to enable hash table join using joinOptions(join_strategy = 'hash_table')(#11696)

  • Improvements to dispatch exception handling (#11688)

  • Allow malformed dateTime string to return default value configurable in the function signature (#11258)

    • fromDateTime(colContainsMalformedStr, '<dateTimeFormat>', '<timezone>', <default_value>)
  • Improvement in multi-stage aggregation to directly store column index as identifier (#11617)

  • Perf optimization to avoid unnecessary rows conversion in aggregation (#11607)

  • Enhance SegmentPartitionMetadataManager to handle new segment (#11585)

  • Optimize mailbox info in query plan to reduce memory footprint (#12382)

    • This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version.

  • Optimizations to query plan serialization (#12370)

  • Optimization for parallel execution of Ser/de stage plan (#12363)

  • Optimizations in query dispatch (#12358)

  • Perf optimization for group-by and join for single key scenario (#11630)

Bugfixes, refactoring, cleanups, tests

  • Bugfix for evaluation of chained literal functions (#12248)

  • Fixes to sort copy rule (#12251 and #12237)

  • Fixes duplicate results for literal queries (#12240)

  • Bugfix to use UTF-8 encoding for default Charset (#12213)

  • Bugfix to escape table name when routing queries (#12212)

  • Refactoring of planner code and removing unnecessary rules (#12070, #12052)

  • Fix to remove unnecessar project after agg during relBuilder (#12058)

  • Fixes issues multi-semi-join (#12038)

  • Fixes leaf limit refactor issue (#12001)

  • Add back filter merge after rule (#11989)

  • Fix operator EOS pull (#11970)

  • Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)

  • Fix to set explicit warning flags set on each stage stats (#11936)

  • Fix mailbox visitor mismatch receive/send (#11908)

  • Fix eliminate multiple exchanges in nested semi-join queries (#11882)

  • Bugfix for multiple consecutive Exchange returning empty response (#11885)

  • Fixing unit-test-2 build (#11889)

  • Fix issue with realtime partition mismatch metric (#11871)

  • Fix the NPE for rebalance retry (#11883)

  • Bugfix to make Agg literal attach happen after BASIC_RULES (#11863)

  • Fix NPE by init execution stats map (#11801)

  • Test cases for special column escape (#11737)

  • Fix StPoint scalar function usage in multi-stage engine intermediate stage (#11731)

  • Clean up for transform function type (#11726)

  • Add capability to ignore test (#11703)

  • Fix custom property naming (#11675)

  • Log warning when multi-stage engine planning throws exception (#11595)

  • Fix usage of metadata overrides (#11587)

  • Test change to enable metadata manager by default for colocated join quickstart (#11579)

  • Tests for IN/NOT-IN operation (#12349)

  • Fix stage id in stage plan (#12366)

  • Bugfix for IN and NOT IN filters within case statements (#12305)

Notable features

Server-level throttling for realtime consumption (#12292)

  • Use server config pinot.server.consumption.rate.limit to enable this feature

  • Server rate limiter is disabled by default (default value 0)

Reduce segment generation disk footprint for Minion Tasks (#12220)

  • Supported in MergeRollupTask and RealtimeToOfflineSegmentsTask minion tasks

  • Use taskConfig segmentMapperFileSizeThresholdInBytes to specify the threshold size

"task": {
  "taskTypeConfigsMap": {
    "<task_name>": {
      "segmentMapperFileSizeThresholdInBytes": "1000000000"
    }
  }
}

Support for swapping of TLS keystore/truststore (#12277, #12325)

  • Security feature that makes the keystore/truststore swappable.

  • Auto-reloads keystore/truststore (without need for a restart) if they are local files

Sticky query routing (#12276)

  • Adds support for deterministic and sticky routing for a query / table / broker. This setting would lead to same server / set of servers (for MultiStageReplicaGroupSelector) being used for all queries of a given table.

  • Query option (takes precedence over fixed routing setting at table / broker config level) SET "useFixedReplica"=true;

  • Table config (takes precedence over fixed routing setting at broker config level)

    "routing": {
       ...          
       "useFixedReplica": true
    }
  • Broker conf - pinot.broker.use.fixed.replica=true

Table Config to disallow duplicate primary key for dimension tables (#12290)

  • Use tableConfig dimensionTableConfig.errorOnDuplicatePrimaryKey=true to enable this behavior

  • Disabled by default

Partition-level ForceCommit for realtime tables (#12088)

  • Support to force-commit specific partitions of a realtime table.

  • Partitions can be specified to the forceCommit API as a comma separated list of partition names or consuming segment names

Support initializing broker tags from config (#12175)

  • Support to give the broker initial tags on startup.

  • Automatically updates brokerResource when broker joins the cluster for the first time

  • Broker tags are provided as comma-separated values in pinot.broker.instance.tags

Support for StreamNative OAuth2 authentication for Pulsar (#12068)

  • StreamNative (the cloud SAAS offering of Pulsar) uses OAuth2 to authenticate clients to their Pulsar clusters.

  • For more information, see how to Configure OAuth2 authentication in Pulsar clients

  • Can be configured by adding the following properties to streamConfigs:

"stream.pulsar.issuerUrl": "https://auth.streamnative.cloud"
"stream.pulsar.credsFilePath": "file:///path/to/private_creds_file"
"stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster"

Introduce low disk mode to table rebalance (#12072)

  • Introduces a new table rebalance boolean config lowDiskMode.Default value is false.

  • Applicable for rebalance with downtime=false.

  • When enabled, segments will first be offloaded from servers, then added to servers after offload is done. It may increase the total time of the rebalance, but can be useful when servers are low on disk space, and we want to scale up the cluster and rebalance the table to more servers.

  • #12112 adds the UI capability to toggle this option

Support vector index and hierarchical navigable small worlds (HNSW) (#11977)

  • Supports Vector Index on float array/multi-value columnz

  • Add predicate and function to retrieve topK closest vector. Example query

SELECT ProductId, UserId, l2_distance(embedding, ARRAY[-0.0013143676,-0.011042999,...]) AS l2_dist, n_tokens, combined
FROM fineFoodReviews
WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.0013143676,-0.011042999,...], 5)  
ORDER by l2_dist ASC 
LIMIT 10
  • The function l2_distance will return a double value where the first parameter is the embedding column and the second parameter is the search term embedding literal.

  • Since VectorSimilarity is a predicate, once config the topK, this predicate will return topk rows per segment. Then if you are using this index with other predicate, you may not get expected number of rows since the records matching other predicate might not in the topk rows.

Support for retention on deleted keys of upsert tables (#12037)

  • Adds an upsert config deletedKeysTTL which will remove deleted keys from in-memory hashmap and mark the validDocID as invalid after the deletedKeysTTL threshold period.

  • Disabled by default. Enabled only if a valid value for deletedKeysTTL is set.

Configurable Lucene analyzer (#12027)

  • Introduces the capability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis.

  • Sample usage

fieldConfigList: [
   {
        "name": "columnName",
        "indexType": "TEXT",
        "indexTypes": [
          "TEXT"
        ],
        "properties": {
          "luceneAnalyzerClass": "org.apache.lucene.analysis.core.KeywordAnalyzer"
        },
      }
  ]
  • Default Behavior falls back to using the standardAnalyzer unless the luceneAnalyzerClass property is specified.

Support for murmur3 as a partition function (#12049)

  • Murmur3 support with optional fields seed and variant for the hash in functionConfig field of columnPartitionMap.Default value for seed is 0.

  • Added support for 2 variants of Murmur3: x86_32 and x64_32 configurable using the variant field in functionConfig. If no variant is provided we choose to keep the x86_32 variant as it was part of the original implementation.

  • Examples of functionConfig;

    "tableIndexConfig": {
          ..
          "segmentPartitionConfig": {
            "columnPartitionMap": {
              "memberId": {
                "functionName": "Murmur3",
                "numPartitions": 3 
              },
              ..
            }
          }

    Here there is no functionConfig configured, so the seed value will be 0 and variant will be x86_32.

    "tableIndexConfig": {
          ..
          "segmentPartitionConfig": {
            "columnPartitionMap": {
              "memberId": {
                "functionName": "Murmur3",
                "numPartitions": 3,
                "functionConfig": {
                   "seed": "9001"
                 },
              },
              ..
            }
          }

    Here the seed is configured as 9001 but as no variant is provided, x86_32 will be picked up.

     "tableIndexConfig": {
          ..
          "segmentPartitionConfig": {
            "columnPartitionMap": {
              "memberId": {
                "functionName": "Murmur3",
                "numPartitions": 3,
                "functionConfig" :{
                   "seed": "9001"
                   "variant": "x64_32"
                 },
              },
              ..
            }
          }

    Here the variant is mentioned so Murmur3 will use the x64_32 variant with 9001 as seed.

  • Note on users using Debezium and Murmur3 as partitioning function :

    • The partitioning key should be set up on either of byte[], String or long[] columns.

    • On pinot variant should be set as x64_32 and seed should be set as 9001.

New optimized MV forward index to only store unique MV values

  • Adds new MV dictionary encoded forward index format that only stores the unique MV entries.

  • This new index format can significantly reduce the index size when the MV entries repeat a lot

  • The new index format can be enabled during index creation, derived column creation, and segment reload

  • To enable the new index format, set the compression codec in the FieldConfig:

    {
      "name": "myCol",
      "encodingType": "DICTIONARY",
      "compressionCodec": "MV_ENTRY_DICT"
    }

    Or use the new index JSON:

    {
      "name": "myCol",
      "encodingType": "DICTIONARY",
      "indexes": {
        "forward": {
          "dictIdCompressionType": "MV_ENTRY_DICT"
        }
      }
    }

Support for explicit null handling modes (#11960)

  • Adds support for 2 possible ways to handle null:

    • Table mode - which already exists

    • Column mode, which means that each column specifies its own nullability in the FieldSpec

  • Column mode can be enabled by the below config.

  • The default value for enableColumnBasedNullHandling is false. When set to true, Pinot will ignore TableConfig.IndexingConfig.nullHandlingEnabled and columns will be nullable if and only if FieldSpec.notNull is false, which is also the default value.

{
  "schemaName": "blablabla",
  "dimensionFieldSpecs": [
    {
      "dataType": "INT",
      "name": "nullableField",
      "notNull": false
    },
    {
      "dataType": "INT",
      "name": "notNullableField",
      "notNull": true
    },
    {
      "dataType": "INT",
      "name": "defaultNullableField"
    },
    ...
  ],
  "enableColumnBasedNullHandling": true/false
}

Support tracking out of order events in Upsert (#11877)

  • Adds a new upsert config outOfOrderRecordColumn

  • When set to a non-null value, we check whether an event is OOO or not and then accordingly update the corresponding column value to true / false.

  • This will help in tracking which event is out-of-order while using skipUpsert

Compression configuration support for aggregationConfigs to StartreeIndexConfigs (#11744)

  • Can be used to save space. For eg: when a functionColumnPairs has a output type of bytes, such as when you use distinctcountrawhll.

  • Sample config

"starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "a",
            "b",
            "c"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [],
          "aggregationConfigs": [
            {
              "columnName": "column1",
              "aggregationFunction": "SUM",
              "compressionCodec": "SNAPPY"
            },
            {
              "columnName": "column2",
              "aggregationFunction": "distinctcounthll",
              "compressionCodec": "LZ4"
            }
          ],
          "maxLeafRecords": 10000
        }
      ]

Preconfiguration based mirror instance assignment (#11578)

  • Supports instance assignment based pre-configured instance assignment map.

  • The assignment will always respect the mirrored servers in the pre-configured map

  • More details here

  • Sample table config

"instanceAssignmentConfigMap": {
  "CONSUMING": {
    "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
    "replicaGroupPartitionConfig": { ... },
     "tagPoolConfig": {
       ...
       "tag": "mt1_REALTIME"
     }
     ...
 }
 "COMPLETED": {
   "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
   "replicaGroupPartitionConfig": { ... },
    "tagPoolConfig": {
       ...
       "tag": "mt1_OFFLINE"
     }
     ...
 },
 "instancePartitionsMap": {
      "CONSUMING": “mt1_CONSUMING"
      "COMPLETED": "mt1_OFFLINE"
 },

Support for listing dimension tables (#11859)

  • Adds dimension as a valid option to table "type" in the /tables controller API

Support in upsert for dropping out of order events (#11811)

  • This patch adds a new config for upsert: dropOutOfOrderRecord

  • If set to true, pinot doesn't persist out-of-order events in the segment.

  • This feature is useful to

    • Save disk-usage

    • Avoid any confusion when using skipUpsert for partial-upsert tables as nulls start showing up for columns where a previous non-null was encountered and we don't know if it's an out-of-order event or not.

Support to retry failed table rebalance tasks (#11740)

  • New configs for the RebalanceChecker periodic task:

    • controller.rebalance.checker.frequencyPeriod: 5min by default ; -1 to disable

    • controller.rebalanceChecker.initialDelayInSeconds: 2min+ by default

  • New configs added for RebalanceConfig:

    • heartbeatIntervalInMs: 300_000 i.e. 5min

    • heartbeatTimeoutInMs: 3600_000 i.e. 1hr

    • maxAttempts: 3 by default, i.e. the original run plus two retries

    • retryInitialDelayInMs: 300_000 i.e. 5min, for exponential backoff w/ jitters

  • New metrics to monitor rebalance and its retries:

    • TABLE_REBALANCE_FAILURE("TableRebalanceFailure", false), emit from TableRebalancer.rebalanceTable()

    • TABLE_REBALANCE_EXECUTION_TIME_MS("tableRebalanceExecutionTimeMs", false), emit from TableRebalancer.rebalanceTable()

    • TABLE_REBALANCE_FAILURE_DETECTED("TableRebalanceFailureDetected", false), emit from RebalanceChecker

    • TABLE_REBALANCE_RETRY("TableRebalanceRetry", false), emit from RebalanceChecker

  • New restful API

    • DELETE /tables/{tableName}/rebalance API to stop rebalance. In comparison, POST /tables/{tableName}/rebalance was used to start one.

Support for UltraLogLog (#11835)

  • UltraLogLog aggregations for Count Distinct (distinctCountULL and distinctCountRawULL)

  • UltraLogLog creation via Transform Function

  • UltraLogLog merging in MergeRollup

  • Support for UltraLogLog in Star-Tree indexes

Support for Apache Datasketches CPC sketch (#11774)

  • Ingestion via transformation function

  • Extracting estimates via query aggregation functions

  • Segment rollup aggregation

  • StarTree aggregation

Support to reduce DirectMemory OOM chances on broker (#11710)

  • Broadly there are two configs that will enable this feature:

    • maxServerResponseSizeBytes: Maximum serialized response size across all servers for a query. This value is equally divided across all servers processing the query.

    • maxQueryResponseSizeBytes: Maximum length of the serialized response per server for a query

  • Configs are available as queryOption, tableConfig and Broker config. The priority of enforcement is as follows:

    The overriding order of priority is:
    1. QueryOption  -> maxServerResponseSizeBytes
    2. QueryOption  -> maxQueryResponseSizeBytes
    3. TableConfig  -> maxServerResponseSizeBytes
    4. TableConfig  -> maxQueryResponseSizeBytes
    5. BrokerConfig -> pinot.broker.max.server.response.size.bytes
    6. BrokerConfig -> pinot.broker.max.query.response.size.bytes

UI support to allow schema to be created with JSON config (#11809)

  • This is helpful when user has the entire JSON handy

  • UI still keeps Form Way to add Schema along with JSON view

Support in JSON index for ignoring values longer than a given length (#11604)

  • Use option maxValueLength in jsonIndexConfig to restrict length of values

  • A value of 0 (or when the key is omitted) means there is no restriction

Support for MultiValue VarByte V4 index writer (#11674)

  • Supports serializing and writing MV columns in VarByteChunkForwardIndexWriterV4

  • Supports V4 reader that can be used to read SV var length, MV fixed length and MV var length buffers encoded with V4 writer

Improved scalar function support for Multivalue columns(#11555, #11654)

arrayIndexOfInt(int[] value, int valToFind)
arrayIndexOfLong(int[] value, long valToFind)
arrayIndexOfFloat(int[] value, float valToFind)
arrayIndexOfDouble(int[] value, double valToFind)
arrayIndexOfString(int[] value, String valToFind)
intersectIndices(int[] values1, int[] values2)

Support for FrequentStringsSketch and FrequentLonsSketch aggregation functions (#11098)

  • Approximation aggregation functions for estimating the frequencies of items a dataset in a memory efficient way. More details in Apache Datasketches library.

FREQUENTLONGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object
FREQUENTSTRINGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object

Controller API for table index (#11576)

  • Table index api to get the aggregate index details of all segments for a table.

    • URL/tables/{tableName}/indexes

  • Response format

    {
        "totalSegments": 31,
        "columnToIndexesCount":
        {
            "col1":
            {
                "dictionary": 31,
                "bloom": 0,
                "null": 0,
                "forward": 31,
                ...
                "inverted": 0,
                "some-dynamically-injected-index-type": 31,
            },
            "col2":
            {
                ...
            }
            ...
    }

Support for configurable rebalance delay at lead controller (#11509)

  • The lead controller rebalance delay is now configurable with controller.resource.rebalance.delay_ms

  • Changing rebalance configurations will now update the lead controller resource

Support for configuration through environment variables (#12307)

  • Adds support for Pinot configuration through ENV variables with Dynamic mapping.

  • More details in issue: #10651

  • Sample configs through ENV

export PINOT_CONTROLLER_HOST=host
export PINOT_SERVER_PROPERTY_WHATEVER=whatever_property
export ANOTHER_VARIABLE=random

Add hyperLogLogPlus aggregation function for distinct count (#11346)

  • HLL++ has higher accuracy than HLL when cardinality of dimension is at 10k-100k.

  • More details here

DISTINCTCOUNTHLLPLUS(some_id, 12)

Support for clpMatch

  • Adds query rewriting logic to transform a "virtual" UDF, clpMatch, into a boolean expression on the columns of a CLP-encoded field.

  • To use the rewriter, modify broker config to add org.apache.pinot.sql.parsers.rewriter.ClpRewriter to pinot.broker.query.rewriter.class.names.

Support for DATETIMECONVERTWINDOWHOP function (#11773)

Support for JSON_EXTRACT_INDEX transform function to leverage json index for json value extraction (#11739)

Support for ArrayAgg aggregation function (#11822)

GenerateData command support for generating data in JSON format (#11778)

Enhancements

SQL

  • Support ARRAY function as a literal evaluation (#12278)

  • Support for ARRAY literal transform functions (#12118)

  • Theta Sketch Aggregation enhancements (#12042)

    • Adds configuration options for DistinctCountThetaSketchAggregationFunction

    • Respects ordering for existing Theta sketches to use "early-stop" optimisations for unions

  • Add query option override for Broker MinGroupTrimSize (#11984)

  • Support for 2 new scalar functions for bytes: toUUIDBytes and fromUUIDBytes (#11988)

  • Config option to make groupBy trim size configurable at Broker (#11958)

  • Pre-aggregation support for distinct count hll++ (#11747)

  • Add float type into literal thrift to preserve literal type conforming to SQL standards (#11697)

  • Enhancement to add query function override for Aggregate functions of multi valued columns (#11307)

  • Perf optimization in IN clause evaluation (#11557)

  • Add TextMatchFilterOptimizer to maximally push down text_match filters to Lucene (#12339

UI

  • Async rendering of UI elements to load UI elements async resulting in faster page loads (#12210)

  • Make the table name link clickable in task details (#12253)

  • Swagger UI enhancements to resumeConsumption API call (#12200)

  • Adds support for CTRL key as a modifier for Query shortcuts (#12087)

  • UI enhancement to show partial index in reload (#11913)

  • UI improvement to add Links to Instance in Table and Segment View (#11807)

  • Fixes reload to use the right indexes API instead of fetching all segment metadata (#11793)

  • Enhancement to add toggle to hide/show query exceptions (#11611)

Misc

  • Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap (#12223)

  • Wire soft upsert delete for Compaction task (12330)

  • Upsert compaction debuggability APIs for validDocId metadata (#12275)

  • Make server resource classes configurable (#12324)

  • Shared aggregations for Startree index - mapping from aggregation used in the query to aggregation used to store pre-aggregated values (#12164)

  • Increased fetch timeout for Kineses to prevent stuck kinesis consumers

  • Metric to track table rebalance (#12270)

  • Allow server-level configs for upsert metadata (#18851)

  • Support to dynamically initialize Kafka client SSL configs (#12249)

  • Optimize segment metadata file creation without having to download full segment (#12255)

  • Allow string / numeric data type for deleteRecordColumn config (#12222)

  • Atomic and Deterministic snapshots of validDocId for upsert tables (#12232, #12246)

  • Observability enhancement to add column name when JSON index building fails (#12151)

  • Creation of DateTimeGenerator for DATE_TIME field type columns (#12206)

  • Add singleton registry for all controller and minion metrics (#12119)

  • Support helm chart server separate liveness and readiness probe endpoints (#11800)

  • Observability enhancement to add metrics for Table Disabled and Consumption Paused (#12000)

  • Support for SegmentGenerationAndPushTask to push segment to realtime table (#12084)

  • Enhancement to make the deep store upload retry async with configurable parallelism (#12017)

  • Optimizations in segment commit to not read partition group metadata (#11943)

  • Replace timer with scheduled executor service in IngestionDelayTracker to reduce number of threads (#11849)

  • Adds an option skipControllerCertValidation to skip controller cert validation in AddTableCommand (#11967)

  • Adds instrumentation for DataTable Creation (#11942)

  • Improve performance of ZkBasicAuthAccessFactory by caching Bcrypt password (#11904)

  • Adds support to to fetch metadata for specific list of segments (#11949)

  • Allow user specify local temp directory for quickstart (#11961)

  • Optimization for server to directly return final result for queries hitting single server (#11938)

  • Explain plan optimization to early release AcquireReleaseColumnsSegmentOperator (#11945)

  • Observability metric to track query timeouts (#11892)

  • Add support for auth in QueryRunner (#11897)

  • Allow users to pass custom RecordTransformers to SegmentProcessorFramework (#11887)

  • Add isPartialResult flag to broker response (#11592)

  • Add new configs to Google Cloud Storage (GCS) connector: jsonKey (#11890)

    • jsonKey is the GCP credential key in string format (either in plain string or base64 encoded string). Refer Creating and managing service account keys to download the keys.

  • Performance enhancement to build segments in column orientation (#11776)

    • Disabled by default. Can be enabled by setting table config columnMajorSegmentBuilderEnabled

  • Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

    • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)

    • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

  • Observability improvement to expose GRPC metrics (#11842)

  • Improvements to response format for reload API to be pretty printed (#11608)

  • Enhancements to support Java 21 (#11672)

  • Add more information in RequestContext class (#11708)

  • Support to read exact buffer byte ranges corresponding to a given forward index doc id (#11729)

  • Enhance Broker reducer to handle expression format change (#11762)

  • Capture build scans on ge.apache.org to benefit from deep build insights (#11767)

  • Performance enhancement in multiple places by updating initial capacity of HashMap (#11709)

  • Support for building indexes post segment file creation, allowing indexes that may depend on a completed segment to be built as part of the segment creation process (#11711)

  • Support excluding time values in SimpleSegmentNameGenerator (#11650)

  • Perf enhancement to reduce cpu usage by avoiding throwing an exception during query execution (#11715)

  • Added framework for supporting nulls in ScalarTransformFunctionWrapper in the future (#11653)

  • Observability change to metrics to export netty direct memory used and max (#11575)

  • Observability change to add a metric to measure total thread cpu time for a table (#11713)

  • Observability change to use SlidingTimeWindowArrayReservoirin dropwizard metrics (#11695)

  • Minor improvements to upsert preload (#11694)

  • Observability changes to expose additional Realtime Ingestion Metrics (#11685)

  • Perf enhancement to remove the global lock in SegmentCompletionManager (#11679)

  • Enhancements to unify tmp file naming format and delete tmp files at a regular cadence by extending the ControllerPeriodicTask (#10815)

    • controller.realtime.segment.tmpFileAsyncDeletionEnabled (default false)

    • controller.realtime.segment.tmpFileRetentionInSeconds (default 3600)

  • Improvements to skip unparseable records in the csv record reader (#11540, #11594)

  • Enhancements to allow override/force options when add schema (#11572)

  • Enhancement to handle direct memory OOM on brokers (#11496)

  • Enhancement to metadata API to return upsert partition to primary key count map for both controller and server APIs (#12334)

  • Enhancements to peer server segment download by retrying both peer discovery and download. (#12317)

  • Helper functions in StarTreeBuilderUtils and StarTreeV2BuilderConfig (#12361)

  • Perf optimizations to release all segments of a table in releaseAndRemoveAllSegments method (#12297)

  • Enhancement to Maintain pool selection for the minimizeDataMovement instance partition assignment strategy (#11953)

  • Upsert enhancement to assign segments for with respect to ideal state (#11628)

  • Observability change to export Additional Upsert Metrics to Prom (#11660)

  • Observibility enhancement to add CPU metrics for minion purge task (#12337)

  • Add HttpHeaders in broker event listener requestContext (#12258)

Bug fixes, refactoring, cleanups, deprecations

  • Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader (#12329)

  • Fix error message format for Preconditions.checks failures(#12327)

  • Bugfix to distribute Pinot as a multi-release JAR (#12131, #12300)

  • Fixes in upsert metadata manager (#12319)

  • Security fix to allow querying tables with table-type suffix (#12310)

  • Bugfix to ensure tagConfigOverride config is null for upsert tables (#12233 and #12311)

  • Increased fetch timeout for Kineses to prevent stuck kinesis consumers(#12214)

  • Fixes to catch-all Regex for JXM -> Prom Exporter (#12073 and #12295)

  • Fixes lucene index errors when using QuickStart (#12289)

  • Null handling bugfix for sketch group-by queries (#12259)

  • Null pointer exception fixes in Controller SQL resource (#12211)

  • Synchronization fixes to replace upsert segments (#12105 and #12241)

  • Bugfix for S3 connection pool error when AWS session tokens expire after an hour (#12221)

  • FileWriter fixes to append headerline only for required formats like csv (#12208)

  • Security bugfix for pulsar OAuth2 authentication (#12195)

  • Bugfix to appropriately compute "segment.flush.threshold.size" when force-committing realtime segments (#12188)

  • Fixes rebalance converge check that reports success before rebalance completes (#12182)

  • Fixes upsertPrimaryKeysCount metric reporting when table is deleted (#12169)

  • Update LICENSE-binary for commons-configuration2 upgrade (#12165)

  • Improve error logging when preloading segments not exist on server (#12153)

  • Fixes to file access resource leaks (#12129)

  • Ingestion bugfix to avoid unnecessary transformers in CompositeTransformer (#12138)

  • Improve logging to print OS name during service statup (#12135)

  • Improve logging in multiple files (#12134, #12137, #12127, #12121)

  • Test fixes for ExprMinMaxRewriterTest.testQueryRewrite (#12047)

  • Fixes default path of log4j in helmchart (#12069, #12083)

  • Fix default brokerUpdateFrequencyInMillis for connector (#12093)

  • Updates to README file (#12075)

  • Fix to remove unnecessary locking during segment preloading (#12077)

  • Fix bug with silently ignoring force commit call failures (#12044)

  • Upsert bugfix to allow optional segments that can be skipped by servers without failing the query (#11978)

  • Fix incorrect handling of consumer creation errors (#12045)

  • Fix the memory leak issue on CommonsConfigurationUtils (#12056)

  • Fix rebalance on upsert table (#12054)

  • Add new Transformer to transform -0.0 and NaN (#12032)

  • Improve inverted index validation in table config to enhance user experience (#12043)

  • Fixes test flakiness by replacing HashSet/HashMap with LinkedHashSet/LinkedHashMap (#11941)

  • Flaky test fix for ServerRoutingStatsManagerTest.testQuerySubmitAndCompletionStats (#12029)

  • Fix derived column from MV column (#12028)

  • Support for leveraging StarTree index in conjunction with filtered aggregations (#11886)

  • Improves tableConfig validation for enabling size based threshold for realtime tables (#12016)

  • Fix flaky PinotTenantRestletResourceTest (#12026)

  • Fix flaky Fix PinotTenantRestletResourceTest (#12019)

  • Fix the race condition of concurrent modification to segment data managers (#12004)

  • Fix the misuse of star-tree when all predicates are always false under OR (#12003)

  • Fix the test failures caused by instance drop failure (#12002)

  • Fix fromULL scalar function (#11995)

  • Fix to exclude module-info.class during shade operations (#11975)

  • Fix the wrong import for Preconditions (#11979)

  • Add check for illegal character '/' in taskName (#11955)

  • Bugfix to only register new segments when it's fully initalized by partitionUpsertMetadataManager (#11964)

  • Obervability fix to add logs to track sequence of events for table creation (#11946)

  • Fix the NPE in minimizeDataMovement instance assignment strategy (#11952)

  • Fix to add catch all logging for exception during DQL/DML process (#11944)

  • Fix bug where we don't handle cases that a upsert table has both upsert deletion and upsert ttl configs (#11791)

  • Removing direct dependencies on commons-logging and replacing with jcl-over-slf4j (#11920)

  • Fix NPE for IN clause on constant STRING dictionary (#11930)

  • Fix flaky OfflineClusterIntegrationTest on server response size tests (#11926)

  • Avoid npe when checking mirror server set assignment (#11915)

  • Deprecate _segmentAssignmentStrategy in favor of SegmentsValidationAndRetentionConfig #11869

  • Bugfix to capture auth phase timing even if access is denied (#11884)

  • Bugfix to mark rows as invalid in case primary time column is out of range (#11907)

  • Fix to radomize server port to avoid port already bind issue (#11861)

  • Add LazyRow abstraction for previously indexed record (#11826)

  • Config Validation for upsert table to not assign COMPLETED segments to another server (#11852)

  • Bugfix to resolve dependency conflict in pinot-protobuf module (#11867)

  • Fix case of useMultistageEngine property reference in JsonAsyncHttpPinotClientTransportFactory (#11820)

  • Bugfix to add woodstox-core to pinot-s3 dependencies and fix stack trace (#11799)

  • Fix to move pinot-segment-local test from unit test suite 1 to 2 (#11865)

  • Observability fix to log upsert config when initializing the metadata manager (#11864)

  • Fix to improve tests when errors are received in the consumer thread (#11858)

  • Fix for flaky ArrayAgg test (#11860)

  • Fix for flaky tests in TupleSelectionTransformFunctionsTest (#11848)

  • Fix for arrayAgg null support (#11853)

  • Fix the bug of reading decimal value stored in int32 or int64 (#11840)

  • Remove duplicate pinot-integration-tests from unit test suite 2 (#11844)

  • Fix for a null handling error in queries (#11829)

  • Fix the way of fetching the segment zk metadata for task generators (#11832)

  • Make testInvalidateCachedControllerLeader times based on getMinInvalidateIntervalMs (#11815)

  • Update doap to reflect latest release (#11827)

  • Clean up integration test pom file (#11817)

  • Bugfix to exclude OFFLINE segments when reading server to segments map (#11818)

  • Add tests for zstd compressed parquet files (#11808)

  • Fix job submission time for reload and foce commit job (#11803)

  • Remove actually unsupported config that selectively enable nullable columns (#10653)

  • Fix LLCRealtimeClusterIntegrationTest.testReset (#11806)

  • Use expected version in api for table config read modify write change (#11782)

  • Move jobId out of rebalanceConfig (#11790)

  • Fix PeerServerSegmentFinder not respecting HTTPS port (#11752)

  • Enhanced geospatial v2 integration tests (#11741)

  • Add integration test for rebalance in upsert tables (#11568)

  • Fix trivy CI issue (#11757)

  • Cleanup rebalance configs by adding a RebalanceConfig class (#11730)

  • Fix a protobuf comment to be more precise (#11735)

  • Move scala dependencies to root pom (#11671)

  • Fix ProtoBuf inputformat plug-in handling for null values (#11723)

  • Bugfix where segment download URI is invalid after same CRC refresh using tar push (#11720)

  • Fix in TableCacheTest (#11717)

  • Add more test for broker jersey bounded thread pool (#11705)

  • Fix bug in gapfill with SumAvgGapfillProcessor. (#11714)

  • Bugfix to allow GcsPinotFS to work with granular permissions (#11655)

  • Fix default log4j2 config file path in helm chart (#11707)

  • Refactor code and doc occurrences of argmin/max -> exprmin/max (#11700)

  • Make constructor and functions public to be used from scheduler plugins (#11699)

  • Bugfix to change json_format to return java null when java null is received (#11673)

  • Fix the potential access to upsert metadata manager after it is closed (#11692)

  • Bugfix to use isOptional instead of the deprecated hasOptional Keyword (#11682)

  • Fix logging issue in RealtimeTableDataManager (#11693)

  • Cleanup some reader/writer logic for raw forward index (#11669)

  • Do not execute spotless in Java 21 (#11670)

  • Update license-maven-plugin (#11665)

  • Bugfix to allow deletion of local files with special characters (#11664)

  • Clean up CaseTransformFunction::constructStatementListLegacy. (#11339)

  • Bugfix to force FileChannel to commit data to disk (#11625)

  • Remove the old deprecated commit end without metadata (#11662)

  • Fix for a jackson vulnerability (#11619)

  • Refactor BasicAuthUtils from pinot-core to pinot-common and remove pinot-core dependency from pinot-jdbc-client (#11620)

  • Bugfix to support several extensions for different indexes (#11600)

  • Fix the alias handling in single-stage engine (#11610)

  • Fix to use constant null place holder (#11615)

  • Refactor to move all BlockValSet into the same package (#11616)

  • Remove deprecated Request class from pinot-java-client (#11614)

  • Refactoring to remove old thirdeye files. (#11609)

  • Testing fix to use builder method in integration test (#11564)

  • Fix the broken Pinot JDBC client. (#11606)

  • Bugfix to change the Forbidden error to Unauthorized (#11501)

  • Fix for schema add UI issue that passing wrong data in the request header (#11602)

  • Remove/Deprecate HLC handling code (#11590)

  • Fix the bug of using push time to identify new created segment (#11599)

  • Bugfix in CSVRecordReader when using line iterator (#11581)

  • Remove split commit and some deprecated config for real-time protocol on controller (#11663)Improved validation for single argument aggregation functions (#11556)

  • Fix to not emit lag once tabledatamanager shutdown (#11534)

  • Bugfix to fail reload if derived columns can't be created (#11559)

  • Fix the double unescape of property value (#12405)

  • Fix for the backward compatible issue that existing metadata may contain unescaped characters (#12393)

  • Skip invalid json string rather than throwing error during json indexing (#12238)

  • Fixing the multiple files concurrent write issue when reloading SSLFactory (#12384)

  • Fix memory leaking issue by making thread local variable static (#12242)

  • Bugfixfor Upsert compaction task generator (#12380)

  • Log information about SSLFactory renewal (#12357)

  • Fixing array literal usage for vector (#12365)

  • Fixing quickstart table baseballStats minion ingestion (#12371)

  • Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction (#12347)

  • Bugfix to skip instead of throwing error on 'getValidDocIdMetadata' (#12360)

  • Fix to clean up segment metadata when the associated segment gets deleted from remote store (#12350)

  • Fix getBigDecimal() scale throwing rounding error (#12326)

  • Workaround fix for the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351)

  • Bugfix for making nonLeaderForTables exhaustive (#12345)

  • Bugfixes for graceful interrupt handling of mutable lucene index (#11558,#12274)

  • Remove split commit and some deprecated config for real-time protocol on controller (#11663)

  • Update the table config in quick start (#11652)

  • Deprecate k8s skaffold scripts and move helm to project root directory (#11648)

  • Fix NPE in SingleColumnKeySelector (#11644)

  • Simplify kafka build and remove old kafka 0.9 files (#11638)

  • Adding comments for docker image tags, make a hyper link of helmChart from root directory (#11646)

  • Improve the error response on controller. (#11624)

  • Simplify authrozation for table config get (#11640)

  • Bugfix to remove segments with empty download url in UpsertCompactionTask (#12320)

  • Test changes to make taskManager resources protected for derived classes to override in their setUp() method. (#12335)

Backward incompatible Changes

  • Fix a race condition for upsert compaction (#12346). Notes on backward incompatibility below:

    • This PR is introducing backward incompatibility for UpsertCompactionTask. Previously, we allowed to configure the compaction task without the snapshot enabled. We found that using in-memory based validDocIds is a bit dangerous as it will not give us the consistency (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).

      We now enforce the enableSnapshot=true for UpsertCompactionTask if the advanced customer wants to run the compaction with the in-memory validDocId bitmap.

      {
        "upsertConfig": {
          "mode": "FULL",
          "enableSnapshot": true
        }
      }
      ...
      "task": {
        "taskTypeConfigsMap": {
          "UpsertCompactionTask": {
            "schedule": "0 */5 * ? * *",
            "bufferTimePeriod": "7d",
            "invalidRecordsThresholdPercent": "30",
            "invalidRecordsThresholdCount": "100000",
            "invalidDocIdsType": "SNAPSHOT/IN_MEMORY/IN_MEMORY_WITH_DELETE"
          }
        }
      }

      Also, we allow to configure invalidDocIdsType to UpsertCompactionTask for advanced user.

      1. snapshot: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.

        1. onHeap: the validDocIds bitmap will be fetched from the server.

        2. onHeapWithDelete: the validDocIds bitmap will be fetched from the server. This will also take account into the deleted documents. UpsertConfig's deleteRecordColumn must be provided for this type.

  • Removal of the feature flag allow.table.name.with.database (#12402)

  • Error handling to throw exception when schema name doesn't match table name during table creation (#11591)

  • Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)

  • Incompatible API fix to remove table state update operation in GET call (#11621)

  • Use string to represent BigDecimal datatype in JSON response (#11716)

  • Single quoted literal will not have its type auto-derived to maintain SQL compatibility (#11763)

  • Changes to always use split commit on server and disables the option to disable it (#11680, #11687)

  • Change to not allow NaN as default value for Float and Double in Schemas (#11661)

  • Code cleanup and refactor that removes TableDataManagerConfig (#12189)

  • Fix partition handling for consistency of values between query and segment (#12115)

  • Changes for migration to commons-configuration2 (#11985)

  • Cleanup to simplify the upsert metadata manager constructor (#12120)

  • Fixes typo in pom.xml (#11997)

  • JDBC Driver fixes to support Jetbrains Intellij/Datagrip database tooling (#11814)

  • Fix regression in ForwardIndexType for noDictionaryConfig and noDictionaryColumns (#11784)

  • Separate pr test scripts and codecov (#11804)

  • Bugfix to make reload status should only count online/consuming segments (#11787)

  • Fix flaky TableViewsTest (#11770)

  • Fix a flaky test (#11771)

  • Cleanup to fee more disk for trivy job (#11780)

  • Fix schema name in table config during controller startup (#11574)

  • Prevent NPE when attempt to fetch partition information fails (#11769)

  • Added UTs for null handling in CaseTransform function. (#11721)

  • Bugfix to disallow peer download when replication is < 2 (#11469)

  • Updates to Docker image and GitHub action scripts (#12378)

  • Enhancements to queries test framework (#12215)

Library upgrades and dependencies

  • Update maven-jar-plugin and maven-enforcer-plugin version (#11637)

  • Update testng as the test provider explicitly instead of relying on the classpath. (#11612)

  • Update compatibility verifier version (#11684)

  • Upgrade Avro dependency to 1.10.2 (#11698)

  • Upgrade testng version to 7.8.0 (#11462)

  • Update lombok version and config (#11742)

  • Upgrading Apache Helix to 1.3.1 version (#11754)

  • Upgrade spark from 3.2 to 3.5 (#11702)

  • Added commons-configuration2 dependency. (#11792)

  • Upgrade confluent libraries to 7.2.6 to fix some errors related to optional proto fields (#11753)

  • Upgrade lucene to 9.8.0 and upgrade text index version (#11857)

  • Upgrade the PinotConfiguartion to commons-configuartion2(#11916)

  • Pre PinotConfig commons-configuartions2 upgrade (#11868)

  • Bump commons-codec:commons-codec from 1.15 to 1.16.0 (#12204)

  • Bump flink.version from 1.12.0 to 1.14.6 (#12202)

  • Bump com.yscope.clp:clp-ffi from 0.4.3 to 0.4.4 (#12203)

  • Bump org.apache.spark:spark-launcher_2.12 from 3.2.1 to 3.5.0 (#12199)

  • Bump io.grpc:grpc-context from 1.59.0 to 1.60.1 (#12198)

  • Bump com.azure:azure-core from 1.37.0 to 1.45.1 (#12193)

  • Bump org.freemarker:freemarker from 2.3.30 to 2.3.32 (#12192)

  • Bump com.google.auto.service:auto-service from 1.0.1 to 1.1.1 (#12183)

  • Bump dropwizard-metrics.version from 4.2.22 to 4.2.23 (#12178)

  • Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 (#12170)

  • Bump com.gradle:common-custom-user-data-maven-extension (#12171)

  • Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 (#12172)

  • Bump org.glassfish.tyrus.bundles:tyrus-standalone-client (#12162)

  • Bump com.google.api.grpc:proto-google-common-protos (#12159)

  • Bump org.apache.datasketches:datasketches-java from 4.1.0 to 5.0.0 (#12161)

  • Bump org.apache.zookeeper:zookeeper from 3.6.3 to 3.7.2 (#12152)

  • Bump org.apache.commons:commons-collections4 from 4.1 to 4.4 (#12149)

  • Bump log4j.version from 2.20.0 to 2.22.0 (#12143)

  • Bump com.github.luben:zstd-jni from 1.5.5-6 to 1.5.5-11 (#12125)

  • Bump com.google.guava:guava from 32.0.1-jre to 32.1.3-jre (#12124)

  • Bump org.apache.avro:avro from 1.10.2 to 1.11.3 (#12116)

  • Bump org.apache.maven.plugins:maven-assembly-plugin from 3.1.1 to 3.6.0 (#12109)

  • Bump net.java.dev.javacc:javacc from 7.0.10 to 7.0.13 (#12103)

  • Bump com.azure:azure-identity from 1.8.1 to 1.11.1 (#12095)

  • Bump xml-apis:xml-apis from 1.4.01 to 2.0.2 (#12082)

  • Bump up the parquet version to 1.13.1 (#12076)

  • Bump io.grpc:grpc-context from 1.14.0 to 1.59.0 (#12034)

  • Bump org.reactivestreams:reactive-streams from 1.0.3 to 1.0.4 (#12033)

  • Bump org.codehaus.mojo:appassembler-maven-plugin from 1.10 to 2.1.0 (#12030)

  • Bump com.google.code.findbugs:jsr305 from 3.0.0 to 3.0.2 (#12031)

  • Bump org.jacoco:jacoco-maven-plugin from 0.8.9 to 0.8.11 (#12024)

  • Bump dropwizard-metrics.version from 4.2.2 to 4.2.22 (#12022)

  • Bump grpc.version from 1.53.0 to 1.59.0 (#12023)

  • Bump com.google.code.gson:gson from 2.2.4 to 2.10.1 (#12009)

  • Bump net.nicoulaj.maven.plugins:checksum-maven-plugin from 1.8 to 1.11 (#12008)

  • Bump circe.version from 0.14.2 to 0.14.6 (#12006)

  • Bump com.mercateo:test-clock from 1.0.2 to 1.0.4 (#12005)

  • Bump simpleclient_common.version from 0.8.1 to 0.16.0 (#11986)

  • Bump com.jayway.jsonpath:json-path from 2.7.0 to 2.8.0 (#11987)

  • Bump commons-net:commons-net from 3.1 to 3.10.0 (#11982)

  • Bump org.scalatest:scalatest-maven-plugin from 1.0 to 2.2.0 (#11973)

  • Bump io.netty:netty-bom from 4.1.94.Final to 4.1.100.Final (#11972)

  • Bump com.google.errorprone:error_prone_annotations from 2.3.4 to 2.23.0 (#11905)

  • Bump net.minidev:json-smart from 2.4.10 to 2.5.0 (#11875)

  • Bump org.yaml:snakeyaml from 2.0 to 2.2 (#11876)

  • Bump browserify-sign in /pinot-controller/src/main/resources (#11896)

  • Bump org.easymock:easymock from 4.2 to 5.2.0 (#11854)

  • Bump org.codehaus.mojo:exec-maven-plugin from 1.5.0 to 3.1.0 (#11856)

  • Bump com.github.luben:zstd-jni from 1.5.2-3 to 1.5.5-6 (#11855)

  • Bump aws.sdk.version from 2.20.94 to 2.20.137 (#11463)

  • Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.4 (#11678)