1 of 19

Releases

The following summarizes Pinot's releases, from the latest one to the earliest one.

Note

Before upgrading from one version to another one, read the release notes. While the Pinot committers strive to keep releases backward-compatible and introduce new features in a compatible manner, your environment may have a unique combination of configurations/data/schema that may have been somehow overlooked. Before you roll out a new release of Pinot on your cluster, it is best that you run the compatibility test suite that Pinot provides. The tests can be easily customized to suit the configurations and tables in your pinot cluster(s). As a good practice, you should build your own test suite, mirroring the table configurations, schema, sample data, and queries that are used in your cluster.

1.0.0 (September 2023)

0.12.1 (March 2023)

0.12.0 (December 2022)

0.11.0 (September 2022)

0.10.0 (March 2022)

0.9.3 (December 2021)

0.9.2 (December 2021)

0.9.1 (December 2021)

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

0.12.1

Summary

This is a bug-fixing release contains:

use legacy case-when format ()

0.9.3

Summary

This is a bug fixing release contains:

Update Log4j to 2.17.0 to address CVE-2021-45105 (#7933)

The release is based on the release 0.9.2 with the following cherry-picks:

0.9.2

Summary

This is a bug fixing release contains:

Upgrade log4j to 2.16.0 to fix ()

0.9.1

Summary

This release fixes the major issue of CVE-2021-44228 and a major bug fixing of pinot admin exit code issue(#7798).

The release is based on the release 0.9.0 with the following cherry-picks:

e44d2e4 af2858a

0.7.1

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations.

Summary

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations and improvements.

It also adds several new APIs to better manage the segments and upload data to the offline table. It also contains many key bug fixes. See details below.

The release was cut from the following commit:

and the following cherry-picks:

Notable New Features

Add a server metric: queriesDisabled to check if queries disabled or not. ()
Optimization on GroupKey to save the overhead of ser/de the group keys () ()
Support validation for jsonExtractKey

Special notes

Pinot controller metrics prefix is fixed to add a missing dot (). This is a backward-incompatible change that JMX query on controller metrics must be updated
Legacy group key delimiter (\t) was removed to be backward-compatible with release 0.5.0 ()
Upgrade zookeeper version to 3.5.8 to fix ZOOKEEPER-2184: Zookeeper Client should re-resolve hosts when connection attempts fail. ()

Major Bug fixes

Fix the SIGSEGV for large index ()
Handle creation of segments with 0 rows so segment creation does not fail if data source has 0 rows. ()
Fix QueryRunner tool for multiple runs ()

0.6.0

Summary

This release introduced some excellent new features, including upsert, tiered storage, pinot-spark-connector, support of having clause, more validations on table config and schema, support of ordinals in GROUP BY and ORDER BY clause, array transform functions, adding push job type of segment metadata only mode, and some new APIs like updating instance tags, new health check endpoint. It also contains many key bug fixes. See details below.

The release was cut from the following commit: e5c9bec and the following cherry-picks:

Notable New Features

Tiered storage ()
Upsert feature (, , , , )
Pre-generate aggregation functions in QueryContext ()

Special notes

Brokers should be upgraded before servers in order to keep backward-compatible:
- Change group key delimiter from '\t' to '\0' ()
- Support for exact distinct count for non int data types ()

Major Bug fixes

Improve performance of DistinctCountThetaSketch by eliminating empty sketches and unions. ()
Enhance VarByteChunkSVForwardIndexReader to directly read from data buffer for uncompressed data ()
Fixing backward-compatible issue of schema fetch call ()

Backward Incompatible Changes

Make real-time threshold property names less ambiguous ()
Enhance DistinctCountThetaSketchAggregationFunction ()
Deep Extraction Support for ORC, Thrift, and ProtoBuf Records ()

0.5.0

This release includes many new features on Pinot ingestion and connectors, query capability and a revamped controller UI.

Summary

This release includes many new features on Pinot ingestion and connectors (e.g., support for filtering during ingestion which is configurable in table config; support for json during ingestion; proto buf input format support and a new Pinot JDBC client), query capability (e.g., a new GROOVY transform function UDF) and admin functions (a revamped Cluster Manager UI & Query Console UI). It also contains many key bug fixes. See details below.

The release was cut from the following commit: d1b4586 and the following cherry-picks:

Notable New Features

Allowing update on an existing instance config: PUT /instances/{instanceName} with Instance object as the pay-load ()
Add PinotServiceManager to start Pinot components ()
Support for protocol buffers input format. ()

Special notes

Changed the stream and metadata interface () — This PR concludes the work for the issue to extend offset support for other streams
TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release ().
Config key enable.case.insensitive.pql in Helix cluster config is deprecated, and replaced with enable.case.insensitive. (

Major Bug fixes

Fix bug in distinctCountRawHLL on SQL path ()
Fix backward incompatibility for existing stream implementations ()
Fix backward incompatibility in StreamFactoryConsumerProvider ()

Backward Incompatible Changes

PQL queries with HAVING clause will no longer be accepted for the following reasons: () — HAVING clause does not apply to PQL GROUP-BY semantic where each aggregation column is ordered individually — The current behavior can produce inaccurate results without any notice — HAVING support will be added for SQL queries in the next release
Because of the standardization of the DistinctCountThetaSketch predicate strings, upgrade Broker before Server. The new Broker can handle both standard and non-standard predicate strings for backward-compatibility. ()

0.4.0

0.4.0 release introduced the theta-sketch based distinct count function, an S3 filesystem plugin, a unified star-tree index implementation, migration from TimeFieldSpec to DateTimeFieldSpec, etc.

Summary

0.4.0 release introduced various new features, including the theta-sketch based distinct count aggregation function, an S3 filesystem plugin, a unified star-tree index implementation, deprecation of TimeFieldSpec in favor of DateTimeFieldSpec, etc. Miscellaneous refactoring, performance improvement and bug fixes were also included in this release. See details below.

Notable New Features

Made DateTimeFieldSpecs mainstream and deprecated TimeFieldSpec (#2756)
- Used time column from table config instead of schema (#5320)
- Included dateTimeFieldSpec in schema columns of Pinot Query Console #5392

Major Bug Fixes

Do not release the PinotDataBuffer when closing the index (#5400)
Handled a no-arg function in query parsing and expression tree (#5375)
Fixed compatibility issues during rolling upgrade due to unknown json fields (#5376)

Work in Progress

Upsert: support overriding data in the real-time table (#4261).
- Add pinot upsert features to pinot common (#5175)
Enhancements for theta-sketch, e.g. multiValue aggregation support, complex predicates, performance tuning, etc

Backward Incompatible Changes

TableConfig no longer support de-serialization from json string of nested json string (i.e. no \" inside the json) (#5194)
The following APIs are changed in AggregationFunction (use TransformExpressionTree instead of String as the key of blockValSetMap) (#5371):

0.2.0

The 0.2.0 release is the first release after the initial one and includes several improvements, reported following.

New Features and Bug Fixes

Added support for Kafka 2.0
Table rebalancer now supports a minimum number of serving replicas during rebalance
Added support for UDF in filter predicates and selection
Added support to use hex string as the representation of byte array for queries (see PR )
Added support for parquet reader (see PR )
Introduced interface stability and audience annotations (see PR )
Refactor HelixBrokerStarter to separate constructor and start() - backwards incompatible (see PR )
Admin tool for listing segments with invalid intervals for offline tables
Migrated to log4j2 (see PR )
Added simple avro msg decoder
Added support for passing headers in Pinot client
Table rebalancer now supports a minimum number of serving replicas during rebalance
Support transform functions with AVG aggregation function (see PR )
Configurations additions/changes
- Allow customized metrics prefix (see PR )
- Controller.enable.batch.message.mode to false by default (see PR )

Work in Progress

We are in the process of separating Helix and Pinot controllers, so that administrators can have the option of running independent Helix controllers and Pinot controllers.
We are in the process of moving towards supporting SQL query format and results.
We are in the process of separating instance and segment assignment using instance pools to optimize the number of Helix state transitions in Pinot clusters with thousands of tables.

Other Notes

Task management does not work correctly in this release, due to bugs in Helix. We will upgrade to Helix 0.9.2 (or later) version to get this fixed.
You must upgrade to this release before moving onto newer versions of Pinot release. The protocol between Pinot-broker and Pinot-server has been changed and this release has the code to retain compatibility moving forward. Skipping this release may (depending on your environment) cause query errors if brokers are upgraded and servers are in the process of being upgraded.
As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

0.1.0

The 0.1.0 is first release of Pinot as an Apache project

New Features

First release
Off-line data ingestion from Apache Hadoop
Real-time data ingestion from Apache Kafka

0.3.0

0.3.0 release of Apache Pinot introduces the concept of plugins that makes it easy to extend and integrate with other systems.

What's the big change?

The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.

For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?

In order to conquer this challenge, below major changes are made:

Refactored common interfaces to pinot-spi module
Concluded four types of modules:
- Pinot input format: How to read records from various data/file formats: e.g. Avro

Now the architecture supports a plug-and-play fashion, where new tools can be supported with little and simple extensions, without affecting big chunks of code. Integrations with new streaming services and data formats can be developed in a much more simple and convenient way.

Notable New Features

SQL Support
- Added Calcite SQL compiler
- Added SQL response format (, )

Major Bug Fixes

Fixed the bug of releasing the segment when there are still threads working on it. ()
Fixed the bug of uneven task distribution for threads ()
Fixed encryption for .tar.gz segment file upload ()

Work in Progress

We are in the process of supporting text search query functionalities.
We are in the process of supporting null value (), currently limited query feature is supported
- Added Presence Vector to represent null value ()

Backward Incompatible Changes

It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Ensure that you upgrade to release 0.2.0 first, then upgrade to this version.
If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. See pinot-admin.sh as an example.

Extract time handling for SegmentProcessorFramework (#7158)
Add Apache Pulsar low level and high level connector (#7026)
Enable parallel builds for compat checker (#7149)
Add controller/server API to fetch aggregated segment metadata ()
Support Dictionary Based Plan For DISTINCT ()
Provide HTTP client to kinesis builder ()
Add datetime function with 2 arguments ()
Adding ability to check ingestion status for Offline Pinot table ()
Add timestamp datatype support in JDBC ()
Allow updating controller and broker helix hostname ()
Cancel running Kinesis consumer tasks when timeout occurs ()
Implement Append merger for partial upsert ()
`* SegmentProcessorFramework Enhancement ()
Added TaskMetricsEmitted periodic controler job ()
Support json path expressions in query. ()
Support data preprocessing for AVRO and ORC formats ()
Add partial upsert config and mergers ()
Add support for range index rule recommendation(#7034) ()
Allow reloading consuming segment by default ()
Add LZ4 Compression Codec (#6804) ([#7035](
))
Make Pinot JDK 11 Compilable (\
Introduce in-Segment Trim for GroupBy OrderBy Query ()
Produce GenericRow file in segment processing mapper ()
Add ago() scalar transform function ()
Add Bloom Filter support for IN predicate(#7005) ()
Add genericRow file reader and writer ()
Normalize LHS and RHS numerical types for >, >=, <, and <= operators. ()
Add Kinesis Stream Ingestion Plugin ()
feature/#6766 JSON and Startree index information in API ()
Support null value fields in generic row ser/de ()
Implement PassThroughTransformOperator to optimize select queries(#6972) ()
Optimize TIME_CONVERT/DATE_TIME_CONVERT predicates ()
Prefetch call to fetch buffers of columns seen in the query ()
Enabling compatibility tests in the script ()
Add collectionToJsonMode to schema inference ()
Add the complex-type support to decoder/reader ()
Adding a new Controller API to retrieve ingestion status for real-time… ()
Add support for Long in Modulo partition function. ()
Enhance PinotSegmentRecordReader to preserve null values ()
add complex-type support to avro-to-pinot schema inference ()
Add correct yaml files for real-time data(#6787) ()
Add complex-type transformation to offline segment creation ()
Add config File support(#6787) ()
Enhance JSON index to support nested array ()
Add debug endpoint for tables. ()
JSON column datatype support. ()
Allow empty string in MV column ()
Add Zstandard compression support with JMH benchmarking(#6804) ()
Normalize LHS and RHS numerical types for = and != operator. ()
Change ConcatCollector implementation to use off-heap ()
[PQL Deprecation] Clean up the old BrokerRequestOptimizer ()
[PQL Deprecation] Do not compile PQL broker request for SQL query ()
Add TIMESTAMP and BOOLEAN data type support ()
Add admin endpoint for Pinot Minon. ()
Remove the usage of PQL compiler ()
Add endpoints in Pinot Controller, Broker and Server to get system and application configs. ()
Support IN predicate in ColumnValue SegmentPruner(#6756) ()
Enable adding new segments to a upsert-enabled real-time table ()
Interface changes for Kinesis connector ()
Pinot Minion SegmentGenerationAndPush task: PinotFS configs inside taskSpec is always temporary and has higher priority than default PinotFS created by the minion server configs ()
DataTable V3 implementation and measure data table serialization cost on server ()
add uploadLLCSegment endpoint in TableResource ()
File-based SegmentWriter implementation ()
Basic Auth for pinot-controller ()
UI integration with Authentication API and added login page ()
Support data ingestion for offline segment in one pass ()
SumPrecision: support all data types and star-tree ()
complete compatibility regression testing ()
Kinesis implementation Part 1: Rename partitionId to partitionGroupId ()
Make Pinot metrics pluggable ()
Recover the segment from controller when LLC table cannot load it ()
Adding a new API for validating specified TableConfig and Schema ()
Introduce a metric for query/response size on broker. ()
Adding a controller periodic task to clean up dead minion instances ()
Adding new validation for Json, TEXT indexing ()
Always return a response from query execution. ()

0.9.0

Summary

This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.

The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9

Support Segment Merge and Roll-up

LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.

To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.

At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.

Major Changes:

Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor ()
Merge/Rollup task scheduler for offline tables. ()
Fix MergeRollupTask uploading segments not updating their metadata ()

UI Improvement

This release also sees improvements to Pinot’s query console UI.

Cmd+Enter shortcut to run query in query console ()
Showing tooltip in SQL Editor ()
Make the SQL Editor box expandable ()

SQL Improvements

There have also been improvements and additions to Pinot’s SQL implementation.

New functions:

IN ()
LASTWITHTIME ()
ID_SET on MV columns ()

New predicates are supported:

LIKE()
REGEXP_EXTRACT()
FILTER()

Query compatibility improvements:

Infer data type for Literal ()
Support logical identifier in predicate ()
Support JSON queries with top-level array path expression. ()

Performance Improvements

This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:

Reduce the disk usage for segment conversion task ()
Simplify association between Java Class and PinotDataType for faster mapping ()
Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation ()

Other Notable New Features and Changes

Human Readable Controller Configs ()
Add the support of geoToH3 function ()
Add Apache Pulsar as Pinot Plugin () ()

Major Bug fixes

Fix null pointer exception for non-existed metric columns in schema for JDBC driver ()
Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD ()
Fixed pinot java client to add zkClient close ()

Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support
concat as || operator (#12025)
Capability for constant filter in QueryContext, with support for server to handle it (#11956)
Tests for filter pushdown ()
Enhancements to query plan tests ()
Refactor PlanFragmenter to make the logic clear ()
Observability enhancements to emit metrics for grpc request and multi-stage leaf stage ()
- pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
- pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)
Security enhancement to add RBAC authorization checks for multi-stage query engine ()
Enhancement to leaf-stage execution stats NPE handling ()
Enhancement to add a framework to back-propagate metadata across opChains ()
Use of BinaryArray to wire proto for multi-stage engine bytes literal handling ()
Enable dynamic broadcast for SEMI joins. Adds a fallback option to enable hash table join using joinOptions(join_strategy = 'hash_table')()
Improvements to dispatch exception handling ()
Allow malformed dateTime string to return default value configurable in the function signature ()
Improvement in multi-stage aggregation to directly store column index as identifier ()
Perf optimization to avoid unnecessary rows conversion in aggregation ()
Enhance SegmentPartitionMetadataManager to handle new segment ()
Optimize mailbox info in query plan to reduce memory footprint ()
- This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version.
Optimizations to query plan serialization ()
Optimization for parallel execution of Ser/de stage plan ()
Optimizations in query dispatch ()
Perf optimization for group-by and join for single key scenario ()

Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap (#12223)
Wire soft upsert delete for Compaction task (12330)
Upsert compaction debuggability APIs for validDocId metadata (#12275)
Make server resource classes configurable ()
Shared aggregations for Startree index - mapping from aggregation used in the query to aggregation used to store pre-aggregated values ()
Increased fetch timeout for Kineses to prevent stuck kinesis consumers
Metric to track table rebalance ()
Allow server-level configs for upsert metadata ()
Support to dynamically initialize Kafka client SSL configs ()
Optimize segment metadata file creation without having to download full segment ()
Allow string / numeric data type for deleteRecordColumn config ()
Atomic and Deterministic snapshots of validDocId for upsert tables (, )
Observability enhancement to add column name when JSON index building fails ()
Creation of DateTimeGenerator for DATE_TIME field type columns ()
Add singleton registry for all controller and minion metrics ()
Support helm chart server separate liveness and readiness probe endpoints ()
Observability enhancement to add metrics for Table Disabled and Consumption Paused ()
Support for SegmentGenerationAndPushTask to push segment to realtime table ()
Enhancement to make the deep store upload retry async with configurable parallelism ()
Optimizations in segment commit to not read partition group metadata ()
Replace timer with scheduled executor service in IngestionDelayTracker to reduce number of threads ()
Adds an option skipControllerCertValidation to skip controller cert validation in AddTableCommand ()
Adds instrumentation for DataTable Creation ()
Improve performance of ZkBasicAuthAccessFactory by caching Bcrypt password ()
Adds support to to fetch metadata for specific list of segments ()
Allow user specify local temp directory for quickstart ()
Optimization for server to directly return final result for queries hitting single server ()
Explain plan optimization to early release AcquireReleaseColumnsSegmentOperator ()
Observability metric to track query timeouts ()
Add support for auth in QueryRunner ()
Allow users to pass custom RecordTransformers to SegmentProcessorFramework ()
Add isPartialResult flag to broker response ()
Add new configs to Google Cloud Storage (GCS) connector: jsonKey ()
- jsonKey is the GCP credential key in string format (either in plain string or base64 encoded string). Refer to download the keys.
Performance enhancement to build segments in column orientation ()
- Disabled by default. Can be enabled by setting table config columnMajorSegmentBuilderEnabled
Observability enhancements to emit metrics for grpc request and multi-stage leaf stage ()
- pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)
- pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)
Observability improvement to expose GRPC metrics ()
Improvements to response format for reload API to be pretty printed ()
Enhancements to support Java 21 ()
Add more information in RequestContext class ()
Support to read exact buffer byte ranges corresponding to a given forward index doc id ()
Enhance Broker reducer to handle expression format change ()
Capture build scans on ge.apache.org to benefit from deep build insights ()
Performance enhancement in multiple places by updating initial capacity of HashMap ()
Support for building indexes post segment file creation, allowing indexes that may depend on a completed segment to be built as part of the segment creation process ()
Support excluding time values in SimpleSegmentNameGenerator ()
Perf enhancement to reduce cpu usage by avoiding throwing an exception during query execution ()
Added framework for supporting nulls in ScalarTransformFunctionWrapper in the future ()
Observability change to metrics to export netty direct memory used and max ()
Observability change to add a metric to measure total thread cpu time for a table ()
Observability change to use SlidingTimeWindowArrayReservoirin dropwizard metrics ()
Minor improvements to upsert preload ()
Observability changes to expose additional Realtime Ingestion Metrics ()
Perf enhancement to remove the global lock in SegmentCompletionManager ()
Enhancements to unify tmp file naming format and delete tmp files at a regular cadence by extending the ControllerPeriodicTask ()
- controller.realtime.segment.tmpFileAsyncDeletionEnabled (default false)
- controller.realtime.segment.tmpFileRetentionInSeconds
Improvements to skip unparseable records in the csv record reader (, )
Enhancements to allow override/force options when add schema ()
Enhancement to handle direct memory OOM on brokers ()
Enhancement to metadata API to return upsert partition to primary key count map for both controller and server APIs ()
Enhancements to peer server segment download by retrying both peer discovery and download. ()
Helper functions in StarTreeBuilderUtils and StarTreeV2BuilderConfig ()
Perf optimizations to release all segments of a table in releaseAndRemoveAllSegments method ()
Enhancement to Maintain pool selection for the minimizeDataMovement instance partition assignment strategy ()
Upsert enhancement to assign segments for with respect to ideal state ()
Observability change to export Additional Upsert Metrics to Prom ()
Observibility enhancement to add CPU metrics for minion purge task ()
Add HttpHeaders in broker event listener requestContext ()

Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader (#12329)
Fix error message format for Preconditions.checks failures(#12327)
Bugfix to distribute Pinot as a multi-release JAR (#12131, #12300)
Fixes in upsert metadata manager ()
Security fix to allow querying tables with table-type suffix ()
Bugfix to ensure tagConfigOverride config is null for upsert tables ( and )
Increased fetch timeout for Kineses to prevent stuck kinesis consumers()
Fixes to catch-all Regex for JXM -> Prom Exporter ( and )
Fixes lucene index errors when using QuickStart ()
Null handling bugfix for sketch group-by queries ()
Null pointer exception fixes in Controller SQL resource ()
Synchronization fixes to replace upsert segments ( and )
Bugfix for S3 connection pool error when AWS session tokens expire after an hour ()
FileWriter fixes to append headerline only for required formats like csv ()
Security bugfix for pulsar OAuth2 authentication ()
Bugfix to appropriately compute "segment.flush.threshold.size" when force-committing realtime segments ()
Fixes rebalance converge check that reports success before rebalance completes ()
Fixes upsertPrimaryKeysCount metric reporting when table is deleted ()
Update LICENSE-binary for commons-configuration2 upgrade ()
Improve error logging when preloading segments not exist on server ()
Fixes to file access resource leaks ()
Ingestion bugfix to avoid unnecessary transformers in CompositeTransformer ()
Improve logging to print OS name during service statup ()
Improve logging in multiple files (, , ,)
Test fixes for ExprMinMaxRewriterTest.testQueryRewrite ()
Fixes default path of log4j in helmchart (, )
Fix default brokerUpdateFrequencyInMillis for connector ()
Updates to README file ()
Fix to remove unnecessary locking during segment preloading ()
Fix bug with silently ignoring force commit call failures ()
Upsert bugfix to allow optional segments that can be skipped by servers without failing the query ()
Fix incorrect handling of consumer creation errors ()
Fix the memory leak issue on CommonsConfigurationUtils ()
Fix rebalance on upsert table ()
Add new Transformer to transform -0.0 and NaN ()
Improve inverted index validation in table config to enhance user experience ()
Fixes test flakiness by replacing HashSet/HashMap with LinkedHashSet/LinkedHashMap ()
Flaky test fix for ServerRoutingStatsManagerTest.testQuerySubmitAndCompletionStats ()
Fix derived column from MV column ()
Support for leveraging StarTree index in conjunction with filtered aggregations ()
Improves tableConfig validation for enabling size based threshold for realtime tables ()
Fix flaky PinotTenantRestletResourceTest ()
Fix flaky Fix PinotTenantRestletResourceTest ()
Fix the race condition of concurrent modification to segment data managers ()
Fix the misuse of star-tree when all predicates are always false under OR ()
Fix the test failures caused by instance drop failure ()
Fix fromULL scalar function ()
Fix to exclude module-info.class during shade operations ()
Fix the wrong import for Preconditions ()
Add check for illegal character '/' in taskName ()
Bugfix to only register new segments when it's fully initalized by partitionUpsertMetadataManager ()
Obervability fix to add logs to track sequence of events for table creation ()
Fix the NPE in minimizeDataMovement instance assignment strategy ()
Fix to add catch all logging for exception during DQL/DML process ()
Fix bug where we don't handle cases that a upsert table has both upsert deletion and upsert ttl configs ()
Removing direct dependencies on commons-logging and replacing with jcl-over-slf4j ()
Fix NPE for IN clause on constant STRING dictionary ()
Fix flaky OfflineClusterIntegrationTest on server response size tests ()
Avoid npe when checking mirror server set assignment ()
Deprecate _segmentAssignmentStrategy in favor of SegmentsValidationAndRetentionConfig
Bugfix to capture auth phase timing even if access is denied ()
Bugfix to mark rows as invalid in case primary time column is out of range ()
Fix to radomize server port to avoid port already bind issue ()
Add LazyRow abstraction for previously indexed record ()
Config Validation for upsert table to not assign COMPLETED segments to another server ()
Bugfix to resolve dependency conflict in pinot-protobuf module ()
Fix case of useMultistageEngine property reference in JsonAsyncHttpPinotClientTransportFactory ()
Bugfix to add woodstox-core to pinot-s3 dependencies and fix stack trace ()
Fix to move pinot-segment-local test from unit test suite 1 to 2 ()
Observability fix to log upsert config when initializing the metadata manager ()
Fix to improve tests when errors are received in the consumer thread ()
Fix for flaky ArrayAgg test ()
Fix for flaky tests in TupleSelectionTransformFunctionsTest ()
Fix for arrayAgg null support ()
Fix the bug of reading decimal value stored in int32 or int64 ()
Remove duplicate pinot-integration-tests from unit test suite 2 ()
Fix for a null handling error in queries ()
Fix the way of fetching the segment zk metadata for task generators ()
Make testInvalidateCachedControllerLeader times based on getMinInvalidateIntervalMs ()
Update doap to reflect latest release ()
Clean up integration test pom file ()
Bugfix to exclude OFFLINE segments when reading server to segments map ()
Add tests for zstd compressed parquet files ()
Fix job submission time for reload and foce commit job ()
Remove actually unsupported config that selectively enable nullable columns ()
Fix LLCRealtimeClusterIntegrationTest.testReset ()
Use expected version in api for table config read modify write change ()
Move jobId out of rebalanceConfig ()
Fix PeerServerSegmentFinder not respecting HTTPS port ()
Enhanced geospatial v2 integration tests ()
Add integration test for rebalance in upsert tables ()
Fix trivy CI issue ()
Cleanup rebalance configs by adding a RebalanceConfig class ()
Fix a protobuf comment to be more precise ()
Move scala dependencies to root pom ()
Fix ProtoBuf inputformat plug-in handling for null values ()
Bugfix where segment download URI is invalid after same CRC refresh using tar push ()
Fix in TableCacheTest ()
Add more test for broker jersey bounded thread pool ()
Fix bug in gapfill with SumAvgGapfillProcessor. ()
Bugfix to allow GcsPinotFS to work with granular permissions ()
Fix default log4j2 config file path in helm chart ()
Refactor code and doc occurrences of argmin/max -> exprmin/max ()
Make constructor and functions public to be used from scheduler plugins ()
Bugfix to change json_format to return java null when java null is received ()
Fix the potential access to upsert metadata manager after it is closed ()
Bugfix to use isOptional instead of the deprecated hasOptional Keyword ()
Fix logging issue in RealtimeTableDataManager ()
Cleanup some reader/writer logic for raw forward index ()
Do not execute spotless in Java 21 ()
Update license-maven-plugin ()
Bugfix to allow deletion of local files with special characters ()
Clean up CaseTransformFunction::constructStatementListLegacy. ()
Bugfix to force FileChannel to commit data to disk ()
Remove the old deprecated commit end without metadata ()
Fix for a jackson vulnerability ()
Refactor BasicAuthUtils from pinot-core to pinot-common and remove pinot-core dependency from pinot-jdbc-client ()
Bugfix to support several extensions for different indexes ()
Fix the alias handling in single-stage engine ()
Fix to use constant null place holder ()
Refactor to move all BlockValSet into the same package ()
Remove deprecated Request class from pinot-java-client ()
Refactoring to remove old thirdeye files. ()
Testing fix to use builder method in integration test ()
Fix the broken Pinot JDBC client. ()
Bugfix to change the Forbidden error to Unauthorized ()
Fix for schema add UI issue that passing wrong data in the request header ()
Remove/Deprecate HLC handling code ()
Fix the bug of using push time to identify new created segment ()
Bugfix in CSVRecordReader when using line iterator ()
Remove split commit and some deprecated config for real-time protocol on controller (#11663)Improved validation for single argument aggregation functions ()
Fix to not emit lag once tabledatamanager shutdown ()
Bugfix to fail reload if derived columns can't be created ()
Fix the double unescape of property value ()
Fix for the backward compatible issue that existing metadata may contain unescaped characters ()
Skip invalid json string rather than throwing error during json indexing ()
Fixing the multiple files concurrent write issue when reloading SSLFactory ()
Fix memory leaking issue by making thread local variable static ()
Bugfixfor Upsert compaction task generator ()
Log information about SSLFactory renewal ()
Fixing array literal usage for vector ()
Fixing quickstart table baseballStats minion ingestion ()
Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction ()
Bugfix to skip instead of throwing error on 'getValidDocIdMetadata' ()
Fix to clean up segment metadata when the associated segment gets deleted from remote store ()
Fix getBigDecimal() scale throwing rounding error ()
Workaround fix for the problem of Helix sending 2 transitions for CONSUMING -> DROPPED ()
Bugfix for making nonLeaderForTables exhaustive ()
Bugfixes for graceful interrupt handling of mutable lucene index (,)
Remove split commit and some deprecated config for real-time protocol on controller ()
Update the table config in quick start ()
Deprecate k8s skaffold scripts and move helm to project root directory ()
Fix NPE in SingleColumnKeySelector ()
Simplify kafka build and remove old kafka 0.9 files ()
Adding comments for docker image tags, make a hyper link of helmChart from root directory ()
Improve the error response on controller. ()
Simplify authrozation for table config get ()
Bugfix to remove segments with empty download url in UpsertCompactionTask ()
Test changes to make taskManager resources protected for derived classes to override in their setUp() method. ()

Fix a race condition for upsert compaction (#12346). Notes on backward incompatibility below:
- This PR is introducing backward incompatibility for UpsertCompactionTask. Previously, we allowed to configure the compaction task without the snapshot enabled. We found that using in-memory based validDocIds is a bit dangerous as it will not give us the consistency (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).
  We now enforce the enableSnapshot=true for UpsertCompactionTask if the advanced customer wants to run the compaction with the in-memory validDocId bitmap.
  Also, we allow to configure invalidDocIdsType to UpsertCompactionTask for advanced user.
  1. snapshot: Default validDocIds type. This indicates that the validDocIds bitmap is loaded from the snapshot from the Pinot segment. UpsertConfig's enableSnapshot must be enabled for this type.
    onHeap: the validDocIds bitmap will be fetched from the server.
Removal of the feature flag allow.table.name.with.database ()
Error handling to throw exception when schema name doesn't match table name during table creation ()
Fix type cast issue with dateTimeConvert scalar function (, )
Incompatible API fix to remove table state update operation in GET call ()
Use string to represent BigDecimal datatype in JSON response ()
Single quoted literal will not have its type auto-derived to maintain SQL compatibility ()
Changes to always use split commit on server and disables the option to disable it (, )
Change to not allow NaN as default value for Float and Double in Schemas ()
Code cleanup and refactor that removes TableDataManagerConfig ()
Fix partition handling for consistency of values between query and segment ()
Changes for migration to commons-configuration2 ()
Cleanup to simplify the upsert metadata manager constructor ()
Fixes typo in pom.xml ()
JDBC Driver fixes to support Jetbrains Intellij/Datagrip database tooling ()
Fix regression in ForwardIndexType for noDictionaryConfig and noDictionaryColumns ()
Separate pr test scripts and codecov ()
Bugfix to make reload status should only count online/consuming segments ()
Fix flaky TableViewsTest ()
Fix a flaky test ()
Cleanup to fee more disk for trivy job ()
Fix schema name in table config during controller startup ()
Prevent NPE when attempt to fetch partition information fails ()
Added UTs for null handling in CaseTransform function. ()
Bugfix to disallow peer download when replication is < 2 ()
Update s todocker image and github action scripts ()
Enhancements to queries test framework ()

update maven-jar-plugin and maven-enforcer-plugin version (#11637)
Update testng as the test provider explicitly instead of relying on the classpath. (#11612)
Update compatibility verifier version (#11684)
Upgrade Avro dependency to 1.10.2 ()
Upgrade testng version to 7.8.0 ()
Update lombok version and config ()
Upgrading Apache Helix to 1.3.1 version ()
Upgrade spark from 3.2 to 3.5 ()
Added commons-configuration2 dependency. ()
Upgrade confluent libraries to 7.2.6 to fix some errors related to optional proto fields ()
Upgrade lucene to 9.8.0 and upgrade text index version ()
Upgrade the PinotConfiguartion to commons-configuartion2()
Pre PinotConfig commons-configuartions2 upgrade ()
Bump commons-codec:commons-codec from 1.15 to 1.16.0 ()
Bump flink.version from 1.12.0 to 1.14.6 ()
Bump com.yscope.clp:clp-ffi from 0.4.3 to 0.4.4 ()
Bump org.apache.spark:spark-launcher_2.12 from 3.2.1 to 3.5.0 ()
Bump io.grpc:grpc-context from 1.59.0 to 1.60.1 ()
Bump com.azure:azure-core from 1.37.0 to 1.45.1 ()
Bump org.freemarker:freemarker from 2.3.30 to 2.3.32 ()
Bump com.google.auto.service:auto-service from 1.0.1 to 1.1.1 ()
Bump dropwizard-metrics.version from 4.2.22 to 4.2.23 ()
Bump org.apache.yetus:audience-annotations from 0.13.0 to 0.15.0 ()
Bump com.gradle:common-custom-user-data-maven-extension ()
Bump org.apache.httpcomponents:httpclient from 4.5.13 to 4.5.14 ()
Bump org.glassfish.tyrus.bundles:tyrus-standalone-client ()
Bump com.google.api.grpc:proto-google-common-protos ()
Bump org.apache.datasketches:datasketches-java from 4.1.0 to 5.0.0 ()
Bump org.apache.zookeeper:zookeeper from 3.6.3 to 3.7.2 ()
Bump org.apache.commons:commons-collections4 from 4.1 to 4.4 ()
Bump log4j.version from 2.20.0 to 2.22.0 ()
Bump com.github.luben:zstd-jni from 1.5.5-6 to 1.5.5-11 ()
Bump com.google.guava:guava from 32.0.1-jre to 32.1.3-jre ()
Bump org.apache.avro:avro from 1.10.2 to 1.11.3 ()
Bump org.apache.maven.plugins:maven-assembly-plugin from 3.1.1 to 3.6.0 ()
Bump net.java.dev.javacc:javacc from 7.0.10 to 7.0.13 ()
Bump com.azure:azure-identity from 1.8.1 to 1.11.1 ()
Bump xml-apis:xml-apis from 1.4.01 to 2.0.2 ()
Bump up the parquet version to 1.13.1 ()
Bump io.grpc:grpc-context from 1.14.0 to 1.59.0 ()
Bump org.reactivestreams:reactive-streams from 1.0.3 to 1.0.4 ()
Bump org.codehaus.mojo:appassembler-maven-plugin from 1.10 to 2.1.0 ()
Bump com.google.code.findbugs:jsr305 from 3.0.0 to 3.0.2 ()
Bump org.jacoco:jacoco-maven-plugin from 0.8.9 to 0.8.11 ()
Bump dropwizard-metrics.version from 4.2.2 to 4.2.22 ()
Bump grpc.version from 1.53.0 to 1.59.0 ()
Bump com.google.code.gson:gson from 2.2.4 to 2.10.1 ()
Bump net.nicoulaj.maven.plugins:checksum-maven-plugin from 1.8 to 1.11 ()
Bump circe.version from 0.14.2 to 0.14.6 ()
Bump com.mercateo:test-clock from 1.0.2 to 1.0.4 ()
Bump simpleclient_common.version from 0.8.1 to 0.16.0 ()
Bump com.jayway.jsonpath:json-path from 2.7.0 to 2.8.0 ()
Bump commons-net:commons-net from 3.1 to 3.10.0 ()
Bump org.scalatest:scalatest-maven-plugin from 1.0 to 2.2.0 ()
Bump io.netty:netty-bom from 4.1.94.Final to 4.1.100.Final ()
Bump com.google.errorprone:error_prone_annotations from 2.3.4 to 2.23.0 ()
Bump net.minidev:json-smart from 2.4.10 to 2.5.0 ()
Bump org.yaml:snakeyaml from 2.0 to 2.2 ()
Bump browserify-sign in /pinot-controller/src/main/resources ()
Bump org.easymock:easymock from 4.2 to 5.2.0 ()
Bump org.codehaus.mojo:exec-maven-plugin from 1.5.0 to 3.1.0 ()
Bump com.github.luben:zstd-jni from 1.5.2-3 to 1.5.5-6 ()
Bump aws.sdk.version from 2.20.94 to 2.20.137 ()
Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.4 ()

0.11.0

Summary

Apache Pinot 0.11.0 has introduced many new features to extend the query abilities, e.g. the Multi-Stage query engine enables Pinot to do distributed joins, more sql syntax(DML support), query functions and indexes(Text index, Timestamp index) supported for new use cases. And as always, more integrations with other systems(E.g. Spark3, Flink).

Note: there is a major upgrade for Apache Helix to 1.0.4, so make sure you upgrade the system in the order of:

Helix Controller -> Pinot Controller -> Pinot Broker -> Pinot server

Multi-Stage Query Engine

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as JOIN, OVER window, MATCH_RECOGNIZE and eventually, make Pinot support closer to full ANSI SQL semantics. More to read:

Pause Stream Consumption on Apache Pinot

Pinot operators can pause real-time consumption of events while queries are being executed, and then resume consumption when ready to do so again.\

Gap-filling function

The gapfilling functions allow users to interpolate data and perform powerful aggregations and data processing over time series data. More to read:

Add support for Spark 3.x ()

Long waiting feature for segment generation on Spark 3.x.

Add Flink Pinot connector ()

Similar to the Spark Pinot connector, this allows Flink users to dump data from the Flink application to Pinot.

Show running queries and cancel query by id ()

This feature allows better fine-grained control on pinot queries.

Timestamp Index ()

This allows users to have better query performance on the timestamp column for lower granularity. See:

Native Text Indices ()

Wanna search text in real time? The new text indexing engine in Pinot supports the following capabilities:

New operator: LIKE

New operator: CONTAINS

Native text index, built from the ground up, focusing on Pinot’s time series use cases and utilizing existing Pinot indices and structures(inverted index, bitmap storage).
Real Time Text Index

Adding DML definition and parse SQL InsertFile ()

Now you can use INSERT INTO [database.]table FROM FILE dataDirURI OPTION ( k=v ) [, OPTION (k=v)]* to load data into Pinot from a file using Minion. See:

Deduplication ()

This feature supports enabling deduplication for real-time tables, via a top-level table config. At a high level, primaryKey (as defined in the table schema) hashes are stored into in-memory data structures, and each incoming row is validated against it. Duplicate rows are dropped.

The expectation while using this feature is for the stream to be partitioned by the primary key, strictReplicaGroup routing to be enabled, and the configured stream consumer type to be low level. These requirements are therefore mandated via table config API's input validations.

Functions support and changes:

Add support for functions arrayConcatLong, arrayConcatFloat, arrayConcatDouble ()
Add support for regexpReplace scalar function ()
Add support for Base64 Encode/Decode Scalar Functions ()

The full list of features introduced in this release

add query cancel APIs on controller backed by those on brokers ()
Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. ()
Adding endpoint to download local log files for each component ()

Vulnerability fixs

Pinot has resolved all the high-level vulnerabilities issues:

Add a new workflow to check vulnerabilities using trivy ()
Disable Groovy function by default ()
Upgrade netty due to security vulnerability ()

Bug fixs

Nested arrays and map not handled correctly for complex types ()
Fix empty data block not returning schema ()
Allow mvn build with development webpack; fix instances default value ()

Adding NoopPinotMetricFactory and corresponding changes (#8270)
Allow to specify fixed segment name for SegmentProcessorFramework (#8269)
Move all prestodb dependencies into a separated module (#8266)
Include docIds in Projection and Transform block
Automatically update broker resource on broker changes
Update ScalarFunction annotation from name to names to support function alias.
Implemented BoundedColumnValue partition function
Add copy recursive API to pinotFS
Add Support for Getting Live Brokers for a Table (without type suffix)
Pinot docker image - cache prometheus rules
In BrokerRequestToQueryContextConverter, remove unused filterExpressionContext
Adding retention period to segment delete REST API
Pinot docker image - upgrade prometheus and scope rulesets to components
Allow segment name postfix for SegmentProcessorFramework
Superset docker image - update pinotdb version in superset image
Add retention period to deleted segment files and allow table level overrides
Remove incubator from pinot and superset
Adding table config overrides for disabling groovy
Optimise sorted docId iteration order in mutable segments
Adding secure grpc query server support
Move Tls configs and utils from pinot-core to pinot-common
Reduce allocation rate in LookupTransformFunction
Allow subclass to customize what happens pre/post segment uploading
Enable controller service auto-discovery in Jersey framework
Add support for pushFileNamePattern in pushJobSpec
Add additionalMatchLabels to helm chart
Simulate rsvps after meetup.com retired the feed
Adding more checkstyle rules
Add persistence.extraVolumeMounts and persistence.extraVolumes to Kubernetes statefulsets
Adding scala profile for kafka 2.x build and remove root pom scala dependencies
Allow real-time data providers to accept non-kafka producers
Enhance revertReplaceSegments api
Adding broker level config for disabling Pinot queries with Groovy
Make presto driver query pinot server with SQL
Adding controller config for disabling Groovy in ingestionConfig
Adding main method for LaunchDataIngestionJobCommand for spark-submit command
Add auth token for segment replace rest APIs
Add allowRefresh option to UploadSegment
Add Ingress to Broker and Controller helm charts
Improve progress reporter in SegmentCreationMapper
St_* function error messages + support literal transform functions
Add schema and segment crc to SegmentDirectoryContext
Extend enableParallePushProtection support in UploadSegment API
Support BOOLEAN type in Config Recommendation Engine
Add a broker metric to distinguish exception happens when acquire channel lock or when send request to server
Add pinot.minion prefix on minion configs for consistency
Enable broker service auto-discovery in Jersey framework
Timeout if waiting server channel lock takes a long time
Wire EmptySegmentPruner to routing config
Support for TIMESTAMP data type in Config Recommendation Engine
Listener TLS customization
Add consumption rate limiter for LLConsumer
Implement Real Time Mutable FST
Allow quickstart to get table files from filesystem
Add support for instant segment deletion
Add a config file to override quickstart configs
Add pinot server grpc metadata acl
Move compatibility verifier to a separate module
Move hadoop and spark ingestion libs from plugins directory to external-plugins
Add global strategy for partial upsert
Upgrade kafka to 2.8.1
Created EmptyQuickstart command
Allow SegmentPushUtil to push real-time segment
Add ignoreMerger for partial upsert
Make task timeout and concurrency configurable
Return 503 response from health check on shut down
Pinot-druid-benchmark: set the multiValueDelimiterEnabled to false when importing TPC-H data
Cleanup: Remove remaining occurrences of incubator.
Refactor segment loading logic in BaseTableDataManager to decouple it with local segment directory
Improving segment replacement/revert protocol
PinotConfigProvider interface
Enhance listSegments API to exclude the provided segments from the output
Remove outdated broker metric definitions
Add skip key for realtimeToOffline job validation
Upgrade async-http-client
Allow Reloading Segments with Multiple Threads
Ignore query options in commented out queries
Remove TableConfigCache which does not listen on ZK changes
Switch to zookeeper of helm 3.0x
Use a single react hook for table status modal
Add debug logging for real-time ingestion
Separate the exception for transform and indexing for consuming records
Disable JsonStatementOptimizer
Make index readers/loaders pluggable
Make index creator provision pluggable
Support loading plugins from multiple directories
Update helm charts to honour readinessEnabled probes flags on the Controller, Broker, Server and Minion StatefulSets
Support non-selection-only GRPC server request handler
GRPC broker request handler
Add validator for SDF
Support large payload in zk put API
Push JSON Path evaluation down to storage layer
When upserting new record, index the record before updating the upsert metadata
Add Post-Aggregation Gapfilling functionality.
Clean up deprecated fields from segment metadata
Remove deprecated method from StreamMetadataProvider
Obtain replication factor from tenant configuration in case of dimension table
Use valid bucket end time instead of segment end time for merge/rollup delay metrics
Make pinot start components command extensible
Make upsert inner segment update atomic
Clean up deprecated ZK metadata keys and methods
Add extraEnv, envFrom to statefulset help template
Make openjdk image name configurable
Add getPredicate() to PredicateEvaluator interface
Make split commit the default commit protocol
Pass Pinot connection properties from JDBC driver
Add Pinot client connection config to allow skip fail on broker response exception
Change default range index version to v2
Put thread timer measuring inside of wall clock timer measuring
Add getRevertReplaceSegmentRequest method in FileUploadDownloadClient
Add JAVA_OPTS env var in docker image
Split thread cpu time into three metrics
Add config for enabling real-time offset based consumption status checker
Add timeColumn, timeUnit and totalDocs to the json segment metadata
Set default Dockerfile CMD to -help
Add getName() to PartitionFunction interface
Support Native FST As An Index Subtype for FST Indices
Add forceCleanup option for 'startReplaceSegments' API
Add config for keystore types, switch tls to native implementation, and add authorization for server-broker tls channel
Extend FileUploadDownloadClient to send post request with json body

Fix string comparisons (#8253)
Bugfix for order-by all sorted optimization (#8263)
Fix dockerfile (#8239)
Ensure partition function never return negative partition
Handle indexing failures without corrupting inverted indexes
Fixed broken HashCode partitioning
Fix segment replace test
Fix filtered aggregation when it is mixed with regular aggregation
Fix FST Like query benchmark to remove SQL parsing from the measurement
Do not identify function types by throwing exceptions
Fix regression bug caused by sharing TSerializer across multiple threads
Fix validation before creating a table
Check cron schedules from table configs after subscribing child changes
Disallow duplicate segment name in tar file
Fix storage quota checker NPE for Dimension Tables
Fix TraceContext NPE issue
Update gcloud libraries to fix underlying issue with api's with CMEK
Fix error handling in jsonPathArray
Fix error handling in json functions with default values
Fix controller config validation failure for customized TLS listeners
Validate the numbers of input and output files in HadoopSegmentCreationJob
Broker Side validation for the query with aggregation and col but without group by
Improve the proactive segment clean-up for REVERTED
Allow JSON forward indexes
Fix the PinotLLCRealtimeSegmentManager on segment name check
Always use smallest offset for new partitionGroups
Fix RealtimeToOfflineSegmentsTaskExecutor to handle time gap
Refine segment consistency checks during segment load
Fixes for various JDBC issues
Delete tmp- segment directories on server startup
Fix ByteArray datatype column metadata getMaxValue NPE bug and expose maxNumMultiValues
Fix the issues that Pinot upsert table's uploaded segments get deleted when a server restarts.
Fixed segment upload error return
Fix QuerySchedulerFactory to plug in custom scheduler
Fix the issue with grpc broker request handler not started correctly
Fix real-time ingestion when an entire batch of messages is filtered out
Move decode method before calling acquireSegment to avoid reference count leak
Fix semaphore issue in consuming segments
Add bootstrap mode for PinotServiceManager to avoid glitch for health check
Fix the broker routing when segment is deleted
Fix obfuscator not capturing secretkey and keytab
Fix segment merge delay metric when there is empty bucket
Fix QuickStart by adding types for invalid/missing type
Use oldest offset on newly detected partitions
Fix javadoc to compatible with jdk8 source
Handle null segment lineage ZNRecord for getSelectedSegments API
Handle fields missing in the source in ParquetNativeRecordReader

Apache Pinot™ 1.0.0 release notes

This page covers the latest changes included in the Apache Pinot™ 1.0.0 release, including new features, enhancements, and bug fixes.

1.0.0 (2023-09-19)

This release includes the several new features, enhancements, and bug fixes, including the following highlights:

Multi-stage query engine: , , and . Learn how to or more about how the works.

Multi-stage query engine new features

Support for
- Initial (phase 1) Query runtime for window functions with ORDER BY within the OVER() clause (#10449)

Multi-stage query engine enhancements

Turn on v2 engine by default ()
Introduced the ability to stream leaf stage blocks for more efficient data processing ().
Early terminate SortOperator if there is a limit ()

Multi-stage query engine bug fixes

Fix Predicate Pushdown by Using Rule Collection ()
Try fixing mailbox cancel race condition ()
Catch Throwable to Propagate Proper Error Message ()

Index SPI

Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details

Null value support for pinot queries

NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.

Upsert enhancements

Delete support in upsert enabled tables ()

Support added to extend upserts and allow deleting records from a realtime table. The design details can be found .

Preload segments with upsert snapshots to speedup table loading ()

Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).

TTL configs for upsert primary keys ()

Adds support for specifying expiry TTL for upsert primary key metadata cleanup.

Segment compaction for upsert real-time tables ()

Adds a new minion task to compact segments belonging to a real-time table with upserts.

Pinot Spark Connector for Spark3 ()

Added spark3 support for Pinot Spark Connector ()
Also added support to pass pinot query options to spark connector ()

PinotDataBufferFactory and new PinotDataBuffer implementations ()

Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.

Query functions enhancements

Add PercentileKLL aggregation function ()
Support for ARG_MIN and ARG_MAX Functions ()
refactor argmin/max to exprmin/max and make it calcite compliant ()

JSON and CLP encoded message ingestion and querying

Add clpDecode transform function for decoding CLP-encoded fields. ()
Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. ()
Add SchemaConformingTransformer to transform records with varying keys to fit a table's schema without dropping fields. ()

Tier level index config override ()

Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.

Ingestion connectors and features

Kinesis stream header extraction ()
Extract record keys, headers and metadata from Pulsar sources ()
Realtime pre-aggregation for Distinct Count HLL & Big Decimal ()

UI enhancements

Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires ()
AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. ()

Pinot docker image improvements and enhancements

Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK ()
Support multi-arch pinot docker image ()
Update dockerfile with recent jdk distro changes ()

Operational improvements

Rebalance

Rebalance status API ()
Tenant level rebalance API Tenant rebalance and status tracking APIs ()

Config to use customized broker query thread pool ()

Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.

This feature allows better management of broker resources.

Drop results support ()

Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.

Make column order deterministic in segment ()

In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.

Allow configuring helix timeouts for EV dropped in Instance manager ()

Adds options to configure helix timeouts external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.

Enable case insensitivity by default ()

This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql

Newly added APIs and client methods

Add Server API to get tenant pools ()
Add new broker query point for querying multi-stage engine ()
Add a new controller endpoint for segment deletion with a time window ()

Cleanup and backward incompatible changes

High level consumers are no longer supported

Cleanup HLC code ()
Remove support for High level consumers in Apache Pinot ()

Type information preservation of query literals

[feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform () String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.

Controller job status ZNode path update

Moving Zk updates for reload, force_commit to their own Znodes which … () The status of previously completed reload jobs will not be available after this change is deployed.

Metric names for mutable indexes to change

Implement mutable index using index SPI () Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.

Update in controller API to enable / disable / drop instances

Update getTenantInstances call for controller and separate POST operations on it ()

Change in substring query function definition

Change substring to comply with standard sql definition ()

Full list of features added

Allow queries on multiple tables of same tenant to be executed from controller UI
Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig
[Index SPI] IndexType ()

Vulnerability fixes, bugfixes, cleanups and deprecations

Remove support for High level consumers in Apache Pinot ()
Fix JDBC driver check for username ()
[Clean up] Remove getColumnName() from AggregationFunction interface ()

Releases

hashtagNote

hashtag1.0.0 (September 2023)

hashtag0.12.1 (March 2023)

hashtag0.12.0 (December 2022)

hashtag0.11.0 (September 2022)

hashtag0.10.0 (March 2022)

hashtag0.9.3 (December 2021)

hashtag0.9.2 (December 2021)

hashtag0.9.1 (December 2021)

hashtag0.9.0 (November 2021)

hashtag0.8.0 (August 2021)

hashtag0.7.1 (April 2021)

hashtag0.6.0 (November 2020)

hashtag0.5.0 (September 2020)

hashtag0.4.0 (June 2020)

hashtag0.3.0 (March 2020)

hashtag0.2.0 (November 2019)

hashtag0.1.0 (March 2019, First release)

0.12.1

hashtagSummary

0.9.3

hashtagSummary

0.9.2

hashtagSummary

0.9.1

hashtagSummary

0.7.1

hashtagSummary

hashtagNotable New Features

hashtagSpecial notes

hashtagMajor Bug fixes

0.6.0

hashtagSummary

hashtagNotable New Features

hashtagSpecial notes

hashtagMajor Bug fixes

hashtagBackward Incompatible Changes

0.5.0

hashtagSummary

hashtagNotable New Features

hashtagSpecial notes

hashtagMajor Bug fixes

hashtagBackward Incompatible Changes

0.4.0

hashtagSummary

hashtagNotable New Features

hashtagMajor Bug Fixes

hashtagWork in Progress

hashtagBackward Incompatible Changes

0.2.0

hashtagNew Features and Bug Fixes

hashtagWork in Progress

hashtagOther Notes

0.1.0

hashtagNew Features

Releases

hashtagNote

hashtag1.0.0 (September 2023)

hashtag0.12.1 (March 2023)

hashtag0.12.0 (December 2022)

hashtag0.11.0 (September 2022)

hashtag0.10.0 (March 2022)

hashtag0.9.3 (December 2021)

hashtag0.9.2 (December 2021)

hashtag0.9.1 (December 2021)

hashtag0.9.0 (November 2021)

hashtag0.8.0 (August 2021)

hashtag0.7.1 (April 2021)

hashtag0.6.0 (November 2020)

hashtag0.5.0 (September 2020)

hashtag0.4.0 (June 2020)

hashtag0.3.0 (March 2020)

hashtag0.2.0 (November 2019)

hashtag0.1.0 (March 2019, First release)

0.9.1

hashtagSummary

0.9.3

hashtagSummary

0.12.1

Note

1.0.0 (September 2023)

0.12.1 (March 2023)

0.12.0 (December 2022)

0.11.0 (September 2022)

0.10.0 (March 2022)

0.9.3 (December 2021)

0.9.2 (December 2021)

0.9.1 (December 2021)

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

Summary

Summary

Summary

Summary

Summary

Notable New Features

Special notes

Major Bug fixes

Summary

Notable New Features

Special notes

Major Bug fixes

Backward Incompatible Changes

Summary

Notable New Features

Special notes

Major Bug fixes

Backward Incompatible Changes

Summary

Notable New Features

Major Bug Fixes

Work in Progress

Backward Incompatible Changes

New Features and Bug Fixes

Work in Progress

Other Notes

New Features

Note

1.0.0 (September 2023)

0.12.1 (March 2023)

0.12.0 (December 2022)

0.11.0 (September 2022)

0.10.0 (March 2022)

0.9.3 (December 2021)

0.9.2 (December 2021)

0.9.1 (December 2021)

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

Summary

Summary

Summary

New Features

Summary

New Features and Bug Fixes

Work in Progress

Other Notes

Summary

Notable New Features

Major Bug Fixes

Work in Progress

Backward Incompatible Changes

Summary

Notable New Features

Special notes

Major Bug fixes