This release introduces some new great features, performance enhancements, UI improvements, and bug fixes which are described in details in the following sections. The release was cut from this commit .

The dependency graph for plug-and-play architecture that was introduced in release has been extended and now it contains new nodes for Pinot Segment SPI.

Implement NOT Operator

Add DistinctCountSmartHLLAggregationFunction which automatically store distinct values in Set or HyperLogLog based on cardinality

Add LEAST and GREATEST functions

Handle SELECT * with extra columns

Add FILTER clauses for aggregates

Add ST_Within function

Handle semicolon in query

Add EXPLAIN PLAN

Show Reported Size and Estimated Size in human readable format in UI

Make query console state URL based

Improve query console to not show query result when multiple columns have the same name

Improve Pinot dashboard tenant view to show correct amount of servers and brokers

Fix issue with opening new tabs from Pinot Dashboard

Fix issue with Query console going blank on syntax error

Make query stats always show even there's error

Implement OIDC auth workflow in UI

Add tooltip and modal for table status

Add option to wrap lines in custom code mirror

Add ability to comment out queries with cmd + /

Return exception when unavailable segments on empty broker response

Properly handle the case where segments are missing in externalview

Add TIMESTAMP to datetime column Type

Reuse regex matcher in dictionary based LIKE queries

Early terminate orderby when columns already sorted

Do not do another pass of Query Automaton Minimization

Improve RangeBitmap by upgrading RoaringBitmap

Optimize geometry serializer usage when literal is available

Improve performance of no-dictionary group by

Allocation free DataBlockCache lookups

Prune unselected THEN statements in CaseTransformFunction

Aggregation delay conversion to double

Reduce object allocation rate in ExpressionContext or FunctionContext

Lock free DimensionDataTableManager

Improve json path performance during ingestion by upgrading JsonPath

Reduce allocations and speed up StringUtil.sanitizeString

Faster metric scans - ForwardIndexReader

Unpeel group by 3 ways to enable vectorization

Power of 2 fixed size chunks

Don't use mmap for compression except for huge chunks

Exit group-by marking loop early

Improve performance of base chunk forward index write

Cache JsonPaths to prevent compilation per segment

Use LZ4 as default compression mode

Peel off special case for 1 dimensional groupby

Bump roaringbitmap version to improve range queries performance

Adding NoopPinotMetricFactory and corresponding changes

Allow to specify fixed segment name for SegmentProcessorFramework

Move all prestodb dependencies into a separated module

Include docIds in Projection and Transform block

Automatically update broker resource on broker changes

Update ScalarFunction annotation from name to names to support function alias.

Implemented BoundedColumnValue partition function

Add copy recursive API to pinotFS

Add Support for Getting Live Brokers for a Table (without type suffix)

Pinot docker image - cache prometheus rules

In BrokerRequestToQueryContextConverter, remove unused filterExpressionContext

Adding retention period to segment delete REST API

Pinot docker image - upgrade prometheus and scope rulesets to components

Allow segment name postfix for SegmentProcessorFramework

Superset docker image - update pinotdb version in superset image

Add retention period to deleted segment files and allow table level overrides

Remove incubator from pinot and superset

Adding table config overrides for disabling groovy

Optimise sorted docId iteration order in mutable segments

Adding secure grpc query server support

Move Tls configs and utils from pinot-core to pinot-common

Reduce allocation rate in LookupTransformFunction

Allow subclass to customize what happens pre/post segment uploading

Enable controller service auto-discovery in Jersey framework

Add support for pushFileNamePattern in pushJobSpec

Add additionalMatchLabels to helm chart

Simulate rsvps after meetup.com retired the feed

Adding more checkstyle rules

Add persistence.extraVolumeMounts and persistence.extraVolumes to Kubernetes statefulsets

Adding scala profile for kafka 2.x build and remove root pom scala dependencies

Allow realtime data providers to accept non-kafka producers

Enhance revertReplaceSegments api

Adding broker level config for disabling Pinot queries with Groovy

Make presto driver query pinot server with SQL

Adding controller config for disabling Groovy in ingestionConfig

Adding main method for LaunchDataIngestionJobCommand for spark-submit command

Add auth token for segment replace rest APIs

Add allowRefresh option to UploadSegment

Add Ingress to Broker and Controller helm charts

Improve progress reporter in SegmentCreationMapper

St_* function error messages + support literal transform functions

Add schema and segment crc to SegmentDirectoryContext

Extend enableParallePushProtection support in UploadSegment API

Support BOOLEAN type in Config Recommendation Engine

Add a broker metric to distinguish exception happens when acquire channel lock or when send request to server

Add pinot.minion prefix on minion configs for consistency

Enable broker service auto-discovery in Jersey framework

Timeout if waiting server channel lock takes a long time

Wire EmptySegmentPruner to routing config

Support for TIMESTAMP data type in Config Recommendation Engine

Listener TLS customization

Add consumption rate limiter for LLConsumer

Implement Real Time Mutable FST

Allow quickstart to get table files from filesystem

Add support for instant segment deletion

Add a config file to override quickstart configs

Add pinot server grpc metadata acl

Move compatibility verifier to a separate module

Move hadoop and spark ingestion libs from plugins directory to external-plugins

Add global strategy for partial upsert

Upgrade kafka to 2.8.1

Created EmptyQuickstart command

Allow SegmentPushUtil to push realtime segment

Add ignoreMerger for partial upsert

Make task timeout and concurrency configurable

Return 503 response from health check on shut down

Pinot-druid-benchmark: set the multiValueDelimiterEnabled to false when importing TPC-H data

Cleanup: Remove remaining occurrences of incubator.

Refactor segment loading logic in BaseTableDataManager to decouple it with local segment directory

Improving segment replacement/revert protocol

PinotConfigProvider interface

Enhance listSegments API to exclude the provided segments from the output

Remove outdated broker metric definitions

Add skip key for realtimeToOffline job validation

Upgrade async-http-client

Allow Reloading Segments with Multiple Threads

Ignore query options in commented out queries

Remove TableConfigCache which does not listen on ZK changes

Switch to zookeeper of helm 3.0x

Use a single react hook for table status modal

Add debug logging for realtime ingestion

Separate the exception for transform and indexing for consuming records

Disable JsonStatementOptimizer

Make index readers/loaders pluggable

Make index creator provision pluggable

Support loading plugins from multiple directories

Update helm charts to honour readinessEnabled probes flags on the Controller, Broker, Server and Minion StatefulSets

Support non-selection-only GRPC server request handler

GRPC broker request handler

Add validator for SDF

Support large payload in zk put API

Push JSON Path evaluation down to storage layer

When upserting new record, index the record before updating the upsert metadata

Add Post-Aggregation Gapfilling functionality.

Clean up deprecated fields from segment metadata

Remove deprecated method from StreamMetadataProvider

Obtain replication factor from tenant configuration in case of dimension table

Use valid bucket end time instead of segment end time for merge/rollup delay metrics

Make pinot start components command extensible

Make upsert inner segment update atomic

Clean up deprecated ZK metadata keys and methods

Add extraEnv, envFrom to statefulset help template

Make openjdk image name configurable

Add getPredicate() to PredicateEvaluator interface

Make split commit the default commit protocol

Pass Pinot connection properties from JDBC driver

Add Pinot client connection config to allow skip fail on broker response exception

Change default range index version to v2

Put thread timer measuring inside of wall clock timer measuring

Add getRevertReplaceSegmentRequest method in FileUploadDownloadClient

Add JAVA_OPTS env var in docker image

Split thread cpu time into three metrics

Add config for enabling realtime offset based consumption status checker

Add timeColumn, timeUnit and totalDocs to the json segment metadata

Set default Dockerfile CMD to -help

Add getName() to PartitionFunction interface

Support Native FST As An Index Subtype for FST Indices

Add forceCleanup option for 'startReplaceSegments' API

Add config for keystore types, switch tls to native implementation, and add authorization for server-broker tls channel

Extend FileUploadDownloadClient to send post request with json body

Fix string comparisons

Bugfix for order-by all sorted optimization

Fix dockerfile

Ensure partition function never return negative partition

Handle indexing failures without corrupting inverted indexes

Fixed broken HashCode partitioning

Fix segment replace test

Fix filtered aggregation when it is mixed with regular aggregation

Fix FST Like query benchmark to remove SQL parsing from the measurement

Do not identify function types by throwing exceptions

Fix regression bug caused by sharing TSerializer across multiple threads

Fix validation before creating a table

Check cron schedules from table configs after subscribing child changes

Disallow duplicate segment name in tar file

Fix storage quota checker NPE for Dimension Tables

Fix TraceContext NPE issue

Update gcloud libraries to fix underlying issue with api's with CMEK

Fix error handling in jsonPathArray

Fix error handling in json functions with default values

Fix controller config validation failure for customized TLS listeners

Validate the numbers of input and output files in HadoopSegmentCreationJob

Broker Side validation for the query with aggregation and col but without group by

Improve the proactive segment clean-up for REVERTED

Allow JSON forward indexes

Fix the PinotLLCRealtimeSegmentManager on segment name check

Always use smallest offset for new partitionGroups

Fix RealtimeToOfflineSegmentsTaskExecutor to handle time gap

Refine segment consistency checks during segment load

Fixes for various JDBC issues

Delete tmp- segment directories on server startup

Fix ByteArray datatype column metadata getMaxValue NPE bug and expose maxNumMultiValues

Fix the issues that Pinot upsert table's uploaded segments get deleted when a server restarts.

Fixed segment upload error return

Fix QuerySchedulerFactory to plug in custom scheduler

Fix the issue with grpc broker request handler not started correctly

Fix realtime ingestion when an entire batch of messages is filtered out

Move decode method before calling acquireSegment to avoid reference count leak

Fix semaphore issue in consuming segments

Add bootstrap mode for PinotServiceManager to avoid glitch for health check

Fix the broker routing when segment is deleted

Fix obfuscator not capturing secretkey and keytab

Fix segment merge delay metric when there is empty bucket

Fix QuickStart by adding types for invalid/missing type

Use oldest offset on newly detected partitions

Fix javadoc to compatible with jdk8 source

Handle null segment lineage ZNRecord for getSelectedSegments API

Handle fields missing in the source in ParquetNativeRecordReader

Fix the issue with HashCode partitioning function

Fix the issue with validation on table creation

Change PinotFS API's

The release was cut from the following commit: and the following cherry-picks: ,

Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor ()

Merge/Rollup task scheduler for offline tables. ()

Fix MergeRollupTask uploading segments not updating their metadata ()

MergeRollupTask integration tests ()

Add mergeRollupTask delay metrics ()

MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling ()

Use maxEndTimeMs for merge/roll-up delay metrics. ()

Cmd+Enter shortcut to run query in query console ()

Showing tooltip in SQL Editor ()

Make the SQL Editor box expandable ()

Fix tables ordering by number of segments ()

IN ()

LASTWITHTIME ()

ID_SET on MV columns ()

Raw results for Percentile TDigest and Est (),

Add timezone as argument in function toDateTime ()

LIKE()

REGEXP_EXTRACT()

FILTER()

Infer data type for Literal ()

Support logical identifier in predicate ()

Support JSON queries with top-level array path expression. ()

Support configurable group by trim size to improve results accuracy ()

Reduce the disk usage for segment conversion task ()

Simplify association between Java Class and PinotDataType for faster mapping ()

Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation ()

Replace MINUS with STRCMP ()

Bit-sliced range index for int, long, float, double, dictionarized SV columns ()

Use MethodHandle to access vectorized unsigned comparison on JDK9+ ()

Add option to limit thread usage per query ()

Improved range queries ()

Faster bitmap scans ()

Optimize EmptySegmentPruner to skip pruning when there is no empty segments ()

Map bitmaps through a bounded window to avoid excessive disk pressure ()

Allow RLE compression of bitmaps for smaller file sizes ()

Support raw index properties for columns with JSON and RANGE indexes ()

Enhance BloomFilter rule to include IN predicate() ()

Introduce LZ4_WITH_LENGTH chunk compression type ()

Enhance ColumnValueSegmentPruner and support bloom filter prefetch ()

Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func ()

During segment pruning, release the bloom filter after each segment is processed ()

Fix JSONPath cache inefficient issue ()

Optimize getUnpaddedString with SWAR padding search ()

Lighter weight LiteralTransformFunction, avoid excessive array fills ()

Inline binary comparison ops to prevent function call overhead ()

Memoize literals in query context in order to deduplicate them ()

Human Readable Controller Configs ()

Add the support of geoToH3 function ()

Add Apache Pulsar as Pinot Plugin () ()

Add dropwizard metrics plugin ()

Introduce OR Predicate Execution On Star Tree Index ()

Allow to extract values from array of objects with jsonPathArray ()

Add Realtime table metadata and indexes API. ()

Support array with mixing data types ()

Support force download segment in reload API ()

Show uncompressed znRecord from zk api ()

Add debug endpoint to get minion task status. ()

Validate CSV Header For Configured Delimiter ()

Add auth tokens and user/password support to ingestion job command ()

Add option to store the hash of the upsert primary key ()

Add null support for time column ()

Add mode aggregation function ()

Support disable swagger in Pinot servers ()

Delete metadata properly on table deletion ()

Add basic Obfuscator Support ()

Add AWS sts dependency to enable auth using web identity token. ()()

Mask credentials in debug endpoint /appconfigs ()

Fix /sql query endpoint now compatible with auth ()

Fix case sensitive issue in BasicAuthPrincipal permission check ()

Fix auth token injection in SegmentGenerationAndPushTaskExecutor ()

Add segmentNameGeneratorType config to IndexingConfig ()

Support trigger PeriodicTask manually ()

Add endpoint to check minion task status for a single task. ()

Showing partial status of segment and counting CONSUMING state as good segment status ()

Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule ()

Check schema backward-compatibility when updating schema through addSchema with override ()

Optimize IndexedTable ()

Support indices remove in V3 segment format ()

Optimize TableResizer ()

Introduce resultSize in IndexedTable ()

Offset based realtime consumption status checker ()

Add causes to stack trace return ()

Create controller resource packages config key ()

Enhance TableCache to support schema name different from table name ()

Add validation for realtimeToOffline task ()

Unify CombineOperator multi-threading logic ()

Support no downtime rebalance for table with 1 replica in TableRebalancer ()

Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. ()

Adjust tuner api ()

Adding config for metrics library ()

Add geo type conversion scalar functions ()

Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types ()

Add MV raw forward index and MV BYTES data type ()

Enhance TableRebalancer to offload the segments from most loaded instances first ()

Improve get tenant API to differentiate offline and realtime tenants ()

Refactor query rewriter to interfaces and implementations to allow customization ()

In ServiceStartable, apply global cluster config in ZK to instance config ()

Make dimension tables creation bypass tenant validation ()

Allow Metadata and Dictionary Based Plans for No Op Filters ()

Reject query with identifiers not in schema ()

Round Robin IP addresses when retry uploading/downloading segments ()

Support multi-value derived column in offline table reload ()

Support segmentNamePostfix in segment name ()

Add select segments API ()

Controller getTableInstance() call now returns the list of live brokers of a table. ()

Allow MV Field Support For Raw Columns in Text Indices ()

Allow override distinctCount to segmentPartitionedDistinctCount ()

Add a quick start with both UPSERT and JSON index ()

Add revertSegmentReplacement API ()

Smooth segment reloading with non blocking semantic ()

Clear the reused record in PartitionUpsertMetadataManager ()

Replace args4j with picocli ()

Handle datetime column consistently ()()

Allow to carry headers with query requests () ()

Allow adding JSON data type for dimension column types ()

Separate SegmentDirectoryLoader and tierBackend concepts ()

Implement size balanced V4 raw chunk format ()

Add presto-pinot-driver lib ()

Fix null pointer exception for non-existed metric columns in schema for JDBC driver ()

Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD ()

Fixed pinot java client to add zkClient close ()

Ignore query json parse errors ()

Fix shutdown hook for PinotServiceManager () ()

Make STRING to BOOLEAN data type change as backward compatible schema change ()

Replace gcp hardcoded values with generic annotations ()

Fix segment conversion executor for in-place conversion ()

Fix reporting consuming rate when the Kafka partition level consumer isn't stopped ()

Fix the issue with concurrent modification for segment lineage ()

Fix TableNotFound error message in PinotHelixResourceManager ()

Fix upload LLC segment endpoint truncated download URL ()

Fix task scheduling on table update ()

Fix metric method for ONLINE_MINION_INSTANCES metric ()

Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema ()

Fix currentOffset volatility in consuming segment()

Fix misleading error msg for missing URI ()

Fix the correctness of getColumnIndices method ()

Fix SegmentZKMetadta time handling ()

Fix retention for cleaning up segment lineage ()

Fix segment generator to not return illegal filenames ()

Fix missing LLC segments in segment store by adding controller periodic task to upload them ()

Fix parsing error messages returned to FileUploadDownloadClient ()

Fix manifest scan which drives /version endpoint ()

Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection ()

Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: ()

Fix pql double quote checker exception ()

Fix minion metrics exporter config ()

Fix segment unable to retry issue by catching timeout exception during segment replace ()

Add Exception to Broker Response When Not All Segments Are Available (Partial Response) ()

Fix segment generation commands ()

Return non zero from main with exception ()

Fix parquet plugin shading error ()

Fix the lowest partition id is not 0 for LLC ()

Fix star-tree index map when column name contains '.' ()

Fix cluster manager URLs encoding issue()

Fix fieldConfig nullable validation ()

Fix verifyHostname issue in FileUploadDownloadClient ()

Fix TableCache schema to include the built-in virtual columns ()

Fix DISTINCT with AS function ()

Fix SDF pattern in DataPreprocessingHelper ()

Fix fields missing issue in the source in ParquetNativeRecordReader ()