This release comes with several features, SQL /UI/Perf enhancements Bugfixes across areas ranging from Multistage Query Engine to Ingestion, Storage format, SQL support, etc.
Multi-stage Query Engine
Support RelDistribution-based trait Planning (, )
Adds support for RelDistribution optimization for more accurate leaf-stage direct exchange/shuffle. Also extends partition optimization beyond leaf stage to entire query plan.
Applies optimization based on distribution trait in the mailbox/worker assignment stage
Fixes previous direct exchange which was decided based on the table partition hint. Now direct exchange is decided via distribution trait: it will applied if-and-only-if the trait propagated matches the exchange requirement.
Leaf stage planning with multi-semi join support ()
Solves the limitation of pinotQuery that supports limited amount of PlanNodes.
Splits the ServerRequest planning into 2 stages
First plan as much as possible into PinotQuery
Support for ArrayAgg aggregation function ()
Usage: ArrayAgg(column, 'dataType' [, 'isDistinct'])
Float type column is treated as Double in the multistage engine, so FLOAT type is not supported.
Supports data BOOLEAN, INT, LONG
Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support
concat as || operator ()
Capability for constant filter in QueryContext, with support for server to handle it (
Bugfixes, Refactoring, Cleanups, Tests
Bugfix for evaluation of chained literal functions ()
Fixes to sort copy rule ( and )
Fixes duplicate results for literal queries ()
Notable Features
Server-level throttling for realtime consumption ()
Use server config pinot.server.consumption.rate.limit to enable this feature
Server rate limiter is disabled by default (default value 0)
Reduce segment generation disk footprint for Minion Tasks ()
Supported in MergeRollupTask and RealtimeToOfflineSegmentsTask minion tasks
Use taskConfig segmentMapperFileSizeThresholdInBytes to specify the threshold size
Support for swapping of TLS keystore/truststore (, )
Security feature that makes the keystore/truststore swappable.
Auto-reloads keystore/truststore (without need for a restart) if they are local files
Sticky Query Routing ()
Adds support for deterministic and sticky routing for a query / table / broker. This setting would lead to same server / set of servers (for MultiStageReplicaGroupSelector) being used for all queries of a given table.
Query option (takes precedence over fixed routing setting at table / broker config level)
SET "useFixedReplica"=true;
Table config (takes precedence over fixed routing setting at broker config level)
Table Config to disallow duplicate primary key for dimension tables ()
Use tableConfig dimensionTableConfig.errorOnDuplicatePrimaryKey=true to enable this behavior
Partition-Level ForceCommit for realtime tables ()
Support to force-commit specific partitions of a realtime table.
Partitions can be specified to the forceCommit API as a comma separated list of partition names or consuming segment names
Support initializing broker tags from config ()
Support to give the broker initial tags on startup.
Automatically updates brokerResource when broker joins the cluster for the first time
Broker tags are provided as comma-separated values in pinot.broker.instance.tags
Support for StreamNative OAuth2 authentication for pulsar ()
StreamNative (the cloud SAAS offering of Pulsar) uses OAuth2 to authenticate clients to their Pulsar clusters.
For more information, see how to
Can be configured by adding the following properties to streamConfigs:
Introduce low disk mode to table rebalance ()
Introduces a new table rebalance boolean config lowDiskMode.Default value is false.
Applicable for rebalance with downtime=false.
When enabled, segments will first be offloaded from servers, then added to servers after offload is done. It may increase the total time of the rebalance, but can be useful when servers are low on disk space, and we want to scale up the cluster and rebalance the table to more servers.
Support Vector index and Hierarchical Navigable Small Worlds (HNSW) ()
Supports Vector Index on float array/multi-value columnz
Add predicate and function to retrieve topK closest vector. Example query
The function VectorSimilarity will return a double value where the first parameter is the embedding column and the second parameter is the search term embedding literal.
Since VectorSimilarity is a predicate, once config the topK, this predicate will return topk rows per segment. Then if you are using this index with other predicate, you may not get expected number of rows since the records matching other predicate might not in the topk rows.
Support for retention on deleted keys of upsert tables ()
Adds an upsert config deletedKeysTTL which will remove deleted keys from in-memory hashmap and mark the validDocID as invalid after the deletedKeysTTL threshold period.
Disabled by default. Enabled only if a valid value for deletedKeysTTL is set
Configurable Lucene analyzer ()
Introduces the capability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis.
Default Behavior falls back to using the standardAnalyzer unless the luceneAnalyzerClass property is specified.
Support for murmur3 as a partition function ()
Murmur3 support with optional fields seed and variant for the hash in functionConfig field of columnPartitionMap.Default value for seed is 0.
Added support for 2 variants of Murmur3: x86_32
New optimized MV forward index to only store unique MV values
Adds new MV dictionary encoded forward index format that only stores the unique MV entries.
This new index format can significantly reduce the index size when the MV entries repeat a lot
The new index format can be enabled during index creation, derived column creation, and segment reload
Support for explicit null handling modes ()
Adds support for 2 possible ways to handle null:
Table mode - which already exists
Column mode, which means that each column specifies its own nullability in the FieldSpec
Support tracking out of order events in Upsert ()
Adds a new upsert config outOfOrderRecordColumn
When set to a non-null value, we check whether an event is OOO or not and then accordingly update the corresponding column value to true / false.
This will help in tracking which event is out-of-order while using skipUpsert
Compression configuration support for aggregationConfigs to StartreeIndexConfigs ()
Can be used to save space. For eg: when a functionColumnPairs has a output type of bytes, such as when you use distinctcountrawhll.
Preconfiguration based mirror instance assignment ()
Supports instance assignment based pre-configured instance assignment map.
The assignment will always respect the mirrored servers in the pre-configured map
Support for Listing Dimension Tables ()
Adds dimension as a valid option to table "type" in the /tables controller API
Support in upsert for dropping out of order events ()
This patch adds a new config for upsert: dropOutOfOrderRecord
If set to true, pinot doesn't persist out-of-order events in the segment.
This feature is useful to
Support to retry failed table rebalance tasks ()
New configs for the RebalanceChecker periodic task:
controller.rebalance.checker.frequencyPeriod: 5min by default ; -1 to disable
controller.rebalanceChecker.initialDelayInSeconds
Support for UltraLogLog ()
UltraLogLog aggregations for Count Distinct (distinctCountULL and distinctCountRawULL)
UltraLogLog creation via Transform Function
UltraLogLog merging in MergeRollup
Support for Apache Datasketches CPC sketch ()
Ingestion via transformation function
Extracting estimates via query aggregation functions
Segment rollup aggregation
Support to reduce DirectMemory OOM chances on broker ()
Broadly there are two configs that will enable this feature:
maxServerResponseSizeBytes: Maximum serialized response size across all servers for a query. This value is equally divided across all servers processing the query.
maxQueryResponseSizeBytes: Maximum length of the serialized response per server for a query
UI Support to Allow schema to be created with JSON config ()
This is helpful when user has the entire JSON handy
UI still keeps Form Way to add Schema along with JSON view
Support in JSON index for ignoring values longer than a given length ()
Use option maxValueLength in jsonIndexConfig to restrict length of values
A value of 0 (or when the key is omitted) means there is no restriction
Support for MultiValue VarByte V4 index writer ()
Supports serializing and writing MV columns in VarByteChunkForwardIndexWriterV4
Supports V4 reader that can be used to read SV var length, MV fixed length and MV var length buffers encoded with V4 writer
Improved scalar function support for Multivalue columns(, )
Support for FrequentStringsSketch and FrequentLonsSketch aggregation functions ()
Approximation aggregation functions for estimating the frequencies of items a dataset in a memory efficient way. More details in library.
Controller API for Table Indexe ()
Table index api to get the aggregate index details of all segments for a table.
URL/tables/{tableName}/indexes
Support for configurable rebalance delay at lead controller ()
The lead controller rebalance delay is now configurable with controller.resource.rebalance.delay_ms
Changing rebalance configurations will now update the lead controller resource
Support for configuration through environment variables ()
Adds support for Pinot configuration through ENV variables with Dynamic mapping.
Sample configs through ENV
Add hyperLogLogPlus aggregation function for distinct count ()
HLL++ has higher accuracy than HLL when cardinality of dimension is at 10k-100k.
Support for clpMatch
Adds query rewriting logic to transform a "virtual" UDF, clpMatch, into a boolean expression on the columns of a CLP-encoded field.
To use the rewriter, modify broker config to add org.apache.pinot.sql.parsers.rewriter.ClpRewriter to pinot.broker.query.rewriter.class.names.
Support for DATETIMECONVERTWINDOWHOP function ()
Support for JSON_EXTRACT_INDEX transform function to leverage json index for json value extraction ()
Support for ArrayAgg aggregation function ()
GenerateData command support for generating data in JSON format ()
Support ARRAY function as a literal evaluation ()
Support for ARRAY literal transform functions ()
Theta Sketch Aggregation enhancements ()
Async rendering of UI elements to load UI elements async resulting in faster page loads ()
Make the table name link clickable in task details ()
Swagger UI enhancements to resumeConsumption API call ()
Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap ()
Wire soft upsert delete for Compaction task ()
Upsert compaction debuggability APIs for validDocId metadata ()
Bugfixes, Refactoring, Cleanups, Deprecations
Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader ()
Fix error message format for Preconditions.checks failures()
Bugfix to distribute Pinot as a multi-release JAR (, )
Backward incompatible Changes
Fix a race condition for upsert compaction (). Notes on backward incompatibility below:
This PR is introducing backward incompatibility for UpsertCompactionTask. Previously, we allowed to configure the compaction task without the snapshot enabled. We found that using in-memory based validDocIds is a bit dangerous as it will not give us the consistency (e.g. fetching validDocIds bitmap while the server is restarting & updating validDocIds).
We now enforce the enableSnapshot=true for UpsertCompactionTask if the advanced customer wants to run the compaction with the in-memory validDocId bitmap.
Library Upgrades and dependencies
update maven-jar-plugin and maven-enforcer-plugin version (#11637)
Update testng as the test provider explicitly instead of relying on the classpath. ()
Update compatibility verifier version ()