0.12.0
Multi-Stage Query Engine
New join semantics support
Left join (#9466)
In-equi join (#9448)
Full join (#9907)
Right join (#9907)
Semi join (#9367)
Using keyword (#9373)
New sql semantics support:
Having (#9274)
Order by (#9279)
In/NotIn clause (#9374)
Cast (#9384)
LIke/Rexlike (#9654)
Range predicate (#9445)
Performance enhancement
Thread safe query planning (#9344)
Partial query execution and round robin scheduling (#9753)
Improve data table serde (#9731)
Major updates
Force commit consuming segments by @sajjad-moradi in #9197
Enable Consistent Data Push for Standalone Segment Push Job Runners by @yuanbenson in #9295
Allow server to directly return the final aggregation result by @Jackie-Jiang in #9304
Adaptive Server Selection by @vvivekiyer in #9311
Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356
Do not allow implicit cast for BOOLEAN and TIMESTAMP by @Jackie-Jiang in #9385
Add more options to json index by @Jackie-Jiang in #9543
Add segment level debug API by @saurabhd336 in #9609
Customize stopword for Lucene Index by @jasperjiaguo in #9708
Added Delete Schema/Table pinot admin commands by @bagipriyank in #9857
Other features/changes
Show most recent scheduling errors by @satishwaghela in #9161
Do not use aggregation result for distinct query in IntermediateResultsBlock by @Jackie-Jiang in #9262
Emit metrics for ratio of actual consumption rate to rate limit in real-time tables by @sajjad-moradi in #9201
add metrics entry offlineTableCount by @walterddr in #9270
Optimize ser/de to avoid using output stream by @Jackie-Jiang in #9278
Add Support for Covariance Function by @SabrinaZhaozyf in #9236
Improve server query cancellation and timeout checking during execution by @jasperjiaguo in #9286
Add capabilities to ingest from another stream without disabling the real-time table by @sajjad-moradi in #9289
Add array cardinality function by @walterddr in #9300
Infer parquet reader type based on file metadata by @saurabhd336 in #9294
Add Support for Cast Function on MV Columns by @SabrinaZhaozyf in #9296
delete all related minion task metadata when deleting a table by @zhtaoxiang in #9339
BloomFilterRule should only recommend for supported column type by @yuanbenson in #9364
Improve segment name check in metadata push by @zhtaoxiang in #9359
Enhance
and
filter predicate evaluation efficiency by @jasperjiaguo in #9336Deprecate instanceId Config For Broker/Minion Specific Configs by @ankitsultana in #9308
Optimize combine operator to fully utilize threads by @Jackie-Jiang in #9387
Terminate the query after plan generation if timeout by @jasperjiaguo in #9386
Benchmark thread cpu time by @jasperjiaguo in #9408
Extract the common logic for upsert metadata manager by @Jackie-Jiang in #9435
Make minion task metadata manager methods more generic by @saurabhd336 in #9436
Adaptive Server Selection by @vvivekiyer in #9311
Cache Deleted Segment Names in Server to Avoid SegmentMissingError by @ankitsultana in #9423
spark-connector should use v2/brokers endpoint by @itschrispeck in #9451
Adaptive Server Selection: Address pending review comments by @vvivekiyer in #9462
Decouple ser/de from DataTable by @Jackie-Jiang in #9468
Eliminate explosion of metrics due to gapfill queries by @elonazoulay in #9490
ForwardIndexHandler: Change compressionType during segmentReload by @vvivekiyer in #9454
Add query interruption flag check to broker groupby reduction by @jasperjiaguo in #9499
adding optional client payload by @walterddr in #9465
Check data table version on server only for null handling by @Jackie-Jiang in #9508
Sort scanning based operators by cardinality in AndDocIdSet evaluation by @jasperjiaguo in #9420
Do not fail CI when codecov upload fails by @Jackie-Jiang in #9522
broker filter by @dongxiaoman in #9391
Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356
[GHA] add cache timeout by @walterddr in #9524
Optimize PinotHelixResourceManager.hasTable() by @Jackie-Jiang in #9526
Include exception when upsert metadata manager cannot be created by @Jackie-Jiang in #9532
Remove the wrong warning log in KafkaPartitionLevelConsumer by @Jackie-Jiang in #9536
Make StreamMessage generic and a bug fix by @vvivekiyer in #9544
[Upsert] Skip removing upsert metadata when shutting down the server by @Jackie-Jiang in #9551
add array element at function by @walterddr in #9554
Handle the case when enableNullHandling is true and an aggregation function is used w/ a column that has an empty null bitmap by @nizarhejazi in #9566
Support rewriting forward index upon changing compression type for existing raw MV column by @vvivekiyer in #9510
Support Avro's Fixed data type by @sajjad-moradi in #9642
[feature] [kubernetes] add loadBalancerSourceRanges to service-external.yaml for controller and broker by @jameskelleher in #9494
Limit up to 10 unavailable segments to be printed in the query exception by @Jackie-Jiang in #9617
Do not cache record reader in segment by @Jackie-Jiang in #9604
make first part of user agent header configurable by @rino-kadijk in #9471
Enhance cluster config update API to handle non-string values properly by @Jackie-Jiang in #9635
Reverts recommender REST API back to PUT (reverts PR #9326) by @yuanbenson in #9638
Remove invalid pruner names from server config by @Jackie-Jiang in #9646
Handle unique query id on server by @Jackie-Jiang in #9648
stateless group marker missing several by @walterddr in #9673
Support reloading consuming segment using force commit by @Jackie-Jiang in #9640
Improve star-tree to use star-node when the predicate matches all the non-star nodes by @Jackie-Jiang in #9667
Improve star-tree traversal using ArrayDeque by @Jackie-Jiang in #9688
Handle errors in combine operator by @Jackie-Jiang in #9689
return different error code if old version is not on master by @SabrinaZhaozyf in #9686
Support creating dictionary at runtime for an existing column by @vvivekiyer in #9678
add index key and size map to segment metadata by @walterddr in #9712
Use ideal state as source of truth for segment existence by @Jackie-Jiang in #9735
Add Support for IP Address Function by @SabrinaZhaozyf in #9501
show table metadata info in aggregate index size form by @walterddr in #9733
AdaptiveServerSelection: Update stats for servers that have not responded by @vvivekiyer in #9801
[MergeRollupTask] include partition info into segment name by @zhtaoxiang in #9815
Download offline segments from peers by @wirybeaver in #9710
Thread Level Usage Accounting and Query Killing on Server by @jasperjiaguo in #9727
#9518 added pinot helm 0.2.6 with secure version pinot 0.11.0 by @bagipriyank in #9519
Optimize AdaptiveServerSelection for replicaGroup based routing by @vvivekiyer in #9803
Do not sort the instances in InstancePartitions by @Jackie-Jiang in #9866
Support disabling dictionary at runtime for an existing column by @vvivekiyer in #9868
In Kafka consumer, seek offset only when needed by @Jackie-Jiang in #9896
Allow liveness check during server shutting down by @Jackie-Jiang in #9915
Add back the Helix workaround for missing IS change by @Jackie-Jiang in #9921
Allow collections for MV transform functions by @saurabhd336 in #9908
Construct new IndexLoadingConfig when loading completed real-time segments by @vvivekiyer in #9938
feat: add compressed file support for ORCRecordReader by @etolbakov in #9884
enable MergeRollupTask on real-time tables by @zhtaoxiang in #9890
Update cardinality when converting raw column to dict based by @vvivekiyer in #9875
allow gauge stored in metric registry to be updated by @zhtaoxiang in #9961
Handle in-memory segment metadata for index checking by @Jackie-Jiang in #10017
report minion task metadata last update time as metric by @zhtaoxiang in #9954
emit minion task generation time and error metrics by @zhtaoxiang in #10026
Use the same default time value for all replicas by @Jackie-Jiang in #10029
Reduce the number of segments to wait for convergence when rebalancing by @saurabhd336 in #10028
UI Update & Improvement
Allow hiding query console tab based on cluster config (#9261)
Allow hiding pinot broker swagger UI by config (#9343)
Add UI to show fine-grained minion task progress (#9488)
Add UI to track segment reload progress (#9521)
Show minion task runtime config details in UI (#9652)
Redefine the segment status (#9699)
Show an option to reload the segments during edit schema (#9762)
Load schema UI async (#9781)
Fix blank screen when redirect to unknown app route (#9888)
Library version upgrade
Upgrade h3 lib from 3.7.2 to 4.0.0 to lower glibc requirement (#9335)
Upgrade ZK version to 3.6.3 (#9612)
Upgrade snakeyaml from 1.30 to 1.33 (#9464)
Upgrade RoaringBitmap from 0.9.28 to 0.9.35 (#9730)
Upgrade spotless-maven-plugin from 2.9.0 to 2.28.0 (#9877)
Upgrade decode-uri-component from 0.2.0 to 0.2.2 (#9941)
BugFixes
Fix message count by @Jackie-Jiang in #9271
Fix issue with auth AccessType in Schema REST endpoints by @sajjad-moradi in #9293
Fix PerfBenchmarkRunner to skip the tmp dir by @Jackie-Jiang in #9298
Fix thrift deserializer thread safety issue by @saurabhd336 in #9299
Fix transformation to string for BOOLEAN and TIMESTAMP by @Jackie-Jiang in #9287
[hotfix] Add VARBINARY column to switch case branch by @walterddr in #9313
Fix annotation for "/recommender" endpoint by @sajjad-moradi in #9326
fix histogram null pointer exception by @jasperjiaguo in #9428
Fix thread safety issues with SDF (WIP) by @saurabhd336 in #9425
Fix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator by @Jackie-Jiang in #9434
Fix the bug of hybrid table request using the same request id by @Jackie-Jiang in #9443
Fix the range check for range index on raw column by @Jackie-Jiang in #9453
Fix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction by @ankitsultana in #9461
fix null transform bound check by @walterddr in #9495
Fix JsonExtractScalar when no value is extracted by @Jackie-Jiang in #9500
Fix some type convert scalar functions by @Jackie-Jiang in #9509
Fix timestamp index on column of preserved key by @Jackie-Jiang in #9533
Fix record extractor when ByteBuffer can be reused by @Jackie-Jiang in #9549
Fix time validation when data type needs to be converted by @Jackie-Jiang in #9569
UI: fix incorrect task finish time by @jayeshchoudhary in #9557
Fix the bug where uploaded segments cannot be deleted on real-time table by @Jackie-Jiang in #9579
[bugfix] correct the dir for building segments in FileIngestionHelper by @zhtaoxiang in #9591
Fix NonAggregationGroupByToDistinctQueryRewriter by @Jackie-Jiang in #9605
fix distinct result return by @walterddr in #9582
Fix GcsPinotFS by @lfernandez93 in #9556
fix DataSchema thread-safe issue by @walterddr in #9619
Fix re-uploading segment when the previous upload failed by @Jackie-Jiang in #9631
Fix string split which should be on whole separator by @Jackie-Jiang in #9650
Fix server request sent delay to be non-negative by @Jackie-Jiang in #9656
Fix extra restlet resource test which should be stateless by @Jackie-Jiang in #9674
AdaptiveServerSelection: Fix timer by @vvivekiyer in #9697
Fix the setup for ControllerTest shared cluster by @Jackie-Jiang in #9704
[hotfix]groovy class cache leak by @walterddr in #9716
Fix TIMESTAMP index handling in SegmentMapper by @Jackie-Jiang in #9722
Fix the server admin endpoint cache to reflect the config changes by @Jackie-Jiang in #9734
[bugfix] fix case-when issue by @walterddr in #9702
[bugfix] Let StartControllerCommand also handle "pinot.zk.server", "pinot.cluster.name" in default conf/pinot-controller.conf by @thangnd197 in #9739
[hotfix] semi-join opt by @walterddr in #9779
UI: show segment debug details when segment is in bad state by @jayeshchoudhary in #9700
Fix NPE when reading ZK address from controller config by @Jackie-Jiang in #9751
[bugfix] fix mergeRollupTask metrics by @zhtaoxiang in #9864
Fix issues for real-time table reload by @Jackie-Jiang in #9885
UI: fix segment status color remains same in different table page by @jayeshchoudhary in #9891
Fix bloom filter creation on BYTES by @Jackie-Jiang in #9898
[hotfix] broker selection not using table name by @walterddr in #9902
Fix ser/de for StringLongPair by @Jackie-Jiang in #9985
Bugfix: Use correct exception import in TableRebalancer. by @mayankshriv in #10025
Fix NPE in AbstractMetrics From Race Condition by @ankitsultana in #10022
Last updated