release-1.2.0

0.12.0

Multi-Stage Query Engine

New join semantics support

New sql semantics support:

Performance enhancement

  • Thread safe query planning (#9344)

  • Partial query execution and round robin scheduling (#9753)

  • Improve data table serde (#9731)

Major updates

  • Force commit consuming segments by @sajjad-moradi in #9197

  • add a freshness based consumption status checker by @jadami10 in #9244

  • Add metrics to track controller segment download and upload requests in progress by @gviedma in #9258

  • Adding endpoint to download local log files for each component by @xiangfu0 in #9259

  • [Feature] Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. by @61yao in #9265

  • add query cancel APIs on controller backed by those on brokers by @klsince in #9276

  • Add Spark Job Launcher tool by @KKcorps in #9288

  • Enable Consistent Data Push for Standalone Segment Push Job Runners by @yuanbenson in #9295

  • Allow server to directly return the final aggregation result by @Jackie-Jiang in #9304

  • TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in #9306

  • Adaptive Server Selection by @vvivekiyer in #9311

  • [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in #9312

  • Allow ingestion of errored records with incorrect datatype by @KKcorps in #9320

  • Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356

  • skip late cron job with max allowed delay by @klsince in #9372

  • Do not allow implicit cast for BOOLEAN and TIMESTAMP by @Jackie-Jiang in #9385

  • Add missing properties in CSV plugin by @KKcorps in #9399

  • set MDC so that one can route minion task logs to separate files cleanly by @klsince in #9400

  • Add a new API to fix segment date time in metadata by @KKcorps in #9413

  • Update get bytes to return raw bytes of string and support getBytesMV by @61yao in #9441

  • Exposing consumer's record lag in /consumingSegmentsInfo by @navina in #9515

  • Do not create dictionary for high-cardinality columns by @KKcorps in #9527

  • get task runtime configs tracked in Helix by @klsince in #9540

  • Add more options to json index by @Jackie-Jiang in #9543

  • add SegmentTierAssigner and refine restful APIs to get segment tier info by @klsince in #9598

  • Add segment level debug API by @saurabhd336 in #9609

  • Add record availability lag for Kafka connector by @navina in #9621

  • notify servers that need to move segments to new tiers via SegmentReloadMessage by @klsince in #9624

  • Allow to configure multi-datadirs as instance configs and a Quickstart example about them by @klsince in #9705

  • Customize stopword for Lucene Index by @jasperjiaguo in #9708

  • Add memory optimized dimension table by @KKcorps in #9802

  • ADLS file system upgrade by @xiangfu0 in #9855

  • Added Delete Schema/Table pinot admin commands by @bagipriyank in #9857

  • Adding new ADLSPinotFS auth type: DEFAULT by @xiangfu0 in #9860

  • Add rate limit to Kinesis requests by @KKcorps in #9863

  • Adding configs for zk client timeout by @xiangfu0 in #9975

Other features/changes

  • Show most recent scheduling errors by @satishwaghela in #9161

  • Do not use aggregation result for distinct query in IntermediateResultsBlock by @Jackie-Jiang in #9262

  • Emit metrics for ratio of actual consumption rate to rate limit in real-time tables by @sajjad-moradi in #9201

  • add metrics entry offlineTableCount by @walterddr in #9270

  • refine query cancel resp msg by @klsince in #9242

  • add @ManualAuthorization annotation for non-standard endpoints by @apucher in #9252

  • Optimize ser/de to avoid using output stream by @Jackie-Jiang in #9278

  • Add Support for Covariance Function by @SabrinaZhaozyf in #9236

  • Throw an exception when MV columns are present in the order-by expression list in selection order-by only queries by @somandal in #9078

  • Improve server query cancellation and timeout checking during execution by @jasperjiaguo in #9286

  • Add capabilities to ingest from another stream without disabling the real-time table by @sajjad-moradi in #9289

  • Add minMaxInvalid flag to avoid unnecessary needPreprocess by @npawar in #9238

  • Add array cardinality function by @walterddr in #9300

  • TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by @klsince in #9306

  • Add support for custom null values in CSV record reader by @KKcorps in #9318

  • Infer parquet reader type based on file metadata by @saurabhd336 in #9294

  • Add Support for Cast Function on MV Columns by @SabrinaZhaozyf in #9296

  • Allow ingestion of errored records with incorrect datatype by @KKcorps in #9320

  • [Feature] Not Operator Transformation by @61yao in #9330

  • Handle null string in CSV decoder by @KKcorps in #9340

  • [Feature] Not scalar function by @61yao in #9338

  • Add support for EXTRACT syntax and converts it to appropriate Pinot expression by @tanmesh in #9184

  • Add support for Auth in controller requests in java query client by @KKcorps in #9230

  • delete all related minion task metadata when deleting a table by @zhtaoxiang in #9339

  • BloomFilterRule should only recommend for supported column type by @yuanbenson in #9364

  • Support all the types in ParquetNativeRecordReader by @xiangfu0 in #9352

  • Improve segment name check in metadata push by @zhtaoxiang in #9359

  • Allow expression transformer cotinue on error by @xiangfu0 in #9376

  • skip late cron job with max allowed delay by @klsince in #9372

  • Enhance and filter predicate evaluation efficiency by @jasperjiaguo in #9336

  • Deprecate instanceId Config For Broker/Minion Specific Configs by @ankitsultana in #9308

  • Optimize combine operator to fully utilize threads by @Jackie-Jiang in #9387

  • Terminate the query after plan generation if timeout by @jasperjiaguo in #9386

  • [Feature] Support IsDistinctFrom and IsNotDistinctFrom by @61yao in #9312

  • [Feature] Support Coalesce for Column Names by @61yao in #9327

  • Disable logging for interrupted exceptions in kinesis by @KKcorps in #9405

  • Benchmark thread cpu time by @jasperjiaguo in #9408

  • Use ISODateTimeFormat as default for SIMPLE_DATE_FORMAT by @KKcorps in #9378

  • Extract the common logic for upsert metadata manager by @Jackie-Jiang in #9435

  • Make minion task metadata manager methods more generic by @saurabhd336 in #9436

  • Always pass clientId to kafka's consumer properties by @navina in #9444

  • Adaptive Server Selection by @vvivekiyer in #9311

  • Refine IndexHandler methods a bit to make them reentrant by @klsince in #9440

  • use MinionEventObserver to track finer grained task progress status on worker by @klsince in #9432

  • Allow spaces in input file paths by @KKcorps in #9426

  • Add support for gracefully handling the errors while transformations by @KKcorps in #9377

  • Cache Deleted Segment Names in Server to Avoid SegmentMissingError by @ankitsultana in #9423

  • Handle Invalid timestamps by @KKcorps in #9355

  • refine minion worker event observer to track finer grained progress for tasks by @klsince in #9449

  • spark-connector should use v2/brokers endpoint by @itschrispeck in #9451

  • Remove netty server query support from presto-pinot-driver to remove pinot-core and pinot-segment-local dependencies by @xiangfu0 in #9455

  • Adaptive Server Selection: Address pending review comments by @vvivekiyer in #9462

  • track progress from within segment processor framework by @klsince in #9457

  • Decouple ser/de from DataTable by @Jackie-Jiang in #9468

  • collect file info like mtime, length while listing files for free by @klsince in #9466

  • Extract record keys, headers and metadata from Stream sources by @navina in #9224

  • [pinot-spark-connector] Bump spark connector max inbound message size by @cbalci in #9475

  • refine the minion task progress api a bit by @klsince in #9482

  • add parsing for AT TIME ZONE by @agavra in #9477

  • Eliminate explosion of metrics due to gapfill queries by @elonazoulay in #9490

  • ForwardIndexHandler: Change compressionType during segmentReload by @vvivekiyer in #9454

  • Introduce Segment AssignmentStrategy Interface by @GSharayu in #9309

  • Add query interruption flag check to broker groupby reduction by @jasperjiaguo in #9499

  • adding optional client payload by @walterddr in #9465

  • [feature] distinct from scalar functions by @61yao in #9486

  • Check data table version on server only for null handling by @Jackie-Jiang in #9508

  • Add docId and column name to segment read exception by @KKcorps in #9512

  • Sort scanning based operators by cardinality in AndDocIdSet evaluation by @jasperjiaguo in #9420

  • Do not fail CI when codecov upload fails by @Jackie-Jiang in #9522

  • [Upsert] persist validDocsIndex snapshot for Pinot upsert optimization by @deemoliu in #9062

  • broker filter by @dongxiaoman in #9391

  • [feature] coalesce scalar by @61yao in #9487

  • Allow setting custom time boundary for hybrid table queries by @saurabhd336 in #9356

  • [GHA] add cache timeout by @walterddr in #9524

  • Optimize PinotHelixResourceManager.hasTable() by @Jackie-Jiang in #9526

  • Include exception when upsert metadata manager cannot be created by @Jackie-Jiang in #9532

  • allow to config task expire time by @klsince in #9530

  • expose task finish time via debug API by @klsince in #9534

  • Remove the wrong warning log in KafkaPartitionLevelConsumer by @Jackie-Jiang in #9536

  • starting http server for minion worker conditionally by @klsince in #9542

  • Make StreamMessage generic and a bug fix by @vvivekiyer in #9544

  • Improve primary key serialization performance by @KKcorps in #9538

  • [Upsert] Skip removing upsert metadata when shutting down the server by @Jackie-Jiang in #9551

  • add array element at function by @walterddr in #9554

  • Handle the case when enableNullHandling is true and an aggregation function is used w/ a column that has an empty null bitmap by @nizarhejazi in #9566

  • Support segment storage format without forward index by @somandal in #9333

  • Adding SegmentNameGenerator type inference if not explicitly set in config by @timsants in #9550

  • add version information to JMX metrics & component logs by @agavra in #9578

  • remove unused RecordTransform/RecordFilter classes by @agavra in #9607

  • Support rewriting forward index upon changing compression type for existing raw MV column by @vvivekiyer in #9510

  • Support Avro's Fixed data type by @sajjad-moradi in #9642

  • [feature] [kubernetes] add loadBalancerSourceRanges to service-external.yaml for controller and broker by @jameskelleher in #9494

  • Limit up to 10 unavailable segments to be printed in the query exception by @Jackie-Jiang in #9617

  • remove more unused filter code by @agavra in #9620

  • Do not cache record reader in segment by @Jackie-Jiang in #9604

  • make first part of user agent header configurable by @rino-kadijk in #9471

  • optimize order by sorted ASC, unsorted and order by DESC cases by @gortiz in #8979

  • Enhance cluster config update API to handle non-string values properly by @Jackie-Jiang in #9635

  • Reverts recommender REST API back to PUT (reverts PR #9326) by @yuanbenson in #9638

  • Remove invalid pruner names from server config by @Jackie-Jiang in #9646

  • Using usageHelp instead of deprecated help in picocli commands by @navina in #9608

  • Handle unique query id on server by @Jackie-Jiang in #9648

  • stateless group marker missing several by @walterddr in #9673

  • Support reloading consuming segment using force commit by @Jackie-Jiang in #9640

  • Improve star-tree to use star-node when the predicate matches all the non-star nodes by @Jackie-Jiang in #9667

  • add FetchPlanner interface to decide what column index to prefetch by @klsince in #9668

  • Improve star-tree traversal using ArrayDeque by @Jackie-Jiang in #9688

  • Handle errors in combine operator by @Jackie-Jiang in #9689

  • return different error code if old version is not on master by @SabrinaZhaozyf in #9686

  • Support creating dictionary at runtime for an existing column by @vvivekiyer in #9678

  • check mutable segment explicitly instead of checking existence of indexDir by @klsince in #9718

  • Remove leftover file before downloading segmentTar by @npawar in #9719

  • add index key and size map to segment metadata by @walterddr in #9712

  • Use ideal state as source of truth for segment existence by @Jackie-Jiang in #9735

  • Close Filesystem on exit with Minion Tasks by @KKcorps in #9681

  • render the tables list even as the table sizes are loading by @jadami10 in #9741

  • Add Support for IP Address Function by @SabrinaZhaozyf in #9501

  • bubble up error messages from broker by @agavra in #9754

  • Add support to disable the forward index for existing columns by @somandal in #9740

  • show table metadata info in aggregate index size form by @walterddr in #9733

  • Preprocess immutable segments from REALTIME table conditionally when loading them by @klsince in #9772

  • revert default timeout nano change in QueryConfig by @agavra in #9790

  • AdaptiveServerSelection: Update stats for servers that have not responded by @vvivekiyer in #9801

  • Add null value index for default column by @KKcorps in #9777

  • [MergeRollupTask] include partition info into segment name by @zhtaoxiang in #9815

  • Adding a consumer lag as metric via a periodic task in controller by @navina in #9800

  • Deserialize Hyperloglog objects more optimally by @priyen in #9749

  • Download offline segments from peers by @wirybeaver in #9710

  • Thread Level Usage Accounting and Query Killing on Server by @jasperjiaguo in #9727

  • Add max merger and min mergers for partial upsert by @deemoliu in #9665

  • #9518 added pinot helm 0.2.6 with secure version pinot 0.11.0 by @bagipriyank in #9519

  • Combine the read access for replication config by @snleee in #9849

  • add v1 ingress in helm chart by @jhisse in #9862

  • Optimize AdaptiveServerSelection for replicaGroup based routing by @vvivekiyer in #9803

  • Do not sort the instances in InstancePartitions by @Jackie-Jiang in #9866

  • Merge new columns in existing record with default merge strategy by @navina in #9851

  • Support disabling dictionary at runtime for an existing column by @vvivekiyer in #9868

  • support BOOL_AND and BOOL_OR aggregate functions by @agavra in #9848

  • Use Pulsar AdminClient to delete unused subscriptions by @navina in #9859

  • add table sort function for table size by @jadami10 in #9844

  • In Kafka consumer, seek offset only when needed by @Jackie-Jiang in #9896

  • fallback if no broker found for the specified table name by @klsince in #9914

  • Allow liveness check during server shutting down by @Jackie-Jiang in #9915

  • Allow segment upload via Metadata in MergeRollup Minion task by @KKcorps in #9825

  • Add back the Helix workaround for missing IS change by @Jackie-Jiang in #9921

  • Allow uploading real-time segments via CLI by @KKcorps in #9861

  • Add capability to update and delete table config via CLI by @KKcorps in #9852

  • default to TAR if push mode is not set by @klsince in #9935

  • load startree index via segment reader interface by @klsince in #9828

  • Allow collections for MV transform functions by @saurabhd336 in #9908

  • Construct new IndexLoadingConfig when loading completed real-time segments by @vvivekiyer in #9938

  • Make GET /tableConfigs backwards compatible in case schema does not match raw table name by @timsants in #9922

  • feat: add compressed file support for ORCRecordReader by @etolbakov in #9884

  • Add Variance and Standard Deviation Aggregation Functions by @snleee in #9910

  • enable MergeRollupTask on real-time tables by @zhtaoxiang in #9890

  • Update cardinality when converting raw column to dict based by @vvivekiyer in #9875

  • Add back auth token for UploadSegmentCommand by @timsants in #9960

  • Improving gz support for avro record readers by @snleee in #9951

  • Default column handling of noForwardIndex and regeneration of forward index on reload path by @somandal in #9810

  • [Feature] Support coalesce literal by @61yao in #9958

  • Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly by @npawar in #9988

  • handle pending minion tasks properly when getting the task progress status by @klsince in #9911

  • allow gauge stored in metric registry to be updated by @zhtaoxiang in #9961

  • support case-insensitive query options in SET syntax by @agavra in #9912

  • pin versions-maven-plugin to 2.13.0 by @jadami10 in #9993

  • Pulsar Connection handler should not spin up a consumer / reader by @navina in #9893

  • Handle in-memory segment metadata for index checking by @Jackie-Jiang in #10017

  • Support the cross-account access using IAM role for S3 PinotFS by @snleee in #10009

  • report minion task metadata last update time as metric by @zhtaoxiang in #9954

  • support SKEWNESS and KURTOSIS aggregates by @agavra in #10021

  • emit minion task generation time and error metrics by @zhtaoxiang in #10026

  • Use the same default time value for all replicas by @Jackie-Jiang in #10029

  • Reduce the number of segments to wait for convergence when rebalancing by @saurabhd336 in #10028

UI Update & Improvement

  • Allow hiding query console tab based on cluster config (#9261)

  • Allow hiding pinot broker swagger UI by config (#9343)

  • Add UI to show fine-grained minion task progress (#9488)

  • Add UI to track segment reload progress (#9521)

  • Show minion task runtime config details in UI (#9652)

  • Redefine the segment status (#9699)

  • Show an option to reload the segments during edit schema (#9762)

  • Load schema UI async (#9781)

  • Fix blank screen when redirect to unknown app route (#9888)

Library version upgrade

  • Upgrade h3 lib from 3.7.2 to 4.0.0 to lower glibc requirement (#9335)

  • Upgrade ZK version to 3.6.3 (#9612)

  • Upgrade snakeyaml from 1.30 to 1.33 (#9464)

  • Upgrade RoaringBitmap from 0.9.28 to 0.9.35 (#9730)

  • Upgrade spotless-maven-plugin from 2.9.0 to 2.28.0 (#9877)

  • Upgrade decode-uri-component from 0.2.0 to 0.2.2 (#9941)

BugFixes

Previous0.12.1Next0.11.0