Note

Before upgrading from one version to another one, read the release notes. While the Pinot committers strive to keep releases backward-compatible and introduce new features in a compatible manner, your environment may have a unique combination of configurations/data/schema that may have been somehow overlooked. Before you roll out a new release of Pinot on your cluster, it is best that you run the compatibility test suite that Pinot provides. The tests can be easily customized to suit the configurations and tables in your pinot cluster(s). As a good practice, you should build your own test suite, mirroring the table configurations, schema, sample data, and queries that are used in your cluster.

1.0.0 (September 2023)

0.12.1 (March 2023)

0.12.0 (December 2022)

0.11.0 (September 2022)

0.10.0 (March 2022)

0.9.3 (December 2021)

0.9.2 (December 2021)

0.9.1 (December 2021)

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

Summary

This is a bug-fixing release contains:

• use legacy case-when format (https://github.com/apache/pinot/pull/10291)

The release is based on the release 0.12.0 with the following cherry-picks:

• 6f5a8fc883e1d576117fdb92f09103067672aaca

Summary

This is a bug fixing release contains:

• Update Log4j to 2.17.0 to address CVE-2021-45105 (#7933)

The release is based on the release 0.9.2 with the following cherry-picks:

• 93c0404

1.0.0 (2023-09-19)

This release includes the several new features, enhancements, and bug fixes, including the following highlights:

• Multi-stage query engine: new features, enhancements, and bug fixes. Learn how to enable and use the multi-stage query engine or more about how the multi-stage query engine works.

Multi-stage query engine new features

• Support for window functions

  • Initial (phase 1) Query runtime for window functions with ORDER BY within the OVER() clause (#10449)

  • Support for the ranking ROW_NUMBER() window function (#10527, #10587)

• Set operations support:

  • Support SetOperations (UNION, INTERSECT, MINUS) compilation in query planner (#10535)

• Timestamp and Date Operations

• Support TIMESTAMP type and date ops functions (#11350)

• Aggregate functions

  • Support more aggregation functions that are currently implementable (#11208)

  • Support multi-value aggregation functions (#11216)

• Support Sketch based functions (#11153), (#11517)

• Make Intermediate Stage Worker Assignment Tenant Aware (#10617)

• Evaluate literal expressions during query parsing, enabling more efficient query execution (#11438)

• Added support for partition parallelism in partitioned table scans, allowing for more efficient data retrieval (#11266)

• [multistage]Adding more tuple sketch scalar functions and integration tests (#11517)

Multi-stage query engine enhancements

• Turn on v2 engine by default (#10543)

• Introduced the ability to stream leaf stage blocks for more efficient data processing (#11472).

• Early terminate SortOperator if there is a limit (#11334)

• Implement ordering for SortExchange (#10408)

• Table level Access Validation, QPS Quota, Phase Metrics for multistage queries (#10534)

• Support partition based leaf stage processing (#11234)

• Populate queryOption down to leaf (#10626)

• Pushdown explain plan queries from the controller to the broker (#10505)

• Enhanced the multi-stage group-by executor to support limiting the number of groups,

• improving query performance and resource utilization (#11424).

• Improved resilience and reliability of the multi-stage join operator, now with added support for hash join right table protection (#11401).

Multi-stage query engine bug fixes

• Fix Predicate Pushdown by Using Rule Collection (#10409)

• Try fixing mailbox cancel race condition (#10432)

• Catch Throwable to Propagate Proper Error Message (#10438)

• Fix tenant detection issues (#10546)

• Handle Integer.MIN_VALUE in hashCode based FieldSelectionKeySelector (#10596)

• Improve error message in case of non-existent table queried from the controller (#10599)

• Derive SUM return type to be PostgreSQL compatible (#11151)

Index SPI

• Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details here

Null value support for pinot queries

• NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.

Upsert enhancements

Delete support in upsert enabled tables (#10703)

Support added to extend upserts and allow deleting records from a realtime table. The design details can be found here.

Preload segments with upsert snapshots to speedup table loading (#11020)

Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).

TTL configs for upsert primary keys (#10915)

Adds support for specifying expiry TTL for upsert primary key metadata cleanup.

Segment compaction for upsert real-time tables (#10463)

Adds a new minion task to compact segments belonging to a real-time table with upserts.

Pinot Spark Connector for Spark3 (#10394)

• Added spark3 support for Pinot Spark Connector (#10394)

• Also added support to pass pinot query options to spark connector (#10443)

PinotDataBufferFactory and new PinotDataBuffer implementations (#10528)

Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.

Query functions enhancements

• Add PercentileKLL aggregation function (#10643)

• Support for ARG_MIN and ARG_MAX Functions (#10636)

• refactor argmin/max to exprmin/max and make it calcite compliant (#11296)

• Integer Tuple Sketch support (#10427)

• Adding vector scalar functions (#11222)

• [feature] multi-value datetime transform variants (#10841)

• FUNNEL_COUNT Aggregation Function (#10867)

• [multistage] Add support for RANK and DENSE_RANK ranking window functions (#10700)

• add theta sketch scalar (#11153)

• Register dateTimeConverter,timeConvert,dateTrunc, regexpReplace to v2 functions (#11097)

• Add extract(quarter/dow/doy) support (#11388)

• Funnel Count - Multiple Strategies (no partitioning requisites) (#11092)

• Add Boolean assertion transform functions. (#11547)

JSON and CLP encoded message ingestion and querying

• Add clpDecode transform function for decoding CLP-encoded fields. (#10885)

• Add CLPDecodeRewriter to make it easier to call clpDecode with a column-group name rather than the individual columns. (#11006)

• Add SchemaConformingTransformer to transform records with varying keys to fit a table's schema without dropping fields. (#11210)

Tier level index config override (#10553)

• Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.

Ingestion connectors and features

• Kinesis stream header extraction (#9713)

• Extract record keys, headers and metadata from Pulsar sources (#10995)

• Realtime pre-aggregation for Distinct Count HLL & Big Decimal (#10926)

• Added support to skip unparseable records in the csv record reader (#11487)

• Null support for protobuf ingestion. (#11553)

UI enhancements

• Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires (#10389)

• AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. (#10925)

Pinot docker image improvements and enhancements

• Make Pinot base build and runtime images support Amazon Corretto and MS OpenJDK (#10422)

• Support multi-arch pinot docker image (#10429)

• Update dockerfile with recent jdk distro changes (#10963)

Operational improvements

Rebalance

• Rebalance status API (#10359)

• Tenant level rebalance API Tenant rebalance and status tracking APIs (#11128)

Config to use customized broker query thread pool (#10614)

Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.

pinot.broker.enable.bounded.http.async.executor
pinot.broker.http.async.executor.max.pool.size
pinot.broker.http.async.executor.core.pool.size
pinot.broker.http.async.executor.queue.size

This feature allows better management of broker resources.

Drop results support (#10419)

Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.

Make column order deterministic in segment (#10468)

In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.

Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

Adds options to configure helix timeouts external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.

Enable case insensitivity by default (#10771)

This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql

Newly added APIs and client methods

• Add Server API to get tenant pools (#11273)

• Add new broker query point for querying multi-stage engine (#11341)

• Add a new controller endpoint for segment deletion with a time window (#10758)

• New API to get tenant tags (#10937)

• Instance retag validation check api (#11077)

• Use PUT request to enable/disable table/instance (#11109)

• Update the pinot tenants tables api to support returning broker tagged tables (#11184)

• Add requestId for BrokerResponse in pinot-broker and java-client (#10943)

• Provide results in CompletableFuture for java clients and expose metrics (#10326)

Cleanup and backward incompatible changes

High level consumers are no longer supported

• Cleanup HLC code (#11326)

• Remove support for High level consumers in Apache Pinot (#11017)

Type information preservation of query literals

• [feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform (#10380) String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.

Controller job status ZNode path update

• Moving Zk updates for reload, force_commit to their own Znodes which … (#10451) The status of previously completed reload jobs will not be available after this change is deployed.

Metric names for mutable indexes to change

• Implement mutable index using index SPI (#10687) Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.

Update in controller API to enable / disable / drop instances

• Update getTenantInstances call for controller and separate POST operations on it (#10993)

Change in substring query function definition

• Change substring to comply with standard sql definition (#11502)

Full list of features added

• Allow queries on multiple tables of same tenant to be executed from controller UI #10336

• Encapsulate changes in IndexLoadingConfig and SegmentGeneratorConfig #10352

• [Index SPI] IndexType (#10191)

• Simplify filtered aggregate transform operator creation (#10410)

• Introduce BaseProjectOperator and ValueBlock (#10405)

• Add support to create realtime segment in local (#10433)

• Refactor: Pass context instead on individual arguments to operator (#10413)

• Add "processAll" mode for MergeRollupTask (#10387)

• Upgrade h2 version from 1.x to 2.x (#10456)

• Added optional force param to the table configs update API (#10441)

• Enhance broker reduce to handle different column names from server response (#10454)

• Adding fields to enable/disable dictionary optimization. (#10484)

• Remove converted H2 type NUMERIC(200, 100) from BIG_DECIMAL (#10483)

• Add JOIN support to PinotQuery (#10421)

• Add testng on verifier (#10491)

• Clean up temp consuming segment files during server start (#10489)

• make pinot k8s sts and deployment start command configurable (#10509)

• Fix Bottleneck for Server Bootstrap by Making maxConnsPerRoute Configurable (#10487)

• Type match between resultType and function's dataType (#10472)

• create segment zk metadata cache (#10455)

• Allow ValueBlock length to increase in TransformFunction (#10515)

• Allow configuring helix timeouts for EV dropped in Instance manager (#10510)

• Enhance error reporting (#10531)

• Combine "GET /segments" API & "GET /segments/{tableName}/select" (#10412)

• Exposed the CSV header map as part of CSVRecordReader (#10542)

• Moving Zk updates for reload,force_commit to their own Znodes which will spread out Zk write load across jobTypes (#10451)

• Enabling dictionary override optimization on the segment reload path as well. (#10557)

• Make broker's rest resource packages configurable (#10588)

• Check EV not exist before allowing creating the table (#10593)

• Adding an parameter (toSegments) to the endSegmentReplacement API (#10630)

• update target tier for segments if tierConfigs is provided (#10642)

• Add support for custom compression factor for Percentile TDigest aggregation functions (#10649)

• Utility to convert table config into updated format (#10623)

• Segment lifecycle event listener support (#10536)

• Add server metrics to capture gRPC activity (#10678)

• Separate and parallelize BloomFilter based semgment pruner (#10660)

• API to expose the contract/rules imposed by pinot on tableConfig #10655

• Add description field to metrics in Pinot (#10744)

• changing the dedup store to become pluggable #10639

• Make the TimeUnit in the DATETRUNC function case insensitive. (#10750)

• [feature] Consider tierConfigs when assigning new offline segment #10746

• Compress idealstate according to estimated size #10766

• 10689: Update for pinot helm release version 0.2.7 (#10723)

• Fail the query if a filter's rhs contains NULL. (#11188)

• Support Off Heap for Native Text Indices (#10842)

• refine segment reload executor to avoid creating threads unbounded #10837

• compress nullvector bitmap upon seal (#10852)

• Enable case insensitivity by default (#10771)

• Push out-of-order events metrics for full upsert (#10944)

• [feature] add requestId for BrokerResponse in pinot-broker and java-client #10943

• Provide results in CompletableFuture for java clients and expose metrics #10326

• Add minion observability for segment upload/download failures (#10978)

• Enhance early terminate for combine operator (#10988)

• Add fromController method that accepts a PinotClientTransport (#11013)

• Ensure min/max value generation in the segment metadata. (#10891)

• Apply some allocation optimizations on GrpcSendingMailbox (#11015)

• When enable case-insensitive, don't allow to add newly column name which have the same lowercase name with existed columns. (#10991)

• Replace Long attributes with primitive values to reduce boxing (#11059)

• retry KafkaConsumer creation in KafkaPartitionLevelConnectionHandler.java (#253) (#11040)

• Support for new dataTime format in DateTimeGranularitySpec without explicitly setting size (#11057)

• Returning 403 status code in case of authorization failures (#11136)

• Simplify compatible test to avoid test against itself (#11163)

• Updated code for setting value of segment min/max property. (#10990)

• Add stat to track number of segments that have valid doc id snapshots (#11110)

• Add brokerId and brokerReduceTimeMs to the broker response stats (#11142)

• safely multiply integers to prevent overflow (#11186)

• Move largest comparison value update logic out of map access (#11157)

• Optimize DimensionTableDataManager to abort unnecesarry loading (#11192)

• Refine isNullsLast and isAsc functions. (#11199)

• Update the pinot tenants tables api to support returning broker tagged tables (#11184)

• add multi-value support for native text index (#11204)

• Add percentiles report in QuerySummary (#11299)

• Add meter for broker responses with unavailable segments (#11301)

• Enhance Minion task management (#11315)

• add additional lucene index configs (#11354)

• Add DECIMAL data type to orc record reader (#11377)

• add configuration to fail server startup on non-good status checker (#11347)

• allow passing freshness checker after an idle threshold (#11345)

• Add broker validation for hybrid tableConfig creation (#7908)

• Support partition parallelism for partitioned table scan (#11266)

Vulnerability fixes, bugfixes, cleanups and deprecations

• Remove support for High level consumers in Apache Pinot (#11017)

• Fix JDBC driver check for username (#10416)

• [Clean up] Remove getColumnName() from AggregationFunction interface (#10431)

• fix jersey TerminalWriterInterceptor MessageBodyWriter not found issue (#10462)

• Bug fix: Start counting operator execution time from first NoOp block (#10450)

• Fix unavailable instances issues for StrictReplicaGroup (#10466)

• Change shell to bash (#10469)

• Fix the double destroy of segment data manager during server shutdown (#10475)

• Remove "isSorted()" precondition check in the ForwardIndexHandler (#10476)

• Fix null handling in streaming selection operator (#10453)

• Fix jackson dependencies (#10477)

• Startree index build enhancement (#10905)

• optimize queries where lhs and rhs of predicate are equal (#10444)

• Trivial fix on a warning detected by static checker (#10492)

• wait for full segment commit protocol on force commit (#10479)

• Fix bug and add test for noDict -> Dict conversion for sorted column (#10497)

• Make column order deterministic in segment (#10468)

• Type match between resultType and function's dataType (#10472)

• Allow empty segmentsTo for segment replacement protocol (#10511)

• Use string as default compatible type for coalesce (#10516)

• Use threadlocal variable for genericRow to make the MemoryOptimizedTable threadsafe (#10502)

• Fix shading in spark2 connector pom file (#10490)

• Fix ramping delay caused by long lasting sequence of unfiltered messa… (#10418)

• Do not serialize metrics in each Operator (#10473)

• Make pinot-controller apply webpack production mode when bin-dist profile is used. (#10525)

• Fix FS props handling when using /ingestFromUri (#10480)

• Clean up v0_deprecated batch ingestion jobs (#10532)

• Deprecate kafka 0.9 support (#10522)

• safely multiply integers to prevent overflow (#11186)

• Reduce timeout for codecov and not fail the job in any case (#10547)

• Fix DataTableV3 serde bug for empty array (#10583)

• Do not record operator stats when tracing is enabled (#10447)

• Forward auth token for logger APIs from controller to other controllers and brokers (#10590)

• Bug fix: Partial upsert default strategy is null (#10610)

• Fix flaky test caused by EV check during table creation (#10616)

• Fix withDissabledTrue typo (#10624)

• Cleanup unnecessary mailbox id ser/de (#10629)

• no error metric for queries where all segments are pruned (#10589)

• bug fix: to keep QueryParser thread safe when handling many read requests on class RealtimeLuceneTextIndex (#10620)

• Fix static DictionaryIndexConfig.DEFAULT_OFFHEAP being actually onheap (#10632)

• 10567: [cleanup pinot-integration-test-base], clean query generations and some other refactoring. (#10648)

• Fixes backward incompatability with SegmentGenerationJobSpec for segment push job runners (#10645)

• Bug fix to get the toSegments list correctly (#10659)

• 10661: Fix for failing numeric comparison in where clause for IllegalStateException. (#10662)

• Fixes partial upsert not reflecting multiple comparison column values (#10693)

• Fix Bug in Reporting Timer Value for Min Consuming Freshness (#10690)

• Fix typo of rowSize -> columnSize (#10699)

• update segment target tier before table rebalance (#10695)

• Fix a bug in star-tree filter operator which can incorrecly filter documents (#10707)

• Enhance the instrumentation for a corner case where the query doesn't go through DocIdSetOp (#10729)

• bug fix: add missing properties when edit instance config (#10741)

• Making segmentMapper do the init and cleanup of RecordReader (#10874)

• Fix githubEvents table for quickstart recipes (#10716)

• Minor Realtime Segment Commit Upload Improvements (#10725)

• Return 503 for all interrupted queries. Refactor the query killing code. (#10683)

• Add decoder initialization error to the server's error cache (#10773)

• bug fix: add @JsonProperty to SegmentAssignmentConfig (#10759)

• ensure we wait the full no query timeout before shutting down (#10784)

• Clean up KLL functions with deprecated convention (#10795)

• Redefine the semantics of SEGMENT_STREAMED_DOWNLOAD_UNTAR_FAILURES metric to count individual segment fetch failures. (#10777)

• fix excpetion during exchange routing causes stucked pipeline (#10802)

• [bugfix] fix floating point and integral type backward incompatible issue (#10650)

• [pinot-core] Start consumption after creating segment data manager (#11227)

• Fix IndexOutOfBoundException in filtered aggregation group-by (#11231)

• Fix null pointer exception in segment debug endpoint #11228

• Clean up RangeIndexBasedFilterOperator. (#11219)

• Fix the escape/unescape issue for property value in metadata (#11223)

• Fix a bug in the order by comparator (#10818)

• Keeps nullness attributes of merged in comparison column values (#10704)

• Add required JSON annotation in H3IndexResolution (#10792)

• Fix a bug in SELECT DISTINCT ORDER BY. (#10827)

• jsonPathString should return null instead of string literal "null" (#10855)

• Bug Fix: Segment Purger cannot purge old segments after schema evolution (#10869)

• Fix #10713 by giving metainfo more priority than config (#10851)

• Close PinotFS after Data Manager Shutdowns (#10888)

• bump awssdk version for a bugfix on http conn leakage (#10898)

• Fix MultiNodesOfflineClusterIntegrationTest.testServerHardFailure() (#10909)

• Fix a bug in SELECT DISTINCT ORDER BY LIMIT. (#10887)

• Fix an integer overflow bug. (#10940)

• Return true when _resultSet is not null (#10899)

• Fixing table name extraction for lateral join queries (#10933)

• Fix casting when prefetching mmap'd segment larger than 2GB (#10936)

• Null check before closing reader (#10954)

• Fixes SQL wildcard escaping in LIKE queries (#10897)

• [Clean up] Do not count DISTINCT as aggregation (#10985)

• do not readd lucene readers to queue if segment is destroyed #10989

• Message batch ingestion lag fix (#10983)

• Fix a typo in snapshot lock (#11007)

• When extracting root-level field name for complex type handling, use the whole delimiter (#11005)

• update jersey to fix Denial of Service (DoS) (#11021)

• Update getTenantInstances call for controller and separate POST operations on it (#10993)

• update freemaker to fix Server-side Template Injection (#11019)

• format double 0 properly to compare with h2 results (#11049)

• Fix double-checked locking in ConnectionFactory (#11014)

• Remove presto-pinot-driver and pinot-java-client-jdk8 module (#11051)

• Make RequestUtils always return a string array when getTableNames (#11069)

• Fix BOOL_AND and BOOL_OR result type (#11033)

• [cleanup] Consolidate some query and controller/broker methods in integration tests (#11064)

• Fix grpc regression on multi-stage engine (#11086)

• Delete an obsolete TODO. (#11080)

• Minor fix on AddTableCommand.toString() (#11082)

• Allow using Lucene text indexes on mutable MV columns. (#11093)

• Allow offloading multiple segments from same table in parallel (#11107)

• Added serviceAccount to minion-stateless (#11095)

• Bug fix: TableUpsertMetadataManager is null (#11129)

• Fix reload bug (#11131)

• Allow extra aggregation types in RealtimeToOfflineSegmentsTask (#10982)

• Fix a bug when use range index to solve EQ predicate (#11146)

• Sanitise API inputs used as file path variables (#11132)

• Fix NPE when nested query doesn't have gapfill (#11155)

• Fix the NPE when query response error stream is null (#11154)

• Make interface methods non private, for java 8 compatibility (#11164)

• Increment nextDocId even if geo indexing fails (#11158)

• Fix the issue of consuming segment entering ERROR state due to stream connection errors (#11166)

• In TableRebalancer, remove instance partitions only when reassigning instances (#11169)

• Remove JDK 8 unsupported code (#11176)

• Fix compat test by adding -am flag to build pinot-integration-tests (#11181)

• dont duplicate register scalar function in CalciteSchema (#11190)

• Fix the storage quota check for metadata push (#11193)

• Delete filtering NULL support dead code paths. (#11198)

• [bugfix] Do not move real-time segments to working dir on restart (#11226)

• Fix a bug in ExpressionScanDocIdIterator for multi-value. (#11253)

• Exclude NULLs when PredicateEvaluator::isAlwaysTrue is true. (#11261)

• UI: fix sql query options seperator (#10770)

• Fix a NullPointerException bug in ScalarTransformFunctionWrapper. (#11309)

• [refactor] improve disk read for partial upsert handler (#10927)

• Fix the wrong query time when the response is empty (#11349)

• getMessageAtIndex should actually return the value in the streamMessage for compatibility (#11355)

• Remove presto jdk8 related dependencies (#11285)

• Remove special routing handling for multiple consuming segments (#11371)

• Properly handle shutdown of TableDataManager (#11380)

• Fixing the stale pinot ServerInstance in _tableTenantServersMap (#11386)

• Fix the thread safety issue for mutable forward index (#11392)

• Fix RawStringDistinctExecutor integer overflow (#11403)

• [logging] fix consume rate logging bug to respect 1 minute threshold (#11421)

Summary

This is a bug fixing release contains:

• Upgrade log4j to 2.16.0 to fix ()

• Upgrade swagger-ui to 3.23.11 to fix ()

• Fix the bug that RealtimeToOfflineTask failed to progress with large time bucket gaps ().

The release is based on the release 0.9.1 with the following cherry-picks:

Summary

Apache Pinot 0.11.0 has introduced many new features to extend the query abilities, e.g. the Multi-Stage query engine enables Pinot to do distributed joins, more sql syntax(DML support), query functions and indexes(Text index, Timestamp index) supported for new use cases. And as always, more integrations with other systems(E.g. Spark3, Flink).

Multi-Stage Query Engine

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as JOIN, OVER window, MATCH_RECOGNIZE and eventually, make Pinot support closer to full ANSI SQL semantics. More to read:

Pause Stream Consumption on Apache Pinot

Pinot operators can pause real-time consumption of events while queries are being executed, and then resume consumption when ready to do so again.\

Gap-filling function

Long waiting feature for segment generation on Spark 3.x.

Similar to the Spark Pinot connector, this allows Flink users to dump data from the Flink application to Pinot.

This feature allows better fine-grained control on pinot queries.

Wanna search text in real time? The new text indexing engine in Pinot supports the following capabilities:

1. New operator: LIKE

1. New operator: CONTAINS

1. Native text index, built from the ground up, focusing on Pinot’s time series use cases and utilizing existing Pinot indices and structures(inverted index, bitmap storage).

2. Real Time Text Index

This feature supports enabling deduplication for real-time tables, via a top-level table config. At a high level, primaryKey (as defined in the table schema) hashes are stored into in-memory data structures, and each incoming row is validated against it. Duplicate rows are dropped.

The expectation while using this feature is for the stream to be partitioned by the primary key, strictReplicaGroup routing to be enabled, and the configured stream consumer type to be low level. These requirements are therefore mandated via table config API's input validations.

Functions support and changes:

The full list of features introduced in this release

Vulnerability fixs

Pinot has resolved all the high-level vulnerabilities issues:

Bug fixs

Summary

This release fixes the major issue of and a major bug fixing of pinot admin exit code issue().

The release is based on the release 0.9.0 with the following cherry-picks:

Summary

This release comes with several features, SQL /UI/Perf enhancements Bugfixes across areas ranging from Multistage Query Engine to Ingestion, Storage format, SQL support, etc.

Multi-stage Query Engine

Features

Support RelDistribution-based trait Planning (#11976, #12079)

• Adds support for RelDistribution optimization for more accurate leaf-stage direct exchange/shuffle. Also extends partition optimization beyond leaf stage to entire query plan.

• Applies optimization based on distribution trait in the mailbox/worker assignment stage

  • Fixes previous direct exchange which was decided based on the table partition hint. Now direct exchange is decided via distribution trait: it will applied if-and-only-if the trait propagated matches the exchange requirement.

  • As a side effect, is_colocated_by_join_keysquery option is reintroduced to ensure dynamic broadcast which can also benefit from direct exchange optimization

  • Allows propagation of partition distribution trait info across the tree to be used during Physical Planning phase. It can be used in the following scenarios (will follow up in separate PRs)

• Note on backward incompatbility

  • is_colocated_by_join_keys hint is now required for making colocated joins

    • it should only affect semi-join b/c it is the only one utilizing broadcast exchange but were pulled to act as direct exchange.

    • inner/left/right/full join should automatically apply colocation thus the backward incompatibility should not affect these.

Leaf stage planning with multi-semi join support (#11937)

• Solves the limitation of pinotQuery that supports limited amount of PlanNodes.

• Splits the ServerRequest planning into 2 stages

  • First plan as much as possible into PinotQuery

  • for any remainder nodes that cannot be planned into PinotQuery, will be run together with the LeafStageTransferrableBlockOperator as the input locally.

Support for ArrayAgg aggregation function (#11822)

• Usage: ArrayAgg(column, 'dataType' [, 'isDistinct'])

• Float type column is treated as Double in the multistage engine, so FLOAT type is not supported.

• Supports data BOOLEAN, INT, LONG, FLOAT(only in V1), DOUBLE, STRING, TIMESTAMP. E.g. ArrayAgg(intCol, 'INT') returns ARRAY<INT>

Enhancements

• Canonicalize SqlKind.OTHERS and SqlKind.OTHER_FUNCTIONS and support

  concat as || operator (#12025)

• Capability for constant filter in QueryContext, with support for server to handle it (#11956)

• Tests for filter pushdown (#11994)

• Enhancements to query plan tests (#11966)

• Refactor PlanFragmenter to make the logic clear (#11912)

• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)

  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

• Security enhancement to add RBAC authorization checks for multi-stage query engine (#11830)

• Enhancement to leaf-stage execution stats NPE handling (#11805)

• Enhancement to add a framework to back-propagate metadata across opChains (#11746)

• Use of BinaryArray to wire proto for multi-stage engine bytes literal handling (#11738)

• Enable dynamic broadcast for SEMI joins. Adds a fallback option to enable hash table join using joinOptions(join_strategy = 'hash_table')(#11696)

• Improvements to dispatch exception handling (#11688)

• Allow malformed dateTime string to return default value configurable in the function signature (#11258)

  • Copy

    fromDateTime(colContainsMalformedStr, '<dateTimeFormat>', '<timezone>', <default_value>)

• Improvement in multi-stage aggregation to directly store column index as identifier (#11617)

• Perf optimization to avoid unnecessary rows conversion in aggregation (#11607)

• Enhance SegmentPartitionMetadataManager to handle new segment (#11585)

• Optimize mailbox info in query plan to reduce memory footprint (#12382)

  • This PR changes the proto object structure, which will cause backward incompatibility when broker and server are running different version.

• Optimizations to query plan serialization (#12370)

• Optimization for parallel execution of Ser/de stage plan (#12363)

• Optimizations in query dispatch (#12358)

• Perf optimization for group-by and join for single key scenario (#11630)

Bugfixes, Refactoring, Cleanups, Tests

• Bugfix for evaluation of chained literal functions (#12248)

• Fixes to sort copy rule (#12251 and #12237)

• Fixes duplicate results for literal queries (#12240)

• Bugfix to use UTF-8 encoding for default Charset (#12213)

• Bugfix to escape table name when routing queries (#12212)

• Refactoring of planner code and removing unnecessary rules (#12070, #12052)

• Fix to remove unnecessar project after agg during relBuilder (#12058)

• Fixes issues multi-semi-join (#12038)

• Fixes leaf limit refactor issue (#12001)

• Add back filter merge after rule (#11989)

• Fix operator EOS pull (#11970)

• Fix type cast issue with dateTimeConvert scalar function (#11839, #11971)

• Fix to set explicit warning flags set on each stage stats (#11936)

• Fix mailbox visitor mismatch receive/send (#11908)

• Fix eliminate multiple exchanges in nested semi-join queries (#11882)

• Bugfix for multiple consecutive Exchange returning empty response (#11885)

• Fixing unit-test-2 build (#11889)

• Fix issue with realtime partition mismatch metric (#11871)

• Fix the NPE for rebalance retry (#11883)

• Bugfix to make Agg literal attach happen after BASIC_RULES (#11863)

• Fix NPE by init execution stats map (#11801)

• Test cases for special column escape (#11737)

• Fix StPoint scalar function usage in multi-stage engine intermediate stage (#11731)

• Clean up for transform function type (#11726)

• Add capability to ignore test (#11703)

• Fix custom property naming (#11675)

• Log warning when multi-stage engine planning throws exception (#11595)

• Fix usage of metadata overrides (#11587)

• Test change to enable metadata manager by default for colocated join quickstart (#11579)

• Tests for IN/NOT-IN operation (#12349)

• Fix stage id in stage plan (#12366)

• Bugfix for IN and NOT IN filters within case statements (#12305)

Notable Features

Server-level throttling for realtime consumption (#12292)

• Use server config pinot.server.consumption.rate.limit to enable this feature

• Server rate limiter is disabled by default (default value 0)

Reduce segment generation disk footprint for Minion Tasks (#12220)

• Supported in MergeRollupTask and RealtimeToOfflineSegmentsTask minion tasks

• Use taskConfig segmentMapperFileSizeThresholdInBytes to specify the threshold size

"task": {
  "taskTypeConfigsMap": {
    "<task_name>": {
      "segmentMapperFileSizeThresholdInBytes": "1000000000"
    }
  }
}

Support for swapping of TLS keystore/truststore (#12277, #12325)

• Security feature that makes the keystore/truststore swappable.

• Auto-reloads keystore/truststore (without need for a restart) if they are local files

Sticky Query Routing (#12276)

• Adds support for deterministic and sticky routing for a query / table / broker. This setting would lead to same server / set of servers (for MultiStageReplicaGroupSelector) being used for all queries of a given table.

• Query option (takes precedence over fixed routing setting at table / broker config level) SET "useFixedReplica"=true;

• Table config (takes precedence over fixed routing setting at broker config level)

  Copy

  "routing": {
     ...          
     "useFixedReplica": true
  }

• Broker conf - pinot.broker.use.fixed.replica=true

Table Config to disallow duplicate primary key for dimension tables (#12290)

• Use tableConfig dimensionTableConfig.errorOnDuplicatePrimaryKey=true to enable this behavior

• Disabled by default

Partition-Level ForceCommit for realtime tables (#12088)

• Support to force-commit specific partitions of a realtime table.

• Partitions can be specified to the forceCommit API as a comma separated list of partition names or consuming segment names

Support initializing broker tags from config (#12175)

• Support to give the broker initial tags on startup.

• Automatically updates brokerResource when broker joins the cluster for the first time

• Broker tags are provided as comma-separated values in pinot.broker.instance.tags

Support for StreamNative OAuth2 authentication for pulsar (#12068)

• StreamNative (the cloud SAAS offering of Pulsar) uses OAuth2 to authenticate clients to their Pulsar clusters.

• For more information, see how to Configure OAuth2 authentication in Pulsar clients

• Can be configured by adding the following properties to streamConfigs:

"stream.pulsar.issuerUrl": "https://auth.streamnative.cloud"
"stream.pulsar.credsFilePath": "file:///path/to/private_creds_file
"stream.pulsar.audience": "urn:sn:pulsar:test:test-cluster"

Introduce low disk mode to table rebalance (#12072)

• Introduces a new table rebalance boolean config lowDiskMode.Default value is false.

• Applicable for rebalance with downtime=false.

• When enabled, segments will first be offloaded from servers, then added to servers after offload is done. It may increase the total time of the rebalance, but can be useful when servers are low on disk space, and we want to scale up the cluster and rebalance the table to more servers.

• #12112 adds the UI capability to toggle this option

Support Vector index and Hierarchical Navigable Small Worlds (HNSW) (#11977)

• Supports Vector Index on float array/multi-value columnz

• Add predicate and function to retrieve topK closest vector. Example query

SELECT ProductId, UserId, l2_distance(embedding, ARRAY[-0.0013143676,-0.011042999,...]) AS l2_dist, n_tokens, combined
FROM fineFoodReviews
WHERE VECTOR_SIMILARITY(embedding, ARRAY[-0.0013143676,-0.011042999,...], 5)  
ORDER by l2_dist ASC 
LIMIT 10

• The function VectorSimilarity will return a double value where the first parameter is the embedding column and the second parameter is the search term embedding literal.

• Since VectorSimilarity is a predicate, once config the topK, this predicate will return topk rows per segment. Then if you are using this index with other predicate, you may not get expected number of rows since the records matching other predicate might not in the topk rows.

Support for retention on deleted keys of upsert tables (#12037)

• Adds an upsert config deletedKeysTTL which will remove deleted keys from in-memory hashmap and mark the validDocID as invalid after the deletedKeysTTL threshold period.

• Disabled by default. Enabled only if a valid value for deletedKeysTTL is set

• More details in the design document

Configurable Lucene analyzer (#12027)

• Introduces the capability to specify a custom Lucene analyzer used by text index for indexing and search on an individual column basis.

• Sample usage

fieldConfigList: [
   {
        "name": "columnName",
        "indexType": "TEXT",
        "indexTypes": [
          "TEXT"
        ],
        "properties": {
          "luceneAnalyzerClass": "org.apache.lucene.analysis.core.KeywordAnalyzer"
        },
      }
  ]

• Default Behavior falls back to using the standardAnalyzer unless the luceneAnalyzerClass property is specified.

Support for murmur3 as a partition function (#12049)

• Murmur3 support with optional fields seed and variant for the hash in functionConfig field of columnPartitionMap.Default value for seed is 0.

• Added support for 2 variants of Murmur3: x86_32 and x64_32 configurable using the variant field in functionConfig. If no variant is provided we choose to keep the x86_32 variant as it was part of the original implementation.

• Examples of functionConfig;

  Copy

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3 
            },
            ..
          }
        }

  Here there is no functionConfig configured, so the seed value will be 0 and variant will be x86_32.

  Copy

  "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig": {
                 "seed": "9001"
               },
            },
            ..
          }
        }

  Here the seed is configured as 9001 but as no variant is provided, x86_32 will be picked up.

  Copy

   "tableIndexConfig": {
        ..
        "segmentPartitionConfig": {
          "columnPartitionMap": {
            "memberId": {
              "functionName": "Murmur3",
              "numPartitions": 3,
              "functionConfig" :{
                 "seed": "9001"
                 "variant": "x64_32"
               },
            },
            ..
          }
        }

  Here the variant is mentioned so Murmur3 will use the x64_32 variant with 9001 as seed.

• Note on users using Debezium and Murmur3 as partitioning function :

  • The partitioning key should be set up on either of byte[], String or long[] columns.

  • On pinot variant should be set as x64_32 and seed should be set as 9001.

New optimized MV forward index to only store unique MV values

• Adds new MV dictionary encoded forward index format that only stores the unique MV entries.

• This new index format can significantly reduce the index size when the MV entries repeat a lot

• The new index format can be enabled during index creation, derived column creation, and segment reload

• To enable the new index format, set the compression codec in the FieldConfig:

  Copy

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "compressionCodec": "MV_ENTRY_DICT"
  }

  Or use the new index JSON:

  Copy

  {
    "name": "myCol",
    "encodingType": "DICTIONARY",
    "indexes": {
      "forward": {
        "dictIdCompressionType": "MV_ENTRY_DICT"
      }
    }
  }

Support for explicit null handling modes (#11960)

• Adds support for 2 possible ways to handle null:

  • Table mode - which already exists

  • Column mode, which means that each column specifies its own nullability in the FieldSpec

• Column mode can be enabled by the below config.

• The default value for enableColumnBasedNullHandling is false. When set to true, Pinot will ignore TableConfig.IndexingConfig.nullHandlingEnabled and columns will be nullable if and only if FieldSpec.notNull is false, which is also the default value.

{
  "schemaName": "blablabla",
  "dimensionFieldSpecs": [
    {
      "dataType": "INT",
      "name": "nullableField",
      "notNull": false
    },
    {
      "dataType": "INT",
      "name": "notNullableField",
      "notNull": true
    },
    {
      "dataType": "INT",
      "name": "defaultNullableField"
    },
    ...
  ],
  "enableColumnBasedNullHandling": true/false
}

Support tracking out of order events in Upsert (#11877)

• Adds a new upsert config outOfOrderRecordColumn

• When set to a non-null value, we check whether an event is OOO or not and then accordingly update the corresponding column value to true / false.

• This will help in tracking which event is out-of-order while using skipUpsert

Compression configuration support for aggregationConfigs to StartreeIndexConfigs (#11744)

• Can be used to save space. For eg: when a functionColumnPairs has a output type of bytes, such as when you use distinctcountrawhll.

• Sample config

"starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "a",
            "b",
            "c"
          ],
          "skipStarNodeCreationForDimensions": [],
          "functionColumnPairs": [],
          "aggregationConfigs": [
            {
              "columnName": "column1",
              "aggregationFunction": "SUM",
              "compressionCodec": "SNAPPY"
            },
            {
              "columnName": "column2",
              "aggregationFunction": "distinctcounthll",
              "compressionCodec": "LZ4"
            }
          ],
          "maxLeafRecords": 10000
        }
      ]

Preconfiguration based mirror instance assignment (#11578)

• Supports instance assignment based pre-configured instance assignment map.

• The assignment will always respect the mirrored servers in the pre-configured map

• More details here

• Sample table config

"instanceAssignmentConfigMap": {
  "CONSUMING": {
    "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
    "replicaGroupPartitionConfig": { ... },
     "tagPoolConfig": {
       ...
       "tag": "mt1_REALTIME"
     }
     ...
 }
 "COMPLETED": {
   "partitionSelector": "MIRROR_SERVER_SET_PARTITION_SELECTOR",
   "replicaGroupPartitionConfig": { ... },
    "tagPoolConfig": {
       ...
       "tag": "mt1_OFFLINE"
     }
     ...
 },
 "instancePartitionsMap": {
      "CONSUMING": “mt1_CONSUMING"
      "COMPLETED": "mt1_OFFLINE"
 },

Support for Listing Dimension Tables (#11859)

• Adds dimension as a valid option to table "type" in the /tables controller API

Support in upsert for dropping out of order events (#11811)

• This patch adds a new config for upsert: dropOutOfOrderRecord

• If set to true, pinot doesn't persist out-of-order events in the segment.

• This feature is useful to

  • Save disk-usage

  • Avoid any confusion when using skipUpsert for partial-upsert tables as nulls start showing up for columns where a previous non-null was encountered and we don't know if it's an out-of-order event or not.

Support to retry failed table rebalance tasks (#11740)

• New configs for the RebalanceChecker periodic task:

  • controller.rebalance.checker.frequencyPeriod: 5min by default ; -1 to disable

  • controller.rebalanceChecker.initialDelayInSeconds: 2min+ by default

• New configs added for RebalanceConfig:

  • heartbeatIntervalInMs: 300_000 i.e. 5min

  • heartbeatTimeoutInMs: 3600_000 i.e. 1hr

  • maxAttempts: 3 by default, i.e. the original run plus two retries

  • retryInitialDelayInMs: 300_000 i.e. 5min, for exponential backoff w/ jitters

• New metrics to monitor rebalance and its retries:

  • TABLE_REBALANCE_FAILURE("TableRebalanceFailure", false), emit from TableRebalancer.rebalanceTable()

  • TABLE_REBALANCE_EXECUTION_TIME_MS("tableRebalanceExecutionTimeMs", false), emit from TableRebalancer.rebalanceTable()

  • TABLE_REBALANCE_FAILURE_DETECTED("TableRebalanceFailureDetected", false), emit from RebalanceChecker

  • TABLE_REBALANCE_RETRY("TableRebalanceRetry", false), emit from RebalanceChecker

• New restful API

  • DELETE /tables/{tableName}/rebalance API to stop rebalance. In comparison, POST /tables/{tableName}/rebalance was used to start one.

Support for UltraLogLog (#11835)

• UltraLogLog aggregations for Count Distinct (distinctCountULL and distinctCountRawULL)

• UltraLogLog creation via Transform Function

• UltraLogLog merging in MergeRollup

• Support for UltraLogLog in Star-Tree indexes

Support for Apache Datasketches CPC sketch (#11774)

• Ingestion via transformation function

• Extracting estimates via query aggregation functions

• Segment rollup aggregation

• StarTree aggregation

Support to reduce DirectMemory OOM chances on broker (#11710)

• Broadly there are two configs that will enable this feature:

  • maxServerResponseSizeBytes: Maximum serialized response size across all servers for a query. This value is equally divided across all servers processing the query.

  • maxQueryResponseSizeBytes: Maximum length of the serialized response per server for a query

• Configs are available as queryOption, tableConfig and Broker config. The priority of enforcement is as follows:

  Copy

  The overriding order of priority is:
  1. QueryOption  -> maxServerResponseSizeBytes
  2. QueryOption  -> maxQueryResponseSizeBytes
  3. TableConfig  -> maxServerResponseSizeBytes
  4. TableConfig  -> maxQueryResponseSizeBytes
  5. BrokerConfig -> pinot.broker.max.server.response.size.bytes
  6. BrokerConfig -> pinot.broker.max.query.response.size.bytes

UI Support to Allow schema to be created with JSON config (#11809)

• This is helpful when user has the entire JSON handy

• UI still keeps Form Way to add Schema along with JSON view

Support in JSON index for ignoring values longer than a given length (#11604)

• Use option maxValueLength in jsonIndexConfig to restrict length of values

• A value of 0 (or when the key is omitted) means there is no restriction

Support for MultiValue VarByte V4 index writer (#11674)

• Supports serializing and writing MV columns in VarByteChunkForwardIndexWriterV4

• Supports V4 reader that can be used to read SV var length, MV fixed length and MV var length buffers encoded with V4 writer

Improved scalar function support for Multivalue columns(#11555, #11654)

arrayIndexOfInt(int[] value, int valToFind)
arrayIndexOfLong(int[] value, long valToFind)
arrayIndexOfFloat(int[] value, float valToFind)
arrayIndexOfDouble(int[] value, double valToFind)
arrayIndexOfString(int[] value, String valToFind)
intersectIndices(int[] values1, int[] values2)

Support for FrequentStringsSketch and FrequentLonsSketch aggregation functions (#11098)

• Approximation aggregation functions for estimating the frequencies of items a dataset in a memory efficient way. More details in Apache Datasketches library.

FREQUENTLONGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object
FREQUENTSTRINGSSKETCH(col, maxMapSize=256) -> Base64 encoded sketch object

Controller API for Table Indexe (#11576)

• Table index api to get the aggregate index details of all segments for a table.

  • URL/tables/{tableName}/indexes

• Response format

  Copy

  {
      "totalSegments": 31,
      "columnToIndexesCount":
      {
          "col1":
          {
              "dictionary": 31,
              "bloom": 0,
              "null": 0,
              "forward": 31,
              ...
              "inverted": 0,
              "some-dynamically-injected-index-type": 31,
          },
          "col2":
          {
              ...
          }
          ...
  }

Support for configurable rebalance delay at lead controller (#11509)

• The lead controller rebalance delay is now configurable with controller.resource.rebalance.delay_ms

• Changing rebalance configurations will now update the lead controller resource

Support for configuration through environment variables (#12307)

• Adds support for Pinot configuration through ENV variables with Dynamic mapping.

• More details in issue: #10651

• Sample configs through ENV

export PINOT_CONTROLLER_HOST=host
export PINOT_SERVER_PROPERTY_WHATEVER=whatever_property
export ANOTHER_VARIABLE=random

Add hyperLogLogPlus aggregation function for distinct count (#11346)

• HLL++ has higher accuracy than HLL when cardinality of dimension is at 10k-100k.

• More details here

DISTINCTCOUNTHLLPLUS(some_id, 12)

Support for clpMatch

• Adds query rewriting logic to transform a "virtual" UDF, clpMatch, into a boolean expression on the columns of a CLP-encoded field.

• To use the rewriter, modify broker config to add org.apache.pinot.sql.parsers.rewriter.ClpRewriter to pinot.broker.query.rewriter.class.names.

Support for DATETIMECONVERTWINDOWHOP function (#11773)

Support for JSON_EXTRACT_INDEX transform function to leverage json index for json value extraction (#11739)

Support for ArrayAgg aggregation function (#11822)

GenerateData command support for generating data in JSON format (#11778)

Enhancements

SQL

• Support ARRAY function as a literal evaluation (#12278)

• Support for ARRAY literal transform functions (#12118)

• Theta Sketch Aggregation enhancements (#12042)

  • Adds configuration options for DistinctCountThetaSketchAggregationFunction

  • Respects ordering for existing Theta sketches to use "early-stop" optimisations for unions

• Add query option override for Broker MinGroupTrimSize (#11984)

• Support for 2 new scalar functions for bytes: toUUIDBytes and fromUUIDBytes (#11988)

• Config option to make groupBy trim size configurable at Broker (#11958)

• Pre-aggregation support for distinct count hll++ (#11747)

• Add float type into literal thrift to preserve literal type conforming to SQL standards (#11697)

• Enhancement to add query function override for Aggregate functions of multi valued columns (#11307)

• Perf optimization in IN clause evaluation (#11557)

• Add TextMatchFilterOptimizer to maximally push down text_match filters to Lucene (#12339

UI

• Async rendering of UI elements to load UI elements async resulting in faster page loads (#12210)

• Make the table name link clickable in task details (#12253)

• Swagger UI enhancements to resumeConsumption API call (#12200)

• Adds support for CTRL key as a modifier for Query shortcuts (#12087)

• UI enhancement to show partial index in reload (#11913)

• UI improvement to add Links to Instance in Table and Segment View (#11807)

• Fixes reload to use the right indexes API instead of fetching all segment metadata (#11793)

• Enhancement to add toggle to hide/show query exceptions (#11611)

Misc

• Enhancement to reduce the heap usage of String Dictionaries that are loaded on-heap (#12223)

• Wire soft upsert delete for Compaction task (12330)

• Upsert compaction debuggability APIs for validDocId metadata (#12275)

• Make server resource classes configurable (#12324)

• Shared aggregations for Startree index - mapping from aggregation used in the query to aggregation used to store pre-aggregated values (#12164)

• Increased fetch timeout for Kineses to prevent stuck kinesis consumers

• Metric to track table rebalance (#12270)

• Allow server-level configs for upsert metadata (#18851)

• Support to dynamically initialize Kafka client SSL configs (#12249)

• Optimize segment metadata file creation without having to download full segment (#12255)

• Allow string / numeric data type for deleteRecordColumn config (#12222)

• Atomic and Deterministic snapshots of validDocId for upsert tables (#12232, #12246)

• Observability enhancement to add column name when JSON index building fails (#12151)

• Creation of DateTimeGenerator for DATE_TIME field type columns (#12206)

• Add singleton registry for all controller and minion metrics (#12119)

• Support helm chart server separate liveness and readiness probe endpoints (#11800)

• Observability enhancement to add metrics for Table Disabled and Consumption Paused (#12000)

• Support for SegmentGenerationAndPushTask to push segment to realtime table (#12084)

• Enhancement to make the deep store upload retry async with configurable parallelism (#12017)

• Optimizations in segment commit to not read partition group metadata (#11943)

• Replace timer with scheduled executor service in IngestionDelayTracker to reduce number of threads (#11849)

• Adds an option skipControllerCertValidation to skip controller cert validation in AddTableCommand (#11967)

• Adds instrumentation for DataTable Creation (#11942)

• Improve performance of ZkBasicAuthAccessFactory by caching Bcrypt password (#11904)

• Adds support to to fetch metadata for specific list of segments (#11949)

• Allow user specify local temp directory for quickstart (#11961)

• Optimization for server to directly return final result for queries hitting single server (#11938)

• Explain plan optimization to early release AcquireReleaseColumnsSegmentOperator (#11945)

• Observability metric to track query timeouts (#11892)

• Add support for auth in QueryRunner (#11897)

• Allow users to pass custom RecordTransformers to SegmentProcessorFramework (#11887)

• Add isPartialResult flag to broker response (#11592)

• Add new configs to Google Cloud Storage (GCS) connector: jsonKey (#11890)

  • jsonKey is the GCP credential key in string format (either in plain string or base64 encoded string). Refer Creating and managing service account keys to download the keys.

• Performance enhancement to build segments in column orientation (#11776)

  • Disabled by default. Can be enabled by setting table config columnMajorSegmentBuilderEnabled

• Observability enhancements to emit metrics for grpc request and multi-stage leaf stage (#11838)

  • pinot.server.query.log.maxRatePerSecond: query log max rate (QPS, default 10K)

  • pinot.server.query.log.droppedReportMaxRatePerSecond: dropped query log report max rate (QPS, default 1)

• Observability improvement to expose GRPC metrics (#11842)

• Improvements to response format for reload API to be pretty printed (#11608)

• Enhancements to support Java 21 (#11672)

• Add more information in RequestContext class (#11708)

• Support to read exact buffer byte ranges corresponding to a given forward index doc id (#11729)

• Enhance Broker reducer to handle expression format change (#11762)

• Capture build scans on ge.apache.org to benefit from deep build insights (#11767)

• Performance enhancement in multiple places by updating initial capacity of HashMap (#11709)

• Support for building indexes post segment file creation, allowing indexes that may depend on a completed segment to be built as part of the segment creation process (#11711)

• Support excluding time values in SimpleSegmentNameGenerator (#11650)

• Perf enhancement to reduce cpu usage by avoiding throwing an exception during query execution (#11715)

• Added framework for supporting nulls in ScalarTransformFunctionWrapper in the future (#11653)

• Observability change to metrics to export netty direct memory used and max (#11575)

• Observability change to add a metric to measure total thread cpu time for a table (#11713)

• Observability change to use SlidingTimeWindowArrayReservoirin dropwizard metrics (#11695)

• Minor improvements to upsert preload (#11694)

• Observability changes to expose additional Realtime Ingestion Metrics (#11685)

• Perf enhancement to remove the global lock in SegmentCompletionManager (#11679)

• Enhancements to unify tmp file naming format and delete tmp files at a regular cadence by extending the ControllerPeriodicTask (#10815)

  • controller.realtime.segment.tmpFileAsyncDeletionEnabled (default false)

  • controller.realtime.segment.tmpFileRetentionInSeconds (default 3600)

• Improvements to skip unparseable records in the csv record reader (#11540, #11594)

• Enhancements to allow override/force options when add schema (#11572)

• Enhancement to handle direct memory OOM on brokers (#11496)

• Enhancement to metadata API to return upsert partition to primary key count map for both controller and server APIs (#12334)

• Enhancements to peer server segment download by retrying both peer discovery and download. (#12317)

• Helper functions in StarTreeBuilderUtils and StarTreeV2BuilderConfig (#12361)

• Perf optimizations to release all segments of a table in releaseAndRemoveAllSegments method (#12297)

• Enhancement to Maintain pool selection for the minimizeDataMovement instance partition assignment strategy (#11953)

• Upsert enhancement to assign segments for with respect to ideal state (#11628)

• Observability change to export Additional Upsert Metrics to Prom (#11660)

• Observibility enhancement to add CPU metrics for minion purge task (#12337)

• Add HttpHeaders in broker event listener requestContext (#12258)

Bugfixes, Refactoring, Cleanups, Deprecations

• Upsert bugfix in "rewind()" for CompactedPinotSegmentRecordReader (#12329)

• Fix error message format for Preconditions.checks failures(#12327)

• Bugfix to distribute Pinot as a multi-release JAR (#12131, #12300)

• Fixes in upsert metadata manager (#12319)

• Security fix to allow querying tables with table-type suffix (#12310)

• Bugfix to ensure tagConfigOverride config is null for upsert tables (#12233 and #12311)

• Increased fetch timeout for Kineses to prevent stuck kinesis consumers(#12214)

• Fixes to catch-all Regex for JXM -> Prom Exporter (#12073 and #12295)

• Fixes lucene index errors when using QuickStart (#12289)

• Null handling bugfix for sketch group-by queries (#12259)

• Null pointer exception fixes in Controller SQL resource (#12211)

• Synchronization fixes to replace upsert segments (#12105 and #12241)

• Bugfix for S3 connection pool error when AWS session tokens expire after an hour (#12221)

• FileWriter fixes to append headerline only for required formats like csv (#12208)

• Security bugfix for pulsar OAuth2 authentication (#12195)

• Bugfix to appropriately compute "segment.flush.threshold.size" when force-committing realtime segments (#12188)

• Fixes rebalance converge check that reports success before rebalance completes (#12182)

• Fixes upsertPrimaryKeysCount metric reporting when table is deleted (#12169)

• Update LICENSE-binary for commons-configuration2 upgrade (#12165)

• Improve error logging when preloading segments not exist on server (#12153)

• Fixes to file access resource leaks (#12129)

• Ingestion bugfix to avoid unnecessary transformers in CompositeTransformer (#12138)

• Improve logging to print OS name during service statup (#12135)

• Improve logging in multiple files (#12134, #12137, #12127, #12121)

• Test fixes for ExprMinMaxRewriterTest.testQueryRewrite (#12047)

• Fixes default path of log4j in helmchart (#12069, #12083)

• Fix default brokerUpdateFrequencyInMillis for connector (#12093)

• Updates to README file (#12075)

• Fix to remove unnecessary locking during segment preloading (#12077)

• Fix bug with silently ignoring force commit call failures (#12044)

• Upsert bugfix to allow optional segments that can be skipped by servers without failing the query (#11978)

• Fix incorrect handling of consumer creation errors (#12045)

• Fix the memory leak issue on CommonsConfigurationUtils (#12056)

• Fix rebalance on upsert table (#12054)

• Add new Transformer to transform -0.0 and NaN (#12032)

• Improve inverted index validation in table config to enhance user experience (#12043)

• Fixes test flakiness by replacing HashSet/HashMap with LinkedHashSet/LinkedHashMap (#11941)

• Flaky test fix for ServerRoutingStatsManagerTest.testQuerySubmitAndCompletionStats (#12029)

• Fix derived column from MV column (#12028)

• Support for leveraging StarTree index in conjunction with filtered aggregations (#11886)

• Improves tableConfig validation for enabling size based threshold for realtime tables (#12016)

• Fix flaky PinotTenantRestletResourceTest (#12026)

• Fix flaky Fix PinotTenantRestletResourceTest (#12019)

• Fix the race condition of concurrent modification to segment data managers (#12004)

• Fix the misuse of star-tree when all predicates are always false under OR (#12003)

• Fix the test failures caused by instance drop failure (#12002)

• Fix fromULL scalar function (#11995)

• Fix to exclude module-info.class during shade operations (#11975)

• Fix the wrong import for Preconditions (#11979)

• Add check for illegal character '/' in taskName (#11955)

• Bugfix to only register new segments when it's fully initalized by partitionUpsertMetadataManager (#11964)

• Obervability fix to add logs to track sequence of events for table creation (#11946)

• Fix the NPE in minimizeDataMovement instance assignment strategy (#11952)

• Fix to add catch all logging for exception during DQL/DML process (#11944)

• Fix bug where we don't handle cases that a upsert table has both upsert deletion and upsert ttl configs (#11791)

• Removing direct dependencies on commons-logging and replacing with jcl-over-slf4j (#11920)

• Fix NPE for IN clause on constant STRING dictionary (#11930)

• Fix flaky OfflineClusterIntegrationTest on server response size tests (#11926)

• Avoid npe when checking mirror server set assignment (#11915)

• Deprecate _segmentAssignmentStrategy in favor of SegmentsValidationAndRetentionConfig #11869

• Bugfix to capture auth phase timing even if access is denied (#11884)

• Bugfix to mark rows as invalid in case primary time column is out of range (#11907)

• Fix to radomize server port to avoid port already bind issue (#11861)

• Add LazyRow abstraction for previously indexed record (#11826)

• Config Validation for upsert table to not assign COMPLETED segments to another server (#11852)

• Bugfix to resolve dependency conflict in pinot-protobuf module (#11867)

• Fix case of useMultistageEngine property reference in JsonAsyncHttpPinotClientTransportFactory (#11820)

• Bugfix to add woodstox-core to pinot-s3 dependencies and fix stack trace (#11799)

• Fix to move pinot-segment-local test from unit test suite 1 to 2 (#11865)

• Observability fix to log upsert config when initializing the metadata manager (#11864)

• Fix to improve tests when errors are received in the consumer thread (#11858)

• Fix for flaky ArrayAgg test (#11860)

• Fix for flaky tests in TupleSelectionTransformFunctionsTest (#11848)

• Fix for arrayAgg null support (#11853)

• Fix the bug of reading decimal value stored in int32 or int64 (#11840)

• Remove duplicate pinot-integration-tests from unit test suite 2 (#11844)

• Fix for a null handling error in queries (#11829)

• Fix the way of fetching the segment zk metadata for task generators (#11832)

• Make testInvalidateCachedControllerLeader times based on getMinInvalidateIntervalMs (#11815)

• Update doap to reflect latest release (#11827)

• Clean up integration test pom file (#11817)

• Bugfix to exclude OFFLINE segments when reading server to segments map (#11818)

• Add tests for zstd compressed parquet files (#11808)

• Fix job submission time for reload and foce commit job (#11803)

• Remove actually unsupported config that selectively enable nullable columns (#10653)

• Fix LLCRealtimeClusterIntegrationTest.testReset (#11806)

• Use expected version in api for table config read modify write change (#11782)

• Move jobId out of rebalanceConfig (#11790)

• Fix PeerServerSegmentFinder not respecting HTTPS port (#11752)

• Enhanced geospatial v2 integration tests (#11741)

• Add integration test for rebalance in upsert tables (#11568)

• Fix trivy CI issue (#11757)

• Cleanup rebalance configs by adding a RebalanceConfig class (#11730)

• Fix a protobuf comment to be more precise (#11735)

• Move scala dependencies to root pom (#11671)

• Fix ProtoBuf inputformat plug-in handling for null values (#11723)

• Bugfix where segment download URI is invalid after same CRC refresh using tar push (#11720)

• Fix in TableCacheTest (#11717)

• Add more test for broker jersey bounded thread pool (#11705)

• Fix bug in gapfill with SumAvgGapfillProcessor. (#11714)

• Bugfix to allow GcsPinotFS to work with granular permissions (#11655)

• Fix default log4j2 config file path in helm chart (#11707)

• Refactor code and doc occurrences of argmin/max -> exprmin/max (#11700)

• Make constructor and functions public to be used from scheduler plugins (#11699)

• Bugfix to change json_format to return java null when java null is received (#11673)

• Fix the potential access to upsert metadata manager after it is closed (#11692)

• Bugfix to use isOptional instead of the deprecated hasOptional Keyword (#11682)

• Fix logging issue in RealtimeTableDataManager (#11693)

• Cleanup some reader/writer logic for raw forward index (#11669)

• Do not execute spotless in Java 21 (#11670)

• Update license-maven-plugin (#11665)

• Bugfix to allow deletion of local files with special characters (#11664)

• Clean up CaseTransformFunction::constructStatementListLegacy. (#11339)

• Bugfix to force FileChannel to commit data to disk (#11625)

• Remove the old deprecated commit end without metadata (#11662)

• Fix for a jackson vulnerability (#11619)

• Refactor BasicAuthUtils from pinot-core to pinot-common and remove pinot-core dependency from pinot-jdbc-client (#11620)

• Bugfix to support several extensions for different indexes (#11600)

• Fix the alias handling in single-stage engine (#11610)

• Fix to use constant null place holder (#11615)

• Refactor to move all BlockValSet into the same package (#11616)

• Remove deprecated Request class from pinot-java-client (#11614)

• Refactoring to remove old thirdeye files. (#11609)

• Testing fix to use builder method in integration test (#11564)

• Fix the broken Pinot JDBC client. (#11606)

• Bugfix to change the Forbidden error to Unauthorized (#11501)

• Fix for schema add UI issue that passing wrong data in the request header (#11602)

• Remove/Deprecate HLC handling code (#11590)

• Fix the bug of using push time to identify new created segment (#11599)

• Bugfix in CSVRecordReader when using line iterator (#11581)

• Remove split commit and some deprecated config for real-time protocol on controller (#11663)Improved validation for single argument aggregation functions (#11556)

• Fix to not emit lag once tabledatamanager shutdown (#11534)

• Bugfix to fail reload if derived columns can't be created (#11559)

• Fix the double unescape of property value (#12405)

• Fix for the backward compatible issue that existing metadata may contain unescaped characters (#12393)

• Skip invalid json string rather than throwing error during json indexing (#12238)

• Fixing the multiple files concurrent write issue when reloading SSLFactory (#12384)

• Fix memory leaking issue by making thread local variable static (#12242)

• Bugfixfor Upsert compaction task generator (#12380)

• Log information about SSLFactory renewal (#12357)

• Fixing array literal usage for vector (#12365)

• Fixing quickstart table baseballStats minion ingestion (#12371)

• Fix backward compatible issue in DistinctCountThetaSketchAggregationFunction (#12347)

• Bugfix to skip instead of throwing error on 'getValidDocIdMetadata' (#12360)

• Fix to clean up segment metadata when the associated segment gets deleted from remote store (#12350)

• Fix getBigDecimal() scale throwing rounding error (#12326)

• Workaround fix for the problem of Helix sending 2 transitions for CONSUMING -> DROPPED (#12351)

• Bugfix for making nonLeaderForTables exhaustive (#12345)

• Bugfixes for graceful interrupt handling of mutable lucene index (#11558,#12274)

• Remove split commit and some deprecated config for real-time protocol on controller (#11663)

• Update the table config in quick start (#11652)

• Deprecate k8s skaffold scripts and move helm to project root directory (#11648)

• Fix NPE in SingleColumnKeySelector (#11644)

• Simplify kafka build and remove old kafka 0.9 files (#11638)

• Adding comments for docker image tags, make a hyper link of helmChart from root directory (#11646)

• Improve the error response on controller. (#11624)

• Simplify authrozation for table config get (#11640)

• Bugfix to remove segments with empty download url in UpsertCompactionTask (#12320)

• Test changes to make taskManager resources protected for derived classes to override in their setUp() method. (#12335)