Apache Pinot™ 1.0.0 release notes
This page covers the latest changes included in the Apache Pinot™ 1.0.0 release, including new features, enhancements, and bug fixes.
This release includes the several new features, enhancements, and bug fixes, including the following highlights:
- Multi-stage query engine: new features, enhancements, and bug fixes. Learn how to enable and use the multi-stage query engine or more about how the multi-stage query engine works.
- Set operations support:
- Timestamp and Date Operations
- Added support for partition parallelism in partitioned table scans, allowing for more efficient data retrieval (#11266)
- Enhanced the multi-stage group-by executor to support limiting the number of groups,
- Improved resilience and reliability of the multi-stage join operator, now with added support for hash join right table protection (#11401).
- Add the ability to include new index types at runtime in Apache Pinot. This opens the ability of adding third party indexes, including proprietary indexes. More details here
- NULL support for ORDER BY, DISTINCT, GROUP BY, value transform functions and filtering.
Support added to extend upserts and allow deleting records from a realtime table. The design details can be found here.
Adds a feature to preload segments from a table that uses the upsert snapshot feature. The segments with validDocIds snapshots can be preloaded in a more efficient manner to speed up the table loading (thus server restarts).
Adds support for specifying expiry TTL for upsert primary key metadata cleanup.
Adds a new minion task to compact segments belonging to a real-time table with upserts.
Adds new implementations of PinotDataBuffer that uses Unsafe java APIs and foreign memory APIs. Also added support for PinotDataBufferFactory to allow plugging in custom PinotDataBuffer implementations.
- Allows overriding index configs at tier level, allowing for more flexible index configurations for different tiers.
- Adds persistence of authentication details in the browser session. This means that even if you refresh the app, you will still be logged in until the authentication session expires (#10389)
- AuthProvider logic updated to decode the access token and extract user name and email. This information will now be available in the app for features to consume. (#10925)
Rebalance
Added new configuration options below which allow use of a bounded thread pool and allocate capacities for it.
pinot.broker.enable.bounded.http.async.executor
pinot.broker.http.async.executor.max.pool.size
pinot.broker.http.async.executor.core.pool.size
pinot.broker.http.async.executor.queue.size
This feature allows better management of broker resources.
Adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a query (which may have sensitive data in the result) using metadata only.
In segment metadata and index map, store columns in alphabetical order so that the result is deterministic. Segments generated before/after this PR will have different CRC, so during the upgrade, we might get segments with different CRC from old and new consuming servers. For the segment consumed during the upgrade, some downloads might be needed.
Adds options to configure helix timeouts
external.view.dropped.max.wait.ms`` - The duration of time in milliseconds to wait for the external view to be dropped. Default - 20 minutes. external.view.check.interval.ms`` - The period in milliseconds in which to ping ZK for latest EV state.This PR makes Pinot case insensitive be default, and removes the deprecated property enable.case.insensitive.pql
Newly added APIs and client methods
High level consumers are no longer supported
Type information preservation of query literals
- [feature] [backward-incompat] [null support # 2] Preserve null literal information in literal context and literal transform (#10380) String versions of numerical values are no longer accepted. For example, "123" won't be treated as a numerical anymore.
Controller job status ZNode path update
- Moving Zk updates for reload, force_commit to their own Znodes which … (#10451) The status of previously completed reload jobs will not be available after this change is deployed.
Metric names for mutable indexes to change
- Implement mutable index using index SPI (#10687) Due to a change in the IndexType enum used for some logs and metrics in mutable indexes, the metric names may change slightly.
Update in controller API to enable / disable / drop instances
Change in
substring query function definition- bug fix: to keep QueryParser thread safe when handling many read requests on class RealtimeLuceneTextIndex (#10620)
- 10567: [cleanup pinot-integration-test-base], clean query generations and some other refactoring. (#10648)
- Enhance the instrumentation for a corner case where the query doesn't go through DocIdSetOp (#10729)
- Redefine the semantics of SEGMENT_STREAMED_DOWNLOAD_UNTAR_FAILURES metric to count individual segment fetch failures. (#10777)
Last modified 1d ago