# 1.2.0

This release comes with several Improvements and Bug Fixes for the Multistage Engine, Upserts and Compaction. There are a ton of other small features and general bug fixes.

## Multistage Engine Improvements

### Features

#### New Window Functions: LEAD, LAG, FIRST\_VALUE, LAST\_VALUE [#12878](https://github.com/apache/pinot/pull/12878) [#13340](https://github.com/apache/pinot/pull/13340)

* LEAD allows you to access values after the current row in a frame.
* LAG allows you to access values before the current row in a frame.
* FIRST\_VALUE and LAST\_VALUE return the respective extremal values in the frame.

#### Support for Logical Database in V2 Engine [#12591](https://github.com/apache/pinot/pull/12591) [#12695](https://github.com/apache/pinot/pull/12695/files)

* V2 Engine now supports a "database" construct, enabling table namespace isolation within the same Pinot cluster.
* Improves user experience when multiple users are using the same Pinot Cluster.
* Access control policies can be set at the database level.
* Database can be selected in a query using a SET statement, such as `SET database=my_db;`.

#### Improved Multi-Value (MV) and Array Function Support

* Added array sum aggregation functions for point-wise array operations [#13324](https://github.com/apache/pinot/pull/13324).
* Added support for `valueIn` MV transform function [#13443](https://github.com/apache/pinot/pull/13443).
* Fixed bug in numeric casts for MV columns in filters [#13425](https://github.com/apache/pinot/pull/13425).
* Fixed NPE in ArrayAgg when a column contains no data [#13358](https://github.com/apache/pinot/pull/13358).
* Fixed array literal handling [#13345](https://github.com/apache/pinot/pull/13345).

#### Support for WITHIN GROUP Clause and ListAgg [#13146](https://github.com/apache/pinot/pull/13146)

* `WITHIN GROUP` Clause can be used to process rows in a given order within a group.
* One of the most common use-cases for this is the `ListAgg` function, which when combined with `WITHIN GROUP` can be used to concatenate strings in a given order.

#### Scalar/Transform Function and Set Operation Improvements

* Added Geospatial Scalar Function support for use in intermediate stage in the v2 query engine [#13457](https://github.com/apache/pinot/pull/13457).
* Fix 'WEEK' transform function [#13483](https://github.com/apache/pinot/pull/13483).
* Support `EXTRACT` as a scalar function [#13463](https://github.com/apache/pinot/pull/13463).
* Added support for ALL modifier for INTERSECT and EXCEPT Set Operations [#13151](https://github.com/apache/pinot/pull/13151) [#13166](https://github.com/apache/pinot/pull/13166).

#### Improved Literal Handling Support

* Fixed bug in handling literal arguments in aggregation functions like Percentile [#13282](https://github.com/apache/pinot/pull/13282).
* Allow INT and FLOAT literals [#13078](https://github.com/apache/pinot/pull/13078).
* Fixed literal handling for all types [#13344](https://github.com/apache/pinot/pull/13344) [#13345](https://github.com/apache/pinot/pull/13345).
* Fixed null literal handling for null intolerant functions [#13255](https://github.com/apache/pinot/pull/13255).

#### Metrics Improvements

* Added new metrics for tracking queries executed globally and at the table level [#12982](https://github.com/apache/pinot/pull/12982).
* New metrics to track join counts and window function counts [#13032](https://github.com/apache/pinot/pull/13032).
* Multiple meters and timers to track Multistage Engine Internals [#13035](https://github.com/apache/pinot/pull/13035).

#### Notable Improvements and Bug Fixes

* Improved Window operators resiliency, with new checks to make sure the window doesn't grow too large [#13180](https://github.com/apache/pinot/pull/13180/) [#13428](https://github.com/apache/pinot/pull/13428) [#13441](https://github.com/apache/pinot/pull/13441).
* Optimized Group Key generation [#12394](https://github.com/apache/pinot/pull/12394).
* Fixed `SortedMailboxReceiveOperator` to honor convention of pulling at most 1 EOS block [#12406](https://github.com/apache/pinot/pull/12406).
* Improvement in how execution stats are handled [#12517](https://github.com/apache/pinot/pull/12517) [#12704](https://github.com/apache/pinot/pull/12704) [#13136](https://github.com/apache/pinot/pull/13136).
* Use Protobuf instead of Reflection for Plan Serialization [#13221](https://github.com/apache/pinot/pull/13221).

## Upsert Compaction and Minion Improvements

### Features and Improvements

#### Minion Resource Isolation [#12459](https://github.com/apache/pinot/pull/12459) [#12786](https://github.com/apache/pinot/pull/12786)

* Minions now support resource isolation based on an instance tag.
* Instance tag is configured at table level, and can be set for each task on a table.
* This enables you to implement arbitrary resource isolation strategies, i.e. you can use a set of Minion Nodes for running any set of tasks across any set of tables.

#### Greedy Upsert Compaction Scheduling [#12461](https://github.com/apache/pinot/pull/12461)

* Upsert compaction now schedules segments for compaction based on the number of invalid docs.
* This helps the compaction task to handle arbitrary temporal distribution of invalid docs.

#### Notable Improvements

* Minions can now download segments from servers when deepstore copy is missing. This feature is enabled via a cluster level config `allowDownloadFromServer` [#12960](https://github.com/apache/pinot/pull/12960) [#13247](https://github.com/apache/pinot/pull/13247).
* Added support for TLS Port in Minions [#12943](https://github.com/apache/pinot/pull/12943).
* New metrics added for Minions to track segment/record processing information [#12710](https://github.com/apache/pinot/pull/12710).

### Bug Fixes

* Minions can now handle invalid instance tags in Task Configs gracefully. Prior to this change, Minions would be stuck in `IN_PROGRESS` state until task timeout [#13092](https://github.com/apache/pinot/pull/13092).
* Fix bug to return validDocIDsMetadata from all servers [#12431](https://github.com/apache/pinot/pull/12431).
* Upsert compaction doesn't retain maxLength information and trims string fields [#13157](https://github.com/apache/pinot/pull/13157).

## Upsert Improvements

### Features and Improvements

#### Consistent Table View for Upsert Tables [#12976](https://github.com/apache/pinot/pull/12976)

* Adds different modes of consistency guarantees for Upsert tables.
* Adds a new UpsertConfig called `consistencyMode` which can be set to `NONE, SYNC, SNAPSHOT`.
* `SYNC` is optimized for data freshness but can lead to elevated query latencies and is best for low-qps use-cases. In this mode, the ingestion threads will take a WLock when updating validDocID bitmaps.
* `SNAPSHOT` mode can handle high-qps/high-ingestion use-cases by getting the list of valid docs from a snapshot of validDocID. The snapshot can be refreshed every few seconds and the tolerance can be set via a query option `upsertViewFreshnessMs`.

#### Pluggable Partial Upsert Merger [#11983](https://github.com/apache/pinot/pull/11983)

* Partial Upsert merges the old record and the new incoming record to generate the final ingested record.
* Pinot now allows users to customize how this merge of an old row and the new row is computed.
* This allows a column value in the new row to be an arbitrary function of the old and the new row.

#### Support for Uploading Externally Partitioned Segments for Upsert Backfill [13107](https://github.com/apache/pinot/pull/13107/files)

* Segments uploaded for Upsert Backfill can now explicitly specify the Kafka partition they belong to.
* This enables backfilling an Upsert table where the externally generated segments are partitioned using an arbitrary hash function on an arbitrary primary key.

#### Misc Improvements and Bug Fixes

* Fixed a Bug in Handling Equal Comparison Column Values in Upsert, which could lead to data inconsistency ([#12395](https://github.com/apache/pinot/pull/12395))
* Upsert snapshot will now snapshot only those segments which have updates. [#13285](https://github.com/apache/pinot/pull/13285).

## Notable Features

#### JSON Support Improvements

* JSON Index can now be used for evaluating Regex and Range Predicates. [#12568](https://github.com/apache/pinot/pull/12568)
* `jsonExtractIndex` now supports contextual array filters. [#12683](https://github.com/apache/pinot/pull/12683) [#12531](https://github.com/apache/pinot/pull/12532).
* JSON column type now supports filter predicates like `=`, `!=`, `IN` and `NOT IN`. This is convenient for scenarios where the JSON values are very small. [#13283](https://github.com/apache/pinot/pull/13283).
* `JSON_MATCH` now supports exclusive predicates correctly. For instance, you can use predicates such as `JSON_MATCH(person, '"$.addresses[*].country" != ''us'''` to find all people who have at least one address that is not in the US. [#13139](https://github.com/apache/pinot/pull/13139).
* `jsonExtractIndex` supports extracting Multi-Value JSON Fields, and also supports providing any default value when the key doesn't exist. [#12748](https://github.com/apache/pinot/pull/12748).
* Added `isJson` UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. [#12603](https://github.com/apache/pinot/pull/12603).
* Fix `ArrayIndexOutOfBoundsException` in `jsonExtractIndex`. [#13479](https://github.com/apache/pinot/pull/13479).

#### Lucene and Text Search Improvements

* Improved Segment Build Time for Lucene Text Index by 40-60%. This improvement is realized when a consuming segment commits and changes to an `ImmutableSegment`. This significantly helps in lowering ingestion lag at commit time due to a large text index [#12744](https://github.com/apache/pinot/pull/12744) [#13094](https://github.com/apache/pinot/pull/13094) [#13050](https://github.com/apache/pinot/pull/13050).
* Phrase Search can run 3x faster when the Lucene Index Config `enablePrefixSuffixMatchingInPhraseQueries` is set to `true`. This is achieved by rewriting phrase search query to a wildcard and prefix matching query [#12680](https://github.com/apache/pinot/pull/12680).
* Fixed bug in `TextMatchFilterOptimizer` that was not applying precedence to the filter expressions properly, which could lead to incorrect results. [#13009](https://github.com/apache/pinot/pull/13009).
* Fixed bug in handling `NOT text_match` which could have returned incorrect results. [#12372](https://github.com/apache/pinot/pull/12372).
* Added `SchemaConformingTranformerV2` to enhance text search abilities. [#12788](https://github.com/apache/pinot/pull/12788).
* Added metrics to track Lucene NRT Refresh Delay [#13307](https://github.com/apache/pinot/pull/13307).
* Switched to `NRTCachingDirectory` for Realtime segments and prevented duplicates in the Realtime Lucene Index to avoid `IndexOutOfBounds` query time exceptions. [#13308](https://github.com/apache/pinot/pull/13308).
* Lucene Version is upgraded to 9.11.1. [#13505](https://github.com/apache/pinot/pull/13505).

#### New Funnel Functions [#13176](https://github.com/apache/pinot/pull/13176) [#13231](https://github.com/apache/pinot/pull/13231) [#13228](https://github.com/apache/pinot/pull/13228)

* Added `funnelMaxStep` function which can be used to calculate max funnel steps for a given sliding window .
* Added `funnelCompleteCount` to calculate the number of completed funnels, and `funnelMatchStep` to get the funnel match array.

#### Support for Interning for OnHeapByteDictionary [#12342](https://github.com/apache/pinot/pull/12342)

* This can reduce the heap usage of a dictionary encoded byte column, for a certain distribution of duplicate values. See [#12223](https://github.com/apache/pinot/pull/12223) for details.

#### Column Major Builder On By Default for New Tables [#12770](https://github.com/apache/pinot/pull/12770)

* Prior to this feature, on a segment commit, Pinot would convert all the columnar data from the Mutable Segment to row-major, and then re-build column major Immutable Segments.
* This feature skips the row-major conversion and is expected to be both space and time efficient.
* It can help lower ingestion lag from segment commits, especially helpful when your segments are large.

#### Support for SQL Formatting in Query Editor [#11725](https://github.com/apache/pinot/pull/11725)

* You can now prettify SQL right in the Controller UI!

#### Hash Function for UUID Primary Keys [#12538](https://github.com/apache/pinot/pull/12538)

* Added a new lossless hash-function for Upsert Primary Keys optimized for UUIDs.
* The hash function can reduce Old Gen by up to 30%.
* It maps a UUID to a 16 byte array, vs encoding it in a UTF string which would take 36 bytes.

#### Column Level Index Skip Query Option [#12414](https://github.com/apache/pinot/pull/12414)

* Convenient for debugging impact of indexes on query performance or results.
* You can add the `skipIndexes` option to your query to skip any number of indexes. e.g. `SET skipIndexes=inverted,range;`

#### New UDFs and Scalar Functions

* New GeoHash functions: `encodeGeoHash`, `decodeGeoHash`, `decodeGeoHashLatitude` and `decodeGeoHashLongitude`.
* `dateBin` can be used to align a timestamp to the nearest time bucket.
* `prefixes`, `suffixes` and `uniqueNgrams` UDFs for generating all respective string subsequences from a string input. [#12392](https://github.com/apache/pinot/pull/12392/).
* Added `isJson` UDF which increases your options to handle invalid JSONs. This can be used in queries and for filtering invalid json column values in ingestion. [#12603](https://github.com/apache/pinot/pull/12603).
* `splitPart` UDF has minor improvements. [#12437](https://github.com/apache/pinot/pull/12437/).

#### CLP Compression Codec in Forward Indexes [#12504](https://github.com/apache/pinot/pull/12504)

* [CLP](https://github.com/y-scope/clp) is a compressed log processor which has really high compression ratio for certain log types.
* To enable this, you can set the `compressionCodec` in the `fieldConfigList` of the column you want to target.

## Misc. Improvements

* Enable segment preloading at partition level [#12451](https://github.com/apache/pinot/pull/12451).
* Use Temurin instead of AdoptOpenJdk [#12533](https://github.com/apache/pinot/pull/12533)
* Adding record reader config/context param to record transformer [#12520](https://github.com/apache/pinot/pull/12520)
* Removing legacy commons-lang dependency [#13480](https://github.com/apache/pinot/pull/13480)
* 12508: Feature add segment rows flush config [#12681](https://github.com/apache/pinot/pull/12681)
* ADSS Race Condition and update to client error codes [#13104](https://github.com/apache/pinot/pull/13104)
* Add ExceptionMapper to convert Exception to Response Object for Broker REST API's [#13292](https://github.com/apache/pinot/pull/13292)
* Add FunnelMaxStepAggregationFunction and FunnelCompleteCountAggregationFunction [#13231](https://github.com/apache/pinot/pull/13231)
* Add GZIP Compression Codec (#11434) [#12668](https://github.com/apache/pinot/pull/12668)
* Add PodDisruptionBudgets to the Pinot Helm chart [#13153](https://github.com/apache/pinot/pull/13153)
* Add Postgres compliant name aliasing for String Functions. [#12795](https://github.com/apache/pinot/pull/12795)
* Add SchemaConformingTransformerV2 to enhance text search abilities [#12788](https://github.com/apache/pinot/pull/12788)
* Add a benchmark to measure multi-stage block serde cost [#13336](https://github.com/apache/pinot/pull/13336)
* Add a plan version field to QueryRequest Protobuf Message [#13267](https://github.com/apache/pinot/pull/13267)
* Add a post-validator visitor that verifies there are no cast to bytes [#12475](https://github.com/apache/pinot/pull/12475)
* Add a safe version of `CLStaticHttpHandler` that disallows path traversal. [#13124](https://github.com/apache/pinot/pull/13124)
* Add ability to track filtered messages offset [#12602](https://github.com/apache/pinot/pull/12602)
* Add back 'numRowsResultSet' to BrokerResponse, and retain it when result table id hidden [#13198](https://github.com/apache/pinot/pull/13198)
* Add back profile for shade [#12979](https://github.com/apache/pinot/pull/12979)
* Add back some exclude deps from hadoop-mapreduce-client-core [#12638](https://github.com/apache/pinot/pull/12638)
* Add backward compatibility regression test suite for multi-stage query engine [#13193](https://github.com/apache/pinot/pull/13193)
* Add base class for custom object accumulator [#12685](https://github.com/apache/pinot/pull/12685)
* Add clickstream example table for funnel analysis [#13379](https://github.com/apache/pinot/pull/13379)
* Add config option for timezone [#12386](https://github.com/apache/pinot/pull/12386)
* Add config to skip record ingestion on string column length exceeding configured max schema length [#13103](https://github.com/apache/pinot/pull/13103)
* Add controller API to get allLiveInstances [#12498](https://github.com/apache/pinot/pull/12498)
* Add isJson UDF [#12603](https://github.com/apache/pinot/pull/12603)
* Add list of collaborators to asf.yaml [#13346](https://github.com/apache/pinot/pull/13346)
* Add locking logic to get consistent table view for upsert tables [#12976](https://github.com/apache/pinot/pull/12976)
* Add metric to track number of segments missed in upsert-snapshot [#12581](https://github.com/apache/pinot/pull/12581)
* Add metrics for SEGMENTS\_WITH\_LESS\_REPLICAS monitoring [#12336](https://github.com/apache/pinot/pull/12336)
* Add mode to allow adding dummy events for non-matching steps [#13382](https://github.com/apache/pinot/pull/13382)
* Add offset based lag metrics [#13298](https://github.com/apache/pinot/pull/13298)
* Add protobuf codegen decoder [#12980](https://github.com/apache/pinot/pull/12980)
* Add retry policy to wait for job id to persist during rebalancing [#13372](https://github.com/apache/pinot/pull/13372)
* Add round-robin logic during downloadSegmentFromPeer [#12353](https://github.com/apache/pinot/pull/12353)
* Add schema as input to the decoder. [#12981](https://github.com/apache/pinot/pull/12981)
* Add splitPartWithLimit and splitPartFromEnd UDFs [#12437](https://github.com/apache/pinot/pull/12437)
* Add support for creating raw derived columns during segment reload [#13037](https://github.com/apache/pinot/pull/13037)
* Add support for raw JSON filter predicates [#13283](https://github.com/apache/pinot/pull/13283)
* Add the possibility of configuring ForwardIndexes with compressionCodec [#12218](https://github.com/apache/pinot/pull/12218)
* Add upsert-snapshot timer metric [#12383](https://github.com/apache/pinot/pull/12383)
* Add validation check for forward index disabled if it's a REALTIME table [#12838](https://github.com/apache/pinot/pull/12838)
* Added PR compatability test against release 1.1.0 [#12921](https://github.com/apache/pinot/pull/12921)
* Added kafka partition number to metadata. [#13447](https://github.com/apache/pinot/pull/13447)
* Added pinot-error-code header in query response [#12338](https://github.com/apache/pinot/pull/12338)
* Added tests for additional data types in SegmentPreProcessorTest.java [#12755](https://github.com/apache/pinot/pull/12755)
* Adding a cluster config to enable instance pool and replica group configuration in table config [#13131](https://github.com/apache/pinot/pull/13131)
* Adding batch api support for WindowFunction [#12993](https://github.com/apache/pinot/pull/12993)
* Adding bytes string data type integration tests [#12387](https://github.com/apache/pinot/pull/12387)
* Adding registerExtraComponents to allow registering additional components in various services [#13465](https://github.com/apache/pinot/pull/13465)
* Adding support of insecure TLS [#12416](https://github.com/apache/pinot/pull/12416)
* Adding support to insecure TLS when creating SSLFactory [#12425](https://github.com/apache/pinot/pull/12425)
* Adds AGGREGATE\_CASE\_TO\_FILTER rule [#12643](https://github.com/apache/pinot/pull/12643)
* Adds per-column, query-time index skip option [#12414](https://github.com/apache/pinot/pull/12414)
* Allow Aggregations in Case Expressions [#12613](https://github.com/apache/pinot/pull/12613)
* Allow PintoHelixResourceManager subclasses to be used in the controller starter by providing an overridable PinotHelixResouceManager object creator function [#13495](https://github.com/apache/pinot/pull/13495)
* Allow RequestContext to consider http-headers case-insensitivity [#13169](https://github.com/apache/pinot/pull/13169)
* Allow Server throttling just before executing queries on server to allow max CPU and disk utilization [#12930](https://github.com/apache/pinot/pull/12930)
* Allow all raw index config in star-tree index [#13225](https://github.com/apache/pinot/pull/13225)
* Allow apply both environment variables and system properties to user and table configs, Environment variables take precedence over system properties [#13011](https://github.com/apache/pinot/pull/13011)
* Allow configurable queryWorkerThreads in Pinot server side GrpcQueryServer [#13404](https://github.com/apache/pinot/pull/13404)
* Allow dynamically setting the log level even for loggers that aren't already explicitly configured [#13156](https://github.com/apache/pinot/pull/13156)
* Allow passing custom record reader to be inited/closed in SegmentProcessorFramework [#12529](https://github.com/apache/pinot/pull/12529)
* Allow passing database context through `database` http header [#12417](https://github.com/apache/pinot/pull/12417)
* Allow stop to interrupt the consumer thread and safely release the resource [#13418](https://github.com/apache/pinot/pull/13418)
* Allow user configurable regex library for queries [#13005](https://github.com/apache/pinot/pull/13005)
* Allow using 'serverReturnFinalResult' to optimize server partitioned table [#13208](https://github.com/apache/pinot/pull/13208)
* Assign default value to newly added derived column upon reload [#12648](https://github.com/apache/pinot/pull/12648)
* Avoid port conflict in integration tests [#13390](https://github.com/apache/pinot/pull/13390)
* Better handling of null tableNames [#12654](https://github.com/apache/pinot/pull/12654)
* CLP as a compressionCodec [#12504](https://github.com/apache/pinot/pull/12504)
* Change helm app version to 1.0.0 for Apache Pinot latest release version [#12436](https://github.com/apache/pinot/pull/12436)
* Clean Google Dependencies [#13297](https://github.com/apache/pinot/pull/13297)
* Clean up BrokerRequestHandler and BrokerResponse [#13179](https://github.com/apache/pinot/pull/13179)
* Clean up arbitrary sleep in /GrpcBrokerClusterIntegrationTest [#12379](https://github.com/apache/pinot/pull/12379)
* Cleaning up vector index comments and exceptions [#13150](https://github.com/apache/pinot/pull/13150)
* Cleanup HTTP components dependencies and upgrade Thrift [#12905](https://github.com/apache/pinot/pull/12905)
* Cleanup Javax and Jakarta dependencies [#12760](https://github.com/apache/pinot/pull/12760)
* Cleanup deprecated query options [#13040](https://github.com/apache/pinot/pull/13040)
* Cleanup the consumer interfaces and legacy code [#12697](https://github.com/apache/pinot/pull/12697)
* Cleanup unnecessary dependencies under pinot-s3 [#12904](https://github.com/apache/pinot/pull/12904)
* Cleanup unused aggregate internal hint [#13295](https://github.com/apache/pinot/pull/13295)
* Consistency in API response for live broker [#12201](https://github.com/apache/pinot/pull/12201)
* Consolidate bouncycastle libraries [#12706](https://github.com/apache/pinot/pull/12706)
* Consolidate nimbus-jose-jwt version to 9.37.3 [#12609](https://github.com/apache/pinot/pull/12609)
* ControllerRequestClient accepts headers. Useful for authN tests [#13481](https://github.com/apache/pinot/pull/13481)
* Custom configuration property reader for segment metadata files [#12440](https://github.com/apache/pinot/pull/12440)
* Delete database API [#12765](https://github.com/apache/pinot/pull/12765)
* Deprecate PinotHelixResourceManager#getAllTables() in favour of getAllTables(String databaseName) [#12782](https://github.com/apache/pinot/pull/12782)
* Detect expired messages in Kafka. Log and set a gauge. [#12608](https://github.com/apache/pinot/pull/12608)
* Do not hard code resource class in BaseClusterIntegrationTest [#13400](https://github.com/apache/pinot/pull/13400)
* Do not pause ingestion when upsert snapshot flow errors out [#13257](https://github.com/apache/pinot/pull/13257)
* Don't drop original field during flatten [#13490](https://github.com/apache/pinot/pull/13490)
* Don't enforce -realTimeInstanceCount and -offlineInstanceCount options when creating broker tenants [#13236](https://github.com/apache/pinot/pull/13236)
* Egalpin/skip indexes minor changes [#12514](https://github.com/apache/pinot/pull/12514)
* Emit Metrics for Broker Adaptive Server Selector type [#12482](https://github.com/apache/pinot/pull/12482)
* Emit table size related metrics only in lead controller [#12747](https://github.com/apache/pinot/pull/12747)
* Enable complexType handling in SegmentProcessFramework [#12942](https://github.com/apache/pinot/pull/12942)
* Enable more integration tests to run on the v2 multi-stage query engine [#13467](https://github.com/apache/pinot/pull/13467)
* Enabling avroParquet to read Int96 as bytes [#12484](https://github.com/apache/pinot/pull/12484)
* Enhance Kinesis consumer [#12806](https://github.com/apache/pinot/pull/12806)
* Enhance Parquet Test [#13082](https://github.com/apache/pinot/pull/13082)
* Enhance ProtoSerializationUtils to handle class move [#12946](https://github.com/apache/pinot/pull/12946)
* Enhance Pulsar consumer [#12812](https://github.com/apache/pinot/pull/12812)
* Enhance PulsarConsumerTest [#12948](https://github.com/apache/pinot/pull/12948)
* Enhance commit threshold to accept size threshold without setting rows to 0 [#12684](https://github.com/apache/pinot/pull/12684)
* Enhance json index to support regexp and range predicate evaluation [#12568](https://github.com/apache/pinot/pull/12568)
* Enhancement: Sketch value aggregator performance [#13020](https://github.com/apache/pinot/pull/13020)
* Ensure FieldConfig.getEncodingType() is never null [#12430](https://github.com/apache/pinot/pull/12430)
* Ensure all the lists used in PinotQuery are ArrayList [#13017](https://github.com/apache/pinot/pull/13017)
* Ensure brokerId and requestId are always set in BrokerResponse [#13200](https://github.com/apache/pinot/pull/13200)
* Enter segment preloading at partition level [#12451](https://github.com/apache/pinot/pull/12451)
* Exclude dimensions from star-tree index stored type check [#13355](https://github.com/apache/pinot/pull/13355)
* Expose more helper API in TableDataManager [#13147](https://github.com/apache/pinot/pull/13147)
* Extend compatibility verifier operation timeout from 1m to 2m to reduce flakiness [#13338](https://github.com/apache/pinot/pull/13338)
* Extract json individual array elements from json index for the transform function jsonExtractIndex [#12466](https://github.com/apache/pinot/pull/12466)
* Fetch query quota capacity utilization rate metric in a callback function [#12767](https://github.com/apache/pinot/pull/12767)
* First with time [#12235](https://github.com/apache/pinot/pull/12235)
* GitHub Actions checkout v4 [#12550](https://github.com/apache/pinot/pull/12550)
* Gzip compression, ensure uncompressed size can be calculated from compressed buffer [#12802](https://github.com/apache/pinot/pull/12802)
* Handle errors gracefully during multi-stage stats collection in the broker [#13496](https://github.com/apache/pinot/pull/13496)
* Handle shaded classes in all methods of kafka factory [#13087](https://github.com/apache/pinot/pull/13087)
* Hash Function for UUID Primary Keys [#12538](https://github.com/apache/pinot/pull/12538)
* Ignore case when checking for Direct Memory OOM [#12657](https://github.com/apache/pinot/pull/12657)
* Improve Retention Manager Segment Lineage Clean Up [#13232](https://github.com/apache/pinot/pull/13232)
* Improve error message for max rows in join limit breach [#13394](https://github.com/apache/pinot/pull/13394)
* Improve exception logging when we fail to index / transform message [#12594](https://github.com/apache/pinot/pull/12594)
* Improve logging in range index handler for index updates [#13381](https://github.com/apache/pinot/pull/13381)
* Improve upsert compaction threshold validations [#13424](https://github.com/apache/pinot/pull/13424)
* Improve warn logs for requesting validDocID snapshots [#13280](https://github.com/apache/pinot/pull/13280)
* Improved metrics for server grpc query [#13177](https://github.com/apache/pinot/pull/13177)
* Improved null check for varargs [#12673](https://github.com/apache/pinot/pull/12673)
* Improved segment build time for Lucene text index realtime to offline conversion [#12744](https://github.com/apache/pinot/pull/12744)
* In ClusterTest, make start port higher to avoid potential conflict with Kafka [#13402](https://github.com/apache/pinot/pull/13402)
* Introduce PinotLogicalAggregate and remove internal hint [#13291](https://github.com/apache/pinot/pull/13291)
* Introduce retries while creating stream message decoder for more robustness [#13036](https://github.com/apache/pinot/pull/13036)
* Isolate bad server configs during broker startup phase [#12931](https://github.com/apache/pinot/pull/12931)
* Issue #12367 [#12922](https://github.com/apache/pinot/pull/12922)
* Json extract index filter support [#12683](https://github.com/apache/pinot/pull/12683)
* Json extract index mv [#12532](https://github.com/apache/pinot/pull/12532)
* Keep get tables API with and without database [#12804](https://github.com/apache/pinot/pull/12804)
* Lint failure [#12294](https://github.com/apache/pinot/pull/12294)
* Logging a warn message instead of throwing exception [#12546](https://github.com/apache/pinot/pull/12546)
* Made the error message around dimension table size clearer [#13163](https://github.com/apache/pinot/pull/13163)
* Make Helix state transition handling idempotent [#12886](https://github.com/apache/pinot/pull/12886)
* Make KafkaConsumerFactory method less restrictive to avoid incompatibility [#12815](https://github.com/apache/pinot/pull/12815)
* Make task manager APIs database aware [#12766](https://github.com/apache/pinot/pull/12766)
* Metric for count of tables configured with various tier backends [#12940](https://github.com/apache/pinot/pull/12940)
* Metric for upsert tables count [#12505](https://github.com/apache/pinot/pull/12505)
* Metrics for Realtime Rows Fetched and Stream Consumer Create Exceptions [#12522](https://github.com/apache/pinot/pull/12522)
* Minmaxrange null [#12252](https://github.com/apache/pinot/pull/12252)
* Modify consumingSegmentsInfo endpoint to indicate how many servers failed [#12523](https://github.com/apache/pinot/pull/12523)
* Move offset validation logic to consumer classes [#13015](https://github.com/apache/pinot/pull/13015)
* Move package org.apache.calcite to org.apache.pinot.calcite [#12837](https://github.com/apache/pinot/pull/12837)
* Move resolveComparisonTies from addOrReplaceSegment to base class [#13396](https://github.com/apache/pinot/pull/13396)
* Move some mispositioned tests under pinot-core [#12884](https://github.com/apache/pinot/pull/12884)
* Move wildfly-openssl dependency management to root pom [#12597](https://github.com/apache/pinot/pull/12597)
* Moving deleteSegment call from POST to DELETE call [#12663](https://github.com/apache/pinot/pull/12663)
* Optimize unnecessary extra array allocation and conversion for raw derived column during segment reload [#13115](https://github.com/apache/pinot/pull/13115)
* Pass explicit TypeRef when evaluating MV jsonPath [#12524](https://github.com/apache/pinot/pull/12524)
* Percentile operations supporting null [#12271](https://github.com/apache/pinot/pull/12271)
* Prepare for next development iteration [#12530](https://github.com/apache/pinot/pull/12530)
* Propagate Disable User Agent Config to Http Client [#12479](https://github.com/apache/pinot/pull/12479)
* Properly handle complex type transformer in segment processor framework [#13258](https://github.com/apache/pinot/pull/13258)
* Properly return response if SegmentCompletion is aborted [#13206](https://github.com/apache/pinot/pull/13206)
* Publish helm 0.2.8 [#12465](https://github.com/apache/pinot/pull/12465)
* Publish helm 0.2.9 [#13230](https://github.com/apache/pinot/pull/13230)
* Pull janino dependency to root pom [#12724](https://github.com/apache/pinot/pull/12724)
* Pull pulsar version definitaion into root POM [#13002](https://github.com/apache/pinot/pull/13002)
* Query response opt [#13420](https://github.com/apache/pinot/pull/13420)
* Re-enable the Spotless plugin for Java 21 [#12992](https://github.com/apache/pinot/pull/12992)
* Readme - How to setup Pinot UI for development [#12408](https://github.com/apache/pinot/pull/12408)
* Record enricher [#12243](https://github.com/apache/pinot/pull/12243)
* Refactor PinotTaskManager class [#12964](https://github.com/apache/pinot/pull/12964)
* Refactored CommonsConfigurationUtils for loading properties configuration. [#13201](https://github.com/apache/pinot/pull/13201)
* Refactored compatibility-verifier module [#13359](https://github.com/apache/pinot/pull/13359)
* Refactoring removeSegment flow in upsert [#13449](https://github.com/apache/pinot/pull/13449)
* Refine PeerServerSegmentFinder [#12933](https://github.com/apache/pinot/pull/12933)
* Refine SegmentFetcherFactory [#12936](https://github.com/apache/pinot/pull/12936)
* Replace custom fmpp plugin with fmpp-maven-plugin [#12737](https://github.com/apache/pinot/pull/12737)
* Reposition query submission spot for adaptive server selection [#13327](https://github.com/apache/pinot/pull/13327)
* Reset controller port when stopping the controller in ControllerTest [#13399](https://github.com/apache/pinot/pull/13399)
* Rest Endpoint to Create ZNode [#12497](https://github.com/apache/pinot/pull/12497)
* Return clear error message when no common broker found for multi-stage query with tables from different tenants [#13235](https://github.com/apache/pinot/pull/13235)
* Returning tables names failing authorization in Exception of Multi State Engine Queries [#13195](https://github.com/apache/pinot/pull/13195)
* Revert " Adding record reader config/context param to record transformer (#12520)" [#12526](https://github.com/apache/pinot/pull/12526)
* Revert "Using local copy of segment instead of downloading from remote (#12863)" [#13114](https://github.com/apache/pinot/pull/13114)
* Short circuit SubPlanFragmenter because we don't support multiple sub-plans yet [#13306](https://github.com/apache/pinot/pull/13306)
* Simplify Google dependencies by importing BOM [#12456](https://github.com/apache/pinot/pull/12456)
* Specify version for commons-validator [#12935](https://github.com/apache/pinot/pull/12935)
* Support NOT in StarTree Index [#12988](https://github.com/apache/pinot/pull/12988)
* Support empty strings as json nodes^ [#12555](https://github.com/apache/pinot/pull/12555)
* Supporting human-readable format when configuring broker response size [#12510](https://github.com/apache/pinot/pull/12510)
* Use ArrayList instead of LinkedList in SortOperator [#12783](https://github.com/apache/pinot/pull/12783)
* Use a two server setup for multi-stage query engine backward compatibility regression test suite [#13371](https://github.com/apache/pinot/pull/13371)
* Use more efficient variants of URLEncoder::encode and URLDecoder::decode [#13030](https://github.com/apache/pinot/pull/13030)
* Use parameterized log messages instead of string concatenation [#13145](https://github.com/apache/pinot/pull/13145)
* Use separate action for /tasks/scheduler/jobDetails API [#13054](https://github.com/apache/pinot/pull/13054)
* Use try-with-resources to close file walk stream in LocalPinotFS [#13029](https://github.com/apache/pinot/pull/13029)
* Using local copy of segment instead of downloading from remote [#12863](https://github.com/apache/pinot/pull/12863)
* \[Adaptive Server Selector] Add metrics for Stats Manager Queue Size [#12340](https://github.com/apache/pinot/pull/12340)
* \[Cleanup] Move classes in pinot-common to the correct package [#13478](https://github.com/apache/pinot/pull/13478)
* \[Feature] Add Support for SQL Formatting in Query Editor [#11725](https://github.com/apache/pinot/pull/11725)
* \[HELM]: Added additional probes options and startup probe. [#13165](https://github.com/apache/pinot/pull/13165)
* \[HELM]: Added checksum config annotation in stateful set for broker, controller and server [#13059](https://github.com/apache/pinot/pull/13059)
* \[HELM]: Added namespace support in K8s deployment. [#13380](https://github.com/apache/pinot/pull/13380)
* \[HELM]: zookeeper chart upgrade to version 13.2.0 [#13083](https://github.com/apache/pinot/pull/13083)
* \[Minor] Add Nullable annotation to HttpHeaders in BrokerRequestHandler [#12816](https://github.com/apache/pinot/pull/12816)
* \[Minor] Small refactor of raw index creator constructor to be more clear [#13093](https://github.com/apache/pinot/pull/13093)
* \[Multi-stage] Clean up RelNode to Operator handling [#13325](https://github.com/apache/pinot/pull/13325)
* \[null-aggr] Add null handling support in `mode` aggregation [#12227](https://github.com/apache/pinot/pull/12227)
* \[partial-upsert] configure early release of \_partitionGroupConsumerSemaphore in RealtimeSegmentDataManager [#13256](https://github.com/apache/pinot/pull/13256)
* \[spark-connector] Add option to fail read when there are invalid segments [#13080](https://github.com/apache/pinot/pull/13080)
* add Netty arm64 dependencies [#12493](https://github.com/apache/pinot/pull/12493)
* add Netty unit test [#12486](https://github.com/apache/pinot/pull/12486)
* add SegmentContext to collect validDocIds bitmaps for many segments together [#12694](https://github.com/apache/pinot/pull/12694)
* add `skipUnavailableServers` query option [#13387](https://github.com/apache/pinot/pull/13387)
* add insecure mode when Pinot uses TLS connections [#12525](https://github.com/apache/pinot/pull/12525)
* add instrumentation to json index getMatchingFlattenedDocsMap() [#13164](https://github.com/apache/pinot/pull/13164)
* add jmx to promethues metric exporting rule for realtimeRowsFiltered [#12759](https://github.com/apache/pinot/pull/12759)
* add metrics for IdeaState update [#13266](https://github.com/apache/pinot/pull/13266)
* add some metrics for upsert table preloading [#12722](https://github.com/apache/pinot/pull/12722)
* add some tests on jsonPathString [#12954](https://github.com/apache/pinot/pull/12954)
* add test cases in RequestUtilsTest [#12557](https://github.com/apache/pinot/pull/12557)
* add unit test for JsonAsyncHttpPinotClientTransport [#12633](https://github.com/apache/pinot/pull/12633)
* add unit test for QueryServer [#12599](https://github.com/apache/pinot/pull/12599)
* add unit test for ServerChannels [#12616](https://github.com/apache/pinot/pull/12616)
* add unit test for StringFunctions encodeUrl [#13391](https://github.com/apache/pinot/pull/13391)
* add unit tests for pinot-jdbc-client [#13137](https://github.com/apache/pinot/pull/13137)
* add url assertion to SegmentCompletionProtocolTest [#13373](https://github.com/apache/pinot/pull/13373)
* adjust the llc partition consuming metric reporting logic [#12627](https://github.com/apache/pinot/pull/12627)
* allow passing null http headers object to translateTableName [#12764](https://github.com/apache/pinot/pull/12764)
* allow to set segment when use SegmentProcessorFramework [#13341](https://github.com/apache/pinot/pull/13341)
* auto renew jvm default sslconext when it's loaded from files [#12462](https://github.com/apache/pinot/pull/12462)
* avoid useless intermediate byte array allocation for VarChunkV4Reader's getStringMV [#12978](https://github.com/apache/pinot/pull/12978)
* aws sdk 2.25.3 [#12562](https://github.com/apache/pinot/pull/12562)
* build-helper-maven-plugin 3.5.0 [#12548](https://github.com/apache/pinot/pull/12548)
* cache ssl contexts and reuse them [#12404](https://github.com/apache/pinot/pull/12404)
* clean up jetbrain nullable annotation [#13427](https://github.com/apache/pinot/pull/13427)
* cleanup: maven no transfer progress [#12444](https://github.com/apache/pinot/pull/12444)
* close JDBC connections [#12494](https://github.com/apache/pinot/pull/12494)
* do not fail on duplicate relaxed vars (#13214)z
* dropwizard metrics 4.2.25 [#12600](https://github.com/apache/pinot/pull/12600)
* dynamic chunk sizing for v4 raw forward index [#12945](https://github.com/apache/pinot/pull/12945)
* enable Netty leak detection [#12483](https://github.com/apache/pinot/pull/12483)
* enable parallel Maven in pinot linter script [#12751](https://github.com/apache/pinot/pull/12751)
* ensure inverse And/OrFilterOperator implementations match the query [#13199](https://github.com/apache/pinot/pull/13199)
* exclude .mvn directory from source assembly [#12558](https://github.com/apache/pinot/pull/12558)
* extend CompactedPinotSegmentRecordReader so that it can skip deleteRecord [#13352](https://github.com/apache/pinot/pull/13352)
* get startTime outside the executor task to avoid flaky time checks [#13250](https://github.com/apache/pinot/pull/13250)
* handle absent segments so that catchup checker doesn't get stuck on them [#12883](https://github.com/apache/pinot/pull/12883)
* handle overflow for `MutableOffHeapByteArrayStore` buffer starting size [#13215](https://github.com/apache/pinot/pull/13215)
* handle segments not tracked by partition mgr and add skipUpsertView query option [#13415](https://github.com/apache/pinot/pull/13415)
* handle table name translation on missed api resources [#12792](https://github.com/apache/pinot/pull/12792)
* hash4j version upgrade to 0.17.0 [#12968](https://github.com/apache/pinot/pull/12968)
* including the underlying exception in the logging output [#13248](https://github.com/apache/pinot/pull/13248)
* int96 parity with native parquet reader [#12496](https://github.com/apache/pinot/pull/12496)
* jsonExtractIndex support array of default values [#12748](https://github.com/apache/pinot/pull/12748)
* log the log rate limiter rate for dropped broker logs [#13041](https://github.com/apache/pinot/pull/13041)
* make http listener ssl config swappable [#12455](https://github.com/apache/pinot/pull/12455)
* make reflection calls compatible with 0.9.11 \[#12958]\(<https://github.com/apache/>
* maven: no transfer progress [#12528](https://github.com/apache/pinot/pull/12528)
* missed to delete the temp dir [#12637](https://github.com/apache/pinot/pull/12637)
* move shouldReplaceOnComparisonTie to base class to be more reusable [#13353](https://github.com/apache/pinot/pull/13353)
* reduce Java enum .values() usage in TimerContext [#12579](https://github.com/apache/pinot/pull/12579)
* reduce logging for SpecialValueTransformer [#12970](https://github.com/apache/pinot/pull/12970)
* reduce regex pattern compilation in Pinot jdbc [#13138](https://github.com/apache/pinot/pull/13138)
* refactor TlsUtils class [#12515](https://github.com/apache/pinot/pull/12515)
* refine when to registerSegment while doing addSegment and replaceSegment for upsert tables for better data consistency [#12709](https://github.com/apache/pinot/pull/12709)
* reformat AdminConsoleIntegrationTest.java [#12552](https://github.com/apache/pinot/pull/12552)
* reformat ClusterTest.java [#12531](https://github.com/apache/pinot/pull/12531)
* release segment mgrs more reliably [#13216](https://github.com/apache/pinot/pull/13216)
* replaced getServer with getServers [#12545](https://github.com/apache/pinot/pull/12545)
* report rebalance job status for the early returns like noops [#13281](https://github.com/apache/pinot/pull/13281)
* require noDictionaryColumns with aggregationConfigs [#12464](https://github.com/apache/pinot/pull/12464)
* share the same table config object [#12463](https://github.com/apache/pinot/pull/12463)
* track segments for snapshotting even if they lost all comparisons [#13388](https://github.com/apache/pinot/pull/13388)
* untrack the segment out of TTL [#12449](https://github.com/apache/pinot/pull/12449)
* update ControllerJobType from enum to string [#12518](https://github.com/apache/pinot/pull/12518)
* update RewriterConstants so that expr min max would not collide with columns start with "parent" [#13357](https://github.com/apache/pinot/pull/13357)
* update access control check error handling to catch throwable and log errors [#13209](https://github.com/apache/pinot/pull/13209)

## Bug Fixes

* Use gte(lte) to replace between() which has a bug [#12595](https://github.com/apache/pinot/pull/12595)
* Fix the ConcurrentModificationException for And/Or DocIdSet [#12611](https://github.com/apache/pinot/pull/12611)
* Upgrade RoaringBitmap to 1.0.5 to pick up the fix for RangeBitmap.between() [#12604](https://github.com/apache/pinot/pull/12604)
* bugfix: do not move src ByteBuffer position for LZ4 length prefixed decompress [#12539](https://github.com/apache/pinot/pull/12539)
* Bug Fix createDictionaryForColumn does not take into account inverted index [#13048](https://github.com/apache/pinot/pull/13048)
* fix Cluster Manager error [#12632](https://github.com/apache/pinot/pull/12632)
* fix for quick start Cluster Manager issue [#12610](https://github.com/apache/pinot/pull/12610)
* Adding config for having suffix for client ID for realtime consumer [#13168](https://github.com/apache/pinot/pull/13168)
* Addressed comments and fixed tests from pull request 12389. /uptime and /start-time endpoints working all components [#12512](https://github.com/apache/pinot/pull/12512)
* Bigfix. Added missing paramName [#13060](https://github.com/apache/pinot/pull/13060)
* Bug fix: Do not ignore scheme property [#12332](https://github.com/apache/pinot/pull/12332)
* Bug fix: Handle missing shade config overwrites for Kafka [#13437](https://github.com/apache/pinot/pull/13437)
* BugFix: Fix merge result from more than one server [#12778](https://github.com/apache/pinot/pull/12778)
* Bugfix. Allow tenant rebalance with downtime as true [#13246](https://github.com/apache/pinot/pull/13246)
* Bugfix. Avoid passing null table name input to translation util [#12726](https://github.com/apache/pinot/pull/12726)
* Bugfix. Correct wrong method call from scheduleTask() to scheduleTaskForDatabase() [#12791](https://github.com/apache/pinot/pull/12791)
* Bugfix. Maintain literal data type during function evaluation [#12607](https://github.com/apache/pinot/pull/12607)
* Cleanup: Fix grammar in error message, also improve readability. [#13451](https://github.com/apache/pinot/pull/13451)
* Fix Bug in Handling Equal Comparison Column Values in Upsert [#12395](https://github.com/apache/pinot/pull/12395)
* Fix ColumnMinMaxValueGenerator [#12502](https://github.com/apache/pinot/pull/12502)
* Fix JavaEE related dependencies [#13058](https://github.com/apache/pinot/pull/13058)
* Fix Logging Location for CPU-Based Query Killing [#13318](https://github.com/apache/pinot/pull/13318)
* Fix PulsarUtils to not share buffer [#12671](https://github.com/apache/pinot/pull/12671)
* Fix URI construction so that AddSchema command line tool works when override flag is set to true [#13320](https://github.com/apache/pinot/pull/13320)
* Fix \[Type]ArrayList elements() method usage [#13354](https://github.com/apache/pinot/pull/13354)
* Fix a typo when calculating query freshness [#12947](https://github.com/apache/pinot/pull/12947)
* Fix an overflow in PinotDataBuffer.readFrom [#13152](https://github.com/apache/pinot/pull/13152)
* Fix bug in logging in UpsertCompaction task [#12419](https://github.com/apache/pinot/pull/12419)
* Fix bug to return validDocIDsMetadata from all servers [#12431](https://github.com/apache/pinot/pull/12431)
* Fix connection issues if using JDBC and Hikari (#12267) [#12411](https://github.com/apache/pinot/pull/12411)
* Fix controller host / port / protocol CLI option description for admin commands [#13237](https://github.com/apache/pinot/pull/13237)
* Fix environment variables not applied when creating table [#12560](https://github.com/apache/pinot/pull/12560)
* Fix error message for insufficient number of untagged brokers during tenant creation [#13234](https://github.com/apache/pinot/pull/13234)
* Fix few metric rules which were affected by the database prefix handling [#13290](https://github.com/apache/pinot/pull/13290)
* Fix file handle leaks in Pinot Driver (apache#12263) [#12356](https://github.com/apache/pinot/pull/12356)
* Fix flakiness of ControllerPeriodicTasksIntegrationTest [#13337](https://github.com/apache/pinot/pull/13337)
* Fix issue with startree index metadata loading for columns with '\_\_' in name [#12554](https://github.com/apache/pinot/pull/12554)
* Fix metric rule pattern regex [#12856](https://github.com/apache/pinot/pull/12856)
* Fix pinot-parquet NoClassFound issue [#12615](https://github.com/apache/pinot/pull/12615)
* Fix segment size check in OfflineClusterIntegrationTest [#13389](https://github.com/apache/pinot/pull/13389)
* Fix some resource leak in tests [#12794](https://github.com/apache/pinot/pull/12794)
* Fix the NPE from IS update metrics [#13313](https://github.com/apache/pinot/pull/13313)
* Fix the NPE when metadataTTL is enabled without delete column [#13262](https://github.com/apache/pinot/pull/13262)
* Fix the ServletConfig loading issue with swagger. [#13122](https://github.com/apache/pinot/pull/13122)
* Fix the issue that map flatten shouldn't remove the map field from the record [#13243](https://github.com/apache/pinot/pull/13243)
* Fix the race condition for H3InclusionIndexFilterOperator [#12487](https://github.com/apache/pinot/pull/12487)
* Fix the time segment pruner on TIMESTAMP data type [#12789](https://github.com/apache/pinot/pull/12789)
* Fix time stats in SegmentIndexCreationDriverImpl [#13429](https://github.com/apache/pinot/pull/13429)
* Fixed infer logical type name from avro union schema [#13224](https://github.com/apache/pinot/pull/13224)
* Fixing instance type to resolve [#12677](https://github.com/apache/pinot/issues/12677) and [#12678](https://github.com/apache/pinot/pull/12678)
* Helm: bug fix for chart rendering issue. [#13264](https://github.com/apache/pinot/pull/13264)
* Try to amend kafka common package with pinot shaded package prefix [#13056](https://github.com/apache/pinot/pull/13056)
* Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes [#13314](https://github.com/apache/pinot/pull/13314)
* Upgrade com.microsoft.azure:msal4j from 1.3.5 to 1.3.10 for CVE fixing [#12580](https://github.com/apache/pinot/pull/12580)
* \[bugfix] Handling null value for kafka client id suffix [#13279](https://github.com/apache/pinot/pull/13279)
* bugfix: fixing jdbc client sql feature not supported exception [#12480](https://github.com/apache/pinot/pull/12480)
* bugfix: re-add support for `not text_match` [#12372](https://github.com/apache/pinot/pull/12372)
* bugfix: reduce enum array allocation in QueryLogger [#12478](https://github.com/apache/pinot/pull/12478)
* bugfix: use consumerDir during lucene realtime segment conversion [#13094](https://github.com/apache/pinot/pull/13094)
* cleanup: fix apache rat violation [#12476](https://github.com/apache/pinot/pull/12476)
* fix GuavaRateLimiter acquire method [#12500](https://github.com/apache/pinot/pull/12500)
* fix fieldsToRead class not in decoder [#13186](https://github.com/apache/pinot/pull/13186)
* fix flakey test, avoid early finalization [#13095](https://github.com/apache/pinot/pull/13095)
* fix merging null multi value in partial upsert [#13031](https://github.com/apache/pinot/pull/13031)
* fix race condition in `ScalingThreadPoolExecutor` [#13360](https://github.com/apache/pinot/pull/13360)
* fix shared buffer, tests [#12587](https://github.com/apache/pinot/pull/12587)
* fix(build): update node version to 16 [#12924](https://github.com/apache/pinot/pull/12924)
* fixing CVE critical issues by resolving kerby/jline and wildfly libraries [#12566](https://github.com/apache/pinot/pull/12566)
* fixing pinot-adls high severity CVEs [#12571](https://github.com/apache/pinot/pull/12571)
* fixing swagger setup using localhost as host name [#13254](https://github.com/apache/pinot/pull/13254)
* swagger-ui upgrade to 5.15.0 Fixes [#12908](https://github.com/apache/pinot/pull/12908)
* upgrade jettison version to fix CVE [#12567](https://github.com/apache/pinot/pull/12567)