# 0.12.0

## Multi-Stage Query Engine

### New join semantics support

* Left join ([#9466](https://github.com/apache/pinot/pull/9466))
* In-equi join ([#9448](https://github.com/apache/pinot/pull/9448))
* Full join ([#9907](https://github.com/apache/pinot/pull/9907))
* Right join ([#9907](https://github.com/apache/pinot/pull/9907))
* Semi join ([#9367](https://github.com/apache/pinot/pull/9367))
* Using keyword ([#9373](https://github.com/apache/pinot/pull/9373))

### New sql semantics support:

* Having ([#9274](https://github.com/apache/pinot/pull/9274))
* Order by ([#9279](https://github.com/apache/pinot/pull/9279))
* In/NotIn clause ([#9374](https://github.com/apache/pinot/pull/9374))
* Cast ([#9384](https://github.com/apache/pinot/pull/9384))
* LIke/Rexlike ([#9654](https://github.com/apache/pinot/pull/9654))
* Range predicate ([#9445](https://github.com/apache/pinot/pull/9445))

### Performance enhancement

* Thread safe query planning ([#9344](https://github.com/apache/pinot/pull/9344))
* Partial query execution and round robin scheduling ([#9753](https://github.com/apache/pinot/pull/9753))
* Improve data table serde ([#9731](https://github.com/apache/pinot/pull/9731))

## Major updates

* Force commit consuming segments by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9197](https://github.com/apache/pinot/pull/9197)
* add a freshness based consumption status checker by [@jadami10](https://github.com/jadami10) in [#9244](https://github.com/apache/pinot/pull/9244)
* Add metrics to track controller segment download and upload requests in progress by [@gviedma](https://github.com/gviedma) in [#9258](https://github.com/apache/pinot/pull/9258)
* Adding endpoint to download local log files for each component by [@xiangfu0](https://github.com/xiangfu0) in [#9259](https://github.com/apache/pinot/pull/9259)
* \[Feature] Add an option to search input files recursively in ingestion job. The default is set to true to be backward compatible. by [@61yao](https://github.com/61yao) in [#9265](https://github.com/apache/pinot/pull/9265)
* add query cancel APIs on controller backed by those on brokers by [@klsince](https://github.com/klsince) in [#9276](https://github.com/apache/pinot/pull/9276)
* Add Spark Job Launcher tool by [@KKcorps](https://github.com/KKcorps) in [#9288](https://github.com/apache/pinot/pull/9288)
* Enable Consistent Data Push for Standalone Segment Push Job Runners by [@yuanbenson](https://github.com/yuanbenson) in [#9295](https://github.com/apache/pinot/pull/9295)
* Allow server to directly return the final aggregation result by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9304](https://github.com/apache/pinot/pull/9304)
* TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by [@klsince](https://github.com/klsince) in [#9306](https://github.com/apache/pinot/pull/9306)
* Adaptive Server Selection by [@vvivekiyer](https://github.com/vvivekiyer) in [#9311](https://github.com/apache/pinot/pull/9311)
* \[Feature] Support IsDistinctFrom and IsNotDistinctFrom by [@61yao](https://github.com/61yao) in [#9312](https://github.com/apache/pinot/pull/9312)
* Allow ingestion of errored records with incorrect datatype by [@KKcorps](https://github.com/KKcorps) in [#9320](https://github.com/apache/pinot/pull/9320)
* Allow setting custom time boundary for hybrid table queries by [@saurabhd336](https://github.com/saurabhd336) in [#9356](https://github.com/apache/pinot/pull/9356)
* skip late cron job with max allowed delay by [@klsince](https://github.com/klsince) in [#9372](https://github.com/apache/pinot/pull/9372)
* Do not allow implicit cast for BOOLEAN and TIMESTAMP by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9385](https://github.com/apache/pinot/pull/9385)
* Add missing properties in CSV plugin by [@KKcorps](https://github.com/KKcorps) in [#9399](https://github.com/apache/pinot/pull/9399)
* set MDC so that one can route minion task logs to separate files cleanly by [@klsince](https://github.com/klsince) in [#9400](https://github.com/apache/pinot/pull/9400)
* Add a new API to fix segment date time in metadata by [@KKcorps](https://github.com/KKcorps) in [#9413](https://github.com/apache/pinot/pull/9413)
* Update get bytes to return raw bytes of string and support getBytesMV by [@61yao](https://github.com/61yao) in [#9441](https://github.com/apache/pinot/pull/9441)
* Exposing consumer's record lag in /consumingSegmentsInfo by [@navina](https://github.com/navina) in [#9515](https://github.com/apache/pinot/pull/9515)
* Do not create dictionary for high-cardinality columns by [@KKcorps](https://github.com/KKcorps) in [#9527](https://github.com/apache/pinot/pull/9527)
* get task runtime configs tracked in Helix by [@klsince](https://github.com/klsince) in [#9540](https://github.com/apache/pinot/pull/9540)
* Add more options to json index by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9543](https://github.com/apache/pinot/pull/9543)
* add SegmentTierAssigner and refine restful APIs to get segment tier info by [@klsince](https://github.com/klsince) in [#9598](https://github.com/apache/pinot/pull/9598)
* Add segment level debug API by [@saurabhd336](https://github.com/saurabhd336) in [#9609](https://github.com/apache/pinot/pull/9609)
* Add record availability lag for Kafka connector by [@navina](https://github.com/navina) in [#9621](https://github.com/apache/pinot/pull/9621)
* notify servers that need to move segments to new tiers via SegmentReloadMessage by [@klsince](https://github.com/klsince) in [#9624](https://github.com/apache/pinot/pull/9624)
* Allow to configure multi-datadirs as instance configs and a Quickstart example about them by [@klsince](https://github.com/klsince) in [#9705](https://github.com/apache/pinot/pull/9705)
* Customize stopword for Lucene Index by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9708](https://github.com/apache/pinot/pull/9708)
* Add memory optimized dimension table by [@KKcorps](https://github.com/KKcorps) in [#9802](https://github.com/apache/pinot/pull/9802)
* ADLS file system upgrade by [@xiangfu0](https://github.com/xiangfu0) in [#9855](https://github.com/apache/pinot/pull/9855)
* Added Delete Schema/Table pinot admin commands by [@bagipriyank](https://github.com/bagipriyank) in [#9857](https://github.com/apache/pinot/pull/9857)
* Adding new ADLSPinotFS auth type: DEFAULT by [@xiangfu0](https://github.com/xiangfu0) in [#9860](https://github.com/apache/pinot/pull/9860)
* Add rate limit to Kinesis requests by [@KKcorps](https://github.com/KKcorps) in [#9863](https://github.com/apache/pinot/pull/9863)
* Adding configs for zk client timeout by [@xiangfu0](https://github.com/xiangfu0) in [#9975](https://github.com/apache/pinot/pull/9975)

## Other features/changes

* Show most recent scheduling errors by [@satishwaghela](https://github.com/satishwaghela) in [#9161](https://github.com/apache/pinot/pull/9161)
* Do not use aggregation result for distinct query in IntermediateResultsBlock by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9262](https://github.com/apache/pinot/pull/9262)
* Emit metrics for ratio of actual consumption rate to rate limit in real-time tables by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9201](https://github.com/apache/pinot/pull/9201)
* add metrics entry offlineTableCount by [@walterddr](https://github.com/walterddr) in [#9270](https://github.com/apache/pinot/pull/9270)
* refine query cancel resp msg by [@klsince](https://github.com/klsince) in [#9242](https://github.com/apache/pinot/pull/9242)
* add @ManualAuthorization annotation for non-standard endpoints by [@apucher](https://github.com/apucher) in [#9252](https://github.com/apache/pinot/pull/9252)
* Optimize ser/de to avoid using output stream by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9278](https://github.com/apache/pinot/pull/9278)
* Add Support for Covariance Function by [@SabrinaZhaozyf](https://github.com/SabrinaZhaozyf) in [#9236](https://github.com/apache/pinot/pull/9236)
* Throw an exception when MV columns are present in the order-by expression list in selection order-by only queries by [@somandal](https://github.com/somandal) in [#9078](https://github.com/apache/pinot/pull/9078)
* Improve server query cancellation and timeout checking during execution by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9286](https://github.com/apache/pinot/pull/9286)
* Add capabilities to ingest from another stream without disabling the real-time table by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9289](https://github.com/apache/pinot/pull/9289)
* Add minMaxInvalid flag to avoid unnecessary needPreprocess by [@npawar](https://github.com/npawar) in [#9238](https://github.com/apache/pinot/pull/9238)
* Add array cardinality function by [@walterddr](https://github.com/walterddr) in [#9300](https://github.com/apache/pinot/pull/9300)
* TierBasedSegmentDirectoryLoader to keep segments in multi-datadir by [@klsince](https://github.com/klsince) in [#9306](https://github.com/apache/pinot/pull/9306)
* Add support for custom null values in CSV record reader by [@KKcorps](https://github.com/KKcorps) in [#9318](https://github.com/apache/pinot/pull/9318)
* Infer parquet reader type based on file metadata by [@saurabhd336](https://github.com/saurabhd336) in [#9294](https://github.com/apache/pinot/pull/9294)
* Add Support for Cast Function on MV Columns by [@SabrinaZhaozyf](https://github.com/SabrinaZhaozyf) in [#9296](https://github.com/apache/pinot/pull/9296)
* Allow ingestion of errored records with incorrect datatype by [@KKcorps](https://github.com/KKcorps) in [#9320](https://github.com/apache/pinot/pull/9320)
* \[Feature] Not Operator Transformation by [@61yao](https://github.com/61yao) in [#9330](https://github.com/apache/pinot/pull/9330)
* Handle null string in CSV decoder by [@KKcorps](https://github.com/KKcorps) in [#9340](https://github.com/apache/pinot/pull/9340)
* \[Feature] Not scalar function by [@61yao](https://github.com/61yao) in [#9338](https://github.com/apache/pinot/pull/9338)
* Add support for EXTRACT syntax and converts it to appropriate Pinot expression by [@tanmesh](https://github.com/tanmesh) in [#9184](https://github.com/apache/pinot/pull/9184)
* Add support for Auth in controller requests in java query client by [@KKcorps](https://github.com/KKcorps) in [#9230](https://github.com/apache/pinot/pull/9230)
* delete all related minion task metadata when deleting a table by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9339](https://github.com/apache/pinot/pull/9339)
* BloomFilterRule should only recommend for supported column type by [@yuanbenson](https://github.com/yuanbenson) in [#9364](https://github.com/apache/pinot/pull/9364)
* Support all the types in ParquetNativeRecordReader by [@xiangfu0](https://github.com/xiangfu0) in [#9352](https://github.com/apache/pinot/pull/9352)
* Improve segment name check in metadata push by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9359](https://github.com/apache/pinot/pull/9359)
* Allow expression transformer cotinue on error by [@xiangfu0](https://github.com/xiangfu0) in [#9376](https://github.com/apache/pinot/pull/9376)
* skip late cron job with max allowed delay by [@klsince](https://github.com/klsince) in [#9372](https://github.com/apache/pinot/pull/9372)
* Enhance `and` filter predicate evaluation efficiency by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9336](https://github.com/apache/pinot/pull/9336)
* Deprecate instanceId Config For Broker/Minion Specific Configs by [@ankitsultana](https://github.com/ankitsultana) in [#9308](https://github.com/apache/pinot/pull/9308)
* Optimize combine operator to fully utilize threads by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9387](https://github.com/apache/pinot/pull/9387)
* Terminate the query after plan generation if timeout by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9386](https://github.com/apache/pinot/pull/9386)
* \[Feature] Support IsDistinctFrom and IsNotDistinctFrom by [@61yao](https://github.com/61yao) in [#9312](https://github.com/apache/pinot/pull/9312)
* \[Feature] Support Coalesce for Column Names by [@61yao](https://github.com/61yao) in [#9327](https://github.com/apache/pinot/pull/9327)
* Disable logging for interrupted exceptions in kinesis by [@KKcorps](https://github.com/KKcorps) in [#9405](https://github.com/apache/pinot/pull/9405)
* Benchmark thread cpu time by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9408](https://github.com/apache/pinot/pull/9408)
* Use ISODateTimeFormat as default for SIMPLE\_DATE\_FORMAT by [@KKcorps](https://github.com/KKcorps) in [#9378](https://github.com/apache/pinot/pull/9378)
* Extract the common logic for upsert metadata manager by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9435](https://github.com/apache/pinot/pull/9435)
* Make minion task metadata manager methods more generic by [@saurabhd336](https://github.com/saurabhd336) in [#9436](https://github.com/apache/pinot/pull/9436)
* Always pass clientId to kafka's consumer properties by [@navina](https://github.com/navina) in [#9444](https://github.com/apache/pinot/pull/9444)
* Adaptive Server Selection by [@vvivekiyer](https://github.com/vvivekiyer) in [#9311](https://github.com/apache/pinot/pull/9311)
* Refine IndexHandler methods a bit to make them reentrant by [@klsince](https://github.com/klsince) in [#9440](https://github.com/apache/pinot/pull/9440)
* use MinionEventObserver to track finer grained task progress status on worker by [@klsince](https://github.com/klsince) in [#9432](https://github.com/apache/pinot/pull/9432)
* Allow spaces in input file paths by [@KKcorps](https://github.com/KKcorps) in [#9426](https://github.com/apache/pinot/pull/9426)
* Add support for gracefully handling the errors while transformations by [@KKcorps](https://github.com/KKcorps) in [#9377](https://github.com/apache/pinot/pull/9377)
* Cache Deleted Segment Names in Server to Avoid SegmentMissingError by [@ankitsultana](https://github.com/ankitsultana) in [#9423](https://github.com/apache/pinot/pull/9423)
* Handle Invalid timestamps by [@KKcorps](https://github.com/KKcorps) in [#9355](https://github.com/apache/pinot/pull/9355)
* refine minion worker event observer to track finer grained progress for tasks by [@klsince](https://github.com/klsince) in [#9449](https://github.com/apache/pinot/pull/9449)
* spark-connector should use v2/brokers endpoint by [@itschrispeck](https://github.com/itschrispeck) in [#9451](https://github.com/apache/pinot/pull/9451)
* Remove netty server query support from presto-pinot-driver to remove pinot-core and pinot-segment-local dependencies by [@xiangfu0](https://github.com/xiangfu0) in [#9455](https://github.com/apache/pinot/pull/9455)
* Adaptive Server Selection: Address pending review comments by [@vvivekiyer](https://github.com/vvivekiyer) in [#9462](https://github.com/apache/pinot/pull/9462)
* track progress from within segment processor framework by [@klsince](https://github.com/klsince) in [#9457](https://github.com/apache/pinot/pull/9457)
* Decouple ser/de from DataTable by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9468](https://github.com/apache/pinot/pull/9468)
* collect file info like mtime, length while listing files for free by [@klsince](https://github.com/klsince) in [#9466](https://github.com/apache/pinot/pull/9466)
* Extract record keys, headers and metadata from Stream sources by [@navina](https://github.com/navina) in [#9224](https://github.com/apache/pinot/pull/9224)
* \[pinot-spark-connector] Bump spark connector max inbound message size by [@cbalci](https://github.com/cbalci) in [#9475](https://github.com/apache/pinot/pull/9475)
* refine the minion task progress api a bit by [@klsince](https://github.com/klsince) in [#9482](https://github.com/apache/pinot/pull/9482)
* add parsing for AT TIME ZONE by [@agavra](https://github.com/agavra) in [#9477](https://github.com/apache/pinot/pull/9477)
* Eliminate explosion of metrics due to gapfill queries by [@elonazoulay](https://github.com/elonazoulay) in [#9490](https://github.com/apache/pinot/pull/9490)
* ForwardIndexHandler: Change compressionType during segmentReload by [@vvivekiyer](https://github.com/vvivekiyer) in [#9454](https://github.com/apache/pinot/pull/9454)
* Introduce Segment AssignmentStrategy Interface by [@GSharayu](https://github.com/GSharayu) in [#9309](https://github.com/apache/pinot/pull/9309)
* Add query interruption flag check to broker groupby reduction by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9499](https://github.com/apache/pinot/pull/9499)
* adding optional client payload by [@walterddr](https://github.com/walterddr) in [#9465](https://github.com/apache/pinot/pull/9465)
* \[feature] distinct from scalar functions by [@61yao](https://github.com/61yao) in [#9486](https://github.com/apache/pinot/pull/9486)
* Check data table version on server only for null handling by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9508](https://github.com/apache/pinot/pull/9508)
* Add docId and column name to segment read exception by [@KKcorps](https://github.com/KKcorps) in [#9512](https://github.com/apache/pinot/pull/9512)
* Sort scanning based operators by cardinality in AndDocIdSet evaluation by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9420](https://github.com/apache/pinot/pull/9420)
* Do not fail CI when codecov upload fails by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9522](https://github.com/apache/pinot/pull/9522)
* \[Upsert] persist validDocsIndex snapshot for Pinot upsert optimization by [@deemoliu](https://github.com/deemoliu) in [#9062](https://github.com/apache/pinot/pull/9062)
* broker filter by [@dongxiaoman](https://github.com/dongxiaoman) in [#9391](https://github.com/apache/pinot/pull/9391)
* \[feature] coalesce scalar by [@61yao](https://github.com/61yao) in [#9487](https://github.com/apache/pinot/pull/9487)
* Allow setting custom time boundary for hybrid table queries by [@saurabhd336](https://github.com/saurabhd336) in [#9356](https://github.com/apache/pinot/pull/9356)
* \[GHA] add cache timeout by [@walterddr](https://github.com/walterddr) in [#9524](https://github.com/apache/pinot/pull/9524)
* Optimize PinotHelixResourceManager.hasTable() by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9526](https://github.com/apache/pinot/pull/9526)
* Include exception when upsert metadata manager cannot be created by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9532](https://github.com/apache/pinot/pull/9532)
* allow to config task expire time by [@klsince](https://github.com/klsince) in [#9530](https://github.com/apache/pinot/pull/9530)
* expose task finish time via debug API by [@klsince](https://github.com/klsince) in [#9534](https://github.com/apache/pinot/pull/9534)
* Remove the wrong warning log in KafkaPartitionLevelConsumer by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9536](https://github.com/apache/pinot/pull/9536)
* starting http server for minion worker conditionally by [@klsince](https://github.com/klsince) in [#9542](https://github.com/apache/pinot/pull/9542)
* Make StreamMessage generic and a bug fix by [@vvivekiyer](https://github.com/vvivekiyer) in [#9544](https://github.com/apache/pinot/pull/9544)
* Improve primary key serialization performance by [@KKcorps](https://github.com/KKcorps) in [#9538](https://github.com/apache/pinot/pull/9538)
* \[Upsert] Skip removing upsert metadata when shutting down the server by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9551](https://github.com/apache/pinot/pull/9551)
* add array element at function by [@walterddr](https://github.com/walterddr) in [#9554](https://github.com/apache/pinot/pull/9554)
* Handle the case when enableNullHandling is true and an aggregation function is used w/ a column that has an empty null bitmap by [@nizarhejazi](https://github.com/nizarhejazi) in [#9566](https://github.com/apache/pinot/pull/9566)
* Support segment storage format without forward index by [@somandal](https://github.com/somandal) in [#9333](https://github.com/apache/pinot/pull/9333)
* Adding SegmentNameGenerator type inference if not explicitly set in config by [@timsants](https://github.com/timsants) in [#9550](https://github.com/apache/pinot/pull/9550)
* add version information to JMX metrics & component logs by [@agavra](https://github.com/agavra) in [#9578](https://github.com/apache/pinot/pull/9578)
* remove unused RecordTransform/RecordFilter classes by [@agavra](https://github.com/agavra) in [#9607](https://github.com/apache/pinot/pull/9607)
* Support rewriting forward index upon changing compression type for existing raw MV column by [@vvivekiyer](https://github.com/vvivekiyer) in [#9510](https://github.com/apache/pinot/pull/9510)
* Support Avro's Fixed data type by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9642](https://github.com/apache/pinot/pull/9642)
* \[feature] \[kubernetes] add loadBalancerSourceRanges to service-external.yaml for controller and broker by [@jameskelleher](https://github.com/jameskelleher) in [#9494](https://github.com/apache/pinot/pull/9494)
* Limit up to 10 unavailable segments to be printed in the query exception by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9617](https://github.com/apache/pinot/pull/9617)
* remove more unused filter code by [@agavra](https://github.com/agavra) in [#9620](https://github.com/apache/pinot/pull/9620)
* Do not cache record reader in segment by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9604](https://github.com/apache/pinot/pull/9604)
* make first part of user agent header configurable by [@rino-kadijk](https://github.com/rino-kadijk) in [#9471](https://github.com/apache/pinot/pull/9471)
* optimize `order by sorted ASC, unsorted` and `order by DESC` cases by [@gortiz](https://github.com/gortiz) in [#8979](https://github.com/apache/pinot/pull/8979)
* Enhance cluster config update API to handle non-string values properly by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9635](https://github.com/apache/pinot/pull/9635)
* Reverts recommender REST API back to PUT (reverts PR [#9326](https://github.com/apache/pinot/pull/9326)) by [@yuanbenson](https://github.com/yuanbenson) in [#9638](https://github.com/apache/pinot/pull/9638)
* Remove invalid pruner names from server config by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9646](https://github.com/apache/pinot/pull/9646)
* Using `usageHelp` instead of deprecated `help` in picocli commands by [@navina](https://github.com/navina) in [#9608](https://github.com/apache/pinot/pull/9608)
* Handle unique query id on server by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9648](https://github.com/apache/pinot/pull/9648)
* stateless group marker missing several by [@walterddr](https://github.com/walterddr) in [#9673](https://github.com/apache/pinot/pull/9673)
* Support reloading consuming segment using force commit by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9640](https://github.com/apache/pinot/pull/9640)
* Improve star-tree to use star-node when the predicate matches all the non-star nodes by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9667](https://github.com/apache/pinot/pull/9667)
* add FetchPlanner interface to decide what column index to prefetch by [@klsince](https://github.com/klsince) in [#9668](https://github.com/apache/pinot/pull/9668)
* Improve star-tree traversal using ArrayDeque by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9688](https://github.com/apache/pinot/pull/9688)
* Handle errors in combine operator by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9689](https://github.com/apache/pinot/pull/9689)
* return different error code if old version is not on master by [@SabrinaZhaozyf](https://github.com/SabrinaZhaozyf) in [#9686](https://github.com/apache/pinot/pull/9686)
* Support creating dictionary at runtime for an existing column by [@vvivekiyer](https://github.com/vvivekiyer) in [#9678](https://github.com/apache/pinot/pull/9678)
* check mutable segment explicitly instead of checking existence of indexDir by [@klsince](https://github.com/klsince) in [#9718](https://github.com/apache/pinot/pull/9718)
* Remove leftover file before downloading segmentTar by [@npawar](https://github.com/npawar) in [#9719](https://github.com/apache/pinot/pull/9719)
* add index key and size map to segment metadata by [@walterddr](https://github.com/walterddr) in [#9712](https://github.com/apache/pinot/pull/9712)
* Use ideal state as source of truth for segment existence by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9735](https://github.com/apache/pinot/pull/9735)
* Close Filesystem on exit with Minion Tasks by [@KKcorps](https://github.com/KKcorps) in [#9681](https://github.com/apache/pinot/pull/9681)
* render the tables list even as the table sizes are loading by [@jadami10](https://github.com/jadami10) in [#9741](https://github.com/apache/pinot/pull/9741)
* Add Support for IP Address Function by [@SabrinaZhaozyf](https://github.com/SabrinaZhaozyf) in [#9501](https://github.com/apache/pinot/pull/9501)
* bubble up error messages from broker by [@agavra](https://github.com/agavra) in [#9754](https://github.com/apache/pinot/pull/9754)
* Add support to disable the forward index for existing columns by [@somandal](https://github.com/somandal) in [#9740](https://github.com/apache/pinot/pull/9740)
* show table metadata info in aggregate index size form by [@walterddr](https://github.com/walterddr) in [#9733](https://github.com/apache/pinot/pull/9733)
* Preprocess immutable segments from REALTIME table conditionally when loading them by [@klsince](https://github.com/klsince) in [#9772](https://github.com/apache/pinot/pull/9772)
* revert default timeout nano change in QueryConfig by [@agavra](https://github.com/agavra) in [#9790](https://github.com/apache/pinot/pull/9790)
* AdaptiveServerSelection: Update stats for servers that have not responded by [@vvivekiyer](https://github.com/vvivekiyer) in [#9801](https://github.com/apache/pinot/pull/9801)
* Add null value index for default column by [@KKcorps](https://github.com/KKcorps) in [#9777](https://github.com/apache/pinot/pull/9777)
* \[MergeRollupTask] include partition info into segment name by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9815](https://github.com/apache/pinot/pull/9815)
* Adding a consumer lag as metric via a periodic task in controller by [@navina](https://github.com/navina) in [#9800](https://github.com/apache/pinot/pull/9800)
* Deserialize Hyperloglog objects more optimally by [@priyen](https://github.com/priyen) in [#9749](https://github.com/apache/pinot/pull/9749)
* Download offline segments from peers by [@wirybeaver](https://github.com/wirybeaver) in [#9710](https://github.com/apache/pinot/pull/9710)
* Thread Level Usage Accounting and Query Killing on Server by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9727](https://github.com/apache/pinot/pull/9727)
* Add max merger and min mergers for partial upsert by [@deemoliu](https://github.com/deemoliu) in [#9665](https://github.com/apache/pinot/pull/9665)
* [#9518](https://github.com/apache/pinot/issues/9518) added pinot helm 0.2.6 with secure version pinot 0.11.0 by [@bagipriyank](https://github.com/bagipriyank) in [#9519](https://github.com/apache/pinot/pull/9519)
* Combine the read access for replication config by [@snleee](https://github.com/snleee) in [#9849](https://github.com/apache/pinot/pull/9849)
* add v1 ingress in helm chart by [@jhisse](https://github.com/jhisse) in [#9862](https://github.com/apache/pinot/pull/9862)
* Optimize AdaptiveServerSelection for replicaGroup based routing by [@vvivekiyer](https://github.com/vvivekiyer) in [#9803](https://github.com/apache/pinot/pull/9803)
* Do not sort the instances in InstancePartitions by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9866](https://github.com/apache/pinot/pull/9866)
* Merge new columns in existing record with default merge strategy by [@navina](https://github.com/navina) in [#9851](https://github.com/apache/pinot/pull/9851)
* Support disabling dictionary at runtime for an existing column by [@vvivekiyer](https://github.com/vvivekiyer) in [#9868](https://github.com/apache/pinot/pull/9868)
* support BOOL\_AND and BOOL\_OR aggregate functions by [@agavra](https://github.com/agavra) in [#9848](https://github.com/apache/pinot/pull/9848)
* Use Pulsar AdminClient to delete unused subscriptions by [@navina](https://github.com/navina) in [#9859](https://github.com/apache/pinot/pull/9859)
* add table sort function for table size by [@jadami10](https://github.com/jadami10) in [#9844](https://github.com/apache/pinot/pull/9844)
* In Kafka consumer, seek offset only when needed by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9896](https://github.com/apache/pinot/pull/9896)
* fallback if no broker found for the specified table name by [@klsince](https://github.com/klsince) in [#9914](https://github.com/apache/pinot/pull/9914)
* Allow liveness check during server shutting down by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9915](https://github.com/apache/pinot/pull/9915)
* Allow segment upload via Metadata in MergeRollup Minion task by [@KKcorps](https://github.com/KKcorps) in [#9825](https://github.com/apache/pinot/pull/9825)
* Add back the Helix workaround for missing IS change by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9921](https://github.com/apache/pinot/pull/9921)
* Allow uploading real-time segments via CLI by [@KKcorps](https://github.com/KKcorps) in [#9861](https://github.com/apache/pinot/pull/9861)
* Add capability to update and delete table config via CLI by [@KKcorps](https://github.com/KKcorps) in [#9852](https://github.com/apache/pinot/pull/9852)
* default to TAR if push mode is not set by [@klsince](https://github.com/klsince) in [#9935](https://github.com/apache/pinot/pull/9935)
* load startree index via segment reader interface by [@klsince](https://github.com/klsince) in [#9828](https://github.com/apache/pinot/pull/9828)
* Allow collections for MV transform functions by [@saurabhd336](https://github.com/saurabhd336) in [#9908](https://github.com/apache/pinot/pull/9908)
* Construct new IndexLoadingConfig when loading completed real-time segments by [@vvivekiyer](https://github.com/vvivekiyer) in [#9938](https://github.com/apache/pinot/pull/9938)
* Make GET /tableConfigs backwards compatible in case schema does not match raw table name by [@timsants](https://github.com/timsants) in [#9922](https://github.com/apache/pinot/pull/9922)
* feat: add compressed file support for ORCRecordReader by [@etolbakov](https://github.com/etolbakov) in [#9884](https://github.com/apache/pinot/pull/9884)
* Add Variance and Standard Deviation Aggregation Functions by [@snleee](https://github.com/snleee) in [#9910](https://github.com/apache/pinot/pull/9910)
* enable MergeRollupTask on real-time tables by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9890](https://github.com/apache/pinot/pull/9890)
* Update cardinality when converting raw column to dict based by [@vvivekiyer](https://github.com/vvivekiyer) in [#9875](https://github.com/apache/pinot/pull/9875)
* Add back auth token for UploadSegmentCommand by [@timsants](https://github.com/timsants) in [#9960](https://github.com/apache/pinot/pull/9960)
* Improving gz support for avro record readers by [@snleee](https://github.com/snleee) in [#9951](https://github.com/apache/pinot/pull/9951)
* Default column handling of noForwardIndex and regeneration of forward index on reload path by [@somandal](https://github.com/somandal) in [#9810](https://github.com/apache/pinot/pull/9810)
* \[Feature] Support coalesce literal by [@61yao](https://github.com/61yao) in [#9958](https://github.com/apache/pinot/pull/9958)
* Ability to initialize S3PinotFs with serverSideEncryption properties when passing client directly by [@npawar](https://github.com/npawar) in [#9988](https://github.com/apache/pinot/pull/9988)
* handle pending minion tasks properly when getting the task progress status by [@klsince](https://github.com/klsince) in [#9911](https://github.com/apache/pinot/pull/9911)
* allow gauge stored in metric registry to be updated by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9961](https://github.com/apache/pinot/pull/9961)
* support case-insensitive query options in SET syntax by [@agavra](https://github.com/agavra) in [#9912](https://github.com/apache/pinot/pull/9912)
* pin versions-maven-plugin to 2.13.0 by [@jadami10](https://github.com/jadami10) in [#9993](https://github.com/apache/pinot/pull/9993)
* Pulsar Connection handler should not spin up a consumer / reader by [@navina](https://github.com/navina) in [#9893](https://github.com/apache/pinot/pull/9893)
* Handle in-memory segment metadata for index checking by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#10017](https://github.com/apache/pinot/pull/10017)
* Support the cross-account access using IAM role for S3 PinotFS by [@snleee](https://github.com/snleee) in [#10009](https://github.com/apache/pinot/pull/10009)
* report minion task metadata last update time as metric by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9954](https://github.com/apache/pinot/pull/9954)
* support SKEWNESS and KURTOSIS aggregates by [@agavra](https://github.com/agavra) in [#10021](https://github.com/apache/pinot/pull/10021)
* emit minion task generation time and error metrics by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#10026](https://github.com/apache/pinot/pull/10026)
* Use the same default time value for all replicas by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#10029](https://github.com/apache/pinot/pull/10029)
* Reduce the number of segments to wait for convergence when rebalancing by [@saurabhd336](https://github.com/saurabhd336) in [#10028](https://github.com/apache/pinot/pull/10028)

## UI Update & Improvement

* Allow hiding query console tab based on cluster config ([#9261](https://github.com/apache/pinot/pull/9261))
* Allow hiding pinot broker swagger UI by config ([#9343](https://github.com/apache/pinot/pull/9343))
* Add UI to show fine-grained minion task progress ([#9488](https://github.com/apache/pinot/pull/9488))
* Add UI to track segment reload progress ([#9521](https://github.com/apache/pinot/pull/9521))
* Show minion task runtime config details in UI ([#9652](https://github.com/apache/pinot/pull/9652))
* Redefine the segment status ([#9699](https://github.com/apache/pinot/pull/9699))
* Show an option to reload the segments during edit schema ([#9762](https://github.com/apache/pinot/pull/9762))
* Load schema UI async ([#9781](https://github.com/apache/pinot/pull/9781))
* Fix blank screen when redirect to unknown app route ([#9888](https://github.com/apache/pinot/pull/9888))

## Library version upgrade

* Upgrade h3 lib from 3.7.2 to 4.0.0 to lower glibc requirement ([#9335](https://github.com/apache/pinot/pull/9335))
* Upgrade ZK version to 3.6.3 ([#9612](https://github.com/apache/pinot/pull/9612))
* Upgrade snakeyaml from 1.30 to 1.33 ([#9464](https://github.com/apache/pinot/pull/9464))
* Upgrade RoaringBitmap from 0.9.28 to 0.9.35 ([#9730](https://github.com/apache/pinot/pull/9730))
* Upgrade spotless-maven-plugin from 2.9.0 to 2.28.0 ([#9877](https://github.com/apache/pinot/pull/9877))
* Upgrade decode-uri-component from 0.2.0 to 0.2.2 ([#9941](https://github.com/apache/pinot/pull/9941))

## BugFixes

* Fix bug with logging request headers by [@abhs50](https://github.com/abhs50) in [#9247](https://github.com/apache/pinot/pull/9247)
* Fix a UT that only shows up on host with more cores by [@klsince](https://github.com/klsince) in [#9257](https://github.com/apache/pinot/pull/9257)
* Fix message count by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9271](https://github.com/apache/pinot/pull/9271)
* Fix issue with auth AccessType in Schema REST endpoints by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9293](https://github.com/apache/pinot/pull/9293)
* Fix PerfBenchmarkRunner to skip the tmp dir by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9298](https://github.com/apache/pinot/pull/9298)
* Fix thrift deserializer thread safety issue by [@saurabhd336](https://github.com/saurabhd336) in [#9299](https://github.com/apache/pinot/pull/9299)
* Fix transformation to string for BOOLEAN and TIMESTAMP by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9287](https://github.com/apache/pinot/pull/9287)
* \[hotfix] Add VARBINARY column to switch case branch by [@walterddr](https://github.com/walterddr) in [#9313](https://github.com/apache/pinot/pull/9313)
* Fix annotation for "/recommender" endpoint by [@sajjad-moradi](https://github.com/sajjad-moradi) in [#9326](https://github.com/apache/pinot/pull/9326)
* Fix jdk8 build issue due to missing pom dependency by [@somandal](https://github.com/somandal) in [#9351](https://github.com/apache/pinot/pull/9351)
* Fix pom to use pinot-common-jdk8 for pinot-connector jkd8 java client by [@somandal](https://github.com/somandal) in [#9353](https://github.com/apache/pinot/pull/9353)
* Fix log to reflect job type by [@KKcorps](https://github.com/KKcorps) in [#9381](https://github.com/apache/pinot/pull/9381)
* \[Bugfix] schema update bug fix by [@MeihanLi](https://github.com/MeihanLi) in [#9382](https://github.com/apache/pinot/pull/9382)
* fix histogram null pointer exception by [@jasperjiaguo](https://github.com/jasperjiaguo) in [#9428](https://github.com/apache/pinot/pull/9428)
* Fix thread safety issues with SDF (WIP) by [@saurabhd336](https://github.com/saurabhd336) in [#9425](https://github.com/apache/pinot/pull/9425)
* Bug fix: failure status in ingestion jobs doesn't reflect in exit code by [@KKcorps](https://github.com/KKcorps) in [#9410](https://github.com/apache/pinot/pull/9410)
* Fix skip segment logic in MinMaxValueBasedSelectionOrderByCombineOperator by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9434](https://github.com/apache/pinot/pull/9434)
* Fix the bug of hybrid table request using the same request id by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9443](https://github.com/apache/pinot/pull/9443)
* Fix the range check for range index on raw column by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9453](https://github.com/apache/pinot/pull/9453)
* Fix Data-Correctness Bug in GTE Comparison in BinaryOperatorTransformFunction by [@ankitsultana](https://github.com/ankitsultana) in [#9461](https://github.com/apache/pinot/pull/9461)
* extend PinotFS impls with listFilesWithMetadata and some bugfix by [@klsince](https://github.com/klsince) in [#9478](https://github.com/apache/pinot/pull/9478)
* fix null transform bound check by [@walterddr](https://github.com/walterddr) in [#9495](https://github.com/apache/pinot/pull/9495)
* Fix JsonExtractScalar when no value is extracted by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9500](https://github.com/apache/pinot/pull/9500)
* Fix AddTable for real-time tables by [@npawar](https://github.com/npawar) in [#9506](https://github.com/apache/pinot/pull/9506)
* Fix some type convert scalar functions by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9509](https://github.com/apache/pinot/pull/9509)
* fix spammy logs for ConfluentSchemaRegistryRealtimeClusterIntegrationTest \[MINOR] by [@agavra](https://github.com/agavra) in [#9516](https://github.com/apache/pinot/pull/9516)
* Fix timestamp index on column of preserved key by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9533](https://github.com/apache/pinot/pull/9533)
* Fix record extractor when ByteBuffer can be reused by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9549](https://github.com/apache/pinot/pull/9549)
* Fix explain plan ALL\_SEGMENTS\_PRUNED\_ON\_SERVER node by [@somandal](https://github.com/somandal) in [#9572](https://github.com/apache/pinot/pull/9572)
* Fix time validation when data type needs to be converted by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9569](https://github.com/apache/pinot/pull/9569)
* UI: fix incorrect task finish time by [@jayeshchoudhary](https://github.com/jayeshchoudhary) in [#9557](https://github.com/apache/pinot/pull/9557)
* Fix the bug where uploaded segments cannot be deleted on real-time table by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9579](https://github.com/apache/pinot/pull/9579)
* \[bugfix] correct the dir for building segments in FileIngestionHelper by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9591](https://github.com/apache/pinot/pull/9591)
* Fix NonAggregationGroupByToDistinctQueryRewriter by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9605](https://github.com/apache/pinot/pull/9605)
* fix distinct result return by [@walterddr](https://github.com/walterddr) in [#9582](https://github.com/apache/pinot/pull/9582)
* Fix GcsPinotFS by [@lfernandez93](https://github.com/lfernandez93) in [#9556](https://github.com/apache/pinot/pull/9556)
* fix DataSchema thread-safe issue by [@walterddr](https://github.com/walterddr) in [#9619](https://github.com/apache/pinot/pull/9619)
* Bug fix: Add missing table config fetch for /tableConfigs list all by [@timsants](https://github.com/timsants) in [#9603](https://github.com/apache/pinot/pull/9603)
* Fix re-uploading segment when the previous upload failed by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9631](https://github.com/apache/pinot/pull/9631)
* Fix string split which should be on whole separator by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9650](https://github.com/apache/pinot/pull/9650)
* Fix server request sent delay to be non-negative by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9656](https://github.com/apache/pinot/pull/9656)
* bugfix: Add missing BIG\_DECIMAL support for GenericRow serde by [@timsants](https://github.com/timsants) in [#9661](https://github.com/apache/pinot/pull/9661)
* Fix extra restlet resource test which should be stateless by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9674](https://github.com/apache/pinot/pull/9674)
* AdaptiveServerSelection: Fix timer by [@vvivekiyer](https://github.com/vvivekiyer) in [#9697](https://github.com/apache/pinot/pull/9697)
* fix PinotVersion to be compatible with prometheus by [@agavra](https://github.com/agavra) in [#9701](https://github.com/apache/pinot/pull/9701)
* Fix the setup for ControllerTest shared cluster by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9704](https://github.com/apache/pinot/pull/9704)
* \[hotfix]groovy class cache leak by [@walterddr](https://github.com/walterddr) in [#9716](https://github.com/apache/pinot/pull/9716)
* Fix TIMESTAMP index handling in SegmentMapper by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9722](https://github.com/apache/pinot/pull/9722)
* Fix the server admin endpoint cache to reflect the config changes by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9734](https://github.com/apache/pinot/pull/9734)
* \[bugfix] fix case-when issue by [@walterddr](https://github.com/walterddr) in [#9702](https://github.com/apache/pinot/pull/9702)
* \[bugfix] Let StartControllerCommand also handle "pinot.zk.server", "pinot.cluster.name" in default conf/pinot-controller.conf by [@thangnd197](https://github.com/thangnd197) in [#9739](https://github.com/apache/pinot/pull/9739)
* \[hotfix] semi-join opt by [@walterddr](https://github.com/walterddr) in [#9779](https://github.com/apache/pinot/pull/9779)
* Fixing the rebalance issue for real-time table with tier by [@snleee](https://github.com/snleee) in [#9780](https://github.com/apache/pinot/pull/9780)
* UI: show segment debug details when segment is in bad state by [@jayeshchoudhary](https://github.com/jayeshchoudhary) in [#9700](https://github.com/apache/pinot/pull/9700)
* Fix the replication in segment assignment strategy by [@GSharayu](https://github.com/GSharayu) in [#9816](https://github.com/apache/pinot/pull/9816)
* fix potential fd leakage for SegmentProcessorFramework by [@klsince](https://github.com/klsince) in [#9797](https://github.com/apache/pinot/pull/9797)
* Fix NPE when reading ZK address from controller config by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9751](https://github.com/apache/pinot/pull/9751)
* have query table list show search bar; fix InstancesTables filter by [@jadami10](https://github.com/jadami10) in [#9742](https://github.com/apache/pinot/pull/9742)
* \[pinot-spark-connector] Fix empty data table handling in GRPC reader by [@cbalci](https://github.com/cbalci) in [#9837](https://github.com/apache/pinot/pull/9837)
* \[bugfix] fix mergeRollupTask metrics by [@zhtaoxiang](https://github.com/zhtaoxiang) in [#9864](https://github.com/apache/pinot/pull/9864)
* Bug fix: Get correct primary key count by [@KKcorps](https://github.com/KKcorps) in [#9876](https://github.com/apache/pinot/pull/9876)
* Fix issues for real-time table reload by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9885](https://github.com/apache/pinot/pull/9885)
* UI: fix segment status color remains same in different table page by [@jayeshchoudhary](https://github.com/jayeshchoudhary) in [#9891](https://github.com/apache/pinot/pull/9891)
* Fix bloom filter creation on BYTES by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9898](https://github.com/apache/pinot/pull/9898)
* \[hotfix] broker selection not using table name by [@walterddr](https://github.com/walterddr) in [#9902](https://github.com/apache/pinot/pull/9902)
* Fix race condition when 2 segment upload occurred for the same segment by [@jackjlli](https://github.com/jackjlli) in [#9905](https://github.com/apache/pinot/pull/9905)
* fix timezone\_hour/timezone\_minute functions by [@agavra](https://github.com/agavra) in [#9949](https://github.com/apache/pinot/pull/9949)
* \[Bugfix] Move brokerId extraction to BaseBrokerStarter by [@jackjlli](https://github.com/jackjlli) in [#9965](https://github.com/apache/pinot/pull/9965)
* Fix ser/de for StringLongPair by [@Jackie-Jiang](https://github.com/Jackie-Jiang) in [#9985](https://github.com/apache/pinot/pull/9985)
* bugfix dir check for HadoopPinotFS.copyFromLocalDir by [@klsince](https://github.com/klsince) in [#9979](https://github.com/apache/pinot/pull/9979)
* Bugfix: Use correct exception import in TableRebalancer. by [@mayankshriv](https://github.com/mayankshriv) in [#10025](https://github.com/apache/pinot/pull/10025)
* Fix NPE in AbstractMetrics From Race Condition by [@ankitsultana](https://github.com/ankitsultana) in [#10022](https://github.com/apache/pinot/pull/10022)