0.8.0

This release introduced several new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins.

Summary

This release introduced several awesome new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins (AWS Kinesis, Apache Pulsar). It contains a lot of query enhancements such as new timestamp and boolean type support and flexible numerical column comparison. It also includes many key bug fixes. See details below.

The release was cut from the following commit: fe83e95aa9124ee59787c580846793ff7456eaa5

and the following cherry-picks:

• 668b5e0

• ee887b9

• c2f7fcc

• c1ac8a1

• 4da1dae

• 573651b

• c6c407d

• 0d96c7f

• c2637d1

Notable New Features

• Extract time handling for SegmentProcessorFramework (#7158)

• Add Apache Pulsar low level and high level connector (#7026)

• Enable parallel builds for compat checker (#7149)

• Add controller/server API to fetch aggregated segment metadata (#7102)

• Support Dictionary Based Plan For DISTINCT (#7141)

• Provide HTTP client to kinesis builder (#7148)

• Add datetime function with 2 arguments (#7116)

• Adding ability to check ingestion status for Offline Pinot table (#7070)

• Add timestamp datatype support in JDBC (#7117)

• Allow updating controller and broker helix hostname (#7064)

• Cancel running Kinesis consumer tasks when timeout occurs (#7109)

• Implement Append merger for partial upsert (#7087)

  `* SegmentProcessorFramework Enhancement (#7092)

• Added TaskMetricsEmitted periodic controler job (#7091)

• Support json path expressions in query. (#6998)

• Support data preprocessing for AVRO and ORC formats (#7062)

• Add partial upsert config and mergers (#6899)

• Add support for range index rule recommendation(#7034) (#7063)

• Allow reloading consuming segment by default (#7078)

• Add LZ4 Compression Codec (#6804) ([#7035](https://github.com/apache/pinot/pull/7035

  ))

• Make Pinot JDK 11 Compilable (#6424\

• Introduce in-Segment Trim for GroupBy OrderBy Query (#6991)

• Produce GenericRow file in segment processing mapper (#7013)

• Add ago() scalar transform function (#6820)

• Add Bloom Filter support for IN predicate(#7005) (#7007)

• Add genericRow file reader and writer (#6997)

• Normalize LHS and RHS numerical types for >, >=, <, and <= operators. (#6927)

• Add Kinesis Stream Ingestion Plugin (#6661)

• feature/#6766 JSON and Startree index information in API (#6873)

• Support null value fields in generic row ser/de (#6968)

• Implement PassThroughTransformOperator to optimize select queries(#6972) (#6973)

• Optimize TIME_CONVERT/DATE_TIME_CONVERT predicates (#6957)

• Prefetch call to fetch buffers of columns seen in the query (#6967)

• Enabling compatibility tests in the script (#6959)

• Add collectionToJsonMode to schema inference (#6946)

• Add the complex-type support to decoder/reader (#6945)

• Adding a new Controller API to retrieve ingestion status for real-time… (#6890)

• Add support for Long in Modulo partition function. (#6929)

• Enhance PinotSegmentRecordReader to preserve null values (#6922)

• add complex-type support to avro-to-pinot schema inference (#6928)

• Add correct yaml files for real-time data(#6787) (#6916)

• Add complex-type transformation to offline segment creation (#6914)

• Add config File support(#6787) (#6901)

• Enhance JSON index to support nested array (#6877)

• Add debug endpoint for tables. (#6897)

• JSON column datatype support. (#6878)

• Allow empty string in MV column (#6879)

• Add Zstandard compression support with JMH benchmarking(#6804) (#6876)

• Normalize LHS and RHS numerical types for = and != operator. (#6811)

• Change ConcatCollector implementation to use off-heap (#6847)

• [PQL Deprecation] Clean up the old BrokerRequestOptimizer (#6859)

• [PQL Deprecation] Do not compile PQL broker request for SQL query (#6855)

• Add TIMESTAMP and BOOLEAN data type support (#6719)

• Add admin endpoint for Pinot Minon. (#6822)

• Remove the usage of PQL compiler (#6808)

• Add endpoints in Pinot Controller, Broker and Server to get system and application configs. (#6817)

• Support IN predicate in ColumnValue SegmentPruner(#6756) (#6776)

• Enable adding new segments to a upsert-enabled real-time table (#6567)

• Interface changes for Kinesis connector (#6667)

• Pinot Minion SegmentGenerationAndPush task: PinotFS configs inside taskSpec is always temporary and has higher priority than default PinotFS created by the minion server configs (#6744)

• DataTable V3 implementation and measure data table serialization cost on server (#6710)

• add uploadLLCSegment endpoint in TableResource (#6653)

• File-based SegmentWriter implementation (#6718)

• Basic Auth for pinot-controller (#6613)

• UI integration with Authentication API and added login page (#6686)

• Support data ingestion for offline segment in one pass (#6479)

• SumPrecision: support all data types and star-tree (#6668)

• complete compatibility regression testing (#6650)

• Kinesis implementation Part 1: Rename partitionId to partitionGroupId (#6655)

• Make Pinot metrics pluggable (#6640)

• Recover the segment from controller when LLC table cannot load it (#6647)

• Adding a new API for validating specified TableConfig and Schema (#6620)

• Introduce a metric for query/response size on broker. (#6590)

• Adding a controller periodic task to clean up dead minion instances (#6543)

• Adding new validation for Json, TEXT indexing (#6541)

• Always return a response from query execution. (#6596)

Special notes

• After the 0.8.0 release, we will officially support jdk 11, and can now safely start to use jdk 11 features. Code is still compilable with jdk 8 (#6424)

• RealtimeToOfflineSegmentsTask config has some backward incompatible changes (#7158)

  — timeColumnTransformFunction is removed (backward-incompatible, but rollup is not supported anyway)

  — Deprecate collectorType and replace it with mergeType

  — Add roundBucketTimePeriod and partitionBucketTimePeriod to config the time bucket for round and partition

• Regex path for pluggable MinionEventObserverFactory is changed from org.apache.pinot.*.event.* to org.apache.pinot.*.plugin.minion.tasks.* (#6980)

• Moved all pinot built-in minion tasks to the pinot-minion-builtin-tasks module and package them into a shaded jar (#6618)

• Reloading consuming segment flag pinot.server.instance.reload.consumingSegment will be true by default (#7078)

• Move JSON decoder from pinot-kafka to pinot-json package. (#7021)

• Backward incompatible schema change through controller rest API PUT /schemas/{schemaName} will be blocked. (#6737)

• Deprecated /tables/validateTableAndSchema in favor of the new configs/validate API and introduced new APIs for /tableConfigs to operate on the real-time table config, offline table config and schema in one shot. (#6840)

Major Bug fixes

• Fix race condition in MinionInstancesCleanupTask (#7122)

• Fix custom instance id for controller/broker/minion (#7127)

• Fix UpsertConfig JSON deserialization. (#7125)

• Fix the memory issue for selection query with large limit (#7112)

• Fix the deleted segments directory not exist warning (#7097)

• Fixing docker build scripts by providing JDK_VERSION as parameter (#7095)

• Misc fixes for json data type (#7057)

• Fix handling of date time columns in query recommender(#7018) (#7031)

• fixing pinot-hadoop and pinot-spark test (#7030)

• Fixing HadoopPinotFS listFiles method to always contain scheme (#7027)

• fixed GenericRow compare for different _fieldToValueMap size (#6964)

• Fix NPE in NumericalFilterOptimizer due to IS NULL and IS NOT NULL operator. (#7001)

• Fix the race condition in real-time text index refresh thread (#6858) (#6990)

• Fix deep store directory structure (#6976)

• Fix NPE issue when consumed kafka message is null or the record value is null. (#6950)

• Mitigate calcite NPE bug. (#6908)

• Fix the exception thrown in the case that a specified table name does not exist (#6328) (#6765)

• Fix CAST transform function for chained transforms (#6941)

• Fixed failing pinot-controller npm build (#6795)