Summary

This release includes many new features on Pinot ingestion and connectors (e.g., support for filtering during ingestion which is configurable in table config; support for json during ingestion; proto buf input format support and a new Pinot JDBC client), query capability (e.g., a new GROOVY transform function UDF) and admin functions (a revamped Cluster Manager UI & Query Console UI). It also contains many key bug fixes. See details below.

The release was cut from the following commit: d1b4586 and the following cherry-picks:

• 63a4fd4

• a7f7f46

• dafbef1

• ced3a70

• d902c1a

Notable New Features

• Allowing update on an existing instance config: PUT /instances/{instanceName} with Instance object as the pay-load (#PR4952)

• Add PinotServiceManager to start Pinot components (#PR5266)

• Support for protocol buffers input format. (#PR5293)

• Add GenericTransformFunction wrapper for simple ScalarFunctions (PR#5440) — Adding support to invoke any scalar function via GenericTransformFunction

• Add Support for SQL CASE Statement (PR#5461)

• Support distinctCountRawThetaSketch aggregation that returns serialized sketch. (PR#5465)

• Add multi-value support to SegmentDumpTool (PR#5487) — add segment dump tool as part of the pinot-tool.sh script

• Add json_format function to convert json object to string during ingestion. (PR#5492) — Can be used to store complex objects as a json string (which can later be queries using jsonExtractScalar)

• Support escaping single quote for SQL literal (PR#5501) — This is especially useful for DistinctCountThetaSketch because it stores expression as literal E.g. DistinctCountThetaSketch(..., 'foo=''bar''', ...)

• Support expression as the left-hand side for BETWEEN and IN clause (PR#5502)

• Add a new field IngestionConfig in TableConfig — FilterConfig: ingestion level filtering of records, based on filter function. (PR#5597) — TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release (PR#5681).

• Allow star-tree creation during segment load (#PR5641) — Introduced a new boolean config enableDynamicStarTreeCreation in IndexingConfig to enable/disable star-tree creation during segment load.

• Support for Pinot clients using JDBC connection (#PR5602)

• Support customized accuracy for distinctCountHLL, distinctCountHLLMV functions by adding log2m value as the second parameter in the function. (#PR5564) —Adding cluster config: default.hyperloglog.log2m to allow user set default log2m value.

• Add segment encryption on Controller based on table config (PR#5617)

• Add a constraint to the message queue for all instances in Helix, with a large default value of 100000. (PR#5631)

• Support order-by aggregations not present in SELECT (PR#5637) — Example: "select subject from transcript group by subject order by count() desc" This is equivalent to the following query but the return response should not contain count(). "select subject, count() from transcript group by subject order by count() desc"

• Add geo support for Pinot queries (PR#5654) — Added geo-spatial data model and geospatial functions

• Cluster Manager UI & Query Console UI revamp (PR#5684 and PR#5732) — updated cluster manage UI and added table details page and segment details page

• Add Controller API to explore Zookeeper (PR#5687)

• Support BYTES type for dictinctCount and group-by (PR#5701 and PR#5708) —Add BYTES type support to DistinctCountAggregationFunction —Correctly handle BYTES type in DictionaryBasedAggregationOperator for DistinctCount

• Support for ingestion job spec in JSON format (#PR5729)

• Improvements to RealtimeProvisioningHelper command (#PR5737) — Improved docs related to ingestion and plugins

• Added GROOVY transform function UDF (#PR5748) — Ability to run a groovy script in the query as a UDF. e.g. string concatenation: SELECT GROOVY('{"returnType": "INT", "isSingleValue": true}', 'arg0 + " " + arg1', columnA, columnB) FROM myTable

Special notes

• Changed the stream and metadata interface (PR#5542) — This PR concludes the work for the issue #5359 to extend offset support for other streams

• TransformConfig: ingestion level column transformations. This was previously introduced in Schema (FieldSpec#transformFunction), and has now been moved to TableConfig. It continues to remain under schema, but we recommend users to set it in the TableConfig starting this release (PR#5681).

• Config key enable.case.insensitive.pql in Helix cluster config is deprecated, and replaced with enable.case.insensitive. (#PR5546)

• Change default segment load mode to MMAP. (PR#5539) —The load mode for segments currently defaults to heap.

Major Bug fixes

• Fix bug in distinctCountRawHLL on SQL path (#5494)

• Fix backward incompatibility for existing stream implementations (#5549)

• Fix backward incompatibility in StreamFactoryConsumerProvider (#5557)

• Fix logic in isLiteralOnlyExpression. (#5611)

• Fix double memory allocation during operator setup (#5619)

• Allow segment download url in Zookeeper to be deep store uri instead of hardcoded controller uri (#5639)

• Fix a backward compatible issue of converting BrokerRequest to QueryContext when querying from Presto segment splits (#5676)

• Fix the issue that PinotSegmentToAvroConverter does not handle BYTES data type. (#5789)

Backward Incompatible Changes

• PQL queries with HAVING clause will no longer be accepted for the following reasons: (#PR5570) — HAVING clause does not apply to PQL GROUP-BY semantic where each aggregation column is ordered individually — The current behavior can produce inaccurate results without any notice — HAVING support will be added for SQL queries in the next release

• Because of the standardization of the DistinctCountThetaSketch predicate strings, please upgrade Broker before Server. The new Broker can handle both standard and non-standard predicate strings for backward-compatibility. (#PR5613)

Summary

This release introduced some excellent new features, including upsert, tiered storage, pinot-spark-connector, support of having clause, more validations on table config and schema, support of ordinals in GROUP BY and ORDER BY clause, array transform functions, adding push job type of segment metadata only mode, and some new APIs like updating instance tags, new health check endpoint. It also contains many key bug fixes. See details below.

The release was cut from the following commit: e5c9bec and the following cherry-picks:

• d033a11

Notable New Features

• Tiered storage (#5793)

• Upsert feature (#6096, #6113, #6141, #6149, #6167)

• Pre-generate aggregation functions in QueryContext (#5805)

• Adding controller healthcheck endpoint: /health (#5846)

• Add pinot-spark-connector (#5787)

• Support multi-value non-dictionary group by (#5851)

• Support type conversion for all scalar functions (#5849)

• Add additional datetime functionality (#5438)

• Support post-aggregation in ORDER-BY (#5856)

• Support post-aggregation in SELECT (#5867)

• Add RANGE FilterKind to support merging ranges for SQL (#5898)

• Add HAVING support (#58895889)

• Support for exact distinct count for non int data types (#5872)

• Add max qps bucket count (#5922)

• Add Range Indexing support for raw values (#5853)

• Add IdSet and IdSetAggregationFunction (#5926)

• [Deepstore by-pass]Add a Deepstore bypass integration test with minor bug fixes. (#5857)

• Add Hadoop counters for detecting schema mismatch (#5873)

• Add RawThetaSketchAggregationFunction (#5970)

• Instance API to directly updateTags (#5902)

• Add streaming query handler (#5717)

• Add InIdSetTransformFunction (#5973)

• Add ingestion descriptor in the header (#5995)

• Zookeeper put api (#5949)

• Feature/#5390 segment indexing reload status api (#5718)

• Segment processing framework (#5934)

• Support streaming query in QueryExecutor (#6027)

• Add list of allowed tables for emitting table level metrics (#6037)

• Add FilterOptimizer which supports optimizing both PQL and SQL query filter (#6056)

• Adding push job type of segment metadata only mode (#5967)

• Minion taskExecutor for RealtimeToOfflineSegments task (#6050, #6124)

• Adding array transform functions: array_average, array_max, array_min, array_sum (#6084)

• Allow modifying/removing existing star-trees during segment reload (#6100)

• Implement off-heap bloom filter reader (#6118)

• Support for multi-threaded Group By reducer for SQL. (#6044)

• Add OnHeapGuavaBloomFilterReader (#6147)

• Support using ordinals in GROUP BY and ORDER BY clause (#6152)

• Merge common APIs for Dictionary (#6176)

• Add table level lock for segment upload ([#6165])

• Added recursive functions validation check for group by (#6186)

• Add StrictReplicaGroupInstanceSelector (#6208)

• Add IN_SUBQUERY support (#6022)

• Add IN_PARTITIONED_SUBQUERY support (#6043)

• Some UI features (#5810, #5981, #6117, #6215)

Special notes

• Brokers should be upgraded before servers in order to keep backward-compatible:

  • Change group key delimiter from '\t' to '\0' (#5858)

  • Support for exact distinct count for non int data types (#5872)

• Pinot Components have to be deployed in the following order:

  (PinotServiceManager -> Bootstrap services in role ServiceRole.CONTROLLER -> All remaining bootstrap services in parallel)

  • Starts Broker and Server in parallel when using ServiceManager (#5917)

  • New settings introduced and old ones deprecated:

  • Make realtime threshold property names less ambiguous (#5953)

  • Change Signature of Broker API in Controller (#6119)

• This aggregation function is still in beta version. This PR involves change on the format of data sent from server to broker, so it works only when both broker and server are upgraded to the new version:

  • Enhance DistinctCountThetaSketchAggregationFunction (#6004)

Major Bug fixes

• Improve performance of DistinctCountThetaSketch by eliminating empty sketches and unions. (#5798)

• Enhance VarByteChunkSVForwardIndexReader to directly read from data buffer for uncompressed data (#5816)

• Fixing backward-compatible issue of schema fetch call (#5885)

• Fix race condition in MetricsHelper (#5887)

• Fixing the race condition that segment finished before ControllerLeaderLocator created. (#5864)

• Fix CSV and JSON converter on BYTES column (#5931)

• Fixing the issue that transform UDFs are parsed as function name 'OTHER', not the real function names (#5940)

• Incorporating embedded exception while trying to fetch stream offset (#5956)

• Use query timeout for planning phase (#5990)

• Add null check while fetching the schema (#5994)

• Validate timeColumnName when adding/updating schema/tableConfig (#5966)

• Handle the partitioning mismatch between table config and stream (#6031)

• Fix built-in virtual columns for immutable segment (#6042)

• Refresh the routing when realtime segment is committed (#6078)

• Add support for Decimal with Precision Sum aggregation (#6053)

• Fixing the calls to Helix to throw exception if zk connection is broken (#6069)

• Allow modifying/removing existing star-trees during segment reload (#6100)

• Add max length support in schema builder (#6112)

• Enhance star-tree to skip matching-all predicate on non-star-tree dimension (#6109)

Backward Incompatible Changes

• Make realtime threshold property names less ambiguous (#5953)

• Enhance DistinctCountThetaSketchAggregationFunction (#6004)

• Deep Extraction Support for ORC, Thrift, and ProtoBuf Records (#6046)

Summary

0.4.0 release introduced various new features, including the theta-sketch based distinct count aggregation function, an S3 filesystem plugin, a unified star-tree index implementation, deprecation of TimeFieldSpec in favor of DateTimeFieldSpec, etc. Miscellaneous refactoring, performance improvement and bug fixes were also included in this release. See details below.

Notable New Features

• Made DateTimeFieldSpecs mainstream and deprecated TimeFieldSpec (#2756)

  • Used time column from table config instead of schema (#5320)

  • Included dateTimeFieldSpec in schema columns of Pinot Query Console #5392

  • Used DATE_TIME as the primary time column for Pinot tables (#5399)

• Supported range queries using indexes (#5240)

• Supported complex aggregation functions

  • Supported Aggregation functions with multiple arguments (#5261)

  • Added api in AggregationFunction to get compiled input expressions (#5339)

• Added a simple PinotFS benchmark driver (#5160)

• Supported default star-tree (#5147)

• Added an initial implementation for theta-sketch based distinct count aggregation function (#5316)

  • One minor side effect: DataSchemaPruner won't work for DistinctCountThetaSketchAggregatinoFunction (#5382)

• Added access control for Pinot server segment download api (#5260)

• Added Pinot S3 Filesystem Plugin (#5249)

• Text search improvement

  • Pruned stop words for text index (#5297)

  • Used 8byte offsets in chunk based raw index creator (#5285)

  • Derived num docs per chunk from max column value length for varbyte raw index creator (#5256)

  • Added inter segment tests for text search and fixed a bug for Lucene query parser creation (#5226)

  • Made text index query cache a configurable option (#5176)

  • Added Lucene DocId to PinotDocId cache to improve performance (#5177)

  • Removed the construction of second bitmap in text index reader to improve performance (#5199)

• Tooling/usability improvement

  • Added template support for Pinot Ingestion Job Spec (#5341)

  • Allowed user to specify zk data dir and don't do clean up during zk shutdown (#5295)

  • Allowed configuring minion task timeout in the PinotTaskGenerator (#5317)

  • Update JVM settings for scripts (#5127)

  • Added Stream github events demo (#5189)

  • Moved docs link from gitbook to docs.pinot.apache.org (#5193)

• Re-implemented ORCRecordReader (#5267)

• Evaluated schema transform expressions during ingestion (#5238)

• Handled count distinct query in selection list (#5223)

• Enabled async processing in pinot broker query api (#5229)

• Supported bootstrap mode for table rebalance (#5224)

• Supported order-by on BYTES column (#5213)

• Added Nightly publish to binary (#5190)

• Shuffled the segments when rebalancing the table to avoid creating hotspot servers (#5197)

• Supported inbuilt transform functions (#5312)

  • Added date time transform functions (#5326)

• Deepstore by-pass in LLC: introduced segment uploader (#5277, #5314)

• APIs Additions/Changes

  • Added a new server api for download of segments

    • /GET /segments/{tableNameWithType}/{segmentName}

• Upgraded helix to 0.9.7 (#5411)

• Added support to execute functions during query compilation (#5406)

• Other notable refactoring

  • Moved table config into pinot-spi (#5194)

  • Cleaned up integration tests. Standardized the creation of schema, table config and segments (#5385)

  • Added jsonExtractScalar function to extract field from json object (#4597)

  • Added template support for Pinot Ingestion Job Spec #5372

  • Cleaned up AggregationFunctionContext (#5364)

  • Optimized real-time range predicate when cardinality is high (#5331)

  • Made PinotOutputFormat use table config and schema to create segments (#5350)

  • Tracked unavailable segments in InstanceSelector (#5337)

  • Added a new best effort segment uploader with bounded upload time (#5314)

  • In SegmentPurger, used table config to generate the segment (#5325)

  • Decoupled schema from RecordReader and StreamMessageDecoder (#5309)

  • Implemented ARRAYLENGTH UDF for multi-valued columns (#5301)

  • Improved GroupBy query performance (#5291)

  • Optimized ExpressionFilterOperator (#5132)

Major Bug Fixes

• Do not release the PinotDataBuffer when closing the index (#5400)

• Handled a no-arg function in query parsing and expression tree (#5375)

• Fixed compatibility issues during rolling upgrade due to unknown json fields (#5376)

• Fixed missing error message from pinot-admin command (#5305)

• Fixed HDFS copy logic (#5218)

• Fixed spark ingestion issue (#5216)

• Fixed the capacity of the DistinctTable (#5204)

• Fixed various links in the Pinot website

Work in Progress

• Upsert: support overriding data in the real-time table (#4261).

  • Add pinot upsert features to pinot common (#5175)

• Enhancements for theta-sketch, e.g. multiValue aggregation support, complex predicates, performance tuning, etc

Backward Incompatible Changes

• TableConfig no longer support de-serialization from json string of nested json string (i.e. no \" inside the json) (#5194)

• The following APIs are changed in AggregationFunction (use TransformExpressionTree instead of String as the key of blockValSetMap) (#5371):

Note

Before upgrading from one version to another one, please read the release notes. While the Pinot committers strive to keep releases backward-compatible and introduce new features in a compatible manner, your environment may have a unique combination of configurations/data/schema that may have been somehow overlooked. Before you roll out a new release of Pinot on your cluster, it is best that you run the that Pinot provides. The tests can be easily customized to suit the configurations and tables in your pinot cluster(s). As a good practice, you should build your own test suite, mirroring the table configurations, schema, sample data, and queries that are used in your cluster.

0.9.0 (November 2021)

0.8.0 (August 2021)

0.7.1 (April 2021)

0.6.0 (November 2020)

0.5.0 (September 2020)

0.4.0 (June 2020)

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

Summary

This release introduced several awesome new features, including JSON index, lookup-based join support, geospatial support, TLS support for pinot connections, and various performance optimizations and improvements.

It also adds several new APIs to better manage the segments and upload data to the offline table. It also contains many key bug fixes. See details below.

The release was cut from the following commit:

and the following cherry-picks:

Notable New Features

Special notes

• • First, configure alternate ingress ports for https/netty-tls on brokers, controllers, and servers. Restart the components with a rolling strategy to avoid cluster downtime.

  • Second, verify manually that https access to controllers and brokers is live. Then, configure all components to prefer TLS-enabled connections (while still allowing unsecured access). Restart the individual components.

  • Third, disable insecure connections via configuration. You may also have to set controller.vip.protocol and controller.vip.port and update the configuration files of any ingestion jobs. Restart components a final time and verify that insecure ingress via http is not available anymore.

• • Apache Pinot has adopted SQL syntax and semantics. Legacy PQL (Pinot Query Language) is deprecated and no longer supported. Please use SQL syntax to query Pinot on broker endpoint /query/sql and controller endpoint /sql

Major Bug fixes

• Fix license headers and plugin checks

New Features

• First release

• Off-line data ingestion from Apache Hadoop

• Real-time data ingestion from Apache Kafka

New Features and Bug Fixes

• Added support for Kafka 2.0

• Table rebalancer now supports a minimum number of serving replicas during rebalance

• Added support for UDF in filter predicates and selection

• Added support to use hex string as the representation of byte array for queries (see PR #4041)

• Added support for parquet reader (see PR #3852)

• Introduced interface stability and audience annotations (see PR #4063)

• Refactor HelixBrokerStarter to separate constructor and start() - backwards incompatible (see PR #4100)

• Admin tool for listing segments with invalid intervals for offline tables

• Migrated to log4j2 (see PR #4139)

• Added simple avro msg decoder

• Added support for passing headers in Pinot client

• Table rebalancer now supports a minimum number of serving replicas during rebalance

• Support transform functions with AVG aggregation function (see PR #4557)

• Configurations additions/changes

  • Allow customized metrics prefix (see PR #4392)

  • Controller.enable.batch.message.mode to false by default (see PR #3928)

  • RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (see PR #3946)

  • Config to control kafka fetcher size and increase default (see PR #3869)

  • Added a percent threshold to consider startup of services (see PR #4011)

  • Make SingleConnectionBrokerRequestHandler as default (see PR #4048)

  • Always enable default column feature, remove the configuration (see PR #4074)

  • Remove redundant default broker configurations (see PR #4106)

  • Removed some config keys in server(see PR #4222)

  • Add config to disable HLC realtime segment (see PR #4235)

  • Make RetentionManager and OfflineSegmentIntervalChecker initial delays configurable (see PR #3946)

  • The following config variables are deprecated and will be removed in the next release:

    • pinot.broker.requestHandlerType will be removed, in favor of using the "singleConnection" broker request handler. If you have set this configuration, please remove it and use the default type ("singleConnection") for broker request handler.

Work in Progress

• We are in the process of separating Helix and Pinot controllers, so that administrators can have the option of running independent Helix controllers and Pinot controllers.

• We are in the process of moving towards supporting SQL query format and results.

• We are in the process of separating instance and segment assignment using instance pools to optimize the number of Helix state transitions in Pinot clusters with thousands of tables.

Other Notes

• Task management does not work correctly in this release, due to bugs in Helix. We will upgrade to Helix 0.9.2 (or later) version to get this fixed.

• You must upgrade to this release before moving onto newer versions of Pinot release. The protocol between Pinot-broker and Pinot-server has been changed and this release has the code to retain compatibility moving forward. Skipping this release may (depending on your environment) cause query errors if brokers are upgraded and servers are in the process of being upgraded.

• As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

• Pull Request #4100 introduces a backwards incompatible change to Pinot broker. If you use the Java constructor on HelixBrokerStarter class, then you will face a compilation error with this version. You will need to construct the object and call start() method in order to start the broker.

• Pull Request #4139 introduces a backwards incompatible change for log4j configuration. If you used a custom log4j configuration (log4j.xml), you need to write a new log4j2 configuration (log4j2.xml). In addition, you may need to change the arguments on the command line to start Pinot components.

  If you used Pinot-admin command to start Pinot components, you don't need any change. If you used your own commands to start pinot components, you will need to pass the new log4j2 config as a jvm parameter (i.e. substitute -Dlog4j.configuration or -Dlog4j.configurationFile argument with -Dlog4j2.configurationFile=log4j2.xml).

Summary

This release introduced several awesome new features, including compatibility tests, enhanced complex type and Json support, partial upsert support, and new stream ingestion plugins (AWS Kinesis, Apache Pulsar). It contains a lot of query enhancements such as new timestamp and boolean type support and flexible numerical column comparison. It also includes many key bug fixes. See details below.

The release was cut from the following commit: fe83e95aa9124ee59787c580846793ff7456eaa5

and the following cherry-picks:

• 668b5e0

• ee887b9

• c2f7fcc

• c1ac8a1

• 4da1dae

• 573651b

• c6c407d

• 0d96c7f

• c2637d1

Notable New Features

• Extract time handling for SegmentProcessorFramework (#7158)

• Add Apache Pulsar low level and high level connector (#7026)

• Enable parallel builds for compat checker (#7149)

• Add controller/server API to fetch aggregated segment metadata (#7102)

• Support Dictionary Based Plan For DISTINCT (#7141)

• Provide HTTP client to kinesis builder (#7148)

• Add datetime function with 2 arguments (#7116)

• Adding ability to check ingestion status for Offline Pinot table (#7070)

• Add timestamp datatype support in JDBC (#7117)

• Allow updating controller and broker helix hostname (#7064)

• Cancel running Kinesis consumer tasks when timeout occurs (#7109)

• Implement Append merger for partial upsert (#7087)

  `* SegmentProcessorFramework Enhancement (#7092)

• Added TaskMetricsEmitted periodic controler job (#7091)

• Support json path expressions in query. (#6998)

• Support data preprocessing for AVRO and ORC formats (#7062)

• Add partial upsert config and mergers (#6899)

• Add support for range index rule recommendation(#7034) (#7063)

• Allow reloading consuming segment by default (#7078)

• Add LZ4 Compression Codec (#6804) ([#7035](https://github.com/apache/pinot/pull/7035

  ))

• Make Pinot JDK 11 Compilable (#6424\

• Introduce in-Segment Trim for GroupBy OrderBy Query (#6991)

• Produce GenericRow file in segment processing mapper (#7013)

• Add ago() scalar transform function (#6820)

• Add Bloom Filter support for IN predicate(#7005) (#7007)

• Add genericRow file reader and writer (#6997)

• Normalize LHS and RHS numerical types for >, >=, <, and <= operators. (#6927)

• Add Kinesis Stream Ingestion Plugin (#6661)

• feature/#6766 JSON and Startree index information in API (#6873)

• Support null value fields in generic row ser/de (#6968)

• Implement PassThroughTransformOperator to optimize select queries(#6972) (#6973)

• Optimize TIME_CONVERT/DATE_TIME_CONVERT predicates (#6957)

• Prefetch call to fetch buffers of columns seen in the query (#6967)

• Enabling compatibility tests in the script (#6959)

• Add collectionToJsonMode to schema inference (#6946)

• Add the complex-type support to decoder/reader (#6945)

• Adding a new Controller API to retrieve ingestion status for realtime… (#6890)

• Add support for Long in Modulo partition function. (#6929)

• Enhance PinotSegmentRecordReader to preserve null values (#6922)

• add complex-type support to avro-to-pinot schema inference (#6928)

• Add correct yaml files for real time data(#6787) (#6916)

• Add complex-type transformation to offline segment creation (#6914)

• Add config File support(#6787) (#6901)

• Enhance JSON index to support nested array (#6877)

• Add debug endpoint for tables. (#6897)

• JSON column datatype support. (#6878)

• Allow empty string in MV column (#6879)

• Add Zstandard compression support with JMH benchmarking(#6804) (#6876)

• Normalize LHS and RHS numerical types for = and != operator. (#6811)

• Change ConcatCollector implementation to use off-heap (#6847)

• [PQL Deprecation] Clean up the old BrokerRequestOptimizer (#6859)

• [PQL Deprecation] Do not compile PQL broker request for SQL query (#6855)

• Add TIMESTAMP and BOOLEAN data type support (#6719)

• Add admin endpoint for Pinot Minon. (#6822)

• Remove the usage of PQL compiler (#6808)

• Add endpoints in Pinot Controller, Broker and Server to get system and application configs. (#6817)

• Support IN predicate in ColumnValue SegmentPruner(#6756) (#6776)

• Enable adding new segments to a upsert-enabled realtime table (#6567)

• Interface changes for Kinesis connector (#6667)

• Pinot Minion SegmentGenerationAndPush task: PinotFS configs inside taskSpec is always temporary and has higher priority than default PinotFS created by the minion server configs (#6744)

• DataTable V3 implementation and measure data table serialization cost on server (#6710)

• add uploadLLCSegment endpoint in TableResource (#6653)

• File-based SegmentWriter implementation (#6718)

• Basic Auth for pinot-controller (#6613)

• UI integration with Authentication API and added login page (#6686)

• Support data ingestion for offline segment in one pass (#6479)

• SumPrecision: support all data types and star-tree (#6668)

• complete compatibility regression testing (#6650)

• Kinesis implementation Part 1: Rename partitionId to partitionGroupId (#6655)

• Make Pinot metrics pluggable (#6640)

• Recover the segment from controller when LLC table cannot load it (#6647)

• Adding a new API for validating specified TableConfig and Schema (#6620)

• Introduce a metric for query/response size on broker. (#6590)

• Adding a controller periodic task to clean up dead minion instances (#6543)

• Adding new validation for Json, TEXT indexing (#6541)

• Always return a response from query execution. (#6596)

Special notes

• After the 0.8.0 release, we will officially support jdk 11, and can now safely start to use jdk 11 features. Code is still compilable with jdk 8 (#6424)

• RealtimeToOfflineSegmentsTask config has some backward incompatible changes (#7158)

  — timeColumnTransformFunction is removed (backward-incompatible, but rollup is not supported anyway)

  — Deprecate collectorType and replace it with mergeType

  — Add roundBucketTimePeriod and partitionBucketTimePeriod to config the time bucket for round and partition

• Regex path for pluggable MinionEventObserverFactory is changed from org.apache.pinot.*.event.* to org.apache.pinot.*.plugin.minion.tasks.* (#6980)

• Moved all pinot built-in minion tasks to the pinot-minion-builtin-tasks module and package them into a shaded jar (#6618)

• Reloading consuming segment flag pinot.server.instance.reload.consumingSegment will be true by default (#7078)

• Move JSON decoder from pinot-kafka to pinot-json package. (#7021)

• Backward incompatible schema change through controller rest API PUT /schemas/{schemaName} will be blocked. (#6737)

• Deprecated /tables/validateTableAndSchema in favor of the new configs/validate API and introduced new APIs for /tableConfigs to operate on the realtime table config, offline table config and schema in one shot. (#6840)

Major Bug fixes

• Fix race condition in MinionInstancesCleanupTask (#7122)

• Fix custom instance id for controller/broker/minion (#7127)

• Fix UpsertConfig JSON deserialization. (#7125)

• Fix the memory issue for selection query with large limit (#7112)

• Fix the deleted segments directory not exist warning (#7097)

• Fixing docker build scripts by providing JDK_VERSION as parameter (#7095)

• Misc fixes for json data type (#7057)

• Fix handling of date time columns in query recommender(#7018) (#7031)

• fixing pinot-hadoop and pinot-spark test (#7030)

• Fixing HadoopPinotFS listFiles method to always contain scheme (#7027)

• fixed GenericRow compare for different _fieldToValueMap size (#6964)

• Fix NPE in NumericalFilterOptimizer due to IS NULL and IS NOT NULL operator. (#7001)

• Fix the race condition in realtime text index refresh thread (#6858) (#6990)

• Fix deep store directory structure (#6976)

• Fix NPE issue when consumed kafka message is null or the record value is null. (#6950)

• Mitigate calcite NPE bug. (#6908)

• Fix the exception thrown in the case that a specified table name does not exist (#6328) (#6765)

• Fix CAST transform function for chained transforms (#6941)

• Fixed failing pinot-controller npm build (#6795)

Summary

This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.

The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9

Support Segment Merge and Roll-up

LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.

To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.

At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.

Major Changes:

• Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)

• Merge/Rollup task scheduler for offline tables. (#7178)

• Fix MergeRollupTask uploading segments not updating their metadata (#7289)

• MergeRollupTask integration tests (#7283)

• Add mergeRollupTask delay metrics (#7368)

• MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)

• Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)

UI Improvement

This release also sees improvements to Pinot’s query console UI.

• Cmd+Enter shortcut to run query in query console (#7359)

• Showing tooltip in SQL Editor (#7387)

• Make the SQL Editor box expandable (#7381)

• Fix tables ordering by number of segments (#7564)

SQL Improvements

There have also been improvements and additions to Pinot’s SQL implementation.

New functions:

• IN (#7542)

• LASTWITHTIME (#7584)

• ID_SET on MV columns (#7355)

• Raw results for Percentile TDigest and Est (#7226),

• Add timezone as argument in function toDateTime (#7552)

New predicates are supported:

• LIKE(#7214)

• REGEXP_EXTRACT(#7114)

• FILTER(#7566)

Query compatibility improvements:

• Infer data type for Literal (#7332)

• Support logical identifier in predicate (#7347)

• Support JSON queries with top-level array path expression. (#7511)

• Support configurable group by trim size to improve results accuracy (#7241)

Performance Improvements

This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:

• Reduce the disk usage for segment conversion task (#7193)

• Simplify association between Java Class and PinotDataType for faster mapping (#7402)

• Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)

• Replace MINUS with STRCMP (#7394)

• Bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)

• Use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)

• Add option to limit thread usage per query (#7492)

• Improved range queries (#7513)

• Faster bitmap scans (#7530)

• Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)

• Map bitmaps through a bounded window to avoid excessive disk pressure (#7535)

• Allow RLE compression of bitmaps for smaller file sizes (#7582)

• Support raw index properties for columns with JSON and RANGE indexes (#7615)

• Enhance BloomFilter rule to include IN predicate(#7444) (#7624)

• Introduce LZ4_WITH_LENGTH chunk compression type (#7655)

• Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)

• Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)

• During segment pruning, release the bloom filter after each segment is processed (#7668)

• Fix JSONPath cache inefficient issue (#7409)

• Optimize getUnpaddedString with SWAR padding search (#7708)

• Lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)

• Inline binary comparison ops to prevent function call overhead (#7709)

• Memoize literals in query context in order to deduplicate them (#7720)

Other Notable New Features and Changes

• Human Readable Controller Configs (#7173)

• Add the support of geoToH3 function (#7182)

• Add Apache Pulsar as Pinot Plugin (#7223) (#7247)

• Add dropwizard metrics plugin (#7263)

• Introduce OR Predicate Execution On Star Tree Index (#7184)

• Allow to extract values from array of objects with jsonPathArray (#7208)

• Add Realtime table metadata and indexes API. (#7169)

• Support array with mixing data types (#7234)

• Support force download segment in reload API (#7249)

• Show uncompressed znRecord from zk api (#7304)

• Add debug endpoint to get minion task status. (#7300)

• Validate CSV Header For Configured Delimiter (#7237)

• Add auth tokens and user/password support to ingestion job command (#7233)

• Add option to store the hash of the upsert primary key (#7246)

• Add null support for time column (#7269)

• Add mode aggregation function (#7318)

• Support disable swagger in Pinot servers (#7341)

• Delete metadata properly on table deletion (#7329)

• Add basic Obfuscator Support (#7407)

• Add AWS sts dependency to enable auth using web identity token. (#7017)(#7445)

• Mask credentials in debug endpoint /appconfigs (#7452)

• Fix /sql query endpoint now compatible with auth (#7230)

• Fix case sensitive issue in BasicAuthPrincipal permission check (#7354)

• Fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)

• Add segmentNameGeneratorType config to IndexingConfig (#7346)

• Support trigger PeriodicTask manually (#7174)

• Add endpoint to check minion task status for a single task. (#7353)

• Showing partial status of segment and counting CONSUMING state as good segment status (#7327)

• Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)

• Check schema backward-compatibility when updating schema through addSchema with override (#7374)

• Optimize IndexedTable (#7373)

• Support indices remove in V3 segment format (#7301)

• Optimize TableResizer (#7392)

• Introduce resultSize in IndexedTable (#7420)

• Offset based realtime consumption status checker (#7267)

• Add causes to stack trace return (#7460)

• Create controller resource packages config key (#7488)

• Enhance TableCache to support schema name different from table name (#7525)

• Add validation for realtimeToOffline task (#7523)

• Unify CombineOperator multi-threading logic (#7450)

• Support no downtime rebalance for table with 1 replica in TableRebalancer (#7532)

• Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)

• Adjust tuner api (#7553)

• Adding config for metrics library (#7551)

• Add geo type conversion scalar functions (#7573)

• Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)

• Add MV raw forward index and MV BYTES data type (#7595)

• Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)

• Improve get tenant API to differentiate offline and realtime tenants (#7548)

• Refactor query rewriter to interfaces and implementations to allow customization (#7576)

• In ServiceStartable, apply global cluster config in ZK to instance config (#7593)

• Make dimension tables creation bypass tenant validation (#7559)

• Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)

• Reject query with identifiers not in schema (#7590)

• Round Robin IP addresses when retry uploading/downloading segments (#7585)

• Support multi-value derived column in offline table reload (#7632)

• Support segmentNamePostfix in segment name (#7646)

• Add select segments API (#7651)

• Controller getTableInstance() call now returns the list of live brokers of a table. (#7556)

• Allow MV Field Support For Raw Columns in Text Indices (#7638)

• Allow override distinctCount to segmentPartitionedDistinctCount (#7664)

• Add a quick start with both UPSERT and JSON index (#7669)

• Add revertSegmentReplacement API (#7662)

• Smooth segment reloading with non blocking semantic (#7675)

• Clear the reused record in PartitionUpsertMetadataManager (#7676)

• Replace args4j with picocli (#7665)

• Handle datetime column consistently (#7645)(#7705)

• Allow to carry headers with query requests (#7696) (#7712)

• Allow adding JSON data type for dimension column types (#7718)

• Separate SegmentDirectoryLoader and tierBackend concepts (#7737)

• Implement size balanced V4 raw chunk format (#7661)

• Add presto-pinot-driver lib (#7384)

Major Bug fixes

• Fix null pointer exception for non-existed metric columns in schema for JDBC driver (#7175)

• Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)

• Fixed pinot java client to add zkClient close (#7196)

• Ignore query json parse errors (#7165)

• Fix shutdown hook for PinotServiceManager (#7251) (#7253)

• Make STRING to BOOLEAN data type change as backward compatible schema change (#7259)

• Replace gcp hardcoded values with generic annotations (#6985)

• Fix segment conversion executor for in-place conversion (#7265)

• Fix reporting consuming rate when the Kafka partition level consumer isn't stopped (#7322)

• Fix the issue with concurrent modification for segment lineage (#7343)

• Fix TableNotFound error message in PinotHelixResourceManager (#7340)

• Fix upload LLC segment endpoint truncated download URL (#7361)

• Fix task scheduling on table update (#7362)

• Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)

• Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema (#7366)

• Fix currentOffset volatility in consuming segment(#7365)

• Fix misleading error msg for missing URI (#7367)

• Fix the correctness of getColumnIndices method (#7370)

• Fix SegmentZKMetadta time handling (#7375)

• Fix retention for cleaning up segment lineage (#7424)

• Fix segment generator to not return illegal filenames (#7085)

• Fix missing LLC segments in segment store by adding controller periodic task to upload them (#6778)

• Fix parsing error messages returned to FileUploadDownloadClient (#7428)

• Fix manifest scan which drives /version endpoint (#7456)

• Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)

• Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)

• Fix pql double quote checker exception (#7485)

• Fix minion metrics exporter config (#7496)

• Fix segment unable to retry issue by catching timeout exception during segment replace (#7509)

• Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)

• Fix segment generation commands (#7527)

• Return non zero from main with exception (#7482)

• Fix parquet plugin shading error (#7570)

• Fix the lowest partition id is not 0 for LLC (#7066)

• Fix star-tree index map when column name contains '.' (#7623)

• Fix cluster manager URLs encoding issue(#7639)

• Fix fieldConfig nullable validation (#7648)

• Fix verifyHostname issue in FileUploadDownloadClient (#7703)

• Fix TableCache schema to include the built-in virtual columns (#7706)

• Fix DISTINCT with AS function (#7678)

• Fix SDF pattern in DataPreprocessingHelper (#7721)

• Fix fields missing issue in the source in ParquetNativeRecordReader (#7742)

What's the big change?

The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.

For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?

In order to conquer this challenge, below major changes are made:

• Refactored common interfaces to pinot-spi module

• Concluded four types of modules:

  • Pinot input format: How to read records from various data/file formats: e.g. Avro/CSV/JSON/ORC/Parquet/Thrift

  • Pinot filesystem: How to operate files on various filesystems: e.g. Azure Data Lake/Google Cloud Storage/S3/HDFS

  • Pinot stream ingestion: How to ingest data stream from various upstream systems, e.g. Kafka/Kinesis/Eventhub

  • Pinot batch ingestion: How to run Pinot batch ingestion jobs in various frameworks, like Standalone, Hadoop, Spark.

• Built shaded jars for each individual plugin

• Added support to dynamically load pinot plugins at server startup time

Now the architecture supports a plug-and-play fashion, where new tools can be supported with little and simple extensions, without affecting big chunks of code. Integrations with new streaming services and data formats can be developed in a much more simple and convenient way.

Notable New Features

• SQL Support

  • Added Calcite SQL compiler

  • Added SQL response format (#4694, #4877)

  • Added support for GROUP BY with ORDER BY (#4602)

  • Query console defaults to use SQL syntax (#4994)

  • Support column alias (#5016, #5033)

  • Added SQL query endpoint: /query/sql (#4964)

  • Support arithmetic operators (#5018)

  • Support non-literal expressions for right-side operand in predicate comparison(#5070)

• Added support for DISTINCT (#4535)

• Added support default value for BYTES column (#4583)

• JDK 11 Support

• Added support to tune size vs accuracy for approximation aggregation functions: DistinctCountHLL, PercentileEst, PercentileTDigest (#4666)

• Added Data Anonymizer Tool (#4747)

• Deprecated pinot-hadoop and pinot-spark modules, replace with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark

• Support STRING and BYTES for no dictionary columns in realtime consuming segments (#4791)

• Make pinot-distribution to build a pinot-all jar and assemble it (#4977)

• Added support for PQL case insensitive (#4983)

• Enhanced TableRebalancer logics

  • Moved to new rebalance strategy (#4695)

  • Supported rebalancing tables under any condition(#4990)

  • Supported reassigning completed segments along with Consuming segments for LLC realtime table (#5015)

• Added experimental support for Text Search‌ (#4993)

• Upgraded Helix to version 0.9.4, task management now works as expected (#5020)

• Added date_trunc transformation function. (#4740)

• Support schema evolution for consuming segment. (#4954)

• APIs Additions/Changes

  • Pinot Admin Command

    • Added -queryType option in PinotAdmin PostQuery subcommand (#4726)

    • Added -schemaFile as option in AddTable command (#4959)

    • Added OperateClusterConfig sub command in PinotAdmin (#5073)

  • Pinot Controller Rest APIs

    • Get Table leader controller resource (#4545)

    • Support HTTP POST/PUT to upload JSON encoded schema (#4639)

    • Table rebalance API now requires both table name and type as parameters. (#4824)

    • Refactored Segments APIs (#4806)

    • Added segment batch deletion REST API (#4828)

    • Update schema API to reload table on schema change when applicable (#4838)

    • Enhance the task related REST APIs (#5054)

    • Added PinotClusterConfig REST APIs (#5073)

      • GET /cluster/configs

      • POST /cluster/configs

      • DELETE /cluster/configs/{configName}

• Configurations Additions/Changes

  • Config: controller.host is now optional in Pinot Controller

  • Added instance config: queriesDisabled to disable query sending to a running server (#4767)

  • Added broker config: pinot.broker.enable.query.limit.override configurable max query response size (#5040)

  • Removed deprecated server configs (#4903)

    • pinot.server.starter.enableSegmentsLoadingCheck

    • pinot.server.starter.timeoutInSeconds

    • pinot.server.instance.enable.shutdown.delay

    • pinot.server.instance.starter.maxShutdownWaitTime

    • pinot.server.instance.starter.checkIntervalTime

  • Decouple server instance id with hostname/port config. (#4995)

  • Add FieldConfig to encapsulate encoding, indexing info for a field.(#5006)

Major Bug Fixes

• Fixed the bug of releasing the segment when there are still threads working on it. (#4764)

• Fixed the bug of uneven task distribution for threads (#4793)

• Fixed encryption for .tar.gz segment file upload (#4855)

• Fixed controller rest API to download segment from non local FS. (#4808)

• Fixed the bug of not releasing segment lock if segment recovery throws exception (#4882)

• Fixed the issue of server not registering state model factory before connecting the Helix manager (#4929)

• Fixed the exception in server instance when Helix starts a new ZK session (#4976)

• Fixed ThreadLocal DocIdSet issue in ExpressionFilterOperator (#5114)

• Fixed the bug in default value provider classes (#5137)

• Fixed the bug when no segment exists in RealtimeSegmentSelector (#5138)

Work in Progress

• We are in the process of supporting text search query functionalities.

• We are in the process of supporting null value (#4230), currently limited query feature is supported

  • Added Presence Vector to represent null value (#4585)

  • Added null predicate support for leaf predicates (#4943)

Backward Incompatible Changes

• It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Please ensure that you upgrade to release 0.2.0 first, then upgrade to this version.

• If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. Please see pinot-admin.sh as an example.

• As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

• Kafka 0.9 is no longer included in the release distribution.

• Pull request #4806 introduces a backward incompatible API change for segments management.

  • Removed segment toggle APIs

  • Removed list all segments in cluster APIs

  • Deprecated below APIs:

    • GET /tables/{tableName}/segments

    • GET /tables/{tableName}/segments/metadata

    • GET /tables/{tableName}/segments/crc

    • GET /tables/{tableName}/segments/{segmentName}

    • GET /tables/{tableName}/segments/{segmentName}/metadata

    • GET /tables/{tableName}/segments/{segmentName}/reload

    • POST /tables/{tableName}/segments/{segmentName}/reload

    • GET /tables/{tableName}/segments/reload

    • POST /tables/{tableName}/segments/reload

• Pull request #5054 deprecated below task related APIs:

  • GET:

    • /tasks/taskqueues: List all task queues

    • /tasks/taskqueuestate/{taskType} -> /tasks/{taskType}/state

    • /tasks/tasks/{taskType} -> /tasks/{taskType}/tasks

    • /tasks/taskstates/{taskType} -> /tasks/{taskType}/taskstates

    • /tasks/taskstate/{taskName} -> /tasks/task/{taskName}/taskstate

    • /tasks/taskconfig/{taskName} -> /tasks/task/{taskName}/taskconfig

  • PUT:

    • /tasks/scheduletasks -> POST /tasks/schedule

    • /tasks/cleanuptasks/{taskType} -> /tasks/{taskType}/cleanup

    • /tasks/taskqueue/{taskType}: Toggle a task queue

  • DELETE:

    • /tasks/taskqueue/{taskType} -> /tasks/{taskType}

• Deprecated modules pinot-hadoop and pinot-spark and replaced with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark.

• Introduced new Pinot batch ingestion jobs and yaml based job specs to define segment generation jobs and segment push jobs.

• You may see exceptions like below in pinot-brokers during cluster upgrade, but it's safe to ignore them.

  Copy

  2020/03/09 23:37:19.879 ERROR [HelixTaskExecutor] [CallbackProcessor@b808af5-pinot] [pinot-broker] [] Message cannot be processed: 78816abe-5288-4f08-88c0-f8aa596114fe, {CREATE_TIMESTAMP=1583797034542, MSG_ID=78816abe-5288-4f08-88c0-f8aa596114fe, MSG_STATE=unprocessable, MSG_SUBTYPE=REFRESH_SEGMENT, MSG_TYPE=USER_DEFINE_MSG, PARTITION_NAME=fooBar_OFFLINE, RESOURCE_NAME=brokerResource, RETRY_COUNT=0, SRC_CLUSTER=pinot, SRC_INSTANCE_TYPE=PARTICIPANT, SRC_NAME=Controller_hostname.domain,com_9000, TGT_NAME=Broker_hostname,domain.com_6998, TGT_SESSION_ID=f6e19a457b80db5, TIMEOUT=-1, segmentName=fooBar_559, tableName=fooBar_OFFLINE}{}{}
  java.lang.UnsupportedOperationException: Unsupported user defined message sub type: REFRESH_SEGMENT
        at org.apache.pinot.broker.broker.helix.TimeboundaryRefreshMessageHandlerFactory.createHandler(TimeboundaryRefreshMessageHandlerFactory.java:68) ~[pinot-broker-0.2.1172.jar:0.3.0-SNAPSHOT-c9d88e47e02d799dc334d7dd1446a38d9ce161a3]
        at org.apache.helix.messaging.handling.HelixTaskExecutor.createMessageHandler(HelixTaskExecutor.java:1096) ~[helix-core-0.9.1.509.jar:0.9.1.509]
        at org.apache.helix.messaging.handling.HelixTaskExecutor.onMessage(HelixTaskExecutor.java:866) [helix-core-0.9.1.509.jar:0.9.1.509]