1 of 4

Releases

The following summarizes Pinot's releases, from the latest one to the earliest one.

Note

Before upgrading from one version to another one, please read the release notes as there may be some incompatibilities between versions.

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

0.3.0

0.3.0 release of Apache Pinot introduces the concept of plugins that makes it easy to extend and integrate with other systems.

What's the big change?

The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.

For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?

0.2.0

The 0.2.0 release is the first release after the initial one and includes several improvements, reported following.

New Features and Bug Fixes

Added support for Kafka 2.0
Table rebalancer now supports a minimum number of serving replicas during rebalance
Added support for UDF in filter predicates and selection
Added support to use hex string as the representation of byte array for queries (see PR )
Added support for parquet reader (see PR )
Introduced interface stability and audience annotations (see PR )
Refactor HelixBrokerStarter to separate constructor and start() - backwards incompatible (see PR )
Admin tool for listing segments with invalid intervals for offline tables
Migrated to log4j2 (see PR )
Added simple avro msg decoder
Added support for passing headers in Pinot client
Table rebalancer now supports a minimum number of serving replicas during rebalance
Support transform functions with AVG aggregation function (see PR )
Configurations additions/changes
- Allow customized metrics prefix (see PR )
- Controller.enable.batch.message.mode to false by default (see PR )

Work in Progress

We are in the process of separating Helix and Pinot controllers, so that administrators can have the option of running independent Helix controllers and Pinot controllers.
We are in the process of moving towards supporting SQL query format and results.
We are in the process of separating instance and segment assignment using instance pools to optimize the number of Helix state transitions in Pinot clusters with thousands of tables.

Other Notes

Task management does not work correctly in this release, due to bugs in Helix. We will upgrade to Helix 0.9.2 (or later) version to get this fixed.
You must upgrade to this release before moving onto newer versions of Pinot release. The protocol between Pinot-broker and Pinot-server has been changed and this release has the code to retain compatibility moving forward. Skipping this release may (depending on your environment) cause query errors if brokers are upgraded and servers are in the process of being upgraded.
As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

0.1.0

The 0.1.0 is first release of Pinot as an Apache project

New Features

First release

0.2.0

The 0.2.0 release is the first release after the initial one and includes several improvements, reported following.

New Features and Bug Fixes

Added support for Kafka 2.0
Table rebalancer now supports a minimum number of serving replicas during rebalance
Added support for UDF in filter predicates and selection
Added support to use hex string as the representation of byte array for queries (see PR )
Added support for parquet reader (see PR )
Introduced interface stability and audience annotations (see PR )
Refactor HelixBrokerStarter to separate constructor and start() - backwards incompatible (see PR )
Admin tool for listing segments with invalid intervals for offline tables
Migrated to log4j2 (see PR )
Added simple avro msg decoder
Added support for passing headers in Pinot client
Table rebalancer now supports a minimum number of serving replicas during rebalance
Support transform functions with AVG aggregation function (see PR )
Configurations additions/changes
- Allow customized metrics prefix (see PR )
- Controller.enable.batch.message.mode to false by default (see PR )

Work in Progress

We are in the process of separating Helix and Pinot controllers, so that administrators can have the option of running independent Helix controllers and Pinot controllers.
We are in the process of moving towards supporting SQL query format and results.
We are in the process of separating instance and segment assignment using instance pools to optimize the number of Helix state transitions in Pinot clusters with thousands of tables.

Other Notes

Task management does not work correctly in this release, due to bugs in Helix. We will upgrade to Helix 0.9.2 (or later) version to get this fixed.
You must upgrade to this release before moving onto newer versions of Pinot release. The protocol between Pinot-broker and Pinot-server has been changed and this release has the code to retain compatibility moving forward. Skipping this release may (depending on your environment) cause query errors if brokers are upgraded and servers are in the process of being upgraded.
As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.

Refactored common interfaces to pinot-spi module
Concluded four types of modules:
- Pinot input format: How to read records from various data/file formats: e.g. Avro/CSV/JSON/ORC/Parquet/Thrift
- Pinot filesystem: How to operate files on various filesystems: e.g. Azure Data Lake/Google Cloud Storage/S3/HDFS
- Pinot stream ingestion: How to ingest data stream from various upstream systems, e.g. Kafka/Kinesis/Eventhub
- Pinot batch ingestion: How to run Pinot batch ingestion jobs in various frameworks, like Standalone, Hadoop, Spark.
Built shaded jars for each individual plugin
Added support to dynamically load pinot plugins at server startup time

SQL Support
- Added Calcite SQL compiler
- Added SQL response format (#4694, #4877)
- Added support for GROUP BY with ORDER BY ()
- Query console defaults to use SQL syntax ()
- Support column alias (, )
- Added SQL query endpoint: /query/sql ()
- Support arithmetic operators ()
- Support non-literal expressions for right-side operand in predicate comparison()
Added support for DISTINCT ()
Added support default value for BYTES column ()
JDK 11 Support
Added support to tune size vs accuracy for approximation aggregation functions: DistinctCountHLL, PercentileEst, PercentileTDigest ()
Added Data Anonymizer Tool ()
Deprecated pinot-hadoop and pinot-spark modules, replace with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark
Support STRING and BYTES for no dictionary columns in realtime consuming segments ()
Make pinot-distribution to build a pinot-all jar and assemble it ()
Added support for PQL case insensitive ()
Enhanced TableRebalancer logics
- Moved to new rebalance strategy ()
- Supported rebalancing tables under any condition()
Added experimental support for Text Search‌ ()
Upgraded Helix to version 0.9.4, task management now works as expected ()
Added date_trunc transformation function. ()
Support schema evolution for consuming segment. ()
APIs Additions/Changes
- Pinot Admin Command
  - Added -queryType option in PinotAdmin PostQuery
Configurations Additions/Changes
- Config: controller.host is now optional in Pinot Controller
- Added instance config: queriesDisabled to disable query sending to a running server (

Fixed the bug of releasing the segment when there are still threads working on it. (#4764)
Fixed the bug of uneven task distribution for threads (#4793)
Fixed encryption for .tar.gz segment file upload (#4855)
Fixed controller rest API to download segment from non local FS. ()
Fixed the bug of not releasing segment lock if segment recovery throws exception ()
Fixed the issue of server not registering state model factory before connecting the Helix manager ()
Fixed the exception in server instance when Helix starts a new ZK session ()
Fixed ThreadLocal DocIdSet issue in ExpressionFilterOperator ()
Fixed the bug in default value provider classes ()
Fixed the bug when no segment exists in RealtimeSegmentSelector ()

It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Please ensure that you upgrade to release 0.2.0 first, then upgrade to this version.
If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. Please see pinot-admin.sh as an example.
As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.
Kafka 0.9 is no longer included in the release distribution.
Pull request introduces a backward incompatible API change for segments management.
- Removed segment toggle APIs
- Removed list all segments in cluster APIs
Pull request deprecated below task related APIs:
- GET:
  - /tasks/taskqueues: List all task queues
Deprecated modules pinot-hadoop and pinot-spark and replaced with pinot-batch-ingestion-hadoop and pinot-batch-ingestion-spark.
Introduced new Pinot batch ingestion jobs and yaml based job specs to define segment generation jobs and segment push jobs.
You may see exceptions like below in pinot-brokers during cluster upgrade, but it's safe to ignore them.

Pinot Controller Rest APIs

Get Table leader controller resource (#4545)
Support HTTP POST/PUT to upload JSON encoded schema (#4639)
Table rebalance API now requires both table name and type as parameters. ()
Refactored Segments APIs ()
Added segment batch deletion REST API ()
Update schema API to reload table on schema change when applicable ()
Enhance the task related REST APIs ()
Added PinotClusterConfig REST APIs ()
- GET /cluster/configs
- POST /cluster/configs

Removed deprecated server configs (#4903)

pinot.server.starter.enableSegmentsLoadingCheck
pinot.server.starter.timeoutInSeconds
pinot.server.instance.enable.shutdown.delay
pinot.server.instance.starter.maxShutdownWaitTime
pinot.server.instance.starter.checkIntervalTime

Deprecated below APIs:

GET /tables/{tableName}/segments
GET /tables/{tableName}/segments/metadata
GET /tables/{tableName}/segments/crc
GET /tables/{tableName}/segments/{segmentName}
GET /tables/{tableName}/segments/{segmentName}/metadata
GET /tables/{tableName}/segments/{segmentName}/reload
POST /tables/{tableName}/segments/{segmentName}/reload
GET /tables/{tableName}/segments/reload
POST /tables/{tableName}/segments/reload

2020/03/09 23:37:19.879 ERROR [HelixTaskExecutor] [CallbackProcessor@b808af5-pinot] [pinot-broker] [] Message cannot be processed: 78816abe-5288-4f08-88c0-f8aa596114fe, {CREATE_TIMESTAMP=1583797034542, MSG_ID=78816abe-5288-4f08-88c0-f8aa596114fe, MSG_STATE=unprocessable, MSG_SUBTYPE=REFRESH_SEGMENT, MSG_TYPE=USER_DEFINE_MSG, PARTITION_NAME=fooBar_OFFLINE, RESOURCE_NAME=brokerResource, RETRY_COUNT=0, SRC_CLUSTER=pinot, SRC_INSTANCE_TYPE=PARTICIPANT, SRC_NAME=Controller_hostname.domain,com_9000, TGT_NAME=Broker_hostname,domain.com_6998, TGT_SESSION_ID=f6e19a457b80db5, TIMEOUT=-1, segmentName=fooBar_559, tableName=fooBar_OFFLINE}{}{}
java.lang.UnsupportedOperationException: Unsupported user defined message sub type: REFRESH_SEGMENT
      at org.apache.pinot.broker.broker.helix.TimeboundaryRefreshMessageHandlerFactory.createHandler(TimeboundaryRefreshMessageHandlerFactory.java:68) ~[pinot-broker-0.2.1172.jar:0.3.0-SNAPSHOT-c9d88e47e02d799dc334d7dd1446a38d9ce161a3]
      at org.apache.helix.messaging.handling.HelixTaskExecutor.createMessageHandler(HelixTaskExecutor.java:1096) ~[helix-core-0.9.1.509.jar:0.9.1.509]
      at org.apache.helix.messaging.handling.HelixTaskExecutor.onMessage(HelixTaskExecutor.java:866) [helix-core-0.9.1.509.jar:0.9.1.509]

2020/03/09 23:37:19.879 ERROR [HelixTaskExecutor] [CallbackProcessor@b808af5-pinot] [pinot-broker] [] Message cannot be processed: 78816abe-5288-4f08-88c0-f8aa596114fe, {CREATE_TIMESTAMP=1583797034542, MSG_ID=78816abe-5288-4f08-88c0-f8aa596114fe, MSG_STATE=unprocessable, MSG_SUBTYPE=REFRESH_SEGMENT, MSG_TYPE=USER_DEFINE_MSG, PARTITION_NAME=fooBar_OFFLINE, RESOURCE_NAME=brokerResource, RETRY_COUNT=0, SRC_CLUSTER=pinot, SRC_INSTANCE_TYPE=PARTICIPANT, SRC_NAME=Controller_hostname.domain,com_9000, TGT_NAME=Broker_hostname,domain.com_6998, TGT_SESSION_ID=f6e19a457b80db5, TIMEOUT=-1, segmentName=fooBar_559, tableName=fooBar_OFFLINE}{}{}
java.lang.UnsupportedOperationException: Unsupported user defined message sub type: REFRESH_SEGMENT
      at org.apache.pinot.broker.broker.helix.TimeboundaryRefreshMessageHandlerFactory.createHandler(TimeboundaryRefreshMessageHandlerFactory.java:68) ~[pinot-broker-0.2.1172.jar:0.3.0-SNAPSHOT-c9d88e47e02d799dc334d7dd1446a38d9ce161a3]
      at org.apache.helix.messaging.handling.HelixTaskExecutor.createMessageHandler(HelixTaskExecutor.java:1096) ~[helix-core-0.9.1.509.jar:0.9.1.509]
      at org.apache.helix.messaging.handling.HelixTaskExecutor.onMessage(HelixTaskExecutor.java:866) [helix-core-0.9.1.509.jar:0.9.1.509]

Releases

Note

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

0.3.0

What's the big change?

0.2.0

New Features and Bug Fixes

Work in Progress

Other Notes

0.1.0

New Features

Releases

Note

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

0.2.0

New Features and Bug Fixes

Work in Progress

Other Notes

0.1.0

New Features

0.3.0

What's the big change?

Notable New Features

Major Bug Fixes

Work in Progress

Backward Incompatible Changes

Releases

hashtagNote

hashtag0.3.0 (March 2020)

hashtag0.2.0 (November 2019)

hashtag0.1.0 (March 2019, First release)

0.3.0

hashtagWhat's the big change?

0.2.0

hashtagNew Features and Bug Fixes

hashtagWork in Progress

hashtagOther Notes

0.1.0

hashtagNew Features

Releases

hashtagNote

hashtag0.3.0 (March 2020)

hashtag0.2.0 (November 2019)

hashtag0.1.0 (March 2019, First release)

0.2.0

hashtagNew Features and Bug Fixes

hashtagWork in Progress

hashtagOther Notes

0.1.0

hashtagNew Features

0.3.0

hashtagWhat's the big change?

hashtagNotable New Features

hashtagMajor Bug Fixes

hashtagWork in Progress

hashtagBackward Incompatible Changes

Note

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

What's the big change?

New Features and Bug Fixes

Work in Progress

Other Notes

New Features

Note

0.3.0 (March 2020)

0.2.0 (November 2019)

0.1.0 (March 2019, First release)

New Features and Bug Fixes

Work in Progress

Other Notes

New Features

What's the big change?

Notable New Features

Major Bug Fixes

Work in Progress

Backward Incompatible Changes