0.3.0
0.3.0 release of Apache Pinot introduces the concept of plugins that makes it easy to extend and integrate with other systems.
What's the big change?
The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.
For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?
In order to conquer this challenge, below major changes are made:
Refactored common interfaces to pinot-spi module
Concluded four types of modules:
Pinot input format: How to read records from various data/file formats: e.g. Avro
Now the architecture supports a plug-and-play fashion, where new tools can be supported with little and simple extensions, without affecting big chunks of code. Integrations with new streaming services and data formats can be developed in a much more simple and convenient way.
Notable New Features
SQL Support
Added Calcite SQL compiler
Added SQL response format (, )
Major Bug Fixes
Fixed the bug of releasing the segment when there are still threads working on it. ()
Fixed the bug of uneven task distribution for threads ()
Fixed encryption for .tar.gz segment file upload ()
Work in Progress
We are in the process of supporting text search query functionalities.
We are in the process of supporting null value (), currently limited query feature is supported
Added Presence Vector to represent null value ()
Backward Incompatible Changes
It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Please ensure that you upgrade to release 0.2.0 first, then upgrade to this version.
If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. Please see pinot-admin.sh as an example.
