0.3.0
0.3.0 release of Apache Pinot introduces the concept of plugins that makes it easy to extend and integrate with other systems.
What's the big change?
The reason behind the architectural change from the previous release (0.2.0) and this release (0.3.0), is the possibility of extending Apache Pinot. The 0.2.0 release was not flexible enough to support new storage types nor new stream types. Basically, inserting a new functionality required to change too much code. Thus, the Pinot team went through an extensive refactoring and improvement of the source code.
For instance, the picture below shows the module dependencies of the 0.2.X or previous releases. If we wanted to support a new storage type, we would have had to change several modules. Pretty bad, huh?
In order to conquer this challenge, below major changes are made:
Refactored common interfaces to
pinot-spi
moduleConcluded four types of modules:
Pinot input format: How to read records from various data/file formats: e.g.
Avro
/CSV
/JSON
/ORC
/Parquet
/Thrift
Pinot filesystem: How to operate files on various filesystems: e.g.
Azure Data Lake
/Google Cloud Storage
/S3
/HDFS
Pinot stream ingestion: How to ingest data stream from various upstream systems, e.g.
Kafka
/Kinesis
/Eventhub
Pinot batch ingestion: How to run Pinot batch ingestion jobs in various frameworks, like
Standalone
,Hadoop
,Spark
.
Built shaded jars for each individual plugin
Added support to dynamically load pinot plugins at server startup time
Now the architecture supports a plug-and-play fashion, where new tools can be supported with little and simple extensions, without affecting big chunks of code. Integrations with new streaming services and data formats can be developed in a much more simple and convenient way.
Notable New Features
SQL Support
Added support for
DISTINCT
(#4535)Added support default value for
BYTES
column (#4583)JDK 11
SupportAdded support to tune size vs accuracy for approximation aggregation functions:
DistinctCountHLL
,PercentileEst
,PercentileTDigest
(#4666)Added Data Anonymizer Tool (#4747)
Deprecated
pinot-hadoop
andpinot-spark
modules, replace withpinot-batch-ingestion-hadoop
andpinot-batch-ingestion-spark
Support
STRING
andBYTES
for no dictionary columns in realtime consuming segments (#4791)Make
pinot-distribution
to build a pinot-all jar and assemble it (#4977)Added support for PQL case insensitive (#4983)
Added experimental support for Text Search (#4993)
Upgraded Helix to version 0.9.4, task management now works as expected (#5020)
Added
date_trunc
transformation function. (#4740)Support schema evolution for consuming segment. (#4954)
APIs Additions/Changes
Pinot Controller Rest APIs
Get Table leader controller resource (#4545)
Support HTTP
POST
/PUT
to upload JSON encoded schema (#4639)Table rebalance API now requires both table name and type as parameters. (#4824)
Refactored Segments APIs (#4806)
Added segment batch deletion REST API (#4828)
Update schema API to reload table on schema change when applicable (#4838)
Enhance the task related REST APIs (#5054)
Added PinotClusterConfig REST APIs (#5073)
GET /cluster/configs
POST /cluster/configs
DELETE /cluster/configs/{configName}
Configurations Additions/Changes
Config:
controller.host
is now optional in Pinot ControllerAdded instance config:
queriesDisabled
to disable query sending to a running server (#4767)Added broker config:
pinot.broker.enable.query.limit.override
configurable max query response size (#5040)Removed deprecated server configs (#4903)
pinot.server.starter.enableSegmentsLoadingCheck
pinot.server.starter.timeoutInSeconds
pinot.server.instance.enable.shutdown.delay
pinot.server.instance.starter.maxShutdownWaitTime
pinot.server.instance.starter.checkIntervalTime
Decouple server instance id with hostname/port config. (#4995)
Add FieldConfig to encapsulate encoding, indexing info for a field.(#5006)
Major Bug Fixes
Fixed the bug of releasing the segment when there are still threads working on it. (#4764)
Fixed the bug of uneven task distribution for threads (#4793)
Fixed encryption for
.tar.gz
segment file upload (#4855)Fixed controller rest API to download segment from non local FS. (#4808)
Fixed the bug of not releasing segment lock if segment recovery throws exception (#4882)
Fixed the issue of server not registering state model factory before connecting the Helix manager (#4929)
Fixed the exception in server instance when Helix starts a new ZK session (#4976)
Fixed ThreadLocal DocIdSet issue in ExpressionFilterOperator (#5114)
Fixed the bug in default value provider classes (#5137)
Fixed the bug when no segment exists in RealtimeSegmentSelector (#5138)
Work in Progress
We are in the process of supporting text search query functionalities.
Backward Incompatible Changes
It’s a disruptive upgrade from version 0.1.0 to this because of the protocol changes between Pinot Broker and Pinot Server. Please ensure that you upgrade to release 0.2.0 first, then upgrade to this version.
If you build your own startable or war without using scripts generated in Pinot-distribution module. For Java 8, an environment variable “plugins.dir” is required for Pinot to find out where to load all the Pinot plugin jars. For Java 11, plugins directory is required to be explicitly set into classpath. Please see
pinot-admin.sh
as an example.As always, we recommend that you upgrade controllers first, and then brokers and lastly the servers in order to have zero downtime in production clusters.
Kafka 0.9 is no longer included in the release distribution.
Pull request #4806 introduces a backward incompatible API change for segments management.
Removed segment toggle APIs
Removed list all segments in cluster APIs
Deprecated below APIs:
GET /tables/{tableName}/segments
GET /tables/{tableName}/segments/metadata
GET /tables/{tableName}/segments/crc
GET /tables/{tableName}/segments/{segmentName}
GET /tables/{tableName}/segments/{segmentName}/metadata
GET /tables/{tableName}/segments/{segmentName}/reload
POST /tables/{tableName}/segments/{segmentName}/reload
GET /tables/{tableName}/segments/reload
POST /tables/{tableName}/segments/reload
Pull request #5054 deprecated below task related APIs:
GET:
/tasks/taskqueues
: List all task queues/tasks/taskqueuestate/{taskType}
->/tasks/{taskType}/state
/tasks/tasks/{taskType}
->/tasks/{taskType}/tasks
/tasks/taskstates/{taskType}
->/tasks/{taskType}/taskstates
/tasks/taskstate/{taskName}
->/tasks/task/{taskName}/taskstate
/tasks/taskconfig/{taskName}
->/tasks/task/{taskName}/taskconfig
PUT:
/tasks/scheduletasks
->POST
/tasks/schedule
/tasks/cleanuptasks/{taskType}
->/tasks/{taskType}/cleanup
/tasks/taskqueue/{taskType}
: Toggle a task queue
DELETE:
/tasks/taskqueue/{taskType}
->/tasks/{taskType}
Deprecated modules
pinot-hadoop
andpinot-spark
and replaced withpinot-batch-ingestion-hadoop
andpinot-batch-ingestion-spark
.Introduced new Pinot batch ingestion jobs and yaml based job specs to define segment generation jobs and segment push jobs.
You may see exceptions like below in pinot-brokers during cluster upgrade, but it's safe to ignore them.
Last updated