0.9.0

Summary
This release introduces a new features: Segment Merge and Rollup to simplify users day to day operational work. A new metrics plugin is added to support dropwizard. As usual, new functionalities and many UI/ Performance improvements.
The release was cut from the following commit: 13c9ee9 and the following cherry-picks: 668b5e0, ee887b9​
Support Segment Merge and Roll-up
LinkedIn operates a large multi-tenant cluster that serves a business metrics dashboard, and noticed that their tables consisted of millions of small segments. This was leading to slow operations in Helix/Zookeeper, long running queries due to having too many tasks to process, as well as using more space because of a lack of compression.
To solve this problem they added the Segment Merge task, which compresses segments based on timestamps and rolls up/aggregates older data. The task can be run on a schedule or triggered manually via the Pinot REST API.
At the moment this feature is only available for offline tables, but will be added for real-time tables in a future release.
Major Changes:
Integrate enhanced SegmentProcessorFramework into MergeRollupTaskExecutor (#7180)
Merge/Rollup task scheduler for offline tables. (#7178)
Fix MergeRollupTask uploading segments not updating their metadata (#7289)
MergeRollupTask integration tests (#7283)
Add mergeRollupTask delay metrics (#7368)
MergeRollupTaskGenerator enhancement: enable parallel buckets scheduling (#7481)
Use maxEndTimeMs for merge/roll-up delay metrics. (#7617)
UI Improvement
This release also sees improvements to Pinot’s query console UI.
Cmd+Enter shortcut to run query in query console (#7359)
Showing tooltip in SQL Editor (#7387)
Make the SQL Editor box expandable (#7381)
Fix tables ordering by number of segments (#7564)
SQL Improvements
There have also been improvements and additions to Pinot’s SQL implementation.
New functions:
IN (#7542)
LASTWITHTIME (#7584)
ID_SET on MV columns (#7355)
Raw results for Percentile TDigest and Est (#7226),
Add timezone as argument in function toDateTime (#7552)
New predicates are supported:
LIKE(#7214)
REGEXP_EXTRACT(#7114)
FILTER(#7566)
Query compatibility improvements:
Infer data type for Literal (#7332)
Support logical identifier in predicate (#7347)
Support JSON queries with top-level array path expression. (#7511)
Support configurable group by trim size to improve results accuracy (#7241)
Performance Improvements
This release contains many performance improvement, you may sense it for you day to day queries. Thanks to all the great contributions listed below:
Reduce the disk usage for segment conversion task (#7193)
Simplify association between Java Class and PinotDataType for faster mapping (#7402)
Avoid creating stateless ParseContextImpl once per jsonpath evaluation, avoid varargs allocation (#7412)
Replace MINUS with STRCMP (#7394)
Bit-sliced range index for int, long, float, double, dictionarized SV columns (#7454)
Use MethodHandle to access vectorized unsigned comparison on JDK9+ (#7487)
Add option to limit thread usage per query (#7492)
Improved range queries (#7513)
Faster bitmap scans (#7530)
Optimize EmptySegmentPruner to skip pruning when there is no empty segments (#7531)
Map bitmaps through a bounded window to avoid excessive disk pressure (#7535)
Allow RLE compression of bitmaps for smaller file sizes (#7582)
Support raw index properties for columns with JSON and RANGE indexes (#7615)
Enhance BloomFilter rule to include IN predicate(#7444) (#7624)
Introduce LZ4_WITH_LENGTH chunk compression type (#7655)
Enhance ColumnValueSegmentPruner and support bloom filter prefetch (#7654)
Apply the optimization on dictIds within the segment to DistinctCountHLL aggregation func (#7630)
During segment pruning, release the bloom filter after each segment is processed (#7668)
Fix JSONPath cache inefficient issue (#7409)
Optimize getUnpaddedString with SWAR padding search (#7708)
Lighter weight LiteralTransformFunction, avoid excessive array fills (#7707)
Inline binary comparison ops to prevent function call overhead (#7709)
Memoize literals in query context in order to deduplicate them (#7720)
Other Notable New Features and Changes
Human Readable Controller Configs (#7173)
Add the support of geoToH3 function (#7182)
Add Apache Pulsar as Pinot Plugin (#7223) (#7247)
Add dropwizard metrics plugin (#7263)
Introduce OR Predicate Execution On Star Tree Index (#7184)
Allow to extract values from array of objects with jsonPathArray (#7208)
Add Realtime table metadata and indexes API. (#7169)
Support array with mixing data types (#7234)
Support force download segment in reload API (#7249)
Show uncompressed znRecord from zk api (#7304)
Add debug endpoint to get minion task status. (#7300)
Validate CSV Header For Configured Delimiter (#7237)
Add auth tokens and user/password support to ingestion job command (#7233)
Add option to store the hash of the upsert primary key (#7246)
Add null support for time column (#7269)
Add mode aggregation function (#7318)
Support disable swagger in Pinot servers (#7341)
Delete metadata properly on table deletion (#7329)
Add basic Obfuscator Support (#7407)
Add AWS sts dependency to enable auth using web identity token. (#7017)(#7445)
Mask credentials in debug endpoint /appconfigs (#7452)
Fix /sql query endpoint now compatible with auth (#7230)
Fix case sensitive issue in BasicAuthPrincipal permission check (#7354)
Fix auth token injection in SegmentGenerationAndPushTaskExecutor (#7464)
Add segmentNameGeneratorType config to IndexingConfig (#7346)
Support trigger PeriodicTask manually (#7174)
Add endpoint to check minion task status for a single task. (#7353)
Showing partial status of segment and counting CONSUMING state as good segment status (#7327)
Add "num rows in segments" and "num segments queried per host" to the output of Realtime Provisioning Rule (#7282)
Check schema backward-compatibility when updating schema through addSchema with override (#7374)
Optimize IndexedTable (#7373)
Support indices remove in V3 segment format (#7301)
Optimize TableResizer (#7392)
Introduce resultSize in IndexedTable (#7420)
Offset based realtime consumption status checker (#7267)
Add causes to stack trace return (#7460)
Create controller resource packages config key (#7488)
Enhance TableCache to support schema name different from table name (#7525)
Add validation for realtimeToOffline task (#7523)
Unify CombineOperator multi-threading logic (#7450)
Support no downtime rebalance for table with 1 replica in TableRebalancer (#7532)
Introduce MinionConf, move END_REPLACE_SEGMENTS_TIMEOUT_MS to minion config instead of task config. (#7516)
Adjust tuner api (#7553)
Adding config for metrics library (#7551)
Add geo type conversion scalar functions (#7573)
Add BOOLEAN_ARRAY and TIMESTAMP_ARRAY types (#7581)
Add MV raw forward index and MV BYTES data type (#7595)
Enhance TableRebalancer to offload the segments from most loaded instances first (#7574)
Improve get tenant API to differentiate offline and realtime tenants (#7548)
Refactor query rewriter to interfaces and implementations to allow customization (#7576)
In ServiceStartable, apply global cluster config in ZK to instance config (#7593)
Make dimension tables creation bypass tenant validation (#7559)
Allow Metadata and Dictionary Based Plans for No Op Filters (#7563)
Reject query with identifiers not in schema (#7590)
Round Robin IP addresses when retry uploading/downloading segments (#7585)
Support multi-value derived column in offline table reload (#7632)
Support segmentNamePostfix in segment name (#7646)
Add select segments API (#7651)
Controller getTableInstance() call now returns the list of live brokers of a table. (#7556)
Allow MV Field Support For Raw Columns in Text Indices (#7638)
Allow override distinctCount to segmentPartitionedDistinctCount (#7664)
Add a quick start with both UPSERT and JSON index (#7669)
Add revertSegmentReplacement API (#7662)
Smooth segment reloading with non blocking semantic (#7675)
Clear the reused record in PartitionUpsertMetadataManager (#7676)
Replace args4j with picocli (#7665)
Handle datetime column consistently (#7645)(#7705)
Allow to carry headers with query requests (#7696) (#7712)
Allow adding JSON data type for dimension column types (#7718)
Separate SegmentDirectoryLoader and tierBackend concepts (#7737)
Implement size balanced V4 raw chunk format (#7661)
Add presto-pinot-driver lib (#7384)
Major Bug fixes
Fix null pointer exception for non-existed metric columns in schema for JDBC driver (#7175)
Fix the config key for TASK_MANAGER_FREQUENCY_PERIOD (#7198)
Fixed pinot java client to add zkClient close (#7196)
Ignore query json parse errors (#7165)
Fix shutdown hook for PinotServiceManager (#7251) (#7253)
Make STRING to BOOLEAN data type change as backward compatible schema change (#7259)
Replace gcp hardcoded values with generic annotations (#6985)
Fix segment conversion executor for in-place conversion (#7265)
Fix reporting consuming rate when the Kafka partition level consumer isn't stopped (#7322)
Fix the issue with concurrent modification for segment lineage (#7343)
Fix TableNotFound error message in PinotHelixResourceManager (#7340)
Fix upload LLC segment endpoint truncated download URL (#7361)
Fix task scheduling on table update (#7362)
Fix metric method for ONLINE_MINION_INSTANCES metric (#7363)
Fix JsonToPinotSchema behavior to be consistent with AvroSchemaToPinotSchema (#7366)
Fix currentOffset volatility in consuming segment(#7365)
Fix misleading error msg for missing URI (#7367)
Fix the correctness of getColumnIndices method (#7370)
Fix SegmentZKMetadta time handling (#7375)
Fix retention for cleaning up segment lineage (#7424)
Fix segment generator to not return illegal filenames (#7085)
Fix missing LLC segments in segment store by adding controller periodic task to upload them (#6778)
Fix parsing error messages returned to FileUploadDownloadClient (#7428)
Fix manifest scan which drives /version endpoint (#7456)
Fix missing rate limiter if brokerResourceEV becomes null due to ZK connection (#7470)
Fix race conditions between segment merge/roll-up and purge (or convertToRawIndex) tasks: (#7427)
Fix pql double quote checker exception (#7485)
Fix minion metrics exporter config (#7496)
Fix segment unable to retry issue by catching timeout exception during segment replace (#7509)
Add Exception to Broker Response When Not All Segments Are Available (Partial Response) (#7397)
Fix segment generation commands (#7527)
Return non zero from main with exception (#7482)
Fix parquet plugin shading error (#7570)
Fix the lowest partition id is not 0 for LLC (#7066)
Fix star-tree index map when column name contains '.' (#7623)
Fix cluster manager URLs encoding issue(#7639)
Fix fieldConfig nullable validation (#7648)
Fix verifyHostname issue in FileUploadDownloadClient (#7703)
Fix TableCache schema to include the built-in virtual columns (#7706)
Fix DISTINCT with AS function (#7678)
Fix SDF pattern in DataPreprocessingHelper (#7721)
Fix fields missing issue in the source in ParquetNativeRecordReader (#7742)
0.9.1
0.8.0
Last modified 5mo ago
Copy link
Contents
Summary
Support Segment Merge and Roll-up
UI Improvement
SQL Improvements
Performance Improvements
Other Notable New Features and Changes