Design Documents

Links to all the design docs

This page contains links to all the design documents

2022

Name	Authors	Date
Peer downloading for offline segments	Xuanyi Li	Nov 2022
Segment format without forward index	Sonam Mandal	Oct 2022
Runtime query killing	Jia Guo	Oct 2022
Forward Index Handler in Segment Reload	Vivek Iyer	Sep 2022
Support multiple data directories for Pinot server	Xiaobing Li	Aug 2022
Reload status API	Saurabh Dubey	Aug 2022
Deduplication during real-time ingestion	Saurabh Dubey	June 2022
Adaptive Server Selection	Vivek Iyer	June 2022
MV column compression evaluation	Sonam Mandal	June 2022
Minimize Data Movement for Instance Assignment	Jialiang Li	June 2022
Server Failure Detector	Jackie Jiang	Apr 2022
Generalized Pre-Aggregation	Evan Noon	Mar 2022
Multi-stage Query Engine	Rong Rong	Feb 2022
Pause/Resume Stream	Sajjad Moradi, Subbu Subramaniam	Feb 2022
Fault Domain Awareness	Jia Guo, Sidd	Feb 2022
Pinot Flink Connector	Yupeng Fu	Jan 2022
Pluggable Indexes	Richard Startin	Jan 2022
Petabyte-Scale Log Storage and Search in Pinot with CLP	Ting Chen	Dec 2022

2021

Name	Authors	Date
Improve raw variable length forward index format	Richard Startin	Nov 2021
Consistent Push and Rollback	Seunghyun Lee, Jialiang Li	Nov 2021
Aggregation Filter	Atri Sharma	Oct 2021
Range encoded bit-sliced indexes	Richard Startin	Sep 2021
Time Series Aggregate Functions	Lakshmanan Velusamy, Weixiang Sun	Sep 2021
Native Text Indices	Atri Sharma	Sep 2021
Geospatial Support	Yupeng Fu	May 2021
Segment Writer API	Neha Pawar	Feb 2021
Partial Upsert	Qiaochu Liu	Feb 2021

2020

Name	Authors	Date
Kinesis Integration in Pinot	Neha Pawar	Dec 2020
JSON Indexing	Kishore Gopalakrishna	Oct 2020
Lookup Join	Dharak Kharod	Oct 2020
Strict Replica-Group Routing	Jackie Jiang	Oct 2020
Cluster Manager UI	Neha Pawar	Sep 2020
Pinot Upsert Revisited	Yupeng Fu	Sep 2020
Use IdSet for Id Filtering	Jackie Jiang	Sep 2020
Pinot managed offline flows	Neha	Aug 2020
Server Health Checker	Chinmay	Aug 2020
HAVING and Post-Aggregation Support	Jackie Jiang	July 2020
Segment Merge and Rollup (Updated)	Seunghyun	June 2020
Filtering during ingestion	Neha	June 2020
Segment Preprocessing Hadoop Job	Jialiang Li	May 2020
Flattening during ingestion	Neha	May 2020
Compatibility Regression Testing	Subbu	May 2020
Refactor pinot-core and pinot-common	Kishore	May 2020
Range Indexing in Pinot	Kishore	Apr 2020
Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream	Neha	Apr 2020
Geospatial support in Pinot	Yupeng Fu	Apr 2020
Column transformation during ingestion	Neha	Mar 2020
Tiered Storage	Neha	Mar 2020
Synthetic Data Generator for Pinot	Alex Pucher	Mar 2020

2019 and earlier

Name	Authors	Date
Text Search	Siddharth Teotia	Nov 2019
Pinot SQL migration Plan	Xiang Fu	Nov 2019
Segment Admin Rest APIs	Jackie Jiang	Nov 2019
Order By in aggregations	Neha	Oct 2019
Pinot Benchmark as a Service	Jialiang Li	Sep 2019
Controller Separation between Helix and Pinot	Jialiang Li	Jun 2019
Upsert	James	Jun 2019
Pinot Freshness Metric	Sunitha Beeram	May 2019
By passing deep-store requirement for Real-time segment completion	Ting, Chinmay	May 2019
Project Tuna: Automatic Inverted Index Recommendation	Jia Guo	May 2019
Segment Completion Enhancement for deep storage support	Subbu	Feb 2019
Segment Merge and Rollup	Seunghyun Lee	2018
Pinot-Minion Service	Jackie Jiang	2017
Consuming and indexing rows in real time	Subbu	2017
Partition Aware Query Routing	Subbu	2017
Query Processing	Subbu	2016
Expressions and UDFs	Subbu	2016