Design Documents

Links to all the design docs

This page contains links to all the design documents
2022
Name
Authors
Date
​Peer downloading for offline segments​
Xuanyi Li
Nov 2022
​Segment format without forward index​
Sonam Mandal
Oct 2022
​Runtime query killing​
Jia Guo
Oct 2022
​Forward Index Handler in Segment Reload​
Vivek Iyer
Sep 2022
​Support multiple data directories for Pinot server​
Xiaobing Li
Aug 2022
​Reload status API​
Saurabh Dubey
Aug 2022
​Deduplication during real-time ingestion​
Saurabh Dubey
June 2022
​Adaptive Server Selection​
Vivek Iyer
June 2022
​MV column compression evaluation​
Sonam Mandal
June 2022
​Minimize Data Movement for Instance Assignment​
Jialiang Li
June 2022
​Server Failure Detector​
Jackie Jiang
Apr 2022
​Generalized Pre-Aggregation​
Evan Noon
Mar 2022
​Multi-stage Query Engine​
Rong Rong
Feb 2022
​Pause/Resume Stream​
Sajjad Moradi, Subbu Subramaniam
Feb 2022
​Fault Domain Awareness​
Jia Guo, Sidd
Feb 2022
​Pinot Flink Connector​
Yupeng Fu
Jan 2022
​Pluggable Indexes​
Richard Startin
Jan 2022
​Petabyte-Scale Log Storage and Search in Pinot with CLP​
Ting Chen
Dec 2022
2021
Name
Authors
Date
​Improve raw variable length forward index format​
Richard Startin
Nov 2021
​Consistent Push and Rollback​
Seunghyun Lee, Jialiang Li
Nov 2021
​Aggregation Filter​
Atri Sharma
Oct 2021
​Range encoded bit-sliced indexes​
Richard Startin
Sep 2021
​Time Series Aggregate Functions​
Lakshmanan Velusamy, Weixiang Sun
Sep 2021
​Native Text Indices​
Atri Sharma
Sep 2021
​Geospatial Support​
Yupeng Fu
May 2021
​Segment Writer API​
Neha Pawar
Feb 2021
​Partial Upsert​
Qiaochu Liu
Feb 2021
2020
Name
Authors
Date
​Kinesis Integration in Pinot​
Neha Pawar
Dec 2020
​JSON Indexing​
Kishore Gopalakrishna
Oct 2020
​Lookup Join​
Dharak Kharod
Oct 2020
​Strict Replica-Group Routing​
Jackie Jiang
Oct 2020
​Cluster Manager UI​
Neha Pawar
Sep 2020
​Pinot Upsert Revisited​
Yupeng Fu
Sep 2020
​Use IdSet for Id Filtering​
Jackie Jiang
Sep 2020
​Pinot managed offline flows​
Neha
Aug 2020
​Server Health Checker​
Chinmay
Aug 2020
​HAVING and Post-Aggregation Support​
Jackie Jiang
July 2020
​Segment Merge and Rollup (Updated)​
Seunghyun
June 2020
​Filtering during ingestion​
Neha
June 2020
​Segment Preprocessing Hadoop Job​
Jialiang Li
May 2020
​Flattening during ingestion​
Neha
May 2020
​Compatibility Regression Testing​
Subbu
May 2020
​Refactor pinot-core and pinot-common​
Kishore
May 2020
​Range Indexing in Pinot​
Kishore
Apr 2020
​Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream​
Neha
Apr 2020
​Geospatial support in Pinot​
Yupeng Fu
Apr 2020
​Column transformation during ingestion​
Neha
Mar 2020
​Tiered Storage​
Neha
Mar 2020
​Synthetic Data Generator for Pinot​
Alex Pucher
Mar 2020
2019 and earlier
Name
Authors
Date
​Text Search​
Siddharth Teotia
Nov 2019
​Pinot SQL migration Plan​
Xiang Fu
Nov 2019
​Segment Admin Rest APIs​
Jackie Jiang
Nov 2019
​Order By in aggregations​
Neha
Oct 2019
​Pinot Benchmark as a Service​
Jialiang Li
Sep 2019
​Controller Separation between Helix and Pinot​
Jialiang Li
Jun 2019
​Upsert​
James
Jun 2019
​Pinot Freshness Metric​
Sunitha Beeram
May 2019
​By passing deep-store requirement for Real-time segment completion​
Ting, Chinmay
May 2019
​Project Tuna: Automatic Inverted Index Recommendation​
Jia Guo
May 2019
​Segment Completion Enhancement for deep storage support​
Subbu
Feb 2019
​Segment Merge and Rollup​
Seunghyun Lee
2018
​Pinot-Minion Service​
Jackie Jiang
2017
​Consuming and indexing rows in real time​
Subbu
2017
​Partition Aware Query Routing​
Subbu
2017
​Query Processing​
Subbu
2016
​Expressions and UDFs​
Subbu
2016

Name	Authors	Date
Peer downloading for offline segments	Xuanyi Li	Nov 2022
Segment format without forward index	Sonam Mandal	Oct 2022
Runtime query killing	Jia Guo	Oct 2022
Forward Index Handler in Segment Reload	Vivek Iyer	Sep 2022
Support multiple data directories for Pinot server	Xiaobing Li	Aug 2022
Reload status API	Saurabh Dubey	Aug 2022
Deduplication during real-time ingestion	Saurabh Dubey	June 2022
Adaptive Server Selection	Vivek Iyer	June 2022
MV column compression evaluation	Sonam Mandal	June 2022
Minimize Data Movement for Instance Assignment	Jialiang Li	June 2022
Server Failure Detector	Jackie Jiang	Apr 2022
Generalized Pre-Aggregation	Evan Noon	Mar 2022
Multi-stage Query Engine	Rong Rong	Feb 2022
Pause/Resume Stream	Sajjad Moradi, Subbu Subramaniam	Feb 2022
Fault Domain Awareness	Jia Guo, Sidd	Feb 2022
Pinot Flink Connector	Yupeng Fu	Jan 2022
Pluggable Indexes	Richard Startin	Jan 2022
Petabyte-Scale Log Storage and Search in Pinot with CLP	Ting Chen	Dec 2022

Name	Authors	Date
Improve raw variable length forward index format	Richard Startin	Nov 2021
Consistent Push and Rollback	Seunghyun Lee, Jialiang Li	Nov 2021
Aggregation Filter	Atri Sharma	Oct 2021
Range encoded bit-sliced indexes	Richard Startin	Sep 2021
Time Series Aggregate Functions	Lakshmanan Velusamy, Weixiang Sun	Sep 2021
Native Text Indices	Atri Sharma	Sep 2021
Geospatial Support	Yupeng Fu	May 2021
Segment Writer API	Neha Pawar	Feb 2021
Partial Upsert	Qiaochu Liu	Feb 2021

Name	Authors	Date
Kinesis Integration in Pinot	Neha Pawar	Dec 2020
JSON Indexing	Kishore Gopalakrishna	Oct 2020
Lookup Join	Dharak Kharod	Oct 2020
Strict Replica-Group Routing	Jackie Jiang	Oct 2020
Cluster Manager UI	Neha Pawar	Sep 2020
Pinot Upsert Revisited	Yupeng Fu	Sep 2020
Use IdSet for Id Filtering	Jackie Jiang	Sep 2020
Pinot managed offline flows	Neha	Aug 2020
Server Health Checker	Chinmay	Aug 2020
HAVING and Post-Aggregation Support	Jackie Jiang	July 2020
Segment Merge and Rollup (Updated)	Seunghyun	June 2020
Filtering during ingestion	Neha	June 2020
Segment Preprocessing Hadoop Job	Jialiang Li	May 2020
Flattening during ingestion	Neha	May 2020
Compatibility Regression Testing	Subbu	May 2020
Refactor pinot-core and pinot-common	Kishore	May 2020
Range Indexing in Pinot	Kishore	Apr 2020
Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream	Neha	Apr 2020
Geospatial support in Pinot	Yupeng Fu	Apr 2020
Column transformation during ingestion	Neha	Mar 2020
Tiered Storage	Neha	Mar 2020
Synthetic Data Generator for Pinot	Alex Pucher	Mar 2020

Name	Authors	Date
Text Search	Siddharth Teotia	Nov 2019
Pinot SQL migration Plan	Xiang Fu	Nov 2019
Segment Admin Rest APIs	Jackie Jiang	Nov 2019
Order By in aggregations	Neha	Oct 2019
Pinot Benchmark as a Service	Jialiang Li	Sep 2019
Controller Separation between Helix and Pinot	Jialiang Li	Jun 2019
Upsert	James	Jun 2019
Pinot Freshness Metric	Sunitha Beeram	May 2019
By passing deep-store requirement for Real-time segment completion	Ting, Chinmay	May 2019
Project Tuna: Automatic Inverted Index Recommendation	Jia Guo	May 2019
Segment Completion Enhancement for deep storage support	Subbu	Feb 2019
Segment Merge and Rollup	Seunghyun Lee	2018
Pinot-Minion Service	Jackie Jiang	2017
Consuming and indexing rows in real time	Subbu	2017
Partition Aware Query Routing	Subbu	2017
Query Processing	Subbu	2016
Expressions and UDFs	Subbu	2016

Stream Ingestion Plugin

Segment Writer API

Last modified 2mo ago