# Design Documents This page contains links to all the design documents ### 2022 | Name | Authors | Date | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | --------- | | [Segment format without forward index](https://docs.google.com/document/d/1MNLLhYCg5e-UFBQ6wTBODd41sDsbjevwRfwoGuNowWw/edit#heading=h.jwlk6va28akf) | Sonam Mandal | Oct 2022 | | [Runtime query killing](https://docs.google.com/document/d/1Z9DYAfKznHQI9Wn8BjTWZYTcNRVGiPP0B8aEP3w_1jQ/edit#heading=h.o3jia8j0k10u) | Jia Guo | Oct 2022 | | [Forward Index Handler in Segment Reload](https://docs.google.com/document/d/1Gai0DHBnyR4joG_8AcoR-27_exEBTpVfTFU04HyPdd8/edit#heading=h.liwwtls82n1z) | Vivek Iyer | Sep 2022 | | [Support multiple data directories for Pinot server](https://docs.google.com/document/d/1B_8k2XGMrxfFAo_g-MEjEPGj84Xy8YW4-yUiZ2BQkuE/edit#heading=h.2ic29gs71g43) | Xiaobing Li | Aug 2022 | | [Reload status API](https://docs.google.com/document/d/1Eqn2FDDIhCr8G2JFlifs5FjT0LsVPfpPTpdJIJvorwI/edit?usp=sharing) | Saurabh Dubey | Aug 2022 | | [Deduplication during realtime ingestion](https://docs.google.com/document/d/17sOSRQ1slff30z7jDc0ec5qKwv0xSfPkDjpMOY07POQ/edit?usp=sharing) | Saurabh Dubey | June 2022 | | [Adaptive Server Selection](https://docs.google.com/document/d/1w8YVpKIj0S62NvwDpf1HgruwxJYJ6ODuKQLjGXupH8w/edit#heading=h.u87pirlgxhmf) | Vivek Iyer | June 2022 | | [MV column compression evaluation](https://docs.google.com/document/d/1BWtNKvxL1Uaydni_BJCgWN8i9_WeSdgL3Ksh4IpY_K0/edit) | Sonam Mandal | June 2022 | | [Minimize Data Movement for Instance Assignment](https://docs.google.com/document/d/1_Fn-yNjt9Ih0SQiIqIhCEvE9BvYugvAhB4tgWH-VoWE/edit) | Jialiang Li | June 2022 | | [Server Failure Detector](https://docs.google.com/document/d/1X32OMT6lC4pCveQVzK6OvRlaW0kE9HZ2vn_EHzesM1w/edit?usp=sharing) | Jackie Jiang | Apr 2022 | | [Generalized Pre-Aggregation](https://docs.google.com/document/d/17nMXwmDa7-eopzSaQ4XhbfnbF7myQMzDshanSPXLK0s/edit?usp=sharing) | Evan Noon | Mar 2022 | | [Multi-stage Query Engine](https://docs.google.com/document/d/10-vL_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit?usp=sharing) | Rong Rong | Feb 2022 | | [Pause/Resume Stream](https://docs.google.com/document/d/1uuXuif0SfMTnH3ykfuzXXNjh-LyszjilodD_mdlilZg/edit) | Sajjad Moradi, Subbu Subramaniam | Feb 2022 | | [Fault Domain Awareness](https://docs.google.com/document/d/1KmJ1DsYXVdzrojj_JYBHRJ2gRMQ5y-o63YqPs7ei7nI/edit) | Jia Guo, Sidd | Feb 2022 | | [Pinot Flink Connector](https://docs.google.com/document/d/1GVoFHOHSDPs1MEDKEmKguKwWMqM1lwQKj2e64RAKDf8/edit#heading=h.uvocz0dwkepo) | Yupeng Fu | Jan 2022 | | [Pluggable Indexes](https://docs.google.com/document/d/1Slu7klgQn_3RcEJWUuHq9AF0v5ZDA9M2Y0dy8iqrtLQ/edit) | Richard Startin | Jan 2022 | ### 2021 | Name | Authors | Date | | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | -------- | | [Improve raw variable length forward index format](https://docs.google.com/document/d/1Y7MyQGmDD2fI7brOOFQtToxd8ML837qRuc3IlNYFvCw/edit) | Richard Startin | Nov 2021 | | [Consistent Push and Rollback](https://docs.google.com/document/d/1PUy4wSUPFyEWEW3a88Mipdug3cPj4EpV__lx-BVUTYk/edit?usp=sharing) | Seunghyun Lee, Jialiang Li | Nov 2021 | | [Aggregation Filter](https://docs.google.com/document/d/1qJzG1CmyVZpLN6rdI8V0zUWEaIjQ86hDL9XwrcPKcTs/edit?usp=sharing) | Atri Sharma | Oct 2021 | | [Range encoded bit-sliced indexes](https://docs.google.com/document/d/1se2OgqXJiD7r7S7U6SUmTIAApO66QIrAYosxvXHEXlw/edit) | Richard Startin | Sep 2021 | | [Time Series Aggregate Functions](https://docs.google.com/document/d/1D5tyd-gFIe5QjL8XYLQmL5_traw-jtKkzeF96G3naes/edit?usp=sharing) | Lakshmanan Velusamy, Weixiang Sun | Sep 2021 | | [Native Text Indices](https://docs.google.com/document/d/1PMhoRy6WF46C4d4mw0LVe9b8Vjqes6vsXZkmxXzMYzw/edit?usp=sharing) | Atri Sharma | Sep 2021 | | [Geospatial Support](https://docs.google.com/document/d/1Mkm5RHS_tof-vIUt5-UNeOgRYSBAN6M_pN-hedV6Q0g/edit?usp=sharing) | Yupeng Fu | May 2021 | | [Segment Writer API](https://docs.google.com/document/d/1f_JlegCkH_Zysm80maLnv7iqgWtD9uPiBLkeLmMUoNg/edit) | Neha Pawar | Feb 2021 | | [Partial Upsert](https://docs.google.com/document/d/1qrTD7x23FlPrAUVIFbWs6GSBtTsztWhWgis-xr1lGMs/edit?usp=sharing) | Qiaochu Liu | Feb 2021 | ### 2020 | Name | Authors | Date | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | --------- | | [Kinesis Integration in Pinot](https://docs.google.com/document/d/1hFbzumQAGALAi8XZMOsVlwVHN-s2t37MA5QUfduL4Yk/edit?usp=sharing) | Neha Pawar | Dec 2020 | | [JSON Indexing](https://docs.google.com/document/d/1ZBkZUjlkTH7pA1dN_hLAUXhuP1pIo4WmtM5YXktUifg/edit?usp=sharing) | Kishore Gopalakrishna | Oct 2020 | | [Lookup Join](https://docs.google.com/document/d/1InWmxbRqwcqIakzvoEWHLxtX4XR9H5L01256EbAUHV8/edit?usp=sharing) | Dharak Kharod | Oct 2020 | | [Strict Replica-Group Routing](https://docs.google.com/document/d/1B5SghG0x5JHfZrKMBjiv_m3Dd969hfyWgc1joKZpJIU/edit?usp=sharing) | Jackie Jiang | Oct 2020 | | [Cluster Manager UI](https://docs.google.com/document/d/1E6OWyt-NvOpbBsOH1qwi5v8DcNSgT4PTLJ21TKYIsoI/edit#heading=h.8xhj1hrxrxl) | Neha Pawar | Sep 2020 | | [Pinot Upsert Revisited](https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit#heading=h.lsfmyoyyxtgt) | Yupeng Fu | Sep 2020 | | [Use IdSet for Id Filtering](https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit?usp=sharing) | Jackie Jiang | Sep 2020 | | [Pinot managed offline flows](https://docs.google.com/document/d/1-e_9aHQB4HXS38ONtofdxNvMsGmAoYfSnc2LP88MbIc/edit#heading=h.60ws8it8iwvp) | Neha | Aug 2020 | | [Server Health Checker](https://docs.google.com/document/d/1PP_RaDuS7KGeF9RnAcRFJRCA8aCVxxVLTJn-c3hg9qQ/edit) | Chinmay | Aug 2020 | | [HAVING and Post-Aggregation Support](https://docs.google.com/document/d/1Dg1KXpxIdl75Tsg2YFCYVeE8sMAIj64ZWoxDcj1cHwo/edit?usp=sharing) | Jackie Jiang | July 2020 | | [Segment Merge and Rollup (Updated)](https://docs.google.com/document/d/1-AKCfXNXdoNjFIvJ87wjWwFM_38gS0NCwFrIYjYsqp8/edit) | Seunghyun | June 2020 | | [Filtering during ingestion](https://docs.google.com/document/d/1Cahnas3nh0XErETH0KHLaecN6xCnRVYWNKO3rDn7qcI/edit) | Neha | June 2020 | | [Segment Preprocessing Hadoop Job](https://docs.google.com/document/d/1BnjjVj3OLuo-vmOt0WjqEFbUC9AZgCDuDxCtLEFPM34/edit?usp=sharing) | Jialiang Li | May 2020 | | [Flattening during ingestion](https://docs.google.com/document/d/1IYCsYCgGn6YMWTDG4-i61Hxbtnac2dCuhvDKUZIxDYg/edit?usp=sharing) | Neha | May 2020 | | [Compatibility Regression Testing](https://docs.google.com/document/d/1yNlvnLKDNUuyRWOKYYF01FWW9weYMGoaLRtU-CueciM/edit#heading=h.sbzlx23tnq14) | Subbu | May 2020 | | [Refactor pinot-core and pinot-common](https://docs.google.com/document/d/1urROfQZuTE8JJmW3IMCeB2i3FYoEyG1TCyPsxvSaNuw/edit?usp=sharing) | Kishore | May 2020 | | [Range Indexing in Pinot](https://docs.google.com/document/d/1eisu7L-ERLs1OZCASOz3qSpzZfoipplKrYgmBXaFobw/edit?usp=sharing) | Kishore | Apr 2020 | | [Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream](https://docs.google.com/document/d/1SU1jCjfsIDSA960fD5YWQbD72p8UdGF0c7CroFNt9Ho/edit#heading=h.qeqkd3x33xzp) | Neha | Apr 2020 | | [Geospatial support in Pinot](https://docs.google.com/document/d/1Mkm5RHS_tof-vIUt5-UNeOgRYSBAN6M_pN-hedV6Q0g/edit?ts=5ea0b8d4#heading=h.i45os595j1sp) | Yupeng Fu | Apr 2020 | | [Column transformation during ingestion](https://docs.google.com/document/d/13BywJncHrLAFLm-qy4kfKaPxXfAg9XE5v3_fk9sGVSo/edit?usp=sharing) | Neha | Mar 2020 | | [Tiered Storage](https://docs.google.com/document/d/1Z4FLg3ezHpqvc6zhy0jR6Wi2OL8wLO_lRC6aLkskFgs/edit?usp=sharing) | Neha | Mar 2020 | | [Synthetic Data Generator for Pinot](https://cwiki.apache.org/confluence/display/PINOT/Synthetic+Data+Generator+for+Pinot) | Alex Pucher | Mar 2020 | ### 2019 and earlier | Name | Authors | Date | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -------- | | [Text Search](https://docs.google.com/document/d/19uLti7wwl7nPlDuy6cUVnLOll2C8u3YtUITbNj0TT5o/edit) | Siddharth Teotia | Nov 2019 | | [Pinot SQL migration Plan](https://docs.google.com/document/d/1uNIq0cybUtVtdtJ38-4ewFNEQorbg-2KYr-CMSj6H_8/edit#heading=h.ejrg0ci2rzol) | Xiang Fu | Nov 2019 | | [Segment Admin Rest APIs](https://cwiki.apache.org/confluence/display/PINOT/Segment+Admin+Rest+APIs) | Jackie Jiang | Nov 2019 | | [Order By in aggregations](https://cwiki.apache.org/confluence/display/PINOT/Order+By) | Neha | Oct 2019 | | [Pinot Benchmark as a Service](https://cwiki.apache.org/confluence/display/PINOT/Pinot+Benchmark+as+a+Service) | Jialiang Li | Sep 2019 | | [Controller Separation between Helix and Pinot](https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot) | Jialiang Li | Jun 2019 | | [Upsert](https://docs.google.com/document/d/1SFFir7ByxCff-aVYxQeTHpNhPXeP5q7P4g_6O2iNGgU/edit?usp=sharing) | James | Jun 2019 | | [Pinot Freshness Metric](https://cwiki.apache.org/confluence/display/PINOT/Pinot+Freshness+Metric) | Sunitha Beeram | May 2019 | | [By passing deep-store requirement for Realtime segment completion](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion) | Ting, Chinmay | May 2019 | | [Project Tuna: Automatic Inverted Index Recommendation](https://cwiki.apache.org/confluence/display/PINOT/Automated+Inverted+Index+Recommendation+for+Pinot) | Jia Guo | May 2019 | | [Segment Completion Enhancement for deep storage support](https://cwiki.apache.org/confluence/display/PINOT/Segment+Completion+Protocol+enhancements+for+Deep+Store+support) | Subbu | Feb 2019 | | [Segment Merge and Rollup](https://cwiki.apache.org/confluence/display/PINOT/Segment+Merge+and+Rollup) | Seunghyun Lee | 2018 | | [Pinot-Minion Service](https://docs.google.com/document/d/1kbK88fCexmEsDcFINebqLvZWtKg8CVQN4kmsLm0s9f8/edit?usp=sharing) | Jackie Jiang | 2017 | | [Consuming and indexing rows in realtime](https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime) | Subbu | 2017 | | [Partition Aware Query Routing](https://cwiki.apache.org/confluence/display/PINOT/Partition+Aware+Query+Routing) | Subbu | 2017 | | [Query Processing](https://cwiki.apache.org/confluence/display/PINOT/Query+Processing) | Subbu | 2016 | | [Expressions and UDFs](https://cwiki.apache.org/confluence/display/PINOT/Expressions+and+UDFs) | Subbu | 2016 |