# Design Documents This page contains links to all the design documents ### 2022 | Name | Authors | Date | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | --------- | | [Peer downloading for offline segments](https://docs.google.com/document/d/1HtU8NsYz84NiAKGh16Yv8WKxyhdXHx-KNsINcWZ7MEI/edit?usp=sharing) | Xuanyi Li | Nov 2022 | | [Segment format without forward index](https://docs.google.com/document/d/1MNLLhYCg5e-UFBQ6wTBODd41sDsbjevwRfwoGuNowWw/edit#heading=h.jwlk6va28akf) | Sonam Mandal | Oct 2022 | | [Runtime query killing](https://docs.google.com/document/d/1Z9DYAfKznHQI9Wn8BjTWZYTcNRVGiPP0B8aEP3w_1jQ/edit#heading=h.o3jia8j0k10u) | Jia Guo | Oct 2022 | | [Forward Index Handler in Segment Reload](https://docs.google.com/document/d/1Gai0DHBnyR4joG_8AcoR-27_exEBTpVfTFU04HyPdd8/edit#heading=h.liwwtls82n1z) | Vivek Iyer | Sep 2022 | | [Support multiple data directories for Pinot server](https://docs.google.com/document/d/1B_8k2XGMrxfFAo_g-MEjEPGj84Xy8YW4-yUiZ2BQkuE/edit#heading=h.2ic29gs71g43) | Xiaobing Li | Aug 2022 | | [Reload status API](https://docs.google.com/document/d/1Eqn2FDDIhCr8G2JFlifs5FjT0LsVPfpPTpdJIJvorwI/edit?usp=sharing) | Saurabh Dubey | Aug 2022 | | [Deduplication during real-time ingestion](https://docs.google.com/document/d/17sOSRQ1slff30z7jDc0ec5qKwv0xSfPkDjpMOY07POQ/edit?usp=sharing) | Saurabh Dubey | June 2022 | | [Adaptive Server Selection](https://docs.google.com/document/d/1w8YVpKIj0S62NvwDpf1HgruwxJYJ6ODuKQLjGXupH8w/edit#heading=h.u87pirlgxhmf) | Vivek Iyer | June 2022 | | [MV column compression evaluation](https://docs.google.com/document/d/1BWtNKvxL1Uaydni_BJCgWN8i9_WeSdgL3Ksh4IpY_K0/edit) | Sonam Mandal | June 2022 | | [Minimize Data Movement for Instance Assignment](https://docs.google.com/document/d/1_Fn-yNjt9Ih0SQiIqIhCEvE9BvYugvAhB4tgWH-VoWE/edit) | Jialiang Li | June 2022 | | [Server Failure Detector](https://docs.google.com/document/d/1X32OMT6lC4pCveQVzK6OvRlaW0kE9HZ2vn_EHzesM1w/edit?usp=sharing) | Jackie Jiang | Apr 2022 | | [Generalized Pre-Aggregation](https://docs.google.com/document/d/17nMXwmDa7-eopzSaQ4XhbfnbF7myQMzDshanSPXLK0s/edit?usp=sharing) | Evan Noon | Mar 2022 | | [Multi-stage Query Engine](https://docs.google.com/document/d/10-vL_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit?usp=sharing) | Rong Rong | Feb 2022 | | [Pause/Resume Stream](https://docs.google.com/document/d/1uuXuif0SfMTnH3ykfuzXXNjh-LyszjilodD_mdlilZg/edit) | Sajjad Moradi, Subbu Subramaniam | Feb 2022 | | [Fault Domain Awareness](https://docs.google.com/document/d/1KmJ1DsYXVdzrojj_JYBHRJ2gRMQ5y-o63YqPs7ei7nI/edit) | Jia Guo, Sidd | Feb 2022 | | [Pinot Flink Connector](https://docs.google.com/document/d/1GVoFHOHSDPs1MEDKEmKguKwWMqM1lwQKj2e64RAKDf8/edit#heading=h.uvocz0dwkepo) | Yupeng Fu | Jan 2022 | | [Pluggable Indexes](https://docs.google.com/document/d/1Slu7klgQn_3RcEJWUuHq9AF0v5ZDA9M2Y0dy8iqrtLQ/edit) | Richard Startin | Jan 2022 | | [Petabyte-Scale Log Storage and Search in Pinot with CLP](https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU/edit#heading=h.j0al1jpfd8eb) | Ting Chen | Dec 2022 | ### 2021 | Name | Authors | Date | | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | -------- | | [Improve raw variable length forward index format](https://docs.google.com/document/d/1Y7MyQGmDD2fI7brOOFQtToxd8ML837qRuc3IlNYFvCw/edit) | Richard Startin | Nov 2021 | | [Consistent Push and Rollback](https://docs.google.com/document/d/1PUy4wSUPFyEWEW3a88Mipdug3cPj4EpV__lx-BVUTYk/edit?usp=sharing) | Seunghyun Lee, Jialiang Li | Nov 2021 | | [Aggregation Filter](https://docs.google.com/document/d/1qJzG1CmyVZpLN6rdI8V0zUWEaIjQ86hDL9XwrcPKcTs/edit?usp=sharing) | Atri Sharma | Oct 2021 | | [Range encoded bit-sliced indexes](https://docs.google.com/document/d/1se2OgqXJiD7r7S7U6SUmTIAApO66QIrAYosxvXHEXlw/edit) | Richard Startin | Sep 2021 | | [Time Series Aggregate Functions](https://docs.google.com/document/d/1D5tyd-gFIe5QjL8XYLQmL5_traw-jtKkzeF96G3naes/edit?usp=sharing) | Lakshmanan Velusamy, Weixiang Sun | Sep 2021 | | [Native Text Indices](https://docs.google.com/document/d/1PMhoRy6WF46C4d4mw0LVe9b8Vjqes6vsXZkmxXzMYzw/edit?usp=sharing) | Atri Sharma | Sep 2021 | | [Geospatial Support](https://docs.google.com/document/d/1Mkm5RHS_tof-vIUt5-UNeOgRYSBAN6M_pN-hedV6Q0g/edit?usp=sharing) | Yupeng Fu | May 2021 | | [Segment Writer API](https://docs.google.com/document/d/1f_JlegCkH_Zysm80maLnv7iqgWtD9uPiBLkeLmMUoNg/edit) | Neha Pawar | Feb 2021 | | [Partial Upsert](https://docs.google.com/document/d/1qrTD7x23FlPrAUVIFbWs6GSBtTsztWhWgis-xr1lGMs/edit?usp=sharing) | Qiaochu Liu | Feb 2021 | ### 2020 | Name | Authors | Date | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | --------- | | [Kinesis Integration in Pinot](https://docs.google.com/document/d/1hFbzumQAGALAi8XZMOsVlwVHN-s2t37MA5QUfduL4Yk/edit?usp=sharing) | Neha Pawar | Dec 2020 | | [JSON Indexing](https://docs.google.com/document/d/1ZBkZUjlkTH7pA1dN_hLAUXhuP1pIo4WmtM5YXktUifg/edit?usp=sharing) | Kishore Gopalakrishna | Oct 2020 | | [Lookup Join](https://docs.google.com/document/d/1InWmxbRqwcqIakzvoEWHLxtX4XR9H5L01256EbAUHV8/edit?usp=sharing) | Dharak Kharod | Oct 2020 | | [Strict Replica-Group Routing](https://docs.google.com/document/d/1B5SghG0x5JHfZrKMBjiv_m3Dd969hfyWgc1joKZpJIU/edit?usp=sharing) | Jackie Jiang | Oct 2020 | | [Cluster Manager UI](https://docs.google.com/document/d/1E6OWyt-NvOpbBsOH1qwi5v8DcNSgT4PTLJ21TKYIsoI/edit#heading=h.8xhj1hrxrxl) | Neha Pawar | Sep 2020 | | [Pinot Upsert Revisited](https://docs.google.com/document/d/1qljEMndPMxbbKtjlVn9mn2toz7Qrk0TGQsHLfI--7h8/edit#heading=h.lsfmyoyyxtgt) | Yupeng Fu | Sep 2020 | | [Use IdSet for Id Filtering](https://docs.google.com/document/d/1s6DZ9eTPqH7vaKQlPjKiWb_OBC3hkkEGICIzcd5gozc/edit?usp=sharing) | Jackie Jiang | Sep 2020 | | [Pinot managed offline flows](https://docs.google.com/document/d/1-e_9aHQB4HXS38ONtofdxNvMsGmAoYfSnc2LP88MbIc/edit#heading=h.60ws8it8iwvp) | Neha | Aug 2020 | | [Server Health Checker](https://docs.google.com/document/d/1PP_RaDuS7KGeF9RnAcRFJRCA8aCVxxVLTJn-c3hg9qQ/edit) | Chinmay | Aug 2020 | | [HAVING and Post-Aggregation Support](https://docs.google.com/document/d/1Dg1KXpxIdl75Tsg2YFCYVeE8sMAIj64ZWoxDcj1cHwo/edit?usp=sharing) | Jackie Jiang | July 2020 | | [Segment Merge and Rollup (Updated)](https://docs.google.com/document/d/1-AKCfXNXdoNjFIvJ87wjWwFM_38gS0NCwFrIYjYsqp8/edit) | Seunghyun | June 2020 | | [Filtering during ingestion](https://docs.google.com/document/d/1Cahnas3nh0XErETH0KHLaecN6xCnRVYWNKO3rDn7qcI/edit) | Neha | June 2020 | | [Segment Preprocessing Hadoop Job](https://docs.google.com/document/d/1BnjjVj3OLuo-vmOt0WjqEFbUC9AZgCDuDxCtLEFPM34/edit?usp=sharing) | Jialiang Li | May 2020 | | [Flattening during ingestion](https://docs.google.com/document/d/1IYCsYCgGn6YMWTDG4-i61Hxbtnac2dCuhvDKUZIxDYg/edit?usp=sharing) | Neha | May 2020 | | [Compatibility Regression Testing](https://docs.google.com/document/d/1yNlvnLKDNUuyRWOKYYF01FWW9weYMGoaLRtU-CueciM/edit#heading=h.sbzlx23tnq14) | Subbu | May 2020 | | [Refactor pinot-core and pinot-common](https://docs.google.com/document/d/1urROfQZuTE8JJmW3IMCeB2i3FYoEyG1TCyPsxvSaNuw/edit?usp=sharing) | Kishore | May 2020 | | [Range Indexing in Pinot](https://docs.google.com/document/d/1eisu7L-ERLs1OZCASOz3qSpzZfoipplKrYgmBXaFobw/edit?usp=sharing) | Kishore | Apr 2020 | | [Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream](https://docs.google.com/document/d/1SU1jCjfsIDSA960fD5YWQbD72p8UdGF0c7CroFNt9Ho/edit#heading=h.qeqkd3x33xzp) | Neha | Apr 2020 | | [Geospatial support in Pinot](https://docs.google.com/document/d/1Mkm5RHS_tof-vIUt5-UNeOgRYSBAN6M_pN-hedV6Q0g/edit?ts=5ea0b8d4#heading=h.i45os595j1sp) | Yupeng Fu | Apr 2020 | | [Column transformation during ingestion](https://docs.google.com/document/d/13BywJncHrLAFLm-qy4kfKaPxXfAg9XE5v3_fk9sGVSo/edit?usp=sharing) | Neha | Mar 2020 | | [Tiered Storage](https://docs.google.com/document/d/1Z4FLg3ezHpqvc6zhy0jR6Wi2OL8wLO_lRC6aLkskFgs/edit?usp=sharing) | Neha | Mar 2020 | | [Synthetic Data Generator for Pinot](https://cwiki.apache.org/confluence/display/PINOT/Synthetic+Data+Generator+for+Pinot) | Alex Pucher | Mar 2020 | ### 2019 and earlier | Name | Authors | Date | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -------- | | [Text Search](https://docs.google.com/document/d/19uLti7wwl7nPlDuy6cUVnLOll2C8u3YtUITbNj0TT5o/edit) | Siddharth Teotia | Nov 2019 | | [Pinot SQL migration Plan](https://docs.google.com/document/d/1uNIq0cybUtVtdtJ38-4ewFNEQorbg-2KYr-CMSj6H_8/edit#heading=h.ejrg0ci2rzol) | Xiang Fu | Nov 2019 | | [Segment Admin Rest APIs](https://cwiki.apache.org/confluence/display/PINOT/Segment+Admin+Rest+APIs) | Jackie Jiang | Nov 2019 | | [Order By in aggregations](https://cwiki.apache.org/confluence/display/PINOT/Order+By) | Neha | Oct 2019 | | [Pinot Benchmark as a Service](https://cwiki.apache.org/confluence/display/PINOT/Pinot+Benchmark+as+a+Service) | Jialiang Li | Sep 2019 | | [Controller Separation between Helix and Pinot](https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot) | Jialiang Li | Jun 2019 | | [Upsert](https://docs.google.com/document/d/1SFFir7ByxCff-aVYxQeTHpNhPXeP5q7P4g_6O2iNGgU/edit?usp=sharing) | James | Jun 2019 | | [Pinot Freshness Metric](https://cwiki.apache.org/confluence/display/PINOT/Pinot+Freshness+Metric) | Sunitha Beeram | May 2019 | | [By passing deep-store requirement for Real-time segment completion](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion) | Ting, Chinmay | May 2019 | | [Project Tuna: Automatic Inverted Index Recommendation](https://cwiki.apache.org/confluence/display/PINOT/Automated+Inverted+Index+Recommendation+for+Pinot) | Jia Guo | May 2019 | | [Segment Completion Enhancement for deep storage support](https://cwiki.apache.org/confluence/display/PINOT/Segment+Completion+Protocol+enhancements+for+Deep+Store+support) | Subbu | Feb 2019 | | [Segment Merge and Rollup](https://cwiki.apache.org/confluence/display/PINOT/Segment+Merge+and+Rollup) | Seunghyun Lee | 2018 | | [Pinot-Minion Service](https://docs.google.com/document/d/1kbK88fCexmEsDcFINebqLvZWtKg8CVQN4kmsLm0s9f8/edit?usp=sharing) | Jackie Jiang | 2017 | | [Consuming and indexing rows in real time](https://cwiki.apache.org/confluence/display/PINOT/Consuming+and+Indexing+rows+in+Realtime) | Subbu | 2017 | | [Partition Aware Query Routing](https://cwiki.apache.org/confluence/display/PINOT/Partition+Aware+Query+Routing) | Subbu | 2017 | | [Query Processing](https://cwiki.apache.org/confluence/display/PINOT/Query+Processing) | Subbu | 2016 | | [Expressions and UDFs](https://cwiki.apache.org/confluence/display/PINOT/Expressions+and+UDFs) | Subbu | 2016 | --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.pinot.apache.org/release-1.2.0/for-developers/design-documents.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.