Design Documents

Index of design documents and how to propose new ones

Design documents capture the motivation, technical approach, and trade-offs behind significant changes to Apache Pinot. They serve as a permanent record so that contributors and operators can understand why a feature works the way it does.

Purpose

• Alignment -- A design doc lets the community review the approach before code is written, reducing rework.

• Onboarding -- New contributors can read design docs to understand the reasoning behind major subsystems.

• Auditability -- Design docs provide a historical record of architectural decisions.

When to write a design document

Write a design doc when your change involves any of the following:

• A new major feature or subsystem (for example, a new index type, a new query engine phase, or a new ingestion path)

• A significant change to an existing subsystem that alters its behavior or data format

• A cross-component change that affects multiple Pinot services (controller, broker, server, minion)

• A public API change that affects backward compatibility

Bug fixes, small enhancements, and documentation changes typically do not need a design doc.

How to propose a design document

1. Open a GitHub issue describing the problem and proposed solution at a high level. Label it with design-doc or PEP (Pinot Enhancement Proposal).

2. Write the design doc in a Google Doc (or equivalent) and share it with the Pinot dev mailing list ([e-mail]). Include:

   • Problem statement and motivation

   • Proposed design with diagrams where helpful

   • API or configuration changes

   • Alternatives considered

   • Rollout and backward-compatibility plan

3. Collect feedback from reviewers on the doc and the GitHub issue.

4. Update this page by adding a row to the index table below once the design is accepted.

5. Implement the feature following the Contribution Guidelines.

Design document index

In-repo design docs

Name	Description
Segment Writer API	API for programmatically building Pinot segments without a full batch ingestion job

2025

Name	Authors	Date
NGram Filtering Index	Ting Chen, Qiaochu Liu	Aug 2025

2024

Name	Authors	Date
Reuse common expressions in a query	Gonzalo Ortiz	Sep 2024

2022

Name	Authors	Date
Peer downloading for offline segments	Xuanyi Li	Nov 2022
Segment format without forward index	Sonam Mandal	Oct 2022
Runtime query killing	Jia Guo	Oct 2022
Forward Index Handler in Segment Reload	Vivek Iyer	Sep 2022
Support multiple data directories for Pinot server	Xiaobing Li	Aug 2022
Reload status API	Saurabh Dubey	Aug 2022
Deduplication during real-time ingestion	Saurabh Dubey	June 2022
Adaptive Server Selection	Vivek Iyer	June 2022
MV column compression evaluation	Sonam Mandal	June 2022
Minimize Data Movement for Instance Assignment	Jialiang Li	June 2022
Server Failure Detector	Jackie Jiang	Apr 2022
Generalized Pre-Aggregation	Evan Noon	Mar 2022
Multi-stage Query Engine	Rong Rong	Feb 2022
Pause/Resume Stream	Sajjad Moradi, Subbu Subramaniam	Feb 2022
Fault Domain Awareness	Jia Guo, Sidd	Feb 2022
Pinot Flink Connector	Yupeng Fu	Jan 2022
Pluggable Indexes	Richard Startin	Jan 2022
Petabyte-Scale Log Storage and Search in Pinot with CLP	Ting Chen	Dec 2022

2021

Name	Authors	Date
Improve raw variable length forward index format	Richard Startin	Nov 2021
Consistent Push and Rollback	Seunghyun Lee, Jialiang Li	Nov 2021
Aggregation Filter	Atri Sharma	Oct 2021
Range encoded bit-sliced indexes	Richard Startin	Sep 2021
Time Series Aggregate Functions	Lakshmanan Velusamy, Weixiang Sun	Sep 2021
Native Text Indices	Atri Sharma	Sep 2021
Geospatial Support	Yupeng Fu	May 2021
Segment Writer API	Neha Pawar	Feb 2021
Partial Upsert	Qiaochu Liu	Feb 2021

2020

Name	Authors	Date
Kinesis Integration in Pinot	Neha Pawar	Dec 2020
JSON Indexing	Kishore Gopalakrishna	Oct 2020
Lookup Join	Dharak Kharod	Oct 2020
Strict Replica-Group Routing	Jackie Jiang	Oct 2020
Cluster Manager UI	Neha Pawar	Sep 2020
Pinot Upsert Revisited	Yupeng Fu	Sep 2020
Use IdSet for Id Filtering	Jackie Jiang	Sep 2020
Pinot managed offline flows	Neha	Aug 2020
Server Health Checker	Chinmay	Aug 2020
HAVING and Post-Aggregation Support	Jackie Jiang	July 2020
Segment Merge and Rollup (Updated)	Seunghyun	June 2020
Filtering during ingestion	Neha	June 2020
Segment Preprocessing Hadoop Job	Jialiang Li	May 2020
Flattening during ingestion	Neha	May 2020
Compatibility Regression Testing	Subbu	May 2020
Refactor pinot-core and pinot-common	Kishore	May 2020
Range Indexing in Pinot	Kishore	Apr 2020
Deprecate TimeFieldSpec, make DateTimeFieldSpec mainstream	Neha	Apr 2020
Geospatial support in Pinot	Yupeng Fu	Apr 2020
Column transformation during ingestion	Neha	Mar 2020
Tiered Storage	Neha	Mar 2020
Synthetic Data Generator for Pinot	Alex Pucher	Mar 2020

2019 and earlier

Name	Authors	Date
Text Search	Siddharth Teotia	Nov 2019
Pinot SQL migration Plan	Xiang Fu	Nov 2019
Segment Admin Rest APIs	Jackie Jiang	Nov 2019
Order By in aggregations	Neha	Oct 2019
Pinot Benchmark as a Service	Jialiang Li	Sep 2019
Controller Separation between Helix and Pinot	Jialiang Li	Jun 2019
Upsert	James	Jun 2019
Pinot Freshness Metric	Sunitha Beeram	May 2019
By passing deep-store requirement for Real-time segment completion	Ting, Chinmay	May 2019
Project Tuna: Automatic Inverted Index Recommendation	Jia Guo	May 2019
Segment Completion Enhancement for deep storage support	Subbu	Feb 2019
Segment Merge and Rollup	Seunghyun Lee	2018
Pinot-Minion Service	Jackie Jiang	2017
Consuming and indexing rows in real time	Subbu	2017
Partition Aware Query Routing	Subbu	2017
Query Processing	Subbu	2016
Expressions and UDFs	Subbu	2016