# V2 Multi-Stage Query Engine

## Overview

The new multi-stage query engine (a.k.a V2 query engine) is designed to support more complex SQL semantics such as `JOIN`, `OVER` window, `MATCH_RECOGNIZE` and eventually, make Pinot support closer to full ANSI SQL semantics. 

<figure><img src="https://630770001-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgSiSBdne4SKGKsqgA5sI%2Fuploads%2FGCufOSjTQZ5CGdQluqXO%2FMulti-Stage-Pinot-Query-Engine-v1.png?alt=media&#x26;token=685b0427-c0fd-4f07-a90b-6079e0e79cad" alt="Scatter-Gather Query Engine"><figcaption><p>Scatter-Gather Query Engine</p></figcaption></figure>

<figure><img src="https://630770001-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgSiSBdne4SKGKsqgA5sI%2Fuploads%2FMNLIkP9EvzYW8WXDSfh4%2FMulti-Stage-Query-Engine-2.png?alt=media&#x26;token=f1b4c33d-cc01-4720-b604-7584d7b88fd2" alt="Pinot Multi-Stage Query Engine"><figcaption><p>Multi-Stage Query Engine</p></figcaption></figure>

It also resolves the bottleneck effect for the broker reduce stage where only a single machine is dedicated to perform heavy lifting such as high cardinality `GROUP BY` result merging; `ORDER BY` sorting, etc.

## How to use the V2 query engine

To enable the V2 engine,&#x20;

1. please make sure to either&#x20;
   * [Building Apache Pinot](https://github.com/apache/pinot#building-pinot) using the latest master commit.
   * Download the latest Apache Pinot docker image using the [official guide](https://docs.pinot.apache.org/basics/getting-started/running-pinot-in-docker).
2. Please add the following configurations to your cluster config:
   * ```
     "pinot.multistage.engine.enabled": "true",
     "pinot.server.instance.currentDataTableVersion": "4",
     "pinot.query.server.port": "8421",
     "pinot.query.runner.port": "8442"
     ```
3. Start the cluster normally, you should see the following window in the controller query page:

   <figure><img src="https://630770001-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FgSiSBdne4SKGKsqgA5sI%2Fuploads%2FuzWf9gp4nlCbjhd7iLkr%2Fimage.png?alt=media&#x26;token=b37da9eb-c475-46c3-a050-30ad1dc12aad" alt=""><figcaption><p>Sample Query Screenshot</p></figcaption></figure>

## Design Details

The overall PEP design doc and discussion can be found in the following links

* [PEP discussion Github Issue](https://github.com/apache/pinot/issues/8260)  and
* [PEP design doc](https://docs.google.com/document/d/10-vL_bUrI-Pi2oYudWyUlQl9Kf0cLrW-Z8hGczkCPik/edit)