Indexing

Pinot supports the following indexing techniques:
​Forward Index​
Dictionary-encoded forward index with bit compression
Raw value forward index
Sorted forward index with run-length encoding
​Inverted Index​
Bitmap inverted index
Sorted inverted index
​Star-tree Index​
​Bloom Filter​
​Range Index​
​Text Index​
​Geospatial​
​JSON Index​
Each of these techniques has advantages in different query scenarios. By default, Pinot creates a dictionary-encoded forward index for each column. 
Enabling indexes
There are 2 ways to create indexes for a Pinot table. 
As part of ingestion, during Pinot segment generation
Indexing is enabled by specifying the desired column names in the table config. More details about how to configure each type of index can be found in the respective index's section above or in the Table Config section.
Dynamically added or removed
Indexes can also be dynamically added to or removed from segments at any point. Update your table config with the latest set of indexes you wish to have. 
For example, if you have an inverted index on the foo field and now want to include the bar field, you would update your table config from this:
1
"tableIndexConfig": {
2
        "invertedIndexColumns": ["foo"],
3
        ...
4
    }
Copied!
To this:
1
"tableIndexConfig": {
2
        "invertedIndexColumns": ["foo", "bar"],
3
        ...
4
    }
Copied!
The updated index config won't be picked up unless we invoke the reload API. This API sends reload messages via Helix to all servers, as part of which indexes are added or removed from the local segments. This happens without any downtime and is completely transparent to the queries. 
When adding an index, only the new index is created and appended to the existing segment. When removing an index, its related states are cleaned up from Pinot servers. You can find this API under the Segments tab on Swagger:
1
curl -X POST 
2
"http://localhost:9000/segments/myTable/reload" 
3
-H "accept: application/json"
Copied!
You can also find this action on the Cluster Manager in the Pinot UI, on the specific table's page.
Tuning Index
The inverted index provides good performance for most use cases, especially if your use case doesn't have a strict low latency requirement. 

You should start by using this, and if your queries aren't fast enough, switch to advanced indices like the sorted or Star-Tree index.

Complex Type (Array, Map) Handling

Forward Index

Last modified 6d ago

Copy link

Contents

Enabling indexes

Tuning Index