FST Index
The FST (Finite State Transducer) index accelerates regex queries on STRING columns by building over dictionary values. It reduces the on-disk index size by 4-6x compared to scanning the full dictionary.
When to use
Use an FST index for case-sensitive REGEXP_LIKE predicates on string columns. It is especially effective for prefix-matching patterns. For case-insensitive LIKE or REGEXP_LIKE predicates, use the IFST index. For full-text search with tokenization, use the text index instead.
Supported column types
STRING columns only
Must be single-valued
Must have dictionary values available. This can come from a dictionary-encoded forward index, or from a standalone dictionary paired with a RAW forward index.
Limitations
Only supports case-sensitive
REGEXP_LIKEpredicates.Only supported on stored or completed segments (not consuming segments in real-time tables).
Only supported on columns with dictionary values available.
Works best for prefix queries. Suffix-only or infix-only patterns may not benefit as much.
For more information on FST construction, see the Lucene FST documentation.
Configuration
To enable the FST index on a column:
The FST index generates one index file (.lucene.fst). If you keep the forward index RAW, declare a dictionary for the column so Pinot can build the FST from dictionary values:
Do not keep the column in legacy tableIndexConfig.noDictionaryColumns or tableIndexConfig.noDictionaryConfig when enabling FST or IFST. Those settings disable the dictionary and Pinot rejects the table config. If an inverted index is also enabled on the column, FST can take advantage of it for faster lookups.
Query examples
These examples are case-sensitive and can use FST.
Prefix match:
Regex match:
Infix match:
Case-insensitive FST index (IFST)
The case-insensitive FST index (IFST) provides the same functionality as the standard FST index but with case-insensitive matching. This eliminates the need to handle case sensitivity manually in queries.
Supports
LIKEpredicates andREGEXP_LIKEpredicates with the'i'flag.Only supported on stored or completed segments (not consuming segments).
Only supported on STRING columns with dictionary values available.
Works best for prefix queries with case-insensitive matching.
Configuration
The case-insensitive FST index generates one index file (.lucene.ifst).
Query examples
Both queries match strings beginning with "Us" regardless of case and can use IFST:
For more information about enabling indexes, see enabling indexes.
Last updated
Was this helpful?

