Table

tableType

Defines the table type: OFFLINE for offline tables or REALTIME for real-time tables. A hybrid table is essentially two table configurations: one of each type, with the same table name.

isDimTable

Boolean field to indicate whether the table is a dimension table

quota

Defines properties related to quotas, such as storage quota and query quota. For details, see the Quota table below.

task

Defines the enabled minion tasks for the table. See Minion for more details.

routing

Defines the properties that determine how the broker selects the servers to route, and how segments can be pruned by the broker based on segment metadata. For details, see the Routing table below.

query

Defines the properties related to query execution. For details, see the Query table below.

segmentsConfig

Defines the properties related to the segments of the table, such as segment push frequency, type, retention, schema, time column etc. For details, see the segmentsConfig table below.

tableIndexConfig

Defines the indexing related information for the Pinot table. For details, see Table indexing config below.

fieldConfigList

Specifies the columns and the type of indices to be created on those columns. See Field config list for sub-properties.

tenants

Defines the server and broker tenant used for this table. For details, see Tenant below.

ingestionConfig

Defines the configurations needed for ingestion level transformations. For details, see Ingestion Level Transformations and Ingestion Level Aggregations.

upsertConfig

Set upset configurations. For details, see Stream ingestion with upsert.

dedupConfig

Set deduplication configurations. For details, see Stream ingestion with Dedup.

tierConfigs

Defines configurations for tiered storage. For details, see Tiered Storage.

metadata

Contains other metadata of the table. There is a string to string map field "customConfigs" under it which is expressed as key-value pairs to hold the custom configurations.

storage

The maximum storage space the table is allowed to use before replication.

For example, in the above table, the storage is 140G and replication is 3, so the maximum storage the table is allowed to use is 140G x 3 = 420G. The space the table uses is calculated by adding up the sizes of all segments from every server hosting this table. Once this limit is reached, offline segment push throws a 403 exception with message, Quota check failed for segment: segment_0 of table: pinotTable.

maxQueriesPerSecond

The maximum queries per second allowed to execute on this table. If query volume exceeds this, a 429 exception with message Request 123 exceeds query quota for table:pinotTable, query:select count(*) from pinotTable will be sent, and a BrokerMetric QUERY_QUOTA_EXCEEDED will be recorded. The application should build an exponential backoff and retry mechanism to react to this exception.

segmentPrunerTypes

The list of segment pruners to be enabled.

The segment pruner prunes the selected segments based on the query.

Supported values:

• partition: Prunes segments based on the partition metadata stored in zookeeper. By default, there is no pruner.

• time: Prune segments for queries filtering on timeColumnName that do not contain data in the query's time range.

instanceSelectorType

The server instances to serve the query based on selected segments. Supported values:

• balanced: Balances the number of segments served by each selected instance. Default.

• replicaGroup: Instance selector for replica group routing strategy.

timeoutMs

Query timeout in milliseconds

disableGroovy

Whether to disable groovy in query. This overrides the broker instance level config (pinot.broker.disable.query.groovy) if configured.

useApproximateFunction

Whether to automatically use approximate function for expensive aggregates, such as DISTINCT_COUNT and PERCENTILE. This overrides the broker instance level config (pinot.broker.use.approximate.function) if configured.

expressionOverrideMap

A map that configures the expressions to override in the query. This can be useful when users cannot control the queries sent to Pinot (e.g. queries auto-generated by some other tools), but want to override the expressions within the query (e.g. override a transform function to a derived column). Example: {"myFunc(a)": "b"}.

schemaName

Name of the schema associated with the table

timeColumnName

The name of the time column for this table. This must match with the time column name in the schema. This is mandatory for tables with push type APPEND, optional for REFRESH. timeColumnName along with timeColumnType is used to manage segment retention and time boundary for offline vs real-time.

replication

Number of replicas for the tables. A replication value of 1 means segments won't be replicated across servers.

retentionTimeUnit

Unit for the retention, such as HOURS or DAYS. This, in combination with retentionTimeValue decides the duration for which to retain the segments.

For example, 365 DAYS means that segments containing data older than 365 days will be deleted periodically by the RetentionManager Controller periodic task. By default, there is no set retention.

retentionTimeValue

A numeric value for the retention. This, in combination with retentionTimeUnit, determines the duration for which to retain the segments

invertedIndexColumns

The list of columns that inverted index should be created on. The name of columns should match the schema. e.g. in the table above, inverted index has been created on three columns foo, bar, moo

createInvertedIndexDuringSegmentGeneration

Boolean to indicate whether to create inverted indexes during the segment creation. By default, false i.e. inverted indexes are created when the segments are loaded on the server

sortedColumn

The column which is sorted in the data and hence will have a sorted index. This does not need to be specified for the offline table, as the segment generation job will automatically detect the sorted column in the data and create a sorted index for it.

bloomFilterColumns

The list of columns to apply bloom filter on. The names of the columns should match the schema. For more details about using bloom filters refer to Bloom Filter.

bloomFilterConfigs

The map from the column to the bloom filter config. The names of the columns should match the schema. For more details about using bloom filters refer to Bloom Filter.

name

Name of the column

encodingType

Should be one of RAW or DICTIONARY

indexTypes

List of indexes to create on this column. Valid values are the ids of the index types (text, fst, h3, etc)

properties

JSON of key-value pairs containing additional properties associated with the index. The following properties are supported currently -

• enableQueryCacheForTextIndex - set to true to enable caching for text index in Lucene

• luceneMaxBufferSizeMB - Lucene IndexWriter buffer max size, defaults to 500

• luceneUseCompoundFile - Lucene IndexWriter file format, defaults to true to use compound files

• rawIndexWriterVersion

• deriveNumDocsPerChunkForRawIndex

• forwardIndexDisabled - set to true to disable the forward index, defaults to false

replicasPerPartition

The number of replicas per partition for the stream

completionMode

determines if segment should be downloaded from other server or built in memory. can be DOWNLOAD or empty

peerSegmentDownloadScheme

protocol to use to download segments from server. can be on of http or https

broker

Broker tenant in which the segment should reside

server

Server tenant in which the segment should reside

tagOverrideConfig

Override the tenant for segment if it fulfills certain conditions. Currently, only support override on realtimeConsuming or realtimeCompleted