# Troubleshooting ## Troubleshooting Use this page to find the right troubleshooting guide for your situation. Start by identifying the type of problem you are experiencing, then follow the link to the relevant guide. ### What kind of problem are you seeing? #### Query issues Queries returning errors, unexpected results, or timing out. | Symptom | Go to | | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `BrokerResourceMissingError`, reserved keyword errors, wrong results, slow queries | [Query FAQ](https://docs.pinot.apache.org/operate-pinot/troubleshooting/query-faq) | | Errors or limitations specific to the multi-stage engine (MSE), including type mismatches, unsupported functions, or timeout errors | [Troubleshoot Multi-Stage Engine (MSE)](https://docs.pinot.apache.org/operate-pinot/troubleshooting/query-faq/troubleshoot-multi-stage-query-engine) | #### Ingestion issues Data not appearing, segments stuck, or ingestion pipelines failing. | Symptom | Go to | | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Segment sizing, partitioning, indexing, Kafka ingestion, data encoding, or real-time ingestion questions | [Ingestion FAQ](https://docs.pinot.apache.org/operate-pinot/troubleshooting/ingestion-faq) | | Kafka partitions stopped consuming, segment commit failures, `Controller response was FAILED` errors | [Realtime Ingestion Stopped](https://docs.pinot.apache.org/operate-pinot/troubleshooting/ingestion-faq/realtime-ingestion-stopped) | #### Operations and cluster issues Cluster instability, memory problems, segment errors, rebalancing, or configuration questions. | Symptom | Go to | | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Heap sizing, backup/restore, schema changes, rebalancing, segment states (BAD/ERROR), tenant configuration, minion tasks, tiered storage | [Operations FAQ](https://docs.pinot.apache.org/operate-pinot/troubleshooting/operations-faq) | | Using the debug API, slow query diagnosis, GC pressure on servers | [Troubleshooting Pinot](https://docs.pinot.apache.org/operate-pinot/troubleshooting/troubleshooting-pinot) | #### Kubernetes issues Problems specific to running Pinot on Kubernetes. | Symptom | Go to | | ------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | Increasing server disk size on AWS EKS, PVC resizing, pod restarts | [Pinot on Kubernetes FAQ](https://docs.pinot.apache.org/operate-pinot/troubleshooting/pinot-on-kubernetes-faq) | #### ZooKeeper issues ZooKeeper errors related to metadata storage limits. | Symptom | Go to | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `packet len is out of range` errors, znode size exceeded, too many segments | [Troubleshoot ZooKeeper](https://docs.pinot.apache.org/operate-pinot/troubleshooting/troubleshoot-zookeeper) | #### General questions Broad questions about Pinot architecture and behavior. | Symptom | Go to | | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | How deep storage works, how Pinot uses ZooKeeper, JDK compatibility, timezone configuration | [General FAQ](https://docs.pinot.apache.org/operate-pinot/troubleshooting/troubleshooting-pinot/general-faq) | ### Quick diagnostic checklist Before diving into a specific guide, gather the following information to speed up diagnosis: 1. **Which component is affected?** Controller, broker, server, or minion? 2. **Check the logs.** All Pinot components log error conditions. Look for stack traces or error messages. 3. **Use the debug API.** The [Table Debug API](https://docs.pinot.apache.org/reference/api-reference/controller-api) surfaces common problems including table size, ingestion status, and state transition errors. 4. **Check metrics.** If you have [monitoring](https://docs.pinot.apache.org/operate-pinot/monitoring) set up, review dashboards for anomalies in query latency, ingestion lag, or GC pressure. 5. **Review recent changes.** Did you recently update a table config, schema, or cluster configuration? ### Related Operate Pinot pages Many troubleshooting issues connect back to operational configuration and tuning. These pages may help: * [Deployment and Monitoring](https://docs.pinot.apache.org/operate-pinot/deployment) -- overview of operating Pinot clusters * [Monitoring](https://docs.pinot.apache.org/operate-pinot/monitoring) -- set up dashboards and alerts to catch issues early * [Rebalance](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance) -- resolve uneven segment distribution or add new servers * [Segment Lifecycle and Repair](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-lifecycle-and-repair) -- understand segment states and repair procedures * [Decoupling Controller from the Data Path](https://docs.pinot.apache.org/operate-pinot/deployment/decoupling-controller-from-the-data-path) -- reduce controller bottlenecks for real-time ingestion * [Tuning](https://docs.pinot.apache.org/operate-pinot/tuning) -- optimize query routing, real-time performance, and segment pruning * [Managing Logs](https://docs.pinot.apache.org/operate-pinot/monitoring/managing-logs) -- configure log levels for debugging * [Upgrading Pinot](https://docs.pinot.apache.org/operate-pinot/upgrades/upgrading-pinot-cluster) -- check upgrade notes before and after cluster upgrades * [Setup Cluster](https://docs.pinot.apache.org/operate-pinot/deployment/setup-cluster) -- verify cluster configuration ### Next step If you cannot resolve your issue using these guides, reach out to the Apache Pinot community: * **Slack**: [Join the Apache Pinot Slack](https://inviter.co/apache-pinot) and ask in the troubleshooting channel * **GitHub Issues**: [apache/pinot](https://github.com/apache/pinot/issues) for bug reports and feature requests * **Mailing list**: for general questions *** ### description: >- This page has a collection of frequently asked questions with answers from the community. ## Frequently Asked Questions (FAQs) {% hint style="info" %} This is a list of questions frequently asked in our troubleshooting channel on Slack. To contribute additional questions and answers, [make a pull request](https://docs.pinot.apache.org/develop-and-contribute/contributing/contributing). {% endhint %} ### Ingestion #### Flatten my JSON Kafka stream The [json\_format(field)](https://docs.pinot.apache.org/build-with-pinot/ingestion/transformations-and-aggregations/ingestion-level-transformations#json-functions) function can store a top level json field as a STRING in Pinot. Then, use these [json functions](https://docs.pinot.apache.org/functions/transformations#json-functions) during query time, to extract fields from the json string. {% hint style="warning" %} **NOTE**\ This works well if some of your fields are nested json, but most of your fields are top level json keys. If all of your fields are within a nested JSON key, you will have to store the entire payload as 1 column, which is not ideal. {% endhint %} ### Indexing #### Set inverted indexes Inverted indexes are set in the table configuration's `tableIndexConfi`g -> `invertedIndexColumns` list. See the documentation for `tableIndexConfig`: [docs](https://docs.pinot.apache.org/reference/configuration-reference/table#table-index-config) which includes a sample table that has set inverted indexes on some columns. Applying inverted indexes to a table configuration will generate an inverted index for all new segments. In order to apply the inverted indexes to all existing segments, see [How to apply inverted index to existing setup?](#apply-inverted-index-to-existing-setup). #### Apply inverted index to existing setup 1. Add the columns you want to index to the `tableIndexConfig-> invertedIndexColumns` list. This sample table config shows inverted indexes set: [docs](https://docs.pinot.apache.org/reference/configuration-reference/table#offline-table). To update the table configuration use the Pinot Swagger API: . 2. Invoke the reload API: . This will trigger a reload operation on each of the servers hosting the table's segments. The API response has a `reloadJobId` which can be used to monitor the status of the reload operation using the segment reload status API: #### Create star-tree indexes Star-tree indexes are configured in the table configuration under the `tableIndexConfig` -> `starTreeIndexConfigs` (list) and `enableDefaultStarTree` (boolean). See here to learn more about how to configure star-tree indexes: [docs](https://docs.pinot.apache.org/readme) The new segments will have star-tree indexes generated after applying the star-tree index configurations to the table configuration. Pinot does not support adding star-tree indexes to the existing segments. ### Querying #### What are all the fields in the Pinot query's JSON response? See this page which explains the Pinot response format: [docs](https://docs.pinot.apache.org/reference/api-reference/query-response-format). #### SQL Query fails with "Encountered 'timestamp' was expecting one of..." "timestamp" is a reserved keyword in SQL. Escape timestamp with double quotes. ```sql select "timestamp" from myTable ``` Other commonly encountered reserved keywords are date, time, table. #### Filtering on STRING column WHERE column = "foo" does not work? For filtering on STRING columns, use single quotes. ```sql SELECT COUNT(*) from myTable WHERE column = 'foo' ``` #### ORDER BY using an alias doesn't work? The fields in the `ORDER BY` clause must be one of the group by clauses or aggregations, ***BEFORE*** applying the alias. Therefore, this will not work: ```sql SELECT count(colA) as aliasA, colA from tableA GROUP BY colA ORDER BY aliasA ``` However, this will work: ```sql SELECT count(colA) as sumA, colA from tableA GROUP BY colA ORDER BY count(colA) ``` #### Does pagination work in GROUP BY queries? No. Pagination only works for SELECTION queries. ### Operations #### How to change the number of replicas of a table? You can change the number of replicas by updating the table configuration's [segmentsConfig](https://docs.pinot.apache.org/reference/configuration-reference/table#segmentsconfig) section. Make sure you have at least as many servers as the replication. For OFFLINE table, update [replication](https://docs.pinot.apache.org/reference/configuration-reference/table#segmentsconfig), as follows: ```json { "tableName": "pinotTable", "tableType": "OFFLINE", "segmentsConfig": { "replication": "3", ... } .. ``` For REALTIME table update [replicasPerPartition](https://docs.pinot.apache.org/reference/configuration-reference/table#segmentsconfig), as follows: ```json { "tableName": "pinotTable", "tableType": "REALTIME", "segmentsConfig": { "replicasPerPartition": "3", ... } .. ``` After changing the replication, run a [table rebalance](#how-to-run-a-rebalance-on-a-table). #### How to run a rebalance on a table? See [Rebalance](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance). #### How to control number of segments generated? The number of segments generated depends on the number of input files. If you provide only one input file, you will get one segment. If you break up the input file into multiple files, you will get as many segments as the input files. ### Tuning and Optimizations #### Do replica groups work for real-time? Yes, replica groups work for real-time. There are two parts to enabling replica groups: 1. Replica groups segment assignment 2. Replica group query routing **Replica group segment assignment** Replica group segment assignment is achieved in realtime, if the number of servers is a multiple of the number of replicas. The partitions get uniformly spread across the servers, creating replica groups. For example, if we have 6 partitions, 2 replicas, and 4 servers: | | r1 | r2 | | -- | -- | -- | | p1 | S0 | S1 | | p2 | S2 | S3 | | p3 | S0 | S1 | | p4 | S2 | S3 | | p5 | S0 | S1 | | p6 | S2 | S3 | In this example, the set (S0, S2) contains r1 of every partition, and (s1, S3) contains r2 of every partition. The query will only be routed to one of the sets, and not span every server.\ If you are are adding/removing servers from an existing table setup, you have to run [rebalance](#how-to-run-a-rebalance-on-a-table) for segment assignment changes to take effect. **Replica group query routing** Once replica group segment assignment is in effect, the query routing can take advantage of it. For replica group based query routing, set the following in the table configuration's [routing](https://docs.pinot.apache.org/reference/configuration-reference/table#routing) section, and then restart brokers ```json { "tableName": "pinotTable", "tableType": "REALTIME", "routing": { "instanceSelectorType": "replicaGroup" } .. } ``` ### Handling time in Pinot #### How does Pinot’s real-time ingestion handle out-of-order events? Pinot does not require ordering of event time stamps. Out of order events are still consumed and indexed into the "currently consuming" segment. In a pathological case, if you have a two-day-old event come in "now", it will still be stored in the segment that is open for consumption "now". There is no strict time-based partitioning for segments, but star-indexes and hybrid tables will handle this as appropriate. See [Components > Broker](https://docs.pinot.apache.org/architecture-and-concepts/components/cluster/broker) for more details about how hybrid tables handle this. Specifically, the time-boundary is computed as `max(OfflineTIme) - 1 unit of granularity`. Pinot stores the min-max time for each segment and uses it for pruning segments, so segments with multiple time intervals may not be perfectly pruned. When generating star-indexes, the time column will be part of the star-tree so the tree can still be efficiently queried for segments with multiple time intervals. #### What is the purpose of a hybrid table not using `max(OfflineTime)` to determine the time-boundary, and instead using an offset? This lets you have an old event up come in without building complex offline pipelines that perfectly partition your events by event timestamps. With this offset, even if your offline data pipeline produces segments with a maximum timestamp, Pinot will not use the offline dataset for that last chunk of segments. The expectation is if you process offline the next time-range of data, your data pipeline will include any late events. #### Why are segments not strictly time-partitioned? It might seem odd that segments are not strictly time-partitioned, unlike similar systems such as Apache Druid. The Pinot way allows real-time ingestion to consume out-of-order events. Even though segments are not strictly time-partitioned, Pinot will still index, prune, and query segments intelligently by time-intervals to for performance of hybrid tables and time-filtered data. When generating offline segments, the segments generated such that segments only contain one time-interval and are well partitioned by the time column.