# Segment Management ## Purpose Segments are the fundamental storage and query unit in Apache Pinot. Every table is divided into segments, and how those segments are assigned to servers, maintained over time, and compacted directly affects query performance, storage cost, and operational resilience. This section covers the full segment lifecycle -- from initial assignment through ongoing maintenance tasks. ## Segment assignment and placement Decide how segments land on servers and how servers are selected for a table. | Page | What it covers | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | [Segment Assignment](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-assignment) | Balanced, replica-group, and partitioned replica-group assignment strategies | | [Instance Assignment](https://docs.pinot.apache.org/operate-pinot/segment-management/instance-assignment) | Tag-based isolation, replica-group instance partitioning, pool-based assignment, and mirroring across tables | ## Segment lifecycle and repair Understand the operations available when segments need to be reset, reloaded, refreshed, or repaired. | Page | What it covers | | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | [Segment Lifecycle and Repair](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-lifecycle-and-repair) | Decision guide for choosing between reset, reload, refresh, rebalance, force commit, purge, and rollback | | [Reload a Table Segment](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-reload) | Step-by-step instructions for reloading segments via the Controller API or Admin Console | ## Rebalance Redistribute segments after capacity changes, config updates, or tenant modifications. | Page | What it covers | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | [Rebalance overview](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance) | When and why to rebalance -- servers, brokers, and tenants | | [Rebalance Servers](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance/rebalance-servers) | Server rebalance API, parameters, and operational guidance | | [Rebalance Servers -- Examples and Scenarios](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance/rebalance-servers/examples-and-scenarios) | Worked examples for common rebalance situations | | [Rebalance Brokers](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance/rebalance-brokers) | Broker rebalance after adding or removing broker instances | | [Rebalance Tenant](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance/rebalance-tenant) | Rebalance all tables belonging to a tenant after tagging changes | ## Tiered storage Move older or less-queried data to cheaper storage tiers while keeping recent data on fast disks. | Page | What it covers | | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | [Separating Data Storage by Age](https://docs.pinot.apache.org/operate-pinot/segment-management/separating-data-storage-by-age) | Overview of tiered storage strategies | | [Moving Segments Across Tenants](https://docs.pinot.apache.org/operate-pinot/segment-management/separating-data-storage-by-age/moving-segments-across-tenants) | Use tag overrides to move completed segments to a different server tier | | [Using Multiple Directories](https://docs.pinot.apache.org/operate-pinot/segment-management/separating-data-storage-by-age/using-multiple-directories) | Configure multiple data directories on a single server to span storage devices | ## Minion tasks for segment maintenance Automate compaction, merging, purging, and ingestion using Pinot Minion. | Page | What it covers | | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | [Pinot Managed Offline Flows](https://docs.pinot.apache.org/operate-pinot/segment-management/pinot-managed-offline-flows) | Automatically move data from real-time tables to offline tables (RealtimeToOfflineSegmentsTask) | | [Merge Rollup Task](https://docs.pinot.apache.org/operate-pinot/segment-management/minion-merge-rollup-task) | Merge small segments into larger time-aligned segments with optional rollup aggregation | | [Segment Generation and Push Task](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-generation-and-push-task) | Batch ingestion via Minion -- read files, build segments, push to the cluster | | [Refresh Segment Task](https://docs.pinot.apache.org/operate-pinot/segment-management/refresh-segment-task) | Automatically rebuild segments when the table config or schema changes | | [Purge Task](https://docs.pinot.apache.org/operate-pinot/segment-management/purge-task) | Remove or modify records for compliance or data-quality reasons | | [Upsert Compaction Task](https://docs.pinot.apache.org/operate-pinot/segment-management/upsert-compaction-task) | Reclaim space by removing invalidated records from upsert-enabled tables | | [Upsert Compact Merge Task](https://docs.pinot.apache.org/operate-pinot/segment-management/upsert-compact-merge-task) | Merge small segments while compacting -- reduces segment count in upsert tables | | [Upsert Merge Compact Task](https://docs.pinot.apache.org/operate-pinot/segment-management/upsert-merge-compact-task) | Alternative merge-compact task for upsert tables | ## Consistent push and rollback Guarantee atomicity when replacing offline segments and quickly revert a bad data push. | Page | What it covers | | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | [Consistent Push and Rollback](https://docs.pinot.apache.org/operate-pinot/segment-management/consistent-push-and-rollback) | Segment lineage protocol for atomic push and one-click rollback of offline table refreshes | ## When to use what | Goal | Recommended action | | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Newly added servers have no segments | Run a [rebalance](https://docs.pinot.apache.org/operate-pinot/segment-management/rebalance/rebalance-servers) | | Segment stuck in ERROR state | [Reset](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-lifecycle-and-repair) the segment, then reload if data is corrupt | | Schema or index config changed | [Reload](https://docs.pinot.apache.org/operate-pinot/segment-management/segment-reload) all segments, or schedule a [RefreshSegmentTask](https://docs.pinot.apache.org/operate-pinot/segment-management/refresh-segment-task) for full rebuild | | Too many small segments | Schedule a [MergeRollupTask](https://docs.pinot.apache.org/operate-pinot/segment-management/minion-merge-rollup-task) or [UpsertCompactMergeTask](https://docs.pinot.apache.org/operate-pinot/segment-management/upsert-compact-merge-task) | | Stale records in upsert table wasting space | Schedule an [UpsertCompactionTask](https://docs.pinot.apache.org/operate-pinot/segment-management/upsert-compaction-task) | | Need to delete specific records (GDPR) | Schedule a [PurgeTask](https://docs.pinot.apache.org/operate-pinot/segment-management/purge-task) | | Bad offline push needs rollback | Use [Consistent Push and Rollback](https://docs.pinot.apache.org/operate-pinot/segment-management/consistent-push-and-rollback) | | Recent data needs fast disks, old data can be on HDDs | Configure [tiered storage](https://docs.pinot.apache.org/operate-pinot/segment-management/separating-data-storage-by-age) |