StarRocks Storage Balancer Administration
When to Use
Load this skill when the user needs to:
- Add new BE nodes to an existing cluster and wait for automatic rebalancing to complete
- Diagnose disk skew alerts where one or more BEs have significantly higher
UsedPctthan others - Decommission a BE node gracefully, ensuring all tablets migrate away before the node is removed
- Recover from a BE failure by understanding how missing replicas are repaired and load redistributed
- Tune balance aggressiveness via
balance_load_score_threshold,max_balancing_tablets, and related FE parameters - Monitor tablet migration progress via
SHOW PROC '/cluster_balance'and Prometheus metrics - Forecast storage capacity and estimate days-to-full based on current disk usage trends
- Move tablets between disks on the same BE when one disk path is hot
Tablet Distribution Concepts
How StarRocks Distributes Tablets
StarRocks splits each table partition into bucket_num tablets. Each tablet has replication_num replicas (default 3). The FE tablet scheduler is responsible for placing and balancing replicas.
Table Partition
└─ Tablet 1 → replica on BE-1, BE-2, BE-3
└─ Tablet 2 → replica on BE-1, BE-3, BE-4
└─ Tablet 3 → replica on BE-2, BE-3, BE-4
...
Placement decisions are made by the tablet scheduler (runs on Leader FE) using two objectives:
- Repair: fix under-replicated, missing, or inconsistent replicas — always has highest priority.
- Balance: redistribute tablets to reduce load score skew across BEs — lower priority than repair.
Load Score Calculation
The FE assigns each BE a load score used to compare relative heaviness:
load_score = w_tablet * (tablet_num / avg_tablet_num)
+ w_disk * (data_used_capacity / total_capacity)
Default weights are equal (w_tablet = w_disk = 0.5). The scheduler migrates tablets from the highest-score BE to the lowest-score BE when the score difference exceeds balance_load_score_threshold (default 0.1 = 10%).
What Triggers Rebalancing
- A new BE is registered: its load score is 0, so it is far below the cluster average — migrations begin immediately.
- A BE is decommissioned: its tablets must move to other BEs.
- A BE recovers after downtime and has missing replicas cloned in, temporarily changing the distribution.
- Ongoing ingestion naturally shifts tablet counts over time; the scheduler periodically re-evaluates balance.
- An operator manually changes
balance_load_score_thresholdto a smaller value.
Detecting Imbalance
SHOW BACKENDS — Per-BE Disk Usage
-- Run from any MySQL client (port 9030)
SHOW BACKENDS\G
Key columns for storage balance:
| Column | Meaning |
|---|---|
BackendId |
Unique numeric BE identifier |
Host |
Hostname or IP |
Alive |
true if heartbeat received within timeout |
SystemDecommissioned |
true if a decommission is in progress |
TabletNum |
Number of tablet replicas on this BE |
DataUsedCapacity |
Bytes of actual tablet data stored |
AvailCapacity |
Bytes of free disk space |
TotalCapacity |
Total disk capacity of all data paths on this BE |
UsedPct |
(TotalCapacity - AvailCapacity) / TotalCapacity * 100 |
MaxDiskUsedPct |
Highest UsedPct across individual disk paths on this BE |
Detecting Imbalanced BEs by Disk Usage
A cluster is considered disk-imbalanced when max(UsedPct) - min(UsedPct) > 20% across alive BEs:
-- Requires information_schema.be_configs (StarRocks 3.1+)
-- For older versions parse SHOW BACKENDS output in the application layer.
SELECT
BackendId,
Host,
TabletNum,
ROUND(DataUsedCapacity / 1073741824.0, 2) AS data_used_gb,
ROUND(TotalCapacity / 1073741824.0, 2) AS total_gb,
ROUND(UsedPct, 2) AS used_pct,
ROUND(MaxDiskUsedPct, 2) AS max_disk_pct
FROM information_schema.be_configs
WHERE Alive = 'true'
ORDER BY UsedPct DESC;
-- Imbalance detection: flag when spread > 20 percentage points
SELECT
MAX(UsedPct) AS max_used_pct,
MIN(UsedPct) AS min_used_pct,
ROUND(MAX(UsedPct) - MIN(UsedPct), 2) AS spread_pct,
CASE
WHEN MAX(UsedPct) - MIN(UsedPct) > 20 THEN 'IMBALANCED'
ELSE 'OK'
END AS disk_balance_status,
ROUND(AVG(UsedPct), 2) AS avg_used_pct,
MAX(TabletNum) AS max_tablets,
MIN(TabletNum) AS min_tablets,
ROUND(AVG(TabletNum), 0) AS avg_tablets,
ROUND(
(MAX(TabletNum) - MIN(TabletNum)) * 100.0 / NULLIF(AVG(TabletNum), 0),
1
) AS tablet_spread_pct
FROM information_schema.be_configs
WHERE Alive = 'true';
Interpretation thresholds:
| Metric | OK | Warning | Critical |
|---|---|---|---|
Disk UsedPct spread |
< 20 pp | 20–40 pp | > 40 pp |
| Tablet count spread | < 30% | 30–60% | > 60% |
Any single MaxDiskUsedPct |
< 75% | 75–85% | > 85% |
SHOW PROC '/cluster_balance' — Sub-paths
-- Top-level overview of the balance subsystem
SHOW PROC '/cluster_balance';
This path lists sub-nodes. Navigate each with SHOW PROC:
| Sub-path | What it shows |
|---|---|
/cluster_balance/overview |
Summary counters: pending tablet count, running migrations, slots per path |
/cluster_balance/working_slots |
Available clone slots per BE disk path (how much concurrency is left) |
/cluster_balance/balance |
In-flight balance migrations currently being executed |
/cluster_balance/pending_tablets |
Tablets queued for migration (waiting for a free slot) |
/cluster_balance/history_tablets |
Ring buffer of recently completed migrations |
-- Summary: how many migrations are queued and running right now
SHOW PROC '/cluster_balance/overview';
-- Key fields: TotalScheduledTabletNum, TotalRunningTabletNum, BalanceSlotNumPerPath
-- A healthy idle cluster: TotalRunningTabletNum = 0 and TotalScheduledTabletNum = 0.
-- After adding a new BE: expect hundreds to thousands of TotalScheduledTabletNum initially.
-- In-flight migrations: source BE → destination BE
SHOW PROC '/cluster_balance/balance';
-- Columns: TabletId, Type (BALANCE), State (PENDING/RUNNING/FINISHED),
-- OrigTabletId, SchemaHash, Src, Dest, CreateTime, ScheduledTime, FinishedTime
-- Pending migrations (waiting for slot)
SHOW PROC '/cluster_balance/pending_tablets';
-- Same columns as /balance but for tablets not yet started.
Balance Configuration
All balance parameters are dynamic FE configuration values. Changes take effect immediately without a restart.
balance_load_score_threshold
Controls how large the load score difference between two BEs must be before a migration is initiated.
-- View current value
ADMIN SHOW FRONTEND CONFIG LIKE '%balance_load_score_threshold%';
-- Default is 0.1 (10%). Lower = more aggressive, more migrations.
-- Raise to 0.2 if the cluster is frequently migrating tablets in a noisy cluster.
-- Lower to 0.05 after adding a large batch of new BEs to speed up initial fill.
ADMIN SET FRONTEND CONFIG ("balance_load_score_threshold" = "0.1");
When to tune:
- Newly added BEs are filling too slowly: lower to
0.05. - Cluster has continuous low-volume migrations that never fully settle: raise to
0.15. - Small cluster (3 BEs) with natural data skew from partitioning: raise to
0.2to avoid futile churn.
tablet_sched_balancer_strategy
Selects what the load score is based on.
ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched_balancer_strategy%';
-- Values:
-- 'disk_and_tablet' (default) — score combines disk usage AND tablet count
-- 'disk' — score based on disk usage only; tablet count is ignored
ADMIN SET FRONTEND CONFIG ("tablet_sched_balancer_strategy" = "disk_and_tablet");
Use disk strategy when tables have very uneven tablet sizes (e.g., range-partitioned tables where recent partitions are much larger) to prioritize equalizing disk usage over tablet count.
max_balancing_tablets
Maximum number of concurrent tablet clone tasks generated by the balance scheduler at any moment.
ADMIN SHOW FRONTEND CONFIG LIKE '%max_balancing_tablets%';
-- Default: 100. Safe range: 50–500.
-- Increase during maintenance windows to fill new BEs faster.
-- Decrease during peak query hours to reduce I/O contention.
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "100");
tablet_sched_max_migration_task_sent_per_minute
Throttle on how many migration tasks the scheduler dispatches per minute to any single BE.
ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched_max_migration_task_sent_per_minute%';
-- Default: 800. Lower to 200–400 on spinning-disk deployments.
-- High values can saturate disk I/O on the source BE during peak hours.
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_migration_task_sent_per_minute" = "800");
Disabling Balance Temporarily
-- Pause all balance migrations (repair tasks still run — they are not affected)
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
-- Re-enable (CRITICAL: always re-enable after the maintenance window)
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");
Viewing All Balance-Related FE Config
ADMIN SHOW FRONTEND CONFIG LIKE '%balance%';
ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched%';
Adding New BE Nodes
Adding a BE node is a zero-downtime operation. Tablets migrate automatically after the node registers.
Step-by-Step
Step 1 — Deploy the BE process on the new host.
# On the new BE host: edit be.conf
# Required settings:
# be_host = <new_be_ip> # must be reachable by all FEs
# be_port = 9060 # Thrift RPC
# webserver_port = 8040 # HTTP Web UI
# brpc_port = 8060 # Binary RPC
# heartbeat_service_port = 9050 # FE→BE heartbeat
# storage_root_path = /data1;/data2 # semi-colon-separated disk paths
# Start the BE process
bin/start_be.sh --daemon
Step 2 — Register the new BE from the Leader FE.
-- Connect to FE Leader (port 9030) as a user with ALTER SYSTEM privilege
ALTER SYSTEM ADD BACKEND 'new_be_host:9050';
-- Use the heartbeat_service_port (9050), NOT the be_port (9060).
Step 3 — Verify the BE has joined.
SHOW BACKENDS\G
-- New BE should appear with:
-- Alive = true
-- TabletNum = 0 (initially — tablets have not migrated yet)
-- SystemDecommissioned = false
-- DataUsedCapacity = 0
Step 4 — Optionally accelerate initial fill.
-- Lower balance threshold and raise concurrency temporarily
ADMIN SET FRONTEND CONFIG ("balance_load_score_threshold" = "0.05");
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "300");
Step 5 — Monitor rebalancing progress.
-- Watch tablet count grow on the new BE over time
SELECT BackendId, Host, TabletNum, ROUND(UsedPct, 2) AS used_pct
FROM information_schema.be_configs
ORDER BY TabletNum DESC;
-- Check pending migration queue shrinking
SHOW PROC '/cluster_balance/overview';
-- Watch in-flight migrations
SHOW PROC '/cluster_balance/balance';
Step 6 — Restore normal balance settings once equilibrium is reached.
ADMIN SET FRONTEND CONFIG ("balance_load_score_threshold" = "0.1");
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "100");
Step 7 — Confirm cluster is balanced.
SELECT
ROUND(MAX(UsedPct) - MIN(UsedPct), 2) AS disk_spread_pp,
ROUND(
(MAX(TabletNum) - MIN(TabletNum)) * 100.0 / AVG(TabletNum),
1
) AS tablet_spread_pct
FROM information_schema.be_configs
WHERE Alive = 'true';
-- Target: disk_spread_pp < 10 and tablet_spread_pct < 20 after equilibrium.
Decommissioning a BE
Decommissioning gracefully migrates all tablets off a BE before it is removed from the cluster. Never kill a BE process before decommissioning; that causes replica loss.
Initiate Decommission
-- Use the heartbeat_service_port (same port as ADD BACKEND)
ALTER SYSTEM DECOMMISSION BACKEND 'be_host:9050';
This sets SystemDecommissioned = true on the BE and triggers the tablet scheduler to migrate all replicas to other BEs. The BE remains alive and serves queries during migration.
Monitor Decommission Progress
-- Check SystemDecommissioned flag and remaining tablet count
SELECT BackendId, Host, Alive, SystemDecommissioned, TabletNum
FROM information_schema.be_configs
ORDER BY SystemDecommissioned DESC, TabletNum DESC;
-- The target BE's TabletNum should decrease to 0.
-- Decommission is complete when TabletNum = 0 AND the BE is removed from the list.
-- This can take minutes to hours depending on data volume and cluster throughput.
-- Watch migration activity
SHOW PROC '/cluster_balance/balance';
-- Source or Dest matching the decommissioning BE's BackendId confirms migrations are running.
-- SHOW PROC '/backends' for per-disk detail (column SystemDecommissioned)
SHOW PROC '/backends';
Cancelling Decommission
If you change your mind before TabletNum reaches 0:
-- Cancel decommission — all remaining tablets stay on this BE
CANCEL DECOMMISSION BACKEND 'be_host:9050';
-- Verify SystemDecommissioned returns to false
SELECT Host, SystemDecommissioned, TabletNum
FROM information_schema.be_configs
WHERE Host = 'be_host';
Requirements for Successful Decommission
- The remaining BEs must have enough free capacity to absorb all tablets from the decommissioned BE. If not, decommission will stall with tablets unable to find a home.
- The cluster must have at least
replication_numalive BEs after removing the target. For a 3-replica table: minimum 3 BEs must remain. - If the cluster has only 3 BEs and
replication_num = 3, decommission will never complete — there is nowhere to place the third replica copy. Add a replacement BE first.
Disk-Level Balancing
Each BE can have multiple disk paths (storage_root_path). StarRocks also balances tablets across disks on the same BE.
Disk-Level Thresholds
-- View disk balance thresholds (FE dynamic config)
ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched_balance_disk%';
| Parameter | Default | Meaning |
|---|---|---|
tablet_sched_balance_disk_threshold_low |
0.5 (50%) |
Below this, the disk is a migration target |
tablet_sched_balance_disk_threshold_high |
0.7 (70%) |
Above this, the disk is a migration source |
When a disk on a BE exceeds the high threshold, the scheduler migrates tablets from it to disks below the low threshold — either on the same BE or on other BEs.
-- Adjust thresholds for dense storage deployments (e.g., NVMe arrays)
ADMIN SET FRONTEND CONFIG ("tablet_sched_balance_disk_threshold_low" = "0.55");
ADMIN SET FRONTEND CONFIG ("tablet_sched_balance_disk_threshold_high" = "0.75");
Viewing Per-Disk Breakdown on a BE
-- SHOW PROC '/backends' returns one row per disk path per BE
SHOW PROC '/backends';
-- Key columns per disk: StorageMedium, StoragePath, UsedCapacity, TotalCapacity, UsedPct,
-- TabletNum, State (ONLINE/OFFLINE/DECOMMISSIONED)
# From shell: list disk paths on a specific BE
mysql -h fe_host -P 9030 -u root --batch \
-e "SHOW PROC '/backends'" \
| awk -F'\t' 'NR==1 || $3=="be_host"'
Tablet Migration Monitoring
SHOW PROC '/cluster_balance/pending_tablets'
SHOW PROC '/cluster_balance/pending_tablets';
Key columns:
| Column | Meaning |
|---|---|
TabletId |
ID of the tablet being scheduled |
Type |
BALANCE (rebalancing) or REPAIR (fixing missing/unhealthy replicas) |
State |
PENDING / RUNNING / FINISHED / CANCELLED |
OrigTabletId |
Original tablet (same as TabletId for non-colocate) |
SchemaHash |
Schema version hash |
Src |
Source BE ID (where tablet is moving from) |
Dest |
Destination BE ID (where tablet is moving to) |
CreateTime |
When the task was created |
ScheduledTime |
When the task was last evaluated |
ErrMsg |
Error message if task failed |
Estimating Completion Time
-- Count remaining pending + running balance tasks
SELECT
State,
COUNT(*) AS task_count
FROM (
-- Parse from SHOW PROC output via application layer or use:
SELECT 'PENDING' AS State -- placeholder; actual query requires SHOW PROC
) t
GROUP BY State;
A practical approach using migration throughput:
-- Step 1: capture TabletNum on the new BE at time T1
-- Step 2: capture again 10 minutes later at T2
-- Step 3: estimate: remaining_tablets / ((T2_tablets - T1_tablets) / 10 min)
-- Example calculation (run in application or Python):
-- T1_tablets = 5000, T2_tablets = 8500 (10 min gap)
-- rate = (8500 - 5000) / 10 = 350 tablets/min
-- target_tablets = 50000 (avg * num_bes / new_be_count)
-- remaining = 50000 - 8500 = 41500
-- ETA = 41500 / 350 = ~118 minutes
-- Monitor rate in real time:
SELECT BackendId, Host, TabletNum, NOW() AS sample_time
FROM information_schema.be_configs
WHERE Alive = 'true'
ORDER BY BackendId;
-- Run this query at T1 and T2; compute the delta.
Prometheus Metrics for Migration
Scrape BE metrics from http://be_host:8040/metrics:
| Metric | Type | Description |
|---|---|---|
starrocks_be_tablet_migration_count |
gauge | Number of tablet migrations currently in progress on this BE |
starrocks_be_clone_task_total |
counter | Cumulative clone tasks completed (includes both repair and balance) |
starrocks_be_clone_bytes_total |
counter | Cumulative bytes transferred by clone operations |
starrocks_be_disk_data_used_capacity |
gauge | path label — bytes per disk path; track per-path fill rate |
starrocks_be_tablet_num |
gauge | Tablet count on this BE; converging to cluster mean = balance in progress |
# Grafana alert rule — migration throughput stalled
- alert: StarRocksBalanceStalled
expr: >
increase(starrocks_be_clone_task_total[30m]) == 0
and
starrocks_be_tablet_num < (
avg(starrocks_be_tablet_num) * 0.7
)
for: 30m
labels:
severity: warning
annotations:
summary: "BE {{ $labels.instance }} has low tablet count and no recent clone activity"
Capacity Forecasting
Project days until a BE (or the cluster) reaches a disk usage threshold using historical growth from information_schema.
Days-to-Full Calculation
This query compares current data size with a reference point. For a single-sample forecast, use the ingestion rate from your load monitoring; for a two-point estimate, run the query twice and compute the delta in your application layer.
-- Current storage snapshot for capacity planning
SELECT
BackendId,
Host,
ROUND(DataUsedCapacity / 1073741824.0, 2) AS data_used_gb,
ROUND(TotalCapacity / 1073741824.0, 2) AS total_gb,
ROUND(AvailCapacity / 1073741824.0, 2) AS avail_gb,
ROUND(UsedPct, 2) AS used_pct,
ROUND(MaxDiskUsedPct, 2) AS max_disk_pct,
-- At 80% threshold: how many GB of headroom remains?
ROUND(
(TotalCapacity * 0.80 - DataUsedCapacity) / 1073741824.0,
2
) AS headroom_to_80pct_gb
FROM information_schema.be_configs
WHERE Alive = 'true'
ORDER BY MaxDiskUsedPct DESC;
-- Two-point forecast (requires storing a snapshot in a staging table)
-- Schema: CREATE TABLE sr_be_snapshots (
-- snapshot_time DATETIME,
-- backend_id BIGINT,
-- data_used_bytes BIGINT
-- );
-- After inserting two snapshots spaced N hours apart:
WITH snapshots AS (
SELECT
backend_id,
MAX(CASE WHEN snapshot_time = (SELECT MAX(snapshot_time) FROM sr_be_snapshots)
THEN data_used_bytes END) AS latest_bytes,
MAX(CASE WHEN snapshot_time = (SELECT MIN(snapshot_time) FROM sr_be_snapshots)
THEN data_used_bytes END) AS earliest_bytes,
TIMESTAMPDIFF(
HOUR,
MIN(snapshot_time),
MAX(snapshot_time)
) AS hours_elapsed
FROM sr_be_snapshots
GROUP BY backend_id
),
growth AS (
SELECT
s.backend_id,
b.Host,
s.latest_bytes,
b.TotalCapacity,
ROUND((s.latest_bytes - s.earliest_bytes) / NULLIF(s.hours_elapsed, 0), 0)
AS bytes_per_hour
FROM snapshots s
JOIN information_schema.be_configs b ON b.BackendId = s.backend_id
WHERE b.Alive = 'true'
)
SELECT
backend_id,
Host,
ROUND(latest_bytes / 1073741824.0, 2) AS current_data_gb,
ROUND(TotalCapacity / 1073741824.0, 2) AS total_gb,
ROUND(bytes_per_hour / 1073741824.0, 4) AS growth_gb_per_hour,
-- Days until 85% full
ROUND(
(TotalCapacity * 0.85 - latest_bytes)
/ NULLIF(bytes_per_hour, 0) / 24.0,
1
) AS days_to_85pct
FROM growth
ORDER BY days_to_85pct ASC NULLS LAST;
Rule of thumb: trigger a capacity expansion when days_to_85pct < 30 on any BE.
Rebalance After BE Failure
When a BE goes dead (process crash, host failure, network partition), the tablet scheduler immediately begins repair — prioritized above normal balance tasks.
Failure Recovery Sequence
BE fails
│
├─ FE heartbeat timeout (default: 30 s) → BE marked Alive = false
│
├─ Tablet scheduler identifies all tablets with a replica on the dead BE
│ → these become UNDER_REPLICATED (2 replicas remain from 3-replica tables)
│
├─ Repair tasks are generated and dispatched to healthy BEs
│ → each repair task clones a replica from a live replica to a healthy BE
│
└─ After repair: all tablets restored to replication_num replicas
→ balance tasks may re-run afterward to normalize distribution
Monitoring Repair After Failure
-- Check how many tablets are under-replicated
SELECT
TABLET_STATUS,
COUNT(*) AS cnt
FROM information_schema.tablets
GROUP BY TABLET_STATUS
ORDER BY cnt DESC;
-- NORMAL: all replicas healthy
-- UNDER_REPLICATED: one or more replicas missing
-- REPLICA_MISSING: all replicas unavailable (data loss risk)
-- Identify which tables have under-replicated tablets
SELECT
TABLE_NAME,
COUNT(*) AS unhealthy_count
FROM information_schema.tablets
WHERE TABLET_STATUS != 'NORMAL'
GROUP BY TABLE_NAME
ORDER BY unhealthy_count DESC
LIMIT 20;
-- Monitor repair progress: CloningTabletNum should decrease over time
SHOW PROC '/statistic';
-- Scan the row with DbName = 'Total':
-- CloningTabletNum = tablets currently being cloned (repair in progress)
-- UnhealthyTabletNum = tablets still needing attention
Accelerating Repair After Failure
-- Prioritize repair for a critical table or partition
ADMIN REPAIR TABLE critical_table PARTITION (p2026_01, p2026_02);
-- Check repair priority queue
SHOW PROC '/cluster_balance/pending_tablets';
-- Tasks with Type = 'REPAIR' are scheduled before Type = 'BALANCE' tasks.
Estimating Recovery Time
recovery_time ≈ (under_replicated_tablet_count × avg_tablet_size_gb)
/ (clone_bandwidth_gb_per_sec × max_repair_task_slots)
Example:
5 000 under-replicated tablets × 0.5 GB average = 2 500 GB to clone
Network bandwidth per BE: ~200 MB/s
max_tablet_sched_task_slots_per_drive = 4 (default)
6 healthy BEs × 4 slots × 200 MB/s = 4 800 MB/s = 4.7 GB/s
Recovery ETA ≈ 2 500 / 4.7 ≈ 532 seconds ≈ ~9 minutes (ideal)
Real world: expect 2–5× longer due to I/O contention with live queries.
-- Tune clone slot concurrency if recovery is too slow (BE config via HTTP API)
-- Increasing this speeds repair but competes with query I/O.
-- (This is a BE-level config; set on each BE individually)
curl -X POST http://be_host:8040/api/update_config \
-d "max_tablet_sched_task_slots_per_drive=8"
# Default is 4. Raise to 8 on NVMe; keep at 4 on HDD to avoid I/O saturation.
If a BE is Permanently Lost
-- If the BE host is unrecoverable, drop it from the cluster
-- Only do this after confirming the host will never return.
-- StarRocks will re-replicate all affected tablets once there are enough healthy BEs.
ALTER SYSTEM DROP BACKEND 'dead_be_host:9050';
-- Verify the BE is removed
SELECT BackendId, Host, Alive
FROM information_schema.be_configs;
-- Then monitor repair progress
SHOW PROC '/statistic';
Warning: DROP BACKEND is irreversible. If you drop a BE and then replication_num = 1 tablets had their only replica there, that data is permanently lost. Always ensure replication_num >= 3 in production.
Anti-Patterns
Do not kill the BE process without decommissioning first.
Abruptly stopping a BE makes all its tablets UNDER_REPLICATED simultaneously. The repair storm can saturate cluster I/O for minutes to hours. Always run ALTER SYSTEM DECOMMISSION BACKEND and wait for TabletNum = 0 before stopping the process.
Do not decommission a BE when the cluster has no capacity headroom.
If the remaining BEs do not have enough free space to absorb the decommissioned BE's tablets, decommission will stall indefinitely with TabletNum > 0. Add replacement BEs or free capacity first.
Do not set balance_load_score_threshold = 0 or near-zero.
This causes the scheduler to migrate tablets continuously for any small imbalance, generating constant background I/O and making the cluster difficult to observe. The minimum practical value is 0.05.
Do not raise max_balancing_tablets to very high values during peak query hours.
Tablet migration competes for disk I/O and network bandwidth with scan queries. Keep max_balancing_tablets <= 100 during business hours; increase to 300–500 only in off-peak maintenance windows.
Do not forget to re-enable balance after a maintenance pause.
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true") is easy to forget. Always add a scheduled step in your runbook to re-enable it. Tablets accumulate on restarted BEs, creating hotspots that degrade query performance.
Do not use ALTER SYSTEM DROP BACKEND to remove a healthy BE.
DROP BACKEND does NOT migrate tablets first. Use DECOMMISSION BACKEND for graceful removal. Use DROP BACKEND only for permanently dead hosts that will never rejoin.
Do not add all new BEs simultaneously to a small cluster during business hours. Adding N new BEs simultaneously generates N × avg_tablets migrations at once. This can briefly saturate disk I/O. In production, stagger BE additions by 15–30 minutes to spread the initial migration wave.
Do not rely on tablet count alone to judge balance.
A cluster with uniform tablet counts but highly skewed tablet sizes will show disk imbalance despite equal TabletNum. Always check both TabletNum spread and UsedPct spread together.
References to Consult When Needed
- StarRocks docs — Tablet scheduler internals:
https://docs.starrocks.io/docs/administration/management/tablet_management/ - StarRocks docs — Scale up/down (add/remove BE):
https://docs.starrocks.io/docs/administration/management/Scale_up_down/ - StarRocks docs — FE configuration reference:
https://docs.starrocks.io/docs/administration/management/FE_configuration/ - StarRocks docs — BE configuration reference:
https://docs.starrocks.io/docs/administration/management/BE_configuration/ - StarRocks docs — Monitoring metrics (Prometheus):
https://docs.starrocks.io/docs/administration/management/monitoring/metrics/ - StarRocks docs — ADMIN commands reference:
https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/ - StarRocks docs —
information_schematables:https://docs.starrocks.io/docs/reference/information_schema/ - StarRocks docs — Cluster management overview:
https://docs.starrocks.io/docs/administration/management/cluster_management/