Starrocks Admin Cluster Health

StarRocks Cluster Health Monitoring

When to Use

Load this skill when the user needs to:

• Perform daily health checks on a StarRocks cluster (FE/BE/CN liveness, disk usage, tablet health)
• Triage incidents: dead nodes, query failures, high latency, replication lag, compaction pressure
• Plan capacity expansion: disk usage trends, tablet distribution imbalance, BE load scores
• Verify FE quorum after a rolling restart or network partition
• Investigate under-replicated or missing tablets following a BE crash or network event
• Monitor compaction backlog that may degrade query performance or trigger write stalls
• Build Prometheus/Grafana alerting on StarRocks cluster health metrics

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    FE Layer (Metadata)                       │
│  Leader FE ──► Follower FE ──► Follower FE                  │
│  (1 leader, 0–N followers, 0–N observers)                   │
│  BDB-JE edit log replicated via Raft-like consensus          │
└──────────────────────────┬──────────────────────────────────┘
                           │  Heartbeat + Tablet Report
┌──────────────────────────▼──────────────────────────────────┐
│              BE/CN Layer (Storage & Compute)                 │
│                                                              │
│  Shared-Nothing mode: each BE owns local disk storage        │
│    BE-1  BE-2  BE-3  ...  BE-N                               │
│    tablets replicated across BEs (default replication = 3)  │
│                                                              │
│  Shared-Data mode (Lake): CN nodes are stateless compute,    │
│    storage is remote object store (S3/HDFS/OSS/COS/GCS)     │
│    CN-1  CN-2  CN-3  ...  CN-N                               │
└─────────────────────────────────────────────────────────────┘

Key concepts:

• FE Leader: accepts DDL writes, coordinates tablet scheduling, runs global metadata.
• FE Follower: participates in quorum; can serve reads and DML.
• FE Observer: read-only replica; does NOT count toward quorum.
• Tablet: the unit of storage and replication. A table is split into tablets; each tablet has N replicas (usually 3) spread across BEs.
• Quorum rule: majority of Follower FEs must be alive. With 3 Followers → need 2 alive; with 5 → need 3.
• CN (Compute Node): shared-data only; no persistent local tablet storage.

FE Health Checks

SHOW FRONTENDS

-- Run from any MySQL-compatible client connected to FE QueryPort (default 9030)
SHOW FRONTENDS\G

Key columns returned:

-- Identify non-alive FEs
SELECT Name, Host, Role, Alive, ErrMsg
FROM information_schema.fe_metrics   -- available in SR 3.1+
WHERE Alive = 'false';

-- Alternatively via SHOW PROC
SHOW PROC '/frontends';

FE Quorum Check

-- Count alive Followers (Observers excluded from quorum)
SELECT
    Role,
    COUNT(*) AS total,
    SUM(CASE WHEN Alive = 'true' THEN 1 ELSE 0 END) AS alive_count
FROM (
    SELECT
        TRIM(Role)  AS Role,
        TRIM(Alive) AS Alive
    FROM information_schema.frontends   -- SR 3.2+; use SHOW FRONTENDS in older versions
) t
WHERE Role IN ('LEADER', 'FOLLOWER')
GROUP BY Role;

Expected: alive_count for (LEADER + FOLLOWER) combined >= ceil((total_followers + 1) / 2).

# Shell one-liner — exits non-zero if fewer than 2 out of 3 FEs are alive
mysql -h fe_host -P 9030 -u root --batch --skip-column-names \
  -e "SHOW FRONTENDS" \
  | awk -F'\t' '$10=="true" && $7!="OBSERVER"' \
  | wc -l \
  | xargs -I{} test {} -ge 2

Edit Log Lag Detection

A Follower whose ReplayedJournalId is far behind the Leader is applying metadata slowly — potential sign of GC pressure or I/O bottleneck on that FE host.

-- Fetch ReplayedJournalId for all FEs (column index 12 in SHOW FRONTENDS)
SHOW FRONTENDS;
-- Compare the Leader's ReplayedJournalId with each Follower's value.
-- A lag > 10 000 journals is worth investigating.

BE Health Checks

SHOW BACKENDS

SHOW BACKENDS\G

Key columns:

-- Dead BE detection
SHOW BACKENDS\G
-- Look for Alive = false rows.

-- Query-based dead BE check
SELECT BackendId, Host, Alive, TabletNum, ErrMsg
FROM information_schema.be_configs   -- use SHOW BACKENDS in older versions
WHERE Alive = 'false';

Disk Usage Alerts

-- Via SHOW BACKENDS output (parse in application layer or script)
-- Thresholds:
--   UsedPct > 75%  => WARNING  (consider rebalancing or adding BEs)
--   UsedPct > 85%  => CRITICAL (compaction stalls likely; add capacity immediately)
--   MaxDiskUsedPct > 90% => EMERGENCY (StarRocks may stop writing to that disk)

-- SQL approach using information_schema (SR 3.1+)
SELECT
    BackendId,
    Host,
    ROUND(DataUsedCapacity / 1073741824, 2)  AS data_used_gb,
    ROUND(TotalCapacity    / 1073741824, 2)  AS total_gb,
    ROUND(UsedPct, 2)                        AS used_pct,
    ROUND(MaxDiskUsedPct, 2)                 AS max_disk_pct,
    CASE
        WHEN MaxDiskUsedPct > 85 THEN 'CRITICAL'
        WHEN MaxDiskUsedPct > 75 THEN 'WARNING'
        ELSE 'OK'
    END AS disk_status
FROM information_schema.be_configs
ORDER BY MaxDiskUsedPct DESC;

# SHOW PROC for detailed per-disk breakdown
# Connect via mysql CLI, then:
mysql -h fe_host -P 9030 -u root -e "SHOW PROC '/backends'"

Tablet Health

Tablets are the fundamental data unit. Each tablet has N replicas. The FE tablet scheduler continuously monitors replica health.

Global Tablet Statistics

-- Overall cluster tablet health summary
SHOW PROC '/statistic';

Columns returned by SHOW PROC '/statistic':

-- Focus on cluster-wide totals
SHOW PROC '/statistic';
-- The last row with DbName='Total' gives aggregates.
-- Healthy cluster: UnhealthyTabletNum = 0, InconsistentTabletNum = 0.

Finding Unhealthy Tablets

-- information_schema.tablets — available SR 2.5+
SELECT
    TABLE_NAME,
    PARTITION_NAME,
    TABLET_ID,
    BACKEND_ID,
    TABLET_STATUS,
    REPLICA_STATUS
FROM information_schema.tablets
WHERE TABLET_STATUS != 'NORMAL'
   OR REPLICA_STATUS != 'NORMAL'
ORDER BY TABLE_NAME, TABLET_ID
LIMIT 200;

-- Drill into a specific tablet
SHOW TABLET 123456789;
-- Returns: DbName, TableName, PartitionName, IndexName, DbId, TableId,
--          PartitionId, IndexId, IsSync, Order, DetailCmd

-- Execute the DetailCmd to see per-replica detail
SHOW PROC '/dbs/<DbId>/<TableId>/partitions/<PartitionId>/<IndexId>/<TabletId>';

Manual Repair Trigger

-- Force the tablet scheduler to prioritize repair of a table/partition
ADMIN REPAIR TABLE sales_fact PARTITION (p2024_01, p2024_02);

-- Check repair progress
SHOW PROC '/statistic';   -- CloningTabletNum decreases as repair completes

-- Cancel repair priority (returns to normal scheduler priority)
ADMIN CANCEL REPAIR TABLE sales_fact PARTITION (p2024_01);

Replica Status

Problem Replicas

-- List tablets with replication problems
SHOW PROC '/replicas/problems';
-- Returns tablets that are: UNDER_REPLICATED, VERSION_INCOMPLETE, COLOCATE_MISMATCH,
-- REPLICA_MISSING, REPLICA_RELOCATING, REDUNDANT, etc.

-- Under-replicated: fewer alive replicas than the replication_num configured for the table
-- Missing: all replicas are dead/unreachable

Manual Replica Status Override

Use ADMIN SET REPLICA STATUS only as a last resort when the scheduler is stuck:

-- Mark a specific replica as BAD so the scheduler will drop and re-clone it
ADMIN SET REPLICA STATUS
    PROPERTIES(
        "tablet_id" = "123456789",
        "backend_id" = "10003",
        "status" = "bad"
    );

-- Mark as OK if a replica was incorrectly flagged
ADMIN SET REPLICA STATUS
    PROPERTIES(
        "tablet_id" = "123456789",
        "backend_id" = "10003",
        "status" = "ok"
    );

-- Count replicas by status across the cluster (SR 3.1+)
SELECT
    REPLICA_STATUS,
    COUNT(*) AS cnt
FROM information_schema.tablets
GROUP BY REPLICA_STATUS
ORDER BY cnt DESC;

Compaction Backlog

StarRocks uses LSM-style compaction to merge small rowsets into larger ones. High compaction backlog indicates write pressure and degrades scan performance.

Compaction Overview

-- Cluster-wide compaction status
SHOW PROC '/compactions';
-- Returns per-BE compaction stats including:
--   BeId, CumulativeCompactionTaskNum, BaseCompactionTaskNum,
--   CumulativeCompactionScore, BaseCompactionScore

High cumulative score means small rowsets are accumulating faster than compaction can merge them — usually caused by high-frequency small writes (many small INSERT batches). High base score usually means large rowsets haven't been merged into the base rowset recently.

Compaction Tuning (BE config — set in be.conf or via HTTP API)

# View current compaction config on a BE
curl http://be_host:8040/api/show_config | grep -i compaction

# Increase parallelism for cumulative compaction (BE config, requires restart or dynamic set)
curl -X POST http://be_host:8040/api/update_config \
  -d "cumulative_compaction_num_threads_per_disk=4"

curl -X POST http://be_host:8040/api/update_config \
  -d "base_compaction_num_threads_per_disk=2"

# Lower the score threshold that triggers base compaction
curl -X POST http://be_host:8040/api/update_config \
  -d "base_compaction_check_interval_seconds=60"

-- Per-tablet compaction info (SR 3.1+)
SELECT
    BACKEND_ID,
    COUNT(*) AS tablet_count,
    MAX(ROW_COUNT) AS max_rows_per_tablet
FROM information_schema.tablets
WHERE TABLET_STATUS = 'NORMAL'
GROUP BY BACKEND_ID
ORDER BY tablet_count DESC;

Storage Balance

StarRocks automatically balances tablets across BEs based on disk usage and tablet count. Imbalance increases after adding new BEs or recovering from a BE failure.

Balance Status

-- View ongoing tablet migration and balance operations
SHOW PROC '/cluster_balance';
-- Sub-paths:
--   /cluster_balance/overview      — summary of pending/running migrations
--   /cluster_balance/pending_tablets   — tablets waiting to be moved
--   /cluster_balance/running_tablets   — tablets currently being cloned for balance
--   /cluster_balance/history_tablets   — recently completed migrations (ring buffer)

SHOW PROC '/cluster_balance/overview';
-- Key fields: TotalScheduledTabletNum, TotalRunningTabletNum, BalanceSlotNumPerPath

-- Detect imbalanced BEs — large spread in TabletNum or DataUsedCapacity
SELECT
    BackendId,
    Host,
    TabletNum,
    ROUND(DataUsedCapacity / 1073741824, 2) AS data_gb,
    ROUND(UsedPct, 2) AS used_pct
FROM information_schema.be_configs
ORDER BY TabletNum DESC;

-- A healthy cluster has tablet counts within ~20% of the mean.
-- Compute coefficient of variation in application code or:
SELECT
    MIN(TabletNum) AS min_tablets,
    MAX(TabletNum) AS max_tablets,
    ROUND(AVG(TabletNum), 0) AS avg_tablets,
    ROUND((MAX(TabletNum) - MIN(TabletNum)) * 100.0 / AVG(TabletNum), 1) AS spread_pct
FROM information_schema.be_configs
WHERE Alive = 'true';
-- spread_pct > 50% after 30 minutes of balance activity warrants investigation.

Balance Configuration (FE dynamic parameters)

-- View current FE configuration (filter for balance-related params)
ADMIN SHOW FRONTEND CONFIG LIKE '%balance%';

-- Adjust balance aggressiveness (default 0.1; higher = more aggressive rebalancing)
ADMIN SET FRONTEND CONFIG ("balance_load_score_threshold" = "0.1");

-- Maximum concurrent tablet clone tasks per BE
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "100");

-- Disable balance temporarily (e.g., during maintenance window)
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
-- Re-enable:
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");

Cluster Health Summary Query

A single query that aggregates key health signals into one result set for dashboards or automated checks:

-- Run against FE QueryPort; requires information_schema access
-- Returns one row per health dimension with status and detail

WITH fe_health AS (
    SELECT
        'FE_ALIVE' AS metric,
        CAST(SUM(CASE WHEN Alive = 'true' THEN 1 ELSE 0 END) AS VARCHAR) AS value,
        CAST(COUNT(*) AS VARCHAR) AS total,
        CASE
            WHEN SUM(CASE WHEN Alive = 'true' AND Role != 'OBSERVER' THEN 1 ELSE 0 END)
                 < CEIL((SUM(CASE WHEN Role != 'OBSERVER' THEN 1 ELSE 0 END) + 1.0) / 2)
            THEN 'CRITICAL'
            WHEN SUM(CASE WHEN Alive = 'false' THEN 1 ELSE 0 END) > 0 THEN 'WARNING'
            ELSE 'OK'
        END AS status
    FROM information_schema.frontends
),
be_alive AS (
    SELECT
        'BE_ALIVE' AS metric,
        CAST(SUM(CASE WHEN Alive = 'true' THEN 1 ELSE 0 END) AS VARCHAR) AS value,
        CAST(COUNT(*) AS VARCHAR) AS total,
        CASE
            WHEN SUM(CASE WHEN Alive = 'false' THEN 1 ELSE 0 END) > 0 THEN 'CRITICAL'
            ELSE 'OK'
        END AS status
    FROM information_schema.be_configs
),
disk_health AS (
    SELECT
        'DISK_MAX_USED_PCT' AS metric,
        CAST(ROUND(MAX(MaxDiskUsedPct), 1) AS VARCHAR) AS value,
        '100' AS total,
        CASE
            WHEN MAX(MaxDiskUsedPct) > 85 THEN 'CRITICAL'
            WHEN MAX(MaxDiskUsedPct) > 75 THEN 'WARNING'
            ELSE 'OK'
        END AS status
    FROM information_schema.be_configs
    WHERE Alive = 'true'
),
tablet_health AS (
    SELECT
        'UNHEALTHY_TABLETS' AS metric,
        CAST(SUM(CASE WHEN TABLET_STATUS != 'NORMAL' THEN 1 ELSE 0 END) AS VARCHAR) AS value,
        CAST(COUNT(*) AS VARCHAR) AS total,
        CASE
            WHEN SUM(CASE WHEN TABLET_STATUS != 'NORMAL' THEN 1 ELSE 0 END) > 0 THEN 'WARNING'
            ELSE 'OK'
        END AS status
    FROM information_schema.tablets
),
balance_check AS (
    SELECT
        'BE_TABLET_SPREAD_PCT' AS metric,
        CAST(ROUND((MAX(TabletNum) - MIN(TabletNum)) * 100.0
                    / NULLIF(AVG(TabletNum), 0), 1) AS VARCHAR) AS value,
        '50' AS total,
        CASE
            WHEN (MAX(TabletNum) - MIN(TabletNum)) * 100.0
                    / NULLIF(AVG(TabletNum), 0) > 50 THEN 'WARNING'
            ELSE 'OK'
        END AS status
    FROM information_schema.be_configs
    WHERE Alive = 'true'
)
SELECT metric, value, total, status
FROM fe_health
UNION ALL SELECT metric, value, total, status FROM be_alive
UNION ALL SELECT metric, value, total, status FROM disk_health
UNION ALL SELECT metric, value, total, status FROM tablet_health
UNION ALL SELECT metric, value, total, status FROM balance_check
ORDER BY FIELD(status, 'CRITICAL', 'WARNING', 'OK'), metric;

Prometheus Metrics

StarRocks FE and BE expose metrics at /metrics (HTTP) for Prometheus scraping.

FE Metrics (port 8030 by default)

BE Metrics (port 8040 by default)

Prometheus Scrape Configuration

# prometheus.yml scrape config
scrape_configs:
  - job_name: starrocks_fe
    metrics_path: /metrics
    static_configs:
      - targets:
          - fe1:8030
          - fe2:8030
          - fe3:8030

  - job_name: starrocks_be
    metrics_path: /metrics
    static_configs:
      - targets:
          - be1:8040
          - be2:8040
          - be3:8040

Key Alertmanager Rules

# alerts/starrocks.yml
groups:
  - name: starrocks_cluster
    rules:
      - alert: StarRocksFEDown
        expr: up{job="starrocks_fe"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "FE node {{ $labels.instance }} is down"

      - alert: StarRocksBEDown
        expr: up{job="starrocks_be"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "BE node {{ $labels.instance }} is down"

      - alert: StarRocksDiskUsageWarning
        expr: >
          starrocks_be_disk_used_capacity / starrocks_be_disk_total_capacity > 0.75
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "BE {{ $labels.instance }} disk {{ $labels.path }} usage > 75%"

      - alert: StarRocksDiskUsageCritical
        expr: >
          starrocks_be_disk_used_capacity / starrocks_be_disk_total_capacity > 0.85
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "BE {{ $labels.instance }} disk {{ $labels.path }} usage > 85%"

      - alert: StarRocksHighCompactionScore
        expr: starrocks_be_compaction_score{type="cumulative"} > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "BE {{ $labels.instance }} cumulative compaction score > 1000"

      - alert: StarRocksQueryErrorRate
        expr: >
          rate(starrocks_fe_query_err[5m])
          / rate(starrocks_fe_query_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "FE {{ $labels.instance }} query error rate > 5%"

Health Check Script

Production-ready Python script that runs SHOW commands, evaluates thresholds, and exits non-zero on critical findings. Suitable for cron, Airflow PythonOperator, or a liveness probe.

#!/usr/bin/env python3
"""
StarRocks cluster health check script.

Exit codes:
  0  = OK (all thresholds within bounds)
  1  = WARNING (soft threshold breached, not yet critical)
  2  = CRITICAL (hard threshold breached, immediate action required)

Usage:
  python starrocks_health_check.py \
      --host fe1.internal \
      --port 9030 \
      --user root \
      --password secret \
      [--warn-disk 75] \
      [--crit-disk 85] \
      [--crit-unhealthy-tablets 0]
"""

import argparse
import sys
from dataclasses import dataclass, field
from typing import List, Tuple

try:
    import mysql.connector
except ImportError:
    print("CRITICAL: mysql-connector-python not installed. Run: pip install mysql-connector-python")
    sys.exit(2)


# ---------------------------------------------------------------------------
# Data containers
# ---------------------------------------------------------------------------

@dataclass
class CheckResult:
    name: str
    status: str          # 'OK', 'WARNING', 'CRITICAL'
    message: str


@dataclass
class HealthReport:
    results: List[CheckResult] = field(default_factory=list)

    def add(self, result: CheckResult) -> None:
        self.results.append(result)

    @property
    def worst_status(self) -> str:
        priority = {"CRITICAL": 2, "WARNING": 1, "OK": 0}
        return max(self.results, key=lambda r: priority.get(r.status, 0)).status

    @property
    def exit_code(self) -> int:
        return {"OK": 0, "WARNING": 1, "CRITICAL": 2}.get(self.worst_status, 2)

    def print_summary(self) -> None:
        priority = {"CRITICAL": 2, "WARNING": 1, "OK": 0}
        for r in sorted(self.results, key=lambda x: -priority.get(x.status, 0)):
            print(f"[{r.status:<8}] {r.name}: {r.message}")


# ---------------------------------------------------------------------------
# Checks
# ---------------------------------------------------------------------------

def check_fe_quorum(cursor) -> CheckResult:
    cursor.execute(
        "SELECT Role, Alive FROM information_schema.frontends"
    )
    rows = cursor.fetchall()
    quorum_nodes = [(r["Role"], r["Alive"]) for r in rows if r["Role"] != "OBSERVER"]
    total = len(quorum_nodes)
    alive = sum(1 for _, a in quorum_nodes if str(a).lower() in ("true", "1", "yes"))
    needed = (total // 2) + 1
    dead = total - alive
    if alive < needed:
        return CheckResult(
            "FE_QUORUM",
            "CRITICAL",
            f"Only {alive}/{total} quorum FEs alive (need {needed}). LEADER election at risk.",
        )
    if dead > 0:
        return CheckResult(
            "FE_QUORUM",
            "WARNING",
            f"{dead} FE(s) are down but quorum maintained ({alive}/{total}).",
        )
    return CheckResult("FE_QUORUM", "OK", f"All {total} quorum FEs alive.")


def check_be_alive(cursor) -> CheckResult:
    cursor.execute(
        "SELECT BackendId, Host, Alive, ErrMsg FROM information_schema.be_configs"
    )
    rows = cursor.fetchall()
    dead = [r for r in rows if str(r["Alive"]).lower() not in ("true", "1", "yes")]
    if dead:
        detail = ", ".join(f"{r['Host']} (id={r['BackendId']})" for r in dead[:5])
        return CheckResult(
            "BE_ALIVE",
            "CRITICAL",
            f"{len(dead)} BE(s) dead: {detail}",
        )
    return CheckResult("BE_ALIVE", "OK", f"All {len(rows)} BEs alive.")


def check_disk_usage(
    cursor,
    warn_pct: float = 75.0,
    crit_pct: float = 85.0,
) -> CheckResult:
    cursor.execute(
        "SELECT BackendId, Host, MaxDiskUsedPct FROM information_schema.be_configs WHERE Alive='true'"
    )
    rows = cursor.fetchall()
    critical_bes = [r for r in rows if float(r["MaxDiskUsedPct"]) > crit_pct]
    warning_bes  = [r for r in rows if crit_pct >= float(r["MaxDiskUsedPct"]) > warn_pct]

    if critical_bes:
        detail = ", ".join(
            f"{r['Host']}={float(r['MaxDiskUsedPct']):.1f}%" for r in critical_bes[:3]
        )
        return CheckResult("DISK_USAGE", "CRITICAL", f"BEs above {crit_pct}%: {detail}")
    if warning_bes:
        detail = ", ".join(
            f"{r['Host']}={float(r['MaxDiskUsedPct']):.1f}%" for r in warning_bes[:3]
        )
        return CheckResult("DISK_USAGE", "WARNING", f"BEs above {warn_pct}%: {detail}")
    max_pct = max(float(r["MaxDiskUsedPct"]) for r in rows) if rows else 0.0
    return CheckResult("DISK_USAGE", "OK", f"Max disk usage {max_pct:.1f}%.")


def check_tablet_health(cursor, crit_threshold: int = 0) -> CheckResult:
    cursor.execute(
        "SELECT COUNT(*) AS cnt FROM information_schema.tablets WHERE TABLET_STATUS != 'NORMAL'"
    )
    row = cursor.fetchone()
    count = int(row["cnt"])
    if count > crit_threshold:
        return CheckResult(
            "TABLET_HEALTH",
            "WARNING" if count <= 10 else "CRITICAL",
            f"{count} unhealthy tablet(s). Run: SELECT * FROM information_schema.tablets "
            f"WHERE TABLET_STATUS != 'NORMAL' LIMIT 20;",
        )
    return CheckResult("TABLET_HEALTH", "OK", "All tablets healthy.")


def check_replica_status(cursor) -> CheckResult:
    cursor.execute(
        "SELECT REPLICA_STATUS, COUNT(*) AS cnt FROM information_schema.tablets "
        "GROUP BY REPLICA_STATUS"
    )
    rows = cursor.fetchall()
    status_map = {r["REPLICA_STATUS"]: int(r["cnt"]) for r in rows}
    bad_statuses = {k: v for k, v in status_map.items() if k not in ("NORMAL", "OK")}
    if bad_statuses:
        detail = ", ".join(f"{k}={v}" for k, v in bad_statuses.items())
        total_bad = sum(bad_statuses.values())
        severity = "CRITICAL" if total_bad > 50 else "WARNING"
        return CheckResult("REPLICA_STATUS", severity, f"Non-normal replicas: {detail}")
    return CheckResult("REPLICA_STATUS", "OK", "All replicas have normal status.")


def check_storage_balance(cursor, spread_warn_pct: float = 50.0) -> CheckResult:
    cursor.execute(
        "SELECT MIN(TabletNum) AS mn, MAX(TabletNum) AS mx, AVG(TabletNum) AS avg_t "
        "FROM information_schema.be_configs WHERE Alive='true'"
    )
    row = cursor.fetchone()
    mn, mx, avg_t = float(row["mn"]), float(row["mx"]), float(row["avg_t"])
    if avg_t == 0:
        return CheckResult("STORAGE_BALANCE", "OK", "No tablets to balance.")
    spread = (mx - mn) * 100.0 / avg_t
    if spread > spread_warn_pct:
        return CheckResult(
            "STORAGE_BALANCE",
            "WARNING",
            f"Tablet spread {spread:.1f}% (min={int(mn)}, max={int(mx)}, avg={avg_t:.0f}). "
            "Consider rebalancing.",
        )
    return CheckResult("STORAGE_BALANCE", "OK", f"Tablet spread {spread:.1f}% (OK).")


# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------

def parse_args() -> argparse.Namespace:
    p = argparse.ArgumentParser(description="StarRocks cluster health check")
    p.add_argument("--host",  default="localhost", help="FE host")
    p.add_argument("--port",  type=int, default=9030, help="FE MySQL port")
    p.add_argument("--user",  default="root")
    p.add_argument("--password", default="")
    p.add_argument("--warn-disk", type=float, default=75.0, help="Disk usage warning %")
    p.add_argument("--crit-disk", type=float, default=85.0, help="Disk usage critical %")
    p.add_argument("--crit-unhealthy-tablets", type=int, default=0)
    p.add_argument("--balance-warn-spread", type=float, default=50.0)
    return p.parse_args()


def main() -> None:
    args = parse_args()
    report = HealthReport()

    try:
        conn = mysql.connector.connect(
            host=args.host,
            port=args.port,
            user=args.user,
            password=args.password,
            database="information_schema",
            connection_timeout=10,
        )
        cursor = conn.cursor(dictionary=True)
    except Exception as exc:
        print(f"[CRITICAL ] CONNECTION: Cannot connect to {args.host}:{args.port} — {exc}")
        sys.exit(2)

    try:
        checks: List[Tuple] = [
            (check_fe_quorum,       (cursor,)),
            (check_be_alive,        (cursor,)),
            (check_disk_usage,      (cursor, args.warn_disk, args.crit_disk)),
            (check_tablet_health,   (cursor, args.crit_unhealthy_tablets)),
            (check_replica_status,  (cursor,)),
            (check_storage_balance, (cursor, args.balance_warn_spread)),
        ]
        for fn, fn_args in checks:
            try:
                report.add(fn(*fn_args))
            except Exception as exc:
                report.add(CheckResult(fn.__name__, "WARNING", f"Check failed: {exc}"))
    finally:
        cursor.close()
        conn.close()

    report.print_summary()
    sys.exit(report.exit_code)


if __name__ == "__main__":
    main()

Run the script:

# Install dependency
pip install mysql-connector-python

# Basic run
python starrocks_health_check.py \
    --host fe1.internal \
    --port 9030 \
    --user root \
    --password "${SR_PASSWORD}" \
    --warn-disk 75 \
    --crit-disk 85

# Integrate with Airflow
# In a DAG: BashOperator(bash_command="python /opt/scripts/starrocks_health_check.py --host ...")
# Task fails automatically on exit code != 0.

# Example output:
# [CRITICAL ] BE_ALIVE: 1 BE(s) dead: be3.internal (id=10005)
# [WARNING  ] DISK_USAGE: BEs above 75%: be1.internal=78.3%, be2.internal=76.1%
# [OK       ] FE_QUORUM: All 3 quorum FEs alive.
# [OK       ] TABLET_HEALTH: All tablets healthy.
# [OK       ] REPLICA_STATUS: All replicas have normal status.
# [OK       ] STORAGE_BALANCE: Tablet spread 12.4% (OK).

Quick Reference — SHOW PROC Paths

Anti-Patterns

Do not run ADMIN REPAIR TABLE cluster-wide without narrowing to a specific partition. ADMIN REPAIR TABLE t without a PARTITION clause escalates priority for all partitions simultaneously, flooding the tablet scheduler with clone tasks and degrading query performance.

Do not use ADMIN SET REPLICA STATUS ... status=bad on large numbers of replicas at once. Marking dozens of replicas bad in rapid succession causes a clone storm. Mark one or two, confirm re-clone completes, then continue.

Do not disable balance (disable_balance=true) and forget to re-enable it. After maintenance windows, operators frequently forget to re-enable balancing. Tablets pile up on restarted BEs with zero load, causing hotspots. Add a scheduled re-enable step to your runbook.

Do not alert only on Alive=false for BEs. A BE can be alive but have MaxDiskUsedPct > 90%, effectively refusing new tablet writes. Always include disk checks alongside liveness checks.

Do not ignore InconsistentTabletNum > 0. Inconsistent tablets have diverging checksums across replicas — silent data corruption. Treat this as CRITICAL and escalate immediately; data verification with ADMIN CHECK TABLE is required.

Do not set max_balancing_tablets to a very large value under production load. Tablet migration competes with user queries for disk I/O and network. Keep max_balancing_tablets <= 100 during peak hours; increase to 500+ only during maintenance windows.

Do not rely solely on SHOW PROC in automation scripts. SHOW PROC output format can shift between minor SR versions. Prefer information_schema tables (be_configs, frontends, tablets) which have stable schemas. Fall back to SHOW PROC only for data not yet exposed in information_schema.

References to Consult When Needed

• StarRocks docs — Cluster Management: https://docs.starrocks.io/docs/administration/management/cluster_management/
• StarRocks docs — Monitoring Metrics: https://docs.starrocks.io/docs/administration/management/monitoring/metrics/
• StarRocks docs — System Variables: https://docs.starrocks.io/docs/reference/System_variable/
• StarRocks docs — information_schema tables: https://docs.starrocks.io/docs/reference/information_schema/
• StarRocks docs — ADMIN commands reference: https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/
• StarRocks docs — Tablet Scheduler internals: https://docs.starrocks.io/docs/administration/management/tablet_management/