Doris Debug Cloud

Cloud (Storage-Compute Separation)

Before proceeding

Confirm cloud mode. Shared-nothing cluster commands (disk rebalance, clone, local tablet repair) are meaningless in cloud mode. Check:

SHOW FRONTENDS\G  -- IsCloudMode field (Doris 3.0+)

Or inspect fe.conf for cloud_unique_id or meta_service_endpoint.

Causes

10 min triage

-- Compute group status
SHOW COMPUTE GROUPS\G

-- File cache hit ratio (Doris 3.0+)
SELECT * FROM information_schema.file_cache_stats;

# BE cache metrics
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "file_cache|local_cache"

# Object store errors in BE log
./scripts/doris-debug log-grep be/log --query-id "$QID"
grep -r "SlowDown\|503\|Throughput.*exceed" be/log/

Cause A — Meta-service latency

In cloud mode, SHOW PROC and DDL go through the meta-service layer, not direct BDBJE. Latency spikes often come from:

1. Meta-service RPC overload — check meta_service_rpc_timeout_ms in fe.conf
2. SHOW PROC '/cluster_health' on a large cluster — walks every tablet via meta-service

# Check meta-service health
curl -s "http://$MS_HOST:$MS_PORT/api/health"

Cause B — Cache miss storm

After bulk ingest (INSERT INTO SELECT or Stream Load), the file cache on compute BEs is cold. First query after ingest hits S3 directly — expect 3-10× latency.

-- Pre-warm cache after bulk ingest
SELECT COUNT(*) FROM new_table;  -- full scan populates file cache

Tune cache capacity:

# be.conf — file cache in cloud mode
file_cache_path = [{"path":"/nvme0/file_cache","total_size":536870912000,"query_limit":107374182400}]
file_cache_type = whole_file_cache  # or sub_file_cache for column-level

Cause C — Object store throttling

S3 has per-prefix request rate limits (~3500 PUT/GET per second per prefix). For high-concurrency cloud workloads:

# be.conf
s3_max_connections = 256          # increase parallel connections
s3_request_timeout_ms = 30000     # increase timeout

Consider sub_file_cache for cold tables with large Parquet files — reduces S3 list/get calls.

Key differences from shared-nothing

Source

• fe/.../cloud/ComputeGroupMgr.java
• fe/.../cloud/MetaServiceClient.java
• be/src/io/fs/s3/S3FileSystem.cpp
• be/src/io/cache/whole_file_cache.cpp
• be/src/io/cache/sub_file_cache.cpp