apache Node Health
Causes
| ID | Cause | Evidence | Source anchor |
|---|---|---|---|
| A | BE OOM / killed | dmesg shows Killed; be.WARNING MemTrackerLimiter |
mem_tracker_limiter.cpp |
| B | FE JVM OOM | GC overhead / heap dump; fe.out shows OutOfMemoryError |
fe.conf JVM opts |
| C | BE crash (SIGSEGV/SIGABRT) | Core dump; be.out stacktrace |
doris_main.cpp signal handler |
| D | False Alive | SHOW BACKENDS Alive=true but queries fail brpc |
heartbeat vs brpc health |
| E | FD exhaustion | Too many open files in be.WARNING |
ulimit -n |
10 min triage
# Memory pressure signals
./scripts/doris-debug log-grep be/log --pack memory
dmesg -T | grep -i "killed process" | tail -5
# FE JVM
jstack $(cat /path/to/fe.pid) > /tmp/fe.jstack
jmap -heap $(cat /path/to/fe.pid) 2>/dev/null | head -30
# FD count
ls /proc/$(cat /path/to/be.pid)/fd | wc -l
cat /proc/$(cat /path/to/be.pid)/limits | grep "open files"
# Resource snapshot
top -b -n 1 -p $(cat /path/to/be.pid) | head -5
free -h
Cause A — BE OOM
Doris tracks memory via MemTracker, but process RSS exceeds tracked memory due to:
- jemalloc arena fragmentation (10-20% overhead)
- brpc buffer pools (outside MemTracker)
- Thread stacks (~8MB each × thread count)
# RSS vs tracked
ps -o rss= -p $BE_PID | awk '{printf "RSS: %.1f GB\n", $1/1024/1024}'
grep "MemTrackerLimiter.*total" be/log/be.WARNING | tail -1
# be.conf — defensive settings
mem_limit = 70% # Leave 30% for overhead
max_segment_cache_size = 0 # Or cap explicitly
enable_je_purge = true # jemalloc dirty page cleanup
If ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc overhead.
Cause B — FE JVM
# FE JVM opts (from fe.conf or start script)
ps aux | grep "PaloFe" | grep -o "\-Xmx[^ ]*"
Common: FE heap grows with catalog metadata (many tables/partitions). Raise -Xmx or reduce hive_meta_cache_ttl_sec / catalog refresh frequency.
Cause D — False Alive
Alive=true means the heartbeat RPC succeeds. This does NOT mean the brpc port (8060) is healthy for data transfer. If queries fail with E1008 while Alive=true:
→ Route to doris-debug-query (Cause D — Exchange / brpc)
→ Check priority_networks, enable_brpc_connection_check, reset_rpc_channel
→ Do not trust SHOW BACKENDS Alive alone
Source
be/src/runtime/memory/mem_tracker_limiter.cppbe/src/service/doris_main.cpp— jemalloc init, signal handlerfe/.../master/HeartbeatHandler.javabe/src/common/config.cpp—mem_limit,enable_je_purge,max_segment_cache_size