# Doris Debug Resource Isolation

> Use for Doris Workload Group / resource tag queue starvation, CPU/memory isolation leaks, and workload policy debugging. Commands: SHOW WORKLOAD GROUPS, EXPLAIN resource.

- Skill: `apache/doris-debug-resource-isolation` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add apache/doris-debug-resource-isolation`
- Raw SKILL.md: https://api.skillmd.com/api/skills/apache/doris-debug-resource-isolation/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: apache (https://skillmd.com/u/apache)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/apache/doris-debug-resource-isolation

---


# Resource Isolation (Workload Group)

## Causes

| ID | Cause | Evidence | Source anchor |
|----|-------|----------|---------------|
| A | Queue starvation | Queries queued in `SHOW WORKLOAD GROUPS`; `QueueTime` growing | `WorkloadGroupMgr.java` |
| B | CPU oversubscription | One WG consuming >90% CPU despite limits | `CgroupCpuCtl.cpp` |
| C | Memory leak across groups | MemTracker shows one WG exceeding `memory_limit` | `WorkloadGroupMemMgr.cpp` |
| D | Tag routing wrong | Query hits wrong BE set; ResourceTag mismatch | `ResourceTag.java`, `Tag.java` |
| E | WG not active / cgroup missing | `CREATE WORKLOAD GROUP` had no effect | `WorkloadGroupMgr.cpp` cgroup init |

## 10 min triage

```sql
-- Workload group status
SHOW WORKLOAD GROUPS\G
-- Check columns: Id, Name, ActiveQueries, QueuedQueries, CpuShares, MemoryLimit

-- Which WG is my query in?
SELECT * FROM information_schema.workload_group_resource_usage;

-- Workload policies (routing rules)
SHOW WORKLOAD POLICY\G
```

```bash
# BE cgroup inspection
cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes

./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "workload_group"
```

## Cause A — Queue starvation

```sql
-- See who's queued
SHOW WORKLOAD GROUPS\G

-- Raise queue concurrency
ALTER WORKLOAD GROUP etl_wg SET (
    "max_concurrency" = "8",
    "max_queue_size" = "100",
    "queue_timeout" = "300"
);
```

Check `be/src/pipeline/task_scheduler.cpp` — the pipeline scheduler respects cgroup shares; if a high-share WG starves a low-share one, the low-share WG's queue backs up.

## Cause B — CPU oversubscription

```sql
-- Create a hard CPU cap
ALTER WORKLOAD GROUP report_wg SET (
    "cpu_share" = "1024",
    "cpu_hard_limit" = "200%"   -- 2 cores max
);

-- Force query into specific WG
SET workload_group = 'report_wg';
```

cgroup v1 vs v2 behavior differs. Confirm which is active on BE hosts:
```bash
mount | grep cgroup
ls /sys/fs/cgroup/cpu/doris/   # cgroup v1
ls /sys/fs/cgroup/doris/       # cgroup v2
```

## Cause D — Tag routing

```sql
-- Check BE tags
SHOW BACKENDS\G  -- look at Tag column

-- Check WG tag constraint
SHOW WORKLOAD GROUPS\G  -- look at ResourceTag
```

If a workload group specifies a tag but no BEs carry that tag, queries queue forever. Fix:
```sql
ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.tag.location" = "high_mem");
```

## Enabling cgroup on BE

```properties
# be.conf
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
enable_workload_group = true
```

Without cgroup configured, `cpu_share` is a cooperative hint, not a hard guarantee.

## Source

- `be/src/common/config.cpp` — `enable_workload_group`, cgroup paths
- `be/src/pipeline/task_scheduler.cpp` — pipeline scheduling
- `fe/.../workloadgroup/WorkloadGroupMgr.java` — WG lifecycle
- `fe/.../resource/Tag.java` — resource tags
- `be/src/util/cgroup_util.cpp` — cgroup v1/v2 adapter