# Running Eval Suite

> running-eval-suite

- Skill: `sgl-project/running-eval-suite` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add sgl-project/running-eval-suite`
- Raw SKILL.md: https://api.skillmd.com/api/skills/sgl-project/running-eval-suite/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: sgl-project (https://skillmd.com/u/sgl-project)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/sgl-project/running-eval-suite

---


# running-eval-suite

## Scope

A contributor on any sglang-omni dev container runs `/running-eval-suite`
to refresh every reference table cell in `benchmarks/eval/benchmark_*.py`
that corresponds to the host's hardware:

1. Auto-detect host hardware (`H200` / `H100` / `H800` / `A100` / ...).
2. Run every row declared in `models/<model>/config.yaml`.
3. For each row, after the client succeeds:
   - If the file has a row whose Source column contains
     `[<hw>, <workload>]` matching this entry → **replace** its cells.
   - Otherwise → **append** a new row at the end of that section's
     table.
4. `Local v1 Pipeline Result` and similar Local-* sections are
   **skipped entirely** — those are contributors' personal experiments,
   not refresh targets.
5. Commit `benchmarks/eval/` to a local commit. **No push.**

The skill never edits anything outside `benchmarks/eval/`. Cell rewrite
is exact-string replacement on the data row line; the audit trail is
plain `git show HEAD`.

## Models

```
.claude/skills/running-eval-suite/models/<model>/config.yaml
```

Today: `qwen3-omni` (covers all 6 benchmarks/eval/*.py — Qwen3-Omni
rows + S2-Pro TTS rows live in the same config; per-row `hf_model_id`
distinguishes the model launched per row).

The linked calibration skill is maintained in a private repository; access is
limited to its maintainers.

## Prerequisites (the skill verifies, it does not create)

- A sglang-omni clone (`benchmarks/eval/` reachable from the working dir).
- A venv with `sglang` + `torch` importable. Default candidates (first existing wins):
  `/sgl-workspace/sglang-omni/omni/bin/python`,
  `/github/home/calibration/omni/bin/python`,
  `/github/home/omni/bin/python` (legacy). Override with
  `--venv-python <path>` or `$EVAL_VENV_PYTHON`.
- HF model + datasets cached under `HF_HOME=/github/home/.cache/huggingface`
  (see `auto_env` in `config.yaml`). precheck reports `✓` / `✗` and prints
  the exact `HF_ENDPOINT=https://hf-mirror.com huggingface-cli download …`
  commands when something's missing.
- Free GPUs. The skill **never** kills another user's processes; if
  GPUs are busy precheck fails with the busy PID list and stops.
- `auto_env` from `config.yaml` is applied at startup (overrides shell) to
  match CI omni-setup: `OMNI_CI_HOME`, `UV_INDEX_URL`, torchinductor slice
  paths, etc. See [`calibrate-h100-ci`][calibration-skill] skill for the full cache layout table.
  All benchmarks share the single **`omni`** venv and `.github/scripts/ci_env.sh`.
- `HF_ENDPOINT` defaults to `https://hf-mirror.com` (matches CI omni-setup).
- Before pytest or eval runs, run `source .github/scripts/ci_env.sh` so
  `TORCHINDUCTOR_CACHE_DIR` stays under `${OMNI_CI_HOME}/.torchinductor`.

## Two-terminal supervision (mandatory — always)

Same contract as [`calibrate-h100-ci`][calibration-skill] — **permanent, non-negotiable**:

| Tab | Role | Shows |
|-----|------|-------|
| **A — Supervision** | `tail -f <log-path>` | **Detailed log** — pytest, router, worker, HTTP, tracebacks |
| **B — Job** | run command / script | **Progress summary only** — START / PASS / FAIL, stage name |

Verbose output goes to the log file (`>> log 2>&1`). **Never `tee` on Tab B** —
that duplicates Tab A and breaks supervision.

Agent **must spawn both tabs** (Shell, `block_until_ms: 0`): Tab A first, Tab B
second. Tell the user which is which.

**Calibration (`tune.py run`):** Tab A = newest `_pytest/*/run*.log`; Tab B =
`tune.py run` on stdout (**never** `>> run.log`). See [`calibrate-h100-ci`][calibration-skill] §
**Calibration (`tune.py run`) — always two tabs**.

**Eval suite commands:**

| Tab | Command |
|-----|---------|
| **A — supervision** | `tail -f <run-dir>/run.log` |
| **B — job** | `cd /sgl-workspace/sglang-omni && python .claude/skills/running-eval-suite/runner.py run ... >> <run-dir>/run.log 2>&1` |

Full diagram, log-path table, and checklist: see [`calibrate-h100-ci`][calibration-skill] §
**Two-terminal supervision (mandatory — always)**.

- First-time venv on the repro host: use `.github/scripts/prepare_omni_venv.sh`
  and `.github/scripts/install_flashinfer_jit_cache.sh` with
  `OMNI_CI_HOME=/github/home/calibration` (documented in [the calibration skill][calibration-skill]).

If anything's off, `precheck` fails with an actionable message; fix it
yourself and retry.

## Invocation

- `/running-eval-suite` — runs everything end-to-end with defaults
  (model `qwen3-omni`, all benchmarks, 1 round, real apply,
  auto-commit).
- `/running-eval-suite --benchmarks mmsu,omni_seedtts` — run only those
  benchmark files. Use `omni_seedtts` / `qwen_seedtts` for Qwen3-Omni
  SeedTTS rows and `tts_seedtts` / `s2pro_seedtts` for S2-Pro rows.
- `/running-eval-suite --rounds 3` — multi-round; only round 1 is used
  for apply (multi-round aggregation is a follow-up).
- `/running-eval-suite --smoke 50` — `--max-samples 50` injected; no
  edits applied. For benchmarks whose full set is already ≤50, the
  smoke flag is effectively a no-op (they run their full set anyway).
- `/running-eval-suite --venv-python /usr/bin/python3` — override the
  default venv detection (also via `$EVAL_VENV_PYTHON`).
- `/running-eval-suite --exclude-ids s2pro-` — skip rows whose id
  contains a substring.
- `/running-eval-suite --retry-failed-from <run-dir>` — rerun rows that
  were not successfully edited in a previous run.
- `/running-eval-suite --gpu-pool 0,1,2` — restrict server and ASR GPU
  allocation to a specific free GPU pool.

I never call `AskUserQuestion`. Defaults handle the full run; optional
flags above are only for filtering, recovery, or explicit GPU placement.
Errors halt the run with a printed reason. This matches the
[`calibrate-h100-ci`][calibration-skill] contract: type `/running-eval-suite` and walk away.

## Steps I follow

1. Apply defaults to the invocation: `model=qwen3-omni`,
   `benchmarks=all`, `rounds=1`, `smoke=null`. **No `AskUserQuestion`.**
   If the user passed unknown flags, stop with the parsing error.

2. Run precheck. The runner creates `<repo>/.eval-runs/<utc-ts>/` on
   first run; reuse the same `<run-dir>` for the rest of the
   invocation:
   ```
   python .claude/skills/running-eval-suite/runner.py \
     --model <M> precheck --benchmarks <names|all> \
     [--exclude-ids <substr>] [--gpu-pool <ids>] \
     --output-dir <run-dir>
   ```
   If exit code != 0, surface the `✗` lines and stop.

3. Run the benchmarks. The runner does the cell rewrites inline as
   each row finishes (no separate apply step):
   ```
   python .claude/skills/running-eval-suite/runner.py \
     --model <M> run --benchmarks <names|all> \
     --rounds <K> [--smoke <N>] \
     [--exclude-ids <substr>] [--gpu-pool <ids>] \
     --output-dir <run-dir>
   ```
   - For each row: launch server (process-group SIGTERM at end, never
     `pkill -f`), run client, capture `result.json`, then immediately
     edit the matching row in `benchmarks/eval/benchmark_*.py`
     (replace existing hw row OR append new hw row at end of the
     section's table; Local sections never touched).
   - Managed-router rows use `router, 2-worker, ...` Source workload
     tags. The first run on a hardware target appends new router-tagged
     rows next to the older direct/V1-pipeline rows; later runs replace
     those router rows in place.
   - Per-row state lands in `<run-dir>/run-state.json`. Stream the
     runner's stdout to the user so they can watch progress.
   - **If a row fails** (server boot timeout / client crash / locator
     mismatch / column missing / JSON path missing / etc.), that row's
     reason is recorded; **the next row continues**. No global halt.

4. After the run finishes, look at `run-state.json`'s summary block
   (also printed to stdout): replaced count + appended count + failed
   runs + failed edits. If anything failed, surface those lines.

5. Auto-commit the refresh:
   ```
   git -C <repo> add benchmarks/eval/
   git -C <repo> diff --cached --quiet && (echo "no changes — nothing to commit"; exit 0)
   git -C <repo> commit -m "[Docs] Refresh reference benchmarks via /running-eval-suite (<utc-ts>)"
   ```
   No `git push`. If `git diff --cached --quiet` says there's nothing
   to commit (every row failed or was skipped), skip the commit and
   print a one-liner.

6. Print three lines for the user:
   - `Review:   git show HEAD`
   - `Revert:   git reset --soft HEAD~1   # keeps the edits unstaged`
   - `Run dir:  <run-dir>` (logs / result.json per row)

## What I do not do

- **Ask the user any questions.** "Type the slash command and walk
  away" is the contract.
- Set up the container / venv / HF cache (covered by
  `docs/contributing/running-eval-suite.md`).
- Touch `Local v1 Pipeline Result` / `Local Speech Pipeline` tables
  inside `benchmark_*.py` — those are contributors' personal
  experiments.
- Kill other users' processes. Busy GPUs → precheck reports busy PID
  list → stop.
- Push. I commit locally so the user can `git show HEAD`,
  rebase/squash/amend, then push to their fork.
- Multi-round aggregation. `--rounds K` runs K rounds (useful for
  variance observation), but inline apply uses round 1 only. Cells
  whose existing Source tag references `n=K mean` are flagged in the
  preview but still updated; user can manually recompute the mean
  before pushing.

## Files

```
.claude/skills/running-eval-suite/
├── SKILL.md
├── runner.py                                # CLI: precheck / run
└── models/
    └── qwen3-omni/
        └── config.yaml                      # 31 rows across all 6 benchmark .py files
```

## Adding a new row

1. Identify the reference table row in some
   `benchmarks/eval/benchmark_*.py`. Note the **section header**
   above the table, the row's **Config** column substring, and the
   **workload tag** inside the row's Source column (the part inside
   the `[…]` brackets, **without** the leading hardware token —
   e.g. `router, 2-worker, full-set, c=8` from
   `[H200, router, 2-worker, full-set, c=8]`).
2. Append a new entry to `models/<model>/config.yaml`:
   ```yaml
   - id: <unique-handle>          # informational; not used to select rows
     file: "benchmarks/eval/benchmark_omni_<name>.py"
     hf_model_id: "Qwen/Qwen3-Omni-30B-A3B-Instruct"   # optional, overrides config-level
     locate:
       section_substring: "Accuracy (accuracy)"
       config_substring: "modalities=text "
       source_workload: "router, 2-worker, full-set, c=8"
     # Server launch is normally inherited from default_server_profile or
     # server_profile_by_hf_model_id. Set server_profile only when a row needs
     # a different managed-router topology.
     server_profile: qwen3_omni_colocated_router
     client: "python benchmarks/eval/benchmark_omni_<name>.py --model qwen3-omni --port {port} --output-dir {output_dir} ..."
     result_json: "<benchmark>_results.json"
     cells:
       <col1>: { path: "<dotted.path>", format: "{v}" }
       <col2>: { paths: { v: "<a>", t: "<b>" }, format: "{v}/{t}" }
   ```
3. Verify the locator matches the intended H200 row (or doesn't yet
   exist for new hw → will append) before committing the config:
   ```bash
   python - <<'EOF'
   import sys; sys.path.insert(0, '.claude/skills/running-eval-suite')
   import runner, yaml; from pathlib import Path
   cfg = yaml.safe_load(open('.claude/skills/running-eval-suite/models/qwen3-omni/config.yaml'))
   for row in cfg['rows']:
       text = Path(row['file']).read_text()
       rows = list(runner._find_table_rows_official(text))
       loc = row['locate']
       full = f"[H200, {loc['source_workload']}]"
       cands = [r for r in rows if loc['section_substring'] in (r.get('section_line') or '')
                and loc['config_substring'] in r['row_text']]
       matched = [r for r in cands if full in r['row_text']]
       print(f"{row['id']}: section_rows={len(cands)} h200_match={len(matched)}")
   EOF
   ```
4. Smoke-test with `--smoke 50 --benchmarks <name>` before
   committing.

Adding a whole new model = drop in `models/<new-model>/config.yaml`
mirroring `qwen3-omni/config.yaml`. No Python changes needed unless
the new benchmark client emits result JSON in a structure that needs
new helpers in `runner.py`.

[calibration-skill]: https://github.com/zhaochenyang20/sglang-omni-calibration/tree/main/skills/calibrate-h100-ci/references/calibration.md