# QA Test Plan

> Author, instantiate, and execute agentic QA test case plans for Elodin releases. Use when the user asks to create a QA plan, test case plan, or release test checklist, add or edit test cases, or run/execute a QA test plan for a release.

- Skill: `elodin-sys/qa-test-plan` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add elodin-sys/qa-test-plan`
- Raw SKILL.md: https://api.skillmd.com/api/skills/elodin-sys/qa-test-plan/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: elodin-sys (https://skillmd.com/u/elodin-sys)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/elodin-sys/qa-test-plan

---


# QA Test Plan

An agentic QA plan is a markdown document that replaces the traditional QA spreadsheet. Each test case carries machine-actionable steps and objective pass criteria so an agent can execute it autonomously; the filled-out document is both the checklist and the report.

Everything for this skill lives in this directory (`.cursor/skills/qa-test-plan/`):

- Reusable template for new plans: [template.md](template.md)
- **Ready-to-run plan — examples smoke/feature test:** [examples/test-plan.md](examples/test-plan.md) (24 cases: `SDK-001` build + one per `examples/` project). Shared helpers live alongside it in [examples/](examples/) (`run_probe.sh`, `probe.py`).
- **Ready-to-run plan — Elodin-DB deep suite:** [elodin-db/test-plan.md](elodin-db/test-plan.md) (23 cases: generate realistic DBs from the examples — drone, logstream, video-stream, sensor-camera, apollo-lander, rc-jet — then exercise every `elodin-db` subcommand, serving mode, replication, exports, and clients on them). Helper: [elodin-db/serve_probe.py](elodin-db/serve_probe.py); reuses the examples probes.
- **Ready-to-run plan — Elodin Editor deep suite:** [elodin-editor/test-plan.md](elodin-editor/test-plan.md) (19 cases: release build, live/recorded DB connect, screenshot gallery across examples covering feature-catalog §16–17, `--kdl` preload, inspector feature check; helpers [elodin-editor/capture.sh](elodin-editor/capture.sh)). Visual cases use `ELODIN_SCREENSHOT*` (see [elodin-editor-dev](../elodin-editor-dev/SKILL.md)).
- Dated run reports and instantiated plans live under gitignored `ai-context/qa-test-plan/<yyyy-mm-dd>-<release>/` (not in this skill folder).

**Source files are never the report.** Do not write Results, Evidence, Notes, Summary ticks, or Plan Header run metadata into `template.md`, `examples/test-plan.md`, `elodin-db/test-plan.md`, or `elodin-editor/test-plan.md`. Those keep AUTHOR-VALIDATED baselines and blank Results. Every execution (and every new custom plan) works on a **dated copy** under `ai-context/`.

## Agent entry point

- **"Run/execute the examples QA plan" (or "smoke-test the examples for a release")** → copy [examples/test-plan.md](examples/test-plan.md) into a dated folder (see *Executing a Plan*), then fill the **copy's** header and run cases top-to-bottom using that document's **Execution Rules**. It is self-contained and every `agent` case has an `AUTHOR-VALIDATED` baseline. Run from the repo root inside `nix develop`. Note the two hard-won prerequisites baked into [examples/run_probe.sh](examples/run_probe.sh): live-run cases bind port `2240` exclusively (never parallelize them), and tearing a run down requires a **process-group kill** (the s10 child restarts on a plain kill).
- **"Run/execute the Elodin-DB QA plan" (or "put elodin-db through its paces")** → copy [elodin-db/test-plan.md](elodin-db/test-plan.md) the same way, then execute the copy. It first generates a stable of realistic DBs under `/tmp/qa-db/` from the examples, then runs the full `elodin-db` command surface over them; its **Hard-won operational rules** section (group-kill teardown, the `tcp+1` asset-server port collision) is required reading before touching live servers.
- **"Run/execute the Elodin Editor QA plan" (or "screenshot / visual-regression the editor")** → copy [elodin-editor/test-plan.md](elodin-editor/test-plan.md) the same way, then execute the copy. Prefer a warm `target/release/elodin` (EDITOR-100). Never parallelize live-editor cases (port 2240). For `agent+visual` criteria, **Read the PNG** — file existence alone is insufficient. Stale `video-stream-db` / voyager DB dirs must be deleted before those captures (handled by [elodin-editor/capture.sh](elodin-editor/capture.sh)).
- **"Create/author a new plan for `<release>`"** → follow *Instantiating a Plan for a Release* below (copy `template.md` into `ai-context/qa-test-plan/`).
- **"Add/edit a test case"** → follow *Authoring Test Cases* below and the case anatomy rules. Persist new cases on the **source** ready-to-run plan and/or `template.md`, not on a dated run report.

There are three workflows: **authoring** test cases (edit sources), **instantiating** a custom plan from `template.md` (dated copy), and **executing** a ready-to-run suite (dated copy of that suite). The execution rules themselves live inside the plan so a copy is self-contained.

## Authoring Test Cases

### Anatomy of a case

Every case is a markdown block with these fields (see template for exact layout):

| Field | Purpose |
|-------|---------|
| Checkbox + ID + name | `#### - [ ] AREA-NNN — Short name` |
| Priority | P0 release-blocking smoke / P1 core functionality / P2 nice-to-have |
| Mode | `agent`, `agent+visual`, or `manual` |
| Requires | Case IDs that must PASS first (keep chains shallow; most cases require only SDK-001) |
| Description | One or two sentences: what behavior this verifies and why it matters |
| Expected duration | Lets the executor set command timeouts sensibly |
| Steps | Exact shell commands, one per line |
| Pass criteria | Objective checklist; every item must be verifiable from command output or artifacts |
| Result / Evidence / Notes | Filled in during execution, left blank when authoring |

### Rules for agent-executable cases

1. **One behavior per case.** If a case verifies two unrelated things, split it.
2. **Steps are exact commands**, run from repo root inside the Nix shell (`nix develop --command ...`). Never write prose steps like "open the editor and check it works".
3. **Commands must terminate.** Beware: `elodin run <sim.py>` and `elodin editor <sim.py>` keep serving the database even after the sim reaches `max_ticks` — they never exit on their own. The terminating form is bench mode: `uv run python <sim.py> bench --ticks N` (honors `ELODIN_DB_PATH`). For servers, background them and kill them in the same case.
4. **Pass criteria are objective**: exit codes, specific output strings, file existence, row counts, screenshot contents. If a criterion can't be checked mechanically or from a screenshot, the case belongs in `manual` mode.
5. **Cases are independent.** No case may rely on another case's side effects except through the declared Requires line. Use `/tmp/qa-run/<case-id>/` for scratch state and clean it up.
6. **Deterministic where possible**: fixed seeds, fixed tick counts, tolerance ranges for anything timing-dependent.
7. **Never pipe the command under test** (`cmd | tail` reports the pipe's exit code, not `cmd`'s). Redirect to a file in the artifacts directory and inspect that instead.
8. **Background processes must be managed inside one wrapper command**: capture the PID, probe, kill by PID, and `exit` with the probe's code (see DB-002 in the template).

### ID and area conventions

IDs are `AREA-NNN`, stable across releases — never renumber or reuse an ID; retired cases keep their number forever and new cases append. Current areas:

| Prefix | Area |
|--------|------|
| SDK | Python SDK build/install |
| SIM | Simulation runtime (headless) |
| DB | Elodin-DB |
| EDITOR | Elodin Editor |
| RUST, LINT | Build area: workspace tests, CI checks |
| ALEPH | Flight computer (usually `manual` — needs hardware) |

Add new areas as needed; record them in this table so IDs stay consistent.

### Execution modes

| Mode | Meaning |
|------|---------|
| `agent` | Fully autonomous, shell-only |
| `agent+visual` | Agent runs it but verification needs a display/screenshot; BLOCKED on headless machines |
| `manual` | A human must perform it (hardware, subjective judgment); agent marks it SKIPPED and reports it |

## Instantiating a Plan for a Release

Use this when the user wants a **new custom plan** (not one of the ready-to-run suites). Copy `template.md` into gitignored `ai-context/`, never edit it in place:

```bash
PLAN_DIR="ai-context/qa-test-plan/$(date +%F)-<release>"
mkdir -p "$PLAN_DIR"
cp .cursor/skills/qa-test-plan/template.md "$PLAN_DIR/test-plan.md"
```

1. Fill the **copy's** Plan Header: release name, `git rev-parse --short HEAD`, `git branch --show-current`, date, environment (note whether a display is available), executor.
2. Scope the copy: delete cases irrelevant to this release, add release-specific cases following the authoring rules. Keep the Summary table in sync with the case blocks — same IDs, same order.
3. If new cases were added that future releases should keep, also add them to [template.md](template.md) (and the relevant ready-to-run suite) so the sources stay the source of truth.

## Executing a Plan

**First action: dated copy. Do not skip this.** Ready-to-run suites (`examples/`, `elodin-db/`, `elodin-editor/`) and `template.md` are templates. Filling Results into them is a bug.

```bash
# <suite> is examples | elodin-db | elodin-editor
# <release> is a short slug the user named, or the suite name if they did not
PLAN_DIR="ai-context/qa-test-plan/$(date +%F)-<release>"
mkdir -p "$PLAN_DIR"
cp ".cursor/skills/qa-test-plan/<suite>/test-plan.md" "$PLAN_DIR/test-plan.md"
```

Work **only** on `$PLAN_DIR/test-plan.md`. Leave the source plan's Status, Summary Results, and case Result/Evidence blank (AUTHOR-VALIDATED notes stay on the source).

The binding rules are in the copied plan itself ("Execution Rules" section) — read them first. Operationally:

1. On the **copy**, set the header Status to IN PROGRESS (commit, branch, date, environment, executor).
2. Work through cases strictly in Summary-table order, one at a time. For each case: check Requires, run the steps, verify every pass criterion, write Evidence (exit codes, matching output lines, artifact paths), then set Result in the case block and mirror it in the Summary row.
3. Long builds are normal (SDK-001 up to 30 min, RUST-001 up to 40 min) — use the case's Expected duration to size timeouts instead of assuming a hang.
4. On FAIL, save output to `<artifacts>/<case-id>-fail.log`, diagnose briefly in Notes, and move on. Never fix code mid-run; a QA run measures the release as it is.
5. When all cases have a result: fill the Run Summary counts, list notable issues and follow-ups, set Status to COMPLETE, and report to the user — lead with FAIL/BLOCKED cases and anything a human must do (`manual` cases marked SKIPPED). Point them at the dated copy path, not the source suite.

## Example: well-formed vs poor case

**Poor (not agent-executable):**

> Steps: Open the editor and load a simulation. Check that everything looks right.

**Well-formed:** see `SIM-001` in [template.md](template.md) — exact command, bounded runtime, three mechanical pass criteria (exit code, summary block present, no traceback/panic).