# Nw Agent Evals

> Lightweight eval method for testing nWave AGENTS and SKILLS (LLM behavior) as a lean alternative to heavy BDD/ATD. An eval = one prompt -> one captured run (trace + artifacts) -> a small set of checks -> a comparable score over time. Load when validating agent behavior, building a regression net for an agent/skill, or reducing agent-test bloat.

- Skill: `nwave-ai/nw-agent-evals` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nwave-ai/nw-agent-evals`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nwave-ai/nw-agent-evals/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: nWave-ai (https://skillmd.com/u/nwave-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nwave-ai/nw-agent-evals

---


# Agent Evals

## Why this exists

Agents are LLMs — non-deterministic. Traditional BDD/unit testing tests them *badly* and breeds bloat (measured: 294K test LOC, 5.5:1 test:src, step reuse 1.10x vs >=4x target). Evals are the lean way to verify agent behavior: a few targeted signals instead of a monolithic spec.

This skill is the **antithesis of ATD over-specification**. Keep it small. The anti-pattern is up-front exhaustive specification — that IS the bloat.

| | `nw-agent-testing` (sibling skill) | `nw-agent-evals` (this skill) |
|---|---|---|
| Form | static 5-layer manual checklist | executable dataset + grader + score over time |
| Use | one-shot design review of a spec | repeatable regression net for behavior |
| Output | pass/fail judgement | comparable score, trend across runs |

Use both: `nw-agent-testing` to vet the spec, `nw-agent-evals` to watch behavior over time.

## What an eval is

One eval = **prompt -> run -> checks -> score**.

- **prompt** — a single input that should (or should NOT) trigger the agent/skill.
- **run** — one dispatch via the Claude Code `Agent` tool, with its trace + artifacts captured.
- **checks** — a small set of targeted assertions (not one monolithic check).
- **score** — a comparable number you can track across runs to catch regressions.

Replaces "vibes" with measurable signals: *did it invoke the right skill, run
the expected tools, respect the conventions and produce the required observable
effect?* Never grade a terminal-text grammar as a behavioral outcome.

## Definition of Done — before you write the eval

Write the success criteria FIRST, before implementing the agent/skill or its eval. Four check categories:

| Category | Question | Graded by |
|---|---|---|
| OUTCOME | Did the task get completed through its public effect or provider-enforced semantic outcome? | deterministic |
| PROCESS | Was the right skill loaded + the expected tool/step sequence run? | deterministic |
| STYLE | Does the output respect nWave conventions (sections, format)? | model-graded |
| EFFICIENCY | No useless commands / no token blowup? | deterministic |

If you cannot state DoD before writing the skill, the skill's job is not yet defined — stop and define it.

## Workflow

Run these steps in order:

1. **Define success first** — write the DoD (4 categories above) as concrete checks. Gate: every check is falsifiable.
2. **Manual trigger probe** — dispatch the agent once by hand to surface hidden assumptions. Gate: you have seen one real trace.
3. **Build the dataset** — 10-20 prompts in a CSV (see Dataset). Include explicit-invocation, implicit-from-description, contextual, and NEGATIVE-CONTROLS (`should_trigger=false`). Gate: >=2 negative controls present.
4. **Deterministic grading** — parse the provider-owned trace (JSONL) -> assert on tools run, files created, step sequence and public effects. Never parse model terminal prose. Gate: grader runs with zero human judgement.
5. **Qualitative grading** — use provider-enforced structured output for the ephemeral STYLE/quality result. Keep narrative feedback diagnostic and unparsed; never ask the grader to print a JSON/YAML grammar. Gate: the adapter validates the provider result.
6. **Grow coverage from failures** — every real failure/manual fix becomes one new eval row. Gate: regression net only grows from observed gaps, never speculatively.

## Capturing the trace (nWave mechanism)

nWave does NOT use `codex exec` — agents are dispatched via the Claude Code **`Agent` tool** (`subagent_type`, `prompt`); resume a spawned agent with **`SendMessage`**. The run is captured from the sub-agent's transcript (the `agent-*.jsonl` files in the transcript dir), which already exists:

- **Transcript JSONL** — each sub-agent run writes a JSONL transcript; the hook payload exposes its path as `agent_transcript_path` (the same field `src/des/.../hooks/skill_tracking_hooks.py:maybe_track_skill_loads` and `deliver_progress_handler.py` already consume). Each line is one event: `tool_use` (name + input), `tool_result`, assistant text.
- **What to parse from it**:
  - skill loaded? -> `Read` tool_use whose path matches `skills/.../SKILL.md` (this is exactly what `skill_tracking_hooks` scans for).
  - expected tools run? -> tool_use `name` values (e.g. a `Bash` call running `des code-fact` present, `Grep` alone absent).
  - files created? -> `Write`/`Edit` tool_use inputs + the artifact on disk.
  - sequence? -> ordered list of tool_use names.
- **Final message** — the agent's last assistant message is diagnostic input for qualitative review only; no deterministic grader parses its headings, fields, JSON or verdict strings.
- **Artifacts** — any file the agent wrote (ADR, review, design doc) is graded by existence + structure.

Capture pattern: dispatch via `Agent`, then read the transcript path + the on-disk artifacts. For a one-off eval you can dispatch and inspect the returned final message + written files directly; for a tracked net, persist the transcript alongside the dataset row.

## Deterministic graders (nWave-native signals)

Parse the trace, assert mechanically. nWave-specific, high-value signals:

| Signal | Assertion | Why it matters |
|---|---|---|
| Right skill loaded | `Read` of the expected `SKILL.md` appears | skill that is catalogued but never loaded = inferior output |
| Code analysis via CLI, not grep | a `des code-fact query.<capability>` Bash call with JSON parse present, `Grep`-only absent | the standing CLI-first preference (degrade-LOUD if AST unavailable). Eval must inspect the JSON envelope: provider + confidence labels in the agent's answer, not raw tool names. |
| Semantic outcome observed | provider-enforced ephemeral outcome plus the required public effect are present; final prose is not parsed | separates semantic judgement from control-plane serialization |
| Gate respected | no bypass marker; expected gate/step trailer present | off-spine dispatch guard |
| Artifact structure | required sections present (grep the written file) | OUTCOME completeness |
| Efficiency | tool_use count within a ceiling; no redundant re-reads | token economy |
| Negative control | for `should_trigger=false`, the skill/tool was NOT invoked | guards against over-eager invocation |

Bind to the code-fact CLI where useful: e.g. assert the agent invoked `des code-fact query.callers-of` rather than relying on a catalog entry (catalogued != wired). Note that feature-level change-scope analysis has no stable CLI today — an eval must not demand a capability the production CLI does not expose, or it grades the tooling rather than the agent.

## Qualitative grader (model-graded rubric)

For STYLE / design-quality / review-quality (not mechanically checkable), bind
the provider's structured-output facility to the existing rubric schema. The
model supplies semantic judgements; the provider validates the ephemeral
result and the adapter maps it to host types. Do not place a JSON template in
the prompt or parse terminal prose. Narrative notes remain diagnostic.

```json
{
  "overall_pass": true,
  "score": 0,
  "checks": [
    {"id": "adr-has-context-section", "pass": true, "notes": ""},
    {"id": "tradeoffs-quantified",   "pass": false, "notes": "no numbers"}
  ]
}
```

Rules: small rubric (3-7 checks), each check single-purpose, `notes` cites evidence. Provider-validate the structured result so an invalid grading turn fails closed rather than passing on vibes.

When the behavior under eval produces a DES handover, the eval crosses the real
boundary: competency -> existing CLI/software producer -> one whole-Request
handover containing the ordered value graph -> downstream consumer -> one
fan-in. Perfect fake `*-RESULT` strings, per-value contracts, and per-slice
review/finalization are not delivery evidence.

## Dataset

10-20 rows, CSV, small on purpose. Minimum columns:

```csv
id,prompt,should_trigger,expected_skill,expected_tools,expected_artifact,notes
ev-01,"Design the ADR for X",true,nw-design-patterns,"Write",docs/.../adr-*.md,explicit
ev-07,"Just fix this typo",false,,,,"negative control - architect must not fire"
```

- Mix: explicit-invocation, implicit-from-description (does `description` alone trigger it?), contextual, and NEGATIVE-CONTROLS (`should_trigger=false`).
- `name` + `description` are the PRIMARY invocation signal — implicit rows test exactly that.
- Coverage grows from real failures, never speculatively.

## Where evals live

```
tests/evals/<agent-or-skill-name>/
  dataset.csv          # the prompt set
  rubric.json          # model-graded rubric (JSON-Schema)
  README.md            # DoD + how to run
  runs/                # captured transcripts + scores per run (gitignored or pruned)
```

`tests/evals/` (sibling to the 5-layer suite), NOT `docs/` — these are executable, not documentation. Keep `runs/` out of the committed bloat; commit the dataset + rubric + scores, not raw transcripts.

## Principles

1. **Define success before you write the skill** — no DoD, no skill.
2. **Small targeted checks beat monolithic ones** — many cheap signals catch regressions early; one giant assertion hides them.
3. **Every manual fix is a future eval** — coverage is earned from observed failures.
4. **Negative controls are mandatory** — an agent that fires when it shouldn't is as broken as one that doesn't fire.
5. **name + description are the invocation contract** — test them, don't bypass them with explicit invocation only.
6. **Least privilege** — eval graders are read-only over traces + artifacts.
7. **Stay lean** — over-specifying up front recreates the ATD bloat this method exists to avoid.

## Example: eval for `nw-solution-architect`

DoD — dispatching the architect on a design prompt must produce a structured ADR via the right skill, using the `des code-fact` CLI for code facts.

Dataset rows (excerpt):

```csv
id,prompt,should_trigger,expected_skill,expected_tools,expected_artifact,notes
sa-01,"Design architecture for the handoff-state-algebra feature; write the ADR",true,nw-design-patterns,"Bash:des code-fact query.callers-of,Write","docs/**/adr-*.md",explicit design
sa-02,"What ADRs exist for the gate layer?",true,,"Bash:des code-fact query.adr-section,Read,Glob",,implicit code-fact lookup via query.adr-section
sa-03,"Rename this variable to camelCase",false,,,,negative control - not an architecture task
```

Deterministic grader (over the captured trace + artifact):

- PROCESS: `Read` of `nw-design-patterns/SKILL.md` present.
- PROCESS: structural code facts came from `des code-fact query.<capability>`, NOT `Grep`/`Bash grep`.
- OUTCOME: an `adr-*.md` was `Write`-n and on disk.
- OUTCOME: artifact contains the required ADR sections (`## Context`, `## Decision`, `## Consequences`).
- NEGATIVE (sa-03): architect did not author an ADR.

Model-graded rubric (STYLE/quality):

```json
{
  "overall_pass": false,
  "score": 70,
  "checks": [
    {"id": "context-states-problem", "pass": true,  "notes": "clear problem framing"},
    {"id": "decision-is-singular",   "pass": true,  "notes": ""},
    {"id": "consequences-quantified","pass": false, "notes": "tradeoffs qualitative only"}
  ]
}
```

Score = deterministic checks (binary, weighted) + rubric `score`, tracked per run. A drop on `code-fact-cli-not-grep` or `adr-sections-present` flags a behavioral regression before it ships.