Nw Agent Evals

Agent Evals

Why this exists

Agents are LLMs — non-deterministic. Traditional BDD/unit testing tests them badly and breeds bloat (measured: 294K test LOC, 5.5:1 test:src, step reuse 1.10x vs >=4x target). Evals are the lean way to verify agent behavior: a few targeted signals instead of a monolithic spec.

This skill is the antithesis of ATD over-specification. Keep it small. The anti-pattern is up-front exhaustive specification — that IS the bloat.

Use both: nw-agent-testing to vet the spec, nw-agent-evals to watch behavior over time.

What an eval is

One eval = prompt -> run -> checks -> score.

• prompt — a single input that should (or should NOT) trigger the agent/skill.
• run — one dispatch via the Claude Code Agent tool, with its trace + artifacts captured.
• checks — a small set of targeted assertions (not one monolithic check).
• score — a comparable number you can track across runs to catch regressions.

Replaces "vibes" with measurable signals: did it invoke the right skill, run the expected tools, respect the conventions and produce the required observable effect? Never grade a terminal-text grammar as a behavioral outcome.

Definition of Done — before you write the eval

Write the success criteria FIRST, before implementing the agent/skill or its eval. Four check categories:

If you cannot state DoD before writing the skill, the skill's job is not yet defined — stop and define it.

Workflow

Run these steps in order:

1. Define success first — write the DoD (4 categories above) as concrete checks. Gate: every check is falsifiable.
2. Manual trigger probe — dispatch the agent once by hand to surface hidden assumptions. Gate: you have seen one real trace.
3. Build the dataset — 10-20 prompts in a CSV (see Dataset). Include explicit-invocation, implicit-from-description, contextual, and NEGATIVE-CONTROLS (should_trigger=false). Gate: >=2 negative controls present.
4. Deterministic grading — parse the provider-owned trace (JSONL) -> assert on tools run, files created, step sequence and public effects. Never parse model terminal prose. Gate: grader runs with zero human judgement.
5. Qualitative grading — use provider-enforced structured output for the ephemeral STYLE/quality result. Keep narrative feedback diagnostic and unparsed; never ask the grader to print a JSON/YAML grammar. Gate: the adapter validates the provider result.
6. Grow coverage from failures — every real failure/manual fix becomes one new eval row. Gate: regression net only grows from observed gaps, never speculatively.

Capturing the trace (nWave mechanism)

nWave does NOT use codex exec — agents are dispatched via the Claude Code Agent tool (subagent_type, prompt); resume a spawned agent with SendMessage. The run is captured from the sub-agent's transcript (the agent-*.jsonl files in the transcript dir), which already exists:

• Transcript JSONL — each sub-agent run writes a JSONL transcript; the hook payload exposes its path as agent_transcript_path (the same field src/des/.../hooks/skill_tracking_hooks.py:maybe_track_skill_loads and deliver_progress_handler.py already consume). Each line is one event: tool_use (name + input), tool_result, assistant text.
• What to parse from it:
  • skill loaded? -> Read tool_use whose path matches skills/.../SKILL.md (this is exactly what skill_tracking_hooks scans for).
  • expected tools run? -> tool_use name values (e.g. a Bash call running des code-fact present, Grep alone absent).
  • files created? -> Write/Edit tool_use inputs + the artifact on disk.
  • sequence? -> ordered list of tool_use names.
• Final message — the agent's last assistant message is diagnostic input for qualitative review only; no deterministic grader parses its headings, fields, JSON or verdict strings.
• Artifacts — any file the agent wrote (ADR, review, design doc) is graded by existence + structure.

Capture pattern: dispatch via Agent, then read the transcript path + the on-disk artifacts. For a one-off eval you can dispatch and inspect the returned final message + written files directly; for a tracked net, persist the transcript alongside the dataset row.

Deterministic graders (nWave-native signals)

Parse the trace, assert mechanically. nWave-specific, high-value signals:

Bind to the code-fact CLI where useful: e.g. assert the agent invoked des code-fact query.callers-of rather than relying on a catalog entry (catalogued != wired). Note that feature-level change-scope analysis has no stable CLI today — an eval must not demand a capability the production CLI does not expose, or it grades the tooling rather than the agent.

Qualitative grader (model-graded rubric)

For STYLE / design-quality / review-quality (not mechanically checkable), bind the provider's structured-output facility to the existing rubric schema. The model supplies semantic judgements; the provider validates the ephemeral result and the adapter maps it to host types. Do not place a JSON template in the prompt or parse terminal prose. Narrative notes remain diagnostic.

{
  "overall_pass": true,
  "score": 0,
  "checks": [
    {"id": "adr-has-context-section", "pass": true, "notes": ""},
    {"id": "tradeoffs-quantified",   "pass": false, "notes": "no numbers"}
  ]
}

Rules: small rubric (3-7 checks), each check single-purpose, notes cites evidence. Provider-validate the structured result so an invalid grading turn fails closed rather than passing on vibes.

When the behavior under eval produces a DES handover, the eval crosses the real boundary: competency -> existing CLI/software producer -> one whole-Request handover containing the ordered value graph -> downstream consumer -> one fan-in. Perfect fake *-RESULT strings, per-value contracts, and per-slice review/finalization are not delivery evidence.

Dataset

10-20 rows, CSV, small on purpose. Minimum columns:

id,prompt,should_trigger,expected_skill,expected_tools,expected_artifact,notes
ev-01,"Design the ADR for X",true,nw-design-patterns,"Write",docs/.../adr-*.md,explicit
ev-07,"Just fix this typo",false,,,,"negative control - architect must not fire"

• Mix: explicit-invocation, implicit-from-description (does description alone trigger it?), contextual, and NEGATIVE-CONTROLS (should_trigger=false).
• name + description are the PRIMARY invocation signal — implicit rows test exactly that.
• Coverage grows from real failures, never speculatively.

Where evals live

tests/evals/<agent-or-skill-name>/
  dataset.csv          # the prompt set
  rubric.json          # model-graded rubric (JSON-Schema)
  README.md            # DoD + how to run
  runs/                # captured transcripts + scores per run (gitignored or pruned)

tests/evals/ (sibling to the 5-layer suite), NOT docs/ — these are executable, not documentation. Keep runs/ out of the committed bloat; commit the dataset + rubric + scores, not raw transcripts.

Principles

1. Define success before you write the skill — no DoD, no skill.
2. Small targeted checks beat monolithic ones — many cheap signals catch regressions early; one giant assertion hides them.
3. Every manual fix is a future eval — coverage is earned from observed failures.
4. Negative controls are mandatory — an agent that fires when it shouldn't is as broken as one that doesn't fire.
5. name + description are the invocation contract — test them, don't bypass them with explicit invocation only.
6. Least privilege — eval graders are read-only over traces + artifacts.
7. Stay lean — over-specifying up front recreates the ATD bloat this method exists to avoid.

Example: eval for nw-solution-architect

DoD — dispatching the architect on a design prompt must produce a structured ADR via the right skill, using the des code-fact CLI for code facts.

Dataset rows (excerpt):

id,prompt,should_trigger,expected_skill,expected_tools,expected_artifact,notes
sa-01,"Design architecture for the handoff-state-algebra feature; write the ADR",true,nw-design-patterns,"Bash:des code-fact query.callers-of,Write","docs/**/adr-*.md",explicit design
sa-02,"What ADRs exist for the gate layer?",true,,"Bash:des code-fact query.adr-section,Read,Glob",,implicit code-fact lookup via query.adr-section
sa-03,"Rename this variable to camelCase",false,,,,negative control - not an architecture task

Deterministic grader (over the captured trace + artifact):

• PROCESS: Read of nw-design-patterns/SKILL.md present.
• PROCESS: structural code facts came from des code-fact query.<capability>, NOT Grep/Bash grep.
• OUTCOME: an adr-*.md was Write-n and on disk.
• OUTCOME: artifact contains the required ADR sections (## Context, ## Decision, ## Consequences).
• NEGATIVE (sa-03): architect did not author an ADR.

Model-graded rubric (STYLE/quality):

{
  "overall_pass": false,
  "score": 70,
  "checks": [
    {"id": "context-states-problem", "pass": true,  "notes": "clear problem framing"},
    {"id": "decision-is-singular",   "pass": true,  "notes": ""},
    {"id": "consequences-quantified","pass": false, "notes": "tradeoffs qualitative only"}
  ]
}

Score = deterministic checks (binary, weighted) + rubric score, tracked per run. A drop on code-fact-cli-not-grep or adr-sections-present flags a behavioral regression before it ships.