novuhq Triage Agent Eval Failures
Diagnose a failing scenario in libs/agent-evals and produce a verdict: is the failure real (the playbook under test regressed) or is the test wrong (grader / tape / scenario / judge), or is it just flaky (model non-determinism)?
The thing under test is the playbook doc (packages/shared/docs/agent-onboarding.md), injected as the agent system prompt. Everything else (graders.ts, catalog.ts, scenario.ts, judge prompts) is test scaffolding. Never fix the playbook to satisfy a broken grader, and never loosen a grader to hide a real playbook regression.
Rule 0: rule out flakiness before changing anything
Scenarios run a live model concurrently, so one red run is one sample, not a verdict. Re-run the single failing scenario 3–5× first:
pnpm --filter @novu/agent-evals exec vitest run --config vitest.evals.config.ts -t <scenario-id>
- Fails every run → deterministic failure, continue triage.
- Fails intermittently → flaky. The cause is usually a non-deterministic judge grader or an over-strict regex. Do not edit the playbook. Tighten the grader/judge prompt or accept variance; consider pass@k rather than single-run gating.
To reproduce judge graders locally:
pnpm --filter @novu/agent-evals exec vitest run --config vitest.evals.config.ts -t <scenario-id>
Step 1: identify which grader failed and its kind
Each scenario registers graders in scenarios/<id>/graders.ts. The kind is the strongest triage signal:
- Deterministic graders (
catalog.*,contains,matches) inspect the structuredRunResult. A fail means the agent's actions/output objectively did not match — or the check is too strict. - Judge graders (
sharedJudgeGraders,judge(...)) call a second LLM pass. A fail is fuzzy and can be the judge prompt's fault, not the agent's.
Find the grader's logic:
| Layer | Location |
|---|---|
| Per-scenario grader wiring | src/suites/agent-onboarding/scenarios/<id>/graders.ts |
| Deterministic grader bodies | src/suites/agent-onboarding/catalog.ts (catalog object) |
| Judge prompts | catalog.ts (judgePrompts) + sharedJudgeGraders |
| Generic helpers | src/core/graders.ts (contains, matches, toolCallsNamed, transcriptText) |
| Judge mechanics | src/core/judge.ts (returns skip on UNKNOWN) |
Step 2: read the RunResult evidence
Graders read fields off RunResult (src/core/types.ts). Map the failing grader to the field it checks and compare against what the agent actually did in the run output:
trackedCommands— raw connect command strings (flag checks like--keyless,--secret-key,--slack-config-token).toolCalls— everyBash/BashOutput/AskUserQuestion/Readcall with args (run_in_background,file_path, pickerselectedId).polledShellIds/killedShellIds— background-polling and kill behavior.capturedUrls/openedFiles— surfaced URLs and opened files (e.g. QR.png, auth-url file).finalText/assistantMessages— user-facing report (transcriptTextjoins these).metadata.description— the drafted agent description (persona / infra-token graders).
Step 3: classify the failure
Walk top-down and stop at the first match:
| Symptom | Verdict | Fix target |
|---|---|---|
| Agent never ran the tracked command / ignored an instruction it should follow | Real — discovery | Playbook agent-onboarding.md (instruction unclear/missing) |
Deterministic grader fails and the RunResult confirms the agent genuinely did the wrong thing |
Real — execution | Playbook agent-onboarding.md |
Deterministic grader fails but RunResult shows the agent behaved correctly (regex too strict, wrong field, valid variant rejected) |
Test bug | catalog.ts grader logic |
Fails only on the scripted CLI path; tape stdout/when/validate or scripted answers are wrong or stale |
Test bug | scenario.ts (tape, scriptedAnswers), connect-parser.ts |
| Judge grader fails but the description/report actually satisfies the criterion | Test bug | Judge prompt in catalog.ts (judgePrompts) |
| Judge verdict flips run-to-run | Flaky judge | Sharpen judge prompt; rely on UNKNOWN→skip escape hatch |
| Passes sometimes, fails sometimes, no clear cause | Flaky | Do not edit playbook; re-run (Rule 0) |
A scenario passes only when every active grader averages ≥ 0.8 (JUDGE_THRESHOLD). A judge returning UNKNOWN becomes skip and scores 1 — it never causes a fail, so an UNKNOWN is not evidence of a real regression.
Step 4: apply one bounded fix, then verify
- Change only the layer the verdict points to — playbook or test, never both to chase green.
- Re-run the single scenario (Step 0 command).
- Confirm the fix holds across the 3–5 re-runs and that no other scenario regressed.
- If editing a deterministic grader, also run the synthetic unit tests so you don't break grader contracts:
pnpm --filter @novu/agent-evals test
Output format
Report the verdict concisely with cited evidence:
Scenario: <id>
Failing grader: <name> (deterministic | judge)
Re-run result: <N/M failed> → real | flaky
Evidence: <RunResult field + actual vs expected>
Verdict: real playbook regression | test bug (<grader|tape|scenario|judge>) | flaky
Fix target: <file path> (or: no change — flaky/UNKNOWN)
Additional resources
For worked triage examples (real regression vs test bug vs flaky judge), see reference.md.