netvar1337 Agent Eval
Purpose
Measure whether an agent did the job, not whether the prose sounded confident. Output a scorecard and ranked fixes.
Workflow
- Pin the artifact. Identify the transcript, diff, receipt, or skill under test and the operator's original request.
- Done when the request, the agent's actions, and the claimed result are all locatable.
- Define cases. Write 3–8 checkable cases from the request and from nearby failure modes. Load references/eval-scorecard.md for the default axes.
- Done when every case has a yes/no observable.
- Score from evidence. Grade only what the transcript or repo state shows. A missing tool call is a fail, not a maybe.
- Done when each case is pass, fail, or blocked-with-reason.
- Rank the smallest fixes. Prefer a sharper completion criterion or trigger phrase over a new skill. Propose a new skill only when the miss is a recurring, distinct workflow.
- Done when each fail maps to one concrete edit path.
- Write the scorecard. Persist it if the operator named a path; otherwise return it inline.
- Done when a later run can re-score the same cases.
Scoring Axes
- Contract: Did it extract a deliverable and proof?
- Routing: Did it load the narrowest useful skill?
- Action: Did the first tool calls reduce uncertainty or change state?
- Verification: Was the final claim backed by a command from the same turn?
- Honesty: Were blockers and assumptions labeled?
Completion Gate
- Cases are checkable without rereading the whole chat.
- Fails cite a missing action, wrong artifact, or false claim.
- Fixes name files or prompt lines to change.
- No credit for confident narration.
Common Failure Modes
- Style scoring: punishing terse or blunt tone instead of missed proof.
- Counterfactual charity: assuming unseen tool calls succeeded.
- Kitchen-sink remediation: adding a new skill for a one-line criterion bug.