Agent Eval

When to invoke

Use when: "evaluate the agent", "build an eval set", "LLM as judge", "prompt regression test", "did the prompt change make it better", "measure prompt quality", "eval harness".

Do not use for web page performance (/benchmark) or for choosing between vendors on a single prompt (/benchmark-models). This skill builds a repeatable test suite for one unit of LLM behavior in the user's own project.

Preamble

eval "$(~/.vibestack/bin/vibe-slug 2>/dev/null)" 2>/dev/null || SLUG="unknown"
_LEARN_FILE="${VIBESTACK_HOME:-$HOME/.vibestack}/projects/${SLUG:-unknown}/learnings.jsonl"
if [ -f "$_LEARN_FILE" ]; then
  _LEARN_COUNT=$(wc -l < "$_LEARN_FILE" 2>/dev/null | tr -d ' ')
  echo "LEARNINGS: $_LEARN_COUNT entries loaded"
  if [ "$_LEARN_COUNT" -gt 5 ] 2>/dev/null; then
    ~/.vibestack/bin/vibe-learnings-search --limit 5 2>/dev/null || true
  fi
else
  echo "LEARNINGS: none yet"
fi

{{include lib/snippets/session-host.md}}

{{include lib/snippets/decision-brief.md}}

{{include lib/snippets/working-protocols.md}}

{{include lib/snippets/state-protocols.md}}

User-invocable

When the user types /agent-eval, run this skill. An optional argument names the unit under test (a file path, a function, or a prompt name); without one, find it in Step 1.

Step 1: Identify the unit under test

Pin down exactly one thing to evaluate. An eval that covers "the whole agent" measures nothing.

Find candidates with Grep and Glob:

• Prompt templates: **/*prompt*, **/prompts/**, string literals containing You are or system:
• Agent loops: files that call a chat completion API in a loop and dispatch on tool calls
• Tool routers: a function that maps a model's tool-call name to a handler
• Existing evals or fixtures: **/eval*, **/golden*, **/fixtures/**, *.jsonl, transcripts under logs/

For the chosen unit, write down its interface before writing any case:

Findings at this step:

• HIGH — the unit has side effects with no stub or dry-run flag. The harness cannot run it safely until one exists. Stop and say so; propose the smallest seam (an injectable client, an env flag).
• MEDIUM — no existing tests, logs, or transcripts. Every case in Step 2 will be hand-written; say so in the plan.
• LOW — prompt text is inlined in code rather than a file. Note it; it makes prompt diffs hard to review but does not block the eval.

Step 2: Build the task set

Store cases as JSONL under evals/<unit>/cases.jsonl, one object per line:

{"id":"gold-001","input":{"message":"Refund order 4412"},"expected":{"tool_calls":[{"name":"lookup_order","args":{"id":"4412"}}]},"tags":["golden","refund"]}

Fields: id (stable, never reused), input (whatever the unit takes), expected (what a correct run produces — a string, a JSON object, a tool-call list, or a rubric note for judged cases), tags (a list; every case has at least one).

Three sources, in this order:

1. Golden cases, written by hand. 10-30. Cover the main paths the unit exists for. Write the expected output before running the unit, not after — copying a live output into expected turns the eval into a snapshot test of whatever the model did today.
2. Mined cases. Pull real inputs from logs or transcripts found in Step 1. Strip PII before saving (names, emails, account ids). Tag them mined. Only fill expected for cases where the correct output is knowable; leave the rest for judge scoring.
3. Adversarial cases. Tag adversarial. Include at least one of each: ambiguous input with two valid readings, a prompt-injection attempt inside a tool result or pasted document, empty input, input near the context limit, input in a language the prompt does not mention, a request the unit should refuse.

Check the set before moving on:

CASES="evals/<unit>/cases.jsonl"
wc -l < "$CASES"
grep -c '"golden"' "$CASES"
grep -c '"adversarial"' "$CASES"
python3 -c 'import collections,json,sys
ids=[json.loads(l)["id"] for l in open(sys.argv[1]) if l.strip()]
print("\n".join(sorted(i for i,n in collections.Counter(ids).items() if n>1)))' "$CASES"

The duplicate check reads each line as JSON and takes only the top-level id. A grep for "id" matches nested ids too — expected.tool_calls[].args.id in the example above — and reports them as duplicate cases.

Findings:

• HIGH — duplicate ids. Fix before running; results keyed by id will silently overwrite.
• MEDIUM — fewer than 10 golden cases, or no adversarial cases. The success rate will be too noisy to gate on. Say what is missing and add cases before Step 6.
• LOW — a tag used by only one case. Per-tag metrics for it are meaningless; merge or add cases.

Step 3: Design the scoring

Deterministic checks first. A judge is the fallback for what no program can check, not the default.

Deterministic scorers (put each in evals/<unit>/scorers/):

• Exact match, after normalizing whitespace and case where the task allows it
• Regex match for "contains the order id" style checks
• JSON schema validation for structured output — invalid JSON is a fail, not a partial
• Tool-call trace equality: same tool names in the same order with the same arguments; allow an ignore_args list per case for arguments that legitimately vary
• Grounding check: every factual claim in the output that matches a pattern (a number, a date, a name) must appear in the provided context; anything else counts toward the hallucination rate in Step 4

LLM-as-judge, only for cases where expected is a rubric note rather than a value:

• Score directly: 1-5 against a written rubric, with the case's threshold deciding pass or fail. The runner produces one output per case, so there is nothing for the judge to compare against. "Did the new prompt improve on the old one" is answered by comparing this run's rates to the committed baseline in Step 6, not by asking the judge to rank two candidates.
• The judge model must differ from the generator model. Same-family models share failure modes and rate their own style higher.
• Write the rubric as a file: evals/<unit>/judge/rubric-v1.md. It states each score level with one concrete example. Include two or three few-shot examples of scored outputs at the top so the judge calibrates on the task rather than on general taste.
• The judge prompt is also a versioned file, evals/<unit>/judge/prompt-v1.md, and the results record its version. Changing the judge invalidates comparison with older runs.
• Calibrate against a human. Sample 20 judged items, score them by hand into evals/<unit>/judge/human-sample.jsonl, and report exact agreement (same score) and within-one agreement. Below 60% exact or 90% within-one, the judge is not measuring what the rubric says — rewrite the rubric before trusting the number.

Findings:

• HIGH — a judged case with no rubric file, or a judge that is the same model and prompt as the generator. The score is not evidence.
• MEDIUM — a rubric with no few-shot examples, or a human sample under 20 items. The agreement number is too noisy to act on.
• LOW — a deterministic check that could replace a judged case (for example, the rubric only asks whether the JSON has a field). Convert it.

Step 4: Define the metrics

Compute every metric overall and per tag. Per-tag is where regressions hide: an overall rate that holds steady while adversarial drops from 80% to 40% is a regression.

Record both price tables used in evals/<unit>/config.json so cost numbers are reproducible after a price change: "price" for the generator and "judge_price" for the judge, each {"in_per_mtok": <usd>, "out_per_mtok": <usd>} — USD per million tokens, the unit vendor price pages publish — and divide by a million wherever the runner turns token counts into dollars. The judge is a paid call like any other: its tokens go on the same ledger as the generator's, so a judge-heavy run cannot slip past the cap or under-report cost.

Findings:

• HIGH — a refusal rate above zero on golden cases, or below 100% on should-refuse cases. Either is a correctness bug, not a tuning detail.
• MEDIUM — p95 latency more than 3x p50. Usually a retry loop or a tool timeout; worth a look before gating.

Step 5: Write the harness

Use the project's own language. Python if the project is Python, TypeScript if it is TypeScript. No new runtime for the eval alone. Reuse the project's HTTP client and API wrapper so the harness exercises the real call path.

Layout:

evals/<unit>/
  cases.jsonl          # Step 2
  config.json          # models, temperature, seed, concurrency, cost cap, per-call token ceilings,
                       #   generator and judge price tables (USD per million tokens)
  scorers/             # Step 3 deterministic checks
  judge/               # rubric-v1.md, prompt-v1.md, human-sample.jsonl
  baseline.json        # Step 6, committed
  baseline-summary.md  # the accepted run's summary, committed next to baseline.json
  runs/                # results-<timestamp>.jsonl + summary-<timestamp>.md, gitignored
  run.py | run.ts      # the runner
  .gitignore           # one line: runs/

Write evals/<unit>/.gitignore containing the single line runs/ while scaffolding the directory, before the first run. Without it the timestamped results and summaries land in git on the first run, and the committed baseline-summary.md of Step 6 stops being the thing that carries provenance.

The runner, in order:

1. Load config.json; refuse to start if cost_cap_usd, max_tokens_in, max_tokens_out, price, or judge_price is missing. The token ceilings are what makes a worst-case call cost knowable in advance.
2. Read cases.jsonl; fail on duplicate ids.
3. For each case, call the unit with a fixed seed and temperature from config, and pass max_tokens_out as the call's output limit so the API enforces the same ceiling the ledger reserves against. Count the input first and fail the case if it is over max_tokens_in rather than dispatching it — the reservation covers both halves of a call, so both halves need a ceiling that holds. Cap concurrency with a semaphore (default 4).
4. Reserve before every paid call — generator and judge alike — and settle after it. The reservation is the worst-case cost of that call: the configured token ceilings at that model's price. Reserving and settling both happen under one lock over a single ledger, so concurrent cases can never each pass a check against the same remaining budget, and no call can start unless its whole worst case fits under the cap. Settling replaces the reservation with the usage the response actually reported, and checks it against the reservation: a call that reports more than its worst case means a ceiling did not reach the API, and the run stops there instead of continuing past the cap.
5. Apply deterministic scorers, then the judge for cases whose expected carries a rubric key. A case with neither a deterministic check nor a rubric is an error, not a pass — stop and name the case. A judged case passes only when its judge score reaches the case's threshold (judge_threshold, default 4).
6. Append one line per case that ran to runs/results-<timestamp>.jsonl: id, tags, output, passed, checks, judge_score, judge_version, latency_ms, tokens_in, tokens_out, judge_tokens_in, judge_tokens_out, cost_usd. cost_usd is the generator and judge cost for that case combined. A case that did not run gets no line — a record without passed and tags is worse than a missing one, because the Step 6 gate would count it.
7. Write summary-<timestamp>.md with the Step 4 table, overall and per tag. If any case did not run — a reservation that did not fit, or a case that raised — head the summary with INCOMPLETE: <ran>/<total> cases (<reason>), print the same line to stderr, and exit non-zero. A truncated run must never be able to satisfy the gate.

A Python skeleton the user can start from; keep the same shape in TypeScript:

#!/usr/bin/env python3
import json, time, sys, asyncio
from pathlib import Path

ROOT = Path(__file__).parent
CFG = json.loads((ROOT / "config.json").read_text())
for key in ("cost_cap_usd", "max_tokens_in", "max_tokens_out", "price", "judge_price"):
    assert key in CFG, f"config.json needs {key}"

def usd(price, tokens_in, tokens_out):
    return (tokens_in * price["in_per_mtok"] + tokens_out * price["out_per_mtok"]) / 1_000_000

WORST_CASE = {role: usd(CFG[key], CFG["max_tokens_in"], CFG["max_tokens_out"])
              for role, key in (("generator", "price"), ("judge", "judge_price"))}

class BudgetExhausted(Exception):
    pass

class CeilingBreached(Exception):
    pass

def check_input(case_id, role, text):
    n = count_tokens(text)
    if n > CFG["max_tokens_in"]:
        raise ValueError(f"{case_id}: {role} input is {n} tokens, over max_tokens_in "
                         f"{CFG['max_tokens_in']}")

class Ledger:
    """One cap covering generator and judge calls alike.

    reserve() claims a call's worst-case cost before it is dispatched;
    settle() swaps that claim for the usage the response reported. Both
    run under the lock, so two cases cannot pass the check against the
    same remaining budget, and no call starts unless its whole worst
    case fits. settle() also refuses a call that came back costing more
    than it reserved, which is the only way the cap can be walked past.
    """
    def __init__(self, cap):
        self.cap, self.committed, self.spent = cap, 0.0, 0.0
        self.lock = asyncio.Lock()

    async def reserve(self, amount):
        async with self.lock:
            if self.committed + amount > self.cap:
                raise BudgetExhausted(f"${amount:.4f} would exceed cap ${self.cap}")
            self.committed += amount

    async def settle(self, reserved, actual):
        async with self.lock:
            self.committed += actual - reserved
            self.spent += actual
        if actual > reserved:
            raise CeilingBreached(
                f"call cost ${actual:.4f} against a ${reserved:.4f} reservation — "
                "the token ceilings did not reach the API")

def load_cases():
    cases = [json.loads(l) for l in (ROOT / "cases.jsonl").read_text().splitlines() if l.strip()]
    ids = [c["id"] for c in cases]
    assert len(ids) == len(set(ids)), "duplicate case ids"
    return cases

async def run_case(case, sem, ledger):
    async with sem:
        expected = case.get("expected")
        judged = isinstance(expected, dict) and "rubric" in expected
        check_input(case["id"], "generator", case["input"])
        await ledger.reserve(WORST_CASE["generator"])
        t0 = time.monotonic()
        out, usage = await call_unit(case["input"], seed=CFG["seed"],
                                     temperature=CFG["temperature"],
                                     max_tokens_out=CFG["max_tokens_out"])
        latency_ms = int((time.monotonic() - t0) * 1000)
        cost = usd(CFG["price"], usage["in"], usage["out"])
        await ledger.settle(WORST_CASE["generator"], cost)

        checks = run_scorers(case, out)
        if not checks and not judged:
            raise ValueError(f"{case['id']}: no deterministic check and no rubric — case is unscored")

        judge_score = judge_version = None
        judge_usage = {"in": 0, "out": 0}
        if judged:
            await ledger.reserve(WORST_CASE["judge"])
            judge_score, judge_version, judge_usage = await run_judge(
                case, out, max_tokens_in=CFG["max_tokens_in"],
                max_tokens_out=CFG["max_tokens_out"])
            judge_cost = usd(CFG["judge_price"], judge_usage["in"], judge_usage["out"])
            await ledger.settle(WORST_CASE["judge"], judge_cost)
            cost += judge_cost

        threshold = case.get("judge_threshold", 4)
        passed = all(checks.values()) and (not judged or judge_score >= threshold)
        return {"id": case["id"], "tags": case["tags"], "output": out, "checks": checks,
                "passed": passed, "judge_score": judge_score, "judge_version": judge_version,
                "latency_ms": latency_ms, "tokens_in": usage["in"], "tokens_out": usage["out"],
                "judge_tokens_in": judge_usage["in"], "judge_tokens_out": judge_usage["out"],
                "cost_usd": round(cost, 6)}

async def main():
    sem = asyncio.Semaphore(CFG.get("concurrency", 4))
    ledger = Ledger(CFG["cost_cap_usd"])
    cases = load_cases()
    settled = await asyncio.gather(*(run_case(c, sem, ledger) for c in cases),
                                   return_exceptions=True)
    results = [r for r in settled if not isinstance(r, Exception)]
    errors = [r for r in settled if isinstance(r, Exception)]
    stamp = time.strftime("%Y%m%dT%H%M%S")
    (ROOT / "runs").mkdir(exist_ok=True)
    with open(ROOT / "runs" / f"results-{stamp}.jsonl", "w") as f:
        for r in results:
            f.write(json.dumps(r) + "\n")
    if any(isinstance(e, CeilingBreached) for e in errors):
        reason = "token ceiling not enforced"
    elif any(isinstance(e, BudgetExhausted) for e in errors):
        reason = "cost cap reached"
    else:
        reason = "case errors"
    incomplete = f"INCOMPLETE: {len(results)}/{len(cases)} cases ({reason})" if errors else None
    write_summary(results, ROOT / "runs" / f"summary-{stamp}.md",
                  incomplete=incomplete, spent_usd=ledger.spent)
    if errors:
        for e in errors:
            print(f"error: {e!r}", file=sys.stderr)
        print(incomplete, file=sys.stderr)
        sys.exit(1)

asyncio.run(main())

call_unit, count_tokens, run_scorers, run_judge, and write_summary are the five functions to fill in against the real project. call_unit must pass max_tokens_out through as the API call's output limit; one that accepts the argument and drops it is what lets a run spend past the cap. count_tokens returns the provider's own count for a string — the tokenizer the model uses, not a word count, since it is what the input half of the reservation rests on. run_judge returns (score, judge_version, usage): the score on the rubric's scale, the version from the judge/prompt-v1.md filename, and the judge call's own token counts, which is what puts the judge on the same ledger as the generator. It applies both ceilings to its own call the way run_case does for the generator — it is the only place that renders the judge prompt, so it is the only place that can check it. write_summary prints the incomplete line as the summary's first line when it is set. Do a dry run with concurrency: 1 and a cost cap of a few cents before the first full run.

Findings:

• HIGH — the runner calls the live unit with real side effects (Step 1 seam missing). Do not run it.
• MEDIUM — no seed or temperature in config. Two runs of the same prompt will differ and the gate will flap.

Step 6: Add the regression gate

Commit evals/<unit>/baseline.json from a run the user has looked at and accepted, and copy that run's summary-<timestamp>.md to evals/<unit>/baseline-summary.md in the same commit. runs/ is gitignored by the .gitignore written in Step 5, so a bare filename there is not something a later reviewer can open — the committed summary is the provenance.

{"run_stamp":"20260901T101500","summary":"baseline-summary.md","judge_version":"v1","success_rate":0.87,"cost_per_task_usd":0.0042,"per_tag":{"golden":0.95,"adversarial":0.70}}

The gate compares a fresh run to the baseline and fails on any of:

• a non-zero runner exit, or a summary headed INCOMPLETE. The run did not cover the case set, so its rates are not comparable — the gate fails without reading them, and such a run can never become a baseline.
• success rate drop greater than N points (default 3) overall or on any tag with 5 or more cases
• cost per task increase greater than M% (default 20)

Add it as a CI step that runs on changes to prompt files, the agent loop, model config, or evals/. Put the thresholds in config.json, not in the CI yaml, so they are reviewed with the code. A gate run needs its own cost cap, lower than a local run.

Updating the baseline is a deliberate act: a separate commit that changes only baseline.json and baseline-summary.md, reviewed by someone other than the author of the prompt change. Never let CI rewrite the baseline on green.

Findings:

• HIGH — no baseline committed, or a CI job that regenerates it automatically. The gate cannot fail.
• MEDIUM — thresholds looser than the noise floor. Run the baseline twice with the same config; if the two runs differ by more than N, N is too tight or the seed is not honored.

Output

Present the plan before writing the harness, then the results after the first run.

AGENT EVAL PLAN
===============

Unit:        <path or name> — <one line: what it takes, what it returns>
Side effects: <none | stubbed via ... | BLOCKED: ...>
Generator:   <model>, temperature <t>, seed <s>
Judge:       <model> (differs from generator), rubric-v1, direct 1-5

Task set:    <N> cases — golden <n>, mined <n>, adversarial <n>
Tags:        <tag list with counts>

Scoring:     deterministic: <exact|regex|schema|trace|grounding>
             judged: <n> cases, threshold >= 4/5
             human sample: 20 items, agreement <pending|0.xx exact / 0.xx within-1>

Metrics:     success rate, tool-call correctness, hallucination rate,
             refusal rate, latency p50/p95, cost per task — per tag
Gate:        fail on success drop > N pts, cost increase > M %
Cost cap:    $<x> per local run, $<y> per CI run


RESULTS — <run file>
====================

Run:     complete | INCOMPLETE <ran>/<total> cases (<reason>) — exit 1, gate fails
Spend:   $<x> of $<cap> cap (generator + judge)

Metric                  Overall   golden   mined   adversarial   vs baseline
--------------------    -------   ------   -----   -----------   -----------
Success rate            0.87      0.95     0.85    0.70          -0.02
Tool-call correctness   0.91      0.97     0.90    0.80          +0.01
Hallucination rate      0.04      0.00     0.06    0.10          +0.01
Refusal rate            0.03      0.00     0.02    0.20          0.00
Latency p50 / p95 ms    1840/4200                                +5% / +12%
Cost per task           $0.0042                                  +3%
Judge agreement         0.65 exact / 0.95 within-1 (20 items)

Gate: PASS | FAIL (<reason>)

Findings
  [HIGH]   <finding> — <where> — <what to do>
  [MEDIUM] ...

Files created
  evals/<unit>/cases.jsonl
  evals/<unit>/config.json
  evals/<unit>/scorers/...
  evals/<unit>/judge/rubric-v1.md
  evals/<unit>/judge/prompt-v1.md
  evals/<unit>/run.py
  evals/<unit>/.gitignore
  evals/<unit>/baseline.json
  evals/<unit>/baseline-summary.md
  .github/workflows/agent-eval.yml (or the project's CI equivalent)

Important Rules

1. Never call a paid API without a run cap. Every run, local or CI, has cost_cap_usd in config. The runner reserves each call's worst-case cost against that cap before dispatch — generator and judge on one ledger — and settles the actual usage afterwards. The same token ceilings go to the call itself: the input is counted and refused if it is over, the output limit is passed to the API, so the worst case reserved is one the call cannot exceed. A missing cap is a refusal to run, not a warning.
2. A run that stops early is a failed run. When a reservation does not fit, or a case raises, keep the lines for the cases that completed, head the summary INCOMPLETE, and exit non-zero. Never write a placeholder record for a case that did not run, and never let a partial run pass the gate or become a baseline.
3. Judge prompts and rubrics are versioned files. Never an inline string. Every result line records the judge version it was scored with.
4. Never score with the generator. The judge is a different model from the one that produced the output, and it never sees the generator's system prompt as its own instructions.
5. Expected outputs are written before the run. Copying a live output into expected is a snapshot, not an eval. Say so if the user asks for it, then do it only for cases they have read.
6. Deterministic before judged. If a program can check it, a program checks it. The judge covers only what remains.
7. Read-only toward everything outside evals/. This skill creates and edits files under evals/<unit>/ and the CI config it names. It does not change the prompt, the agent, or the model config; it never modifies cloud resources, purchases anything, or writes to a customer account. Proposed changes to the unit under test go in the report as findings.
8. Strip PII from mined cases. No names, emails, account ids, or free-text that identifies a person lands in cases.jsonl.
9. Baseline updates are separate commits. One commit, only baseline.json and the baseline-summary.md it points at. CI never rewrites it.
10. Report noise. If two baseline runs disagree by more than the gate threshold, say the gate cannot hold at that N and propose a wider one or a larger case set.

{{include lib/snippets/capture-learnings.md}}