Vat Skill Testing

VAT Skill Testing: Behavioral Downstream Packaging Check

vat skill test answers one question: does this packaged skill actually work when installed in isolation? It stages the built skill into a fresh, context-isolated temp harness — a scrubbed env allowlist and no user/project settings, not an OS security sandbox — and runs a headless behavioral evaluation, no interactive session required.

Boundary: Where vat skill test Fits

vat skill test reuses skill-creator's grading rubric and JSON shapes for grading output, but not its interactive driver. It is a post-packaging validation step, not a skill-authoring aid.

Cost & Cadence — Where This Fits in Your Workflow

A full run is not cheap: on the order of ~38 headless claude sessions (an executor and a grader spawned per eval), ~8–13 minutes wall-clock, and real token spend. Treat vat skill test as a pre-release / nightly / on-demand check — not a gate on every push or commit. Keep the per-commit loop on fast static checks (lint, typecheck, unit tests); reach for vat skill test before a release, on a schedule, or when you've changed the skill's behavior and want a behavioral signal.

To make that cadence budgetable, the run's summary line reports the actual spend it just incurred — a ≈$<total> across <N> sessions suffix that rolls up total_cost_usd across every executor and grader session (so it counts both halves of each eval, not just the skill run). Read it off a real run to size a nightly/pre-release schedule; it's omitted only when no session reported a cost (e.g. a fully mocked test).

Cost-tier fail-fast (see below) helps at the margin, not the order of magnitude. Gating expensive tiers behind cheap ones stops a broken foundation from burning tokens on hard cases, but a full green run through every declared tier is still tens of sessions and minutes — tiering changes when you stop spending on a bad run, not how much a good run costs.

--dry-run is the underused, zero-token way to validate plumbing. It assembles the executor command — staging, model resolution, evals.json parsing, toolExpectations/declaredExecutables resolution — without spawning claude, so it catches config typos and broken wiring before you spend anything:

# No --i-understand-this-runs-skill-code needed — a dry-run never executes skill code
vat skill test run my-skill --dry-run

Run --dry-run first any time you touch evals.json, skills.config.<skill>.test, or executables.

The Execution Loop

VAT owns the loop directly — there is no single "experimenter" agent. For each eval, VAT spawns a blind executor (the skill under test), captures its transcript in memory, then spawns a separate grader over that transcript. VAT — never the model — writes the results.

driver (you)
  └─ vat skill test run [source]
       ├─ 1. Resolve source (local path, git URL, npm)
       ├─ 2. Stage: copy packaged skill + declared deps into an isolated temp harness
       ├─ 3. Preflight: token-free fail-fast checks (see below)
       └─ 4. Per eval (bounded-parallel, --concurrency), lowest cost tier first:
            ├─ Spawn EXECUTOR (headless `claude`, the skill under test, `--model`)
            │    └─ transcript captured IN MEMORY (never written to the sandbox)
            ├─ Spawn GRADER (headless `claude`, `--grader-model`) over that transcript
            │    └─ emits a nonce-stamped fragment (expectations + optional tool verdict)
            └─ VAT merges the fragments → results/ (sole writer; nonce re-verified)

Artifacts produced under <harnessRoot>/results/:

results/ survives every run. The harness root's path is echoed as Harness: and the results directory as Results: — use the latter. On a default run (no --out, --workdir or --keep) everything else under the harness root is removed when the command exits: the staged skill bytes are what cleanup is for, the artifacts are the product. (Harness: is printed only when that root is still on disk — a run that ends before results/ exists, such as the security-ack refusal, has its root removed outright.)

--out/--workdir spare only the harness ROOT. They name a directory you own, so vat never deletes it and the staged untrusted skill bytes inside it stay until you remove them. That is not a blanket "keep everything": three vat-owned directories live outside that root — the grader's output dir, the held eval suite, and the per-eval executor workspaces (every eval's working directory, and everything the evals wrote) — and they are reaped on a rule that ignores both flags. --keep is the only flag that retains the workspaces, which is why the Workspaces: line is printed only under --keep. If you pass --out to inspect what the evals produced, add --keep or the per-eval output is already gone.

Executor transcripts are held in memory and never written to disk — not to results/, not to the sandbox. That is the anti-forgery property below, not an omission; when an eval fails unexpectedly, the grader's per-expectation evidence in grading.json is the quoted transcript excerpt to read.

Anti-forgery model. The executor and grader are separate roles; the transcript never touches the skill's sandbox; the grader runs in a directory outside the harness root; and every grader fragment carries a secret per-run nonce (delivered only via the grader's stdin — never on disk or an argv) that VAT re-verifies before merging. So untrusted skill code running in the sandbox cannot forge its own passing grade, tamper with the transcript, or write a result VAT will accept. Full contract: docs/skill-test-grading-schema.md.

Every spawn passes --no-session-persistence, and that is part of the model above, not housekeeping. Claude Code otherwise writes each headless session to $CLAUDE_CONFIG_DIR/projects/<cwd-slug>/<uuid>.jsonl — plaintext, retained indefinitely — and VAT has to forward CLAUDE_CONFIG_DIR and HOME to the child because that is where auth lives. Those files carry the grading nonce, the eval expected_output, and (in a --baseline run) the treatment arm's whole transcript, all readable by the skill under test, and readable by the next run weeks later. Preflight fails closed (exit 2) on a claude that does not support the flag rather than running without it.

Exit Codes

A dead control arm does not change the exit code. In a --baseline run every failure listed under 1 above — executor spawn error, stall, wall-clock timeout, grader non-zero exit, grader wrote no fragment, unparseable fragment, nonce mismatch, grader returning the wrong number of expectations — is recorded and survived on the skill-withheld arm rather than thrown. That is deliberate: grading.json is the treatment arm, and destroying a completed, already-billed treatment run because the control half died costs more than losing the comparison. The consequence for CI is sharp, and it has been measured: a control executor that hits the watchdog timeout, and a control grader that exits status 3 writing no fragment, both finish exitCode 0, PASS 2/2. The case $? block below cannot see either. What you get instead is a stderr warning as it happens, "delta": null in baselineDelta, and an entry in baselineIntegrity.controlArmFailures naming the eval and the failure.

The taxonomy separates "evals failed" (4) from "the harness broke" (1/2/3) so a CI consumer can tolerate the former while failing closed on the latter:

vat skill test run my-skill --i-understand-this-runs-skill-code
case $? in
  0) ;;              # all evals passed
  4) ;;              # evals failed but the harness is healthy — tolerate/warn
  *) exit 1 ;;       # 1/2/3/unknown — harness broke, fail the build
esac

On a --baseline run that block is necessary but not sufficient. Exit 0 says the treatment arm completed; it says nothing about whether the comparison exists. If your gate is about the lift, read the artifact, not the exit code — require baselineDelta.delta !== null, baselineIntegrity.controlArmFailures empty, baselineIntegrity.comparable true, baselineIntegrity.degraded empty, and baselineIntegrity.contaminated false. Treat a missing or unparseable baseline.json as a failure, exactly as for grading.json.

Which specific evals failed lives in results/grading.json, never in the exit code.

A timed-out run may leave results/ incomplete or a grading.json unparseable. Any consumer reading grading.json must treat an unparseable or missing file as a failure, not crash on it. The exit code (1) already tells you the run did not complete; do not trust a partial artifact.

The default --timeout scales with the suite's declared eval count (roughly 2min + 2min/eval, floored at 5min, capped at 1h) so a correctly-configured multi-eval suite is not truncated by a flat budget. An explicit --timeout always overrides. If a run times out, the message names the declared eval count — raise --timeout (and --stall) for large suites.

Executor vs Grader Models

Two independent models run per eval, and you pick each one:

• --model is passed verbatim to claude --model (no VAT mapping/validation) — pin it for reproducible, cost-controlled runs.
• --grader-model is independent of --model. Keeping the grader fixed while you vary the executor is the point: it holds the yardstick steady.
• --concurrency <n> (default 4) bounds how many evals run their executor→grader pair in parallel (each retries a rate-limit with backoff).

graderModel and concurrency are GLOBAL, not per-skill: they live in a top-level test: node in vibe-agent-toolkit.config.yaml (distinct from the per-skill skills.config.<skill>.test block), because they describe the judge/pipeline, not one skill. vat skill test configure writes the per-skill block only — set the global ones with the --grader-model / --concurrency flags, or by hand-editing the top-level test: node:

# vibe-agent-toolkit.config.yaml
test:                      # GLOBAL — judge + pipeline
  graderModel: claude-sonnet-5
  concurrency: 4

skills:
  config:
    my-skill:
      test:                # PER-SKILL — executor + this skill's knobs
        model: claude-opus-4-8

Precedence for the global knobs: flag > top-level test: config > built-in default.

Bootstrap Flow (Exit 3)

First run with no evals.json:

vat skill test run ./dist/skills/my-skill/ --i-understand-this-runs-skill-code
# Exit 3 — VAT scaffolds evals.json template next to the skill source

Edit evals.json to fill in expected behaviors, then re-run. The template includes annotated comments explaining each field.

Authoring evals.json — the part that actually matters

A run is only as good as its evals. The harness, isolation, and grading are plumbing; the eval suite is where you encode what "working" means for your skill. These practices come straight from Anthropic's skill-creator methodology and its grader rubric — VAT reuses both.

The suite lives at <skill>/evals/evals.json (override with skills.config.<skill>.test.evals); input fixtures live alongside it under <skill>/evals/fixtures/.

The shape

{
  "skill_name": "csv-summarizer",
  "evals": [
    {
      "id": "line-totals-reconcile",
      "category": "recognition-accuracy",
      "prompt": "Can you take a look at this orders export and tell me whether it adds up? I just need to know if unit price times quantity ties to the line total before I send it on.",
      "files": ["fixtures/orders-clean.csv"],
      "expected_output": "Recognizes the file as a monthly orders export (8 rows), runs the reconciliation, and reports in plain English that unit_price x quantity ties to line_total for every row.",
      "expectations": [
        "The output identifies the file as a monthly orders export with 8 rows.",
        "The output states unit_price x quantity reconciles to line_total for every row.",
        "The output does NOT claim any row fails to add up."
      ],
      "tier": 0,
      "toolExpectations": {
        "mustRun": ["csvsum"],
        "mustNotRun": ["rm"],
        "sequence": ["csvsum parses the orders export", "csvsum reports the reconciliation"]
      }
    }
  ]
}

• id — unique identifier; an integer or a descriptive string (descriptive ids read better in results).
• category — optional grouping label you choose; VAT carries it through untouched.
• files — optional input paths relative to the evals.json directory; each eval's files are staged into its own isolated working directory.
• expectations is required and needs at least one entry — it is what the grader scores (pass/fail per entry).
• expected_output is optional prose describing a correct result; when present the grader receives it as context informing its judgment, but the verdict is still decided per expectations entry.
• tier (optional, integer) — cost tier for fail-fast ordering (see Cost tiers below). Omit to leave an eval in the default tier 0.
• toolExpectations (optional) — assert which tools/executables the skill should (and shouldn't) invoke (see Tool expectations below).

Write blind, realistic prompts

The executor is never told it's being tested — and it shouldn't be able to tell. Phrase prompts exactly the way a real user would ("Quick sanity check on this month's orders export, please."), never "Test that the skill computes line totals." This is not cosmetic: models behave measurably differently when they sense an eval, so a benchmark-flavored prompt measures the wrong thing.

Write discriminating expectations

Every expectation must fail for a wrong output. The classic trap — which the grader is explicitly told to flag — is checking mere presence: "the output mentions Acme Freight" also passes for a hallucinated document. Check correctness, tied to the input, not presence. And pair positive assertions with negative ones ("does NOT claim every row reconciles"): one-sided evals create one-sided optimization. If an assertion would pass for an obviously broken result, it's not pulling its weight.

The harness itself is only as sharp as the evals you write — a passing toolExpectations/expectations grade tells you nothing if the assertion couldn't have failed. Antipattern: a presence-only check with no negative counterpart. "the output mentions the order id" or "the output includes a reconciliation summary", on its own, passes for a hallucinated order id or a summary that reaches the wrong conclusion — it can't distinguish "did the right thing" from "said the right words." vat skill test emits an advisory lint warning when every expectation in an eval is presence-only ("mentions/includes/contains/references/appears/states…") with no discriminating or negative cue and no toolExpectations declared. It's a nudge, not a gate — it never blocks a run or changes the exit code — so treat it as a prompt to strengthen the eval, not a substitute for writing discriminating expectations in the first place.

A second advisory in the same family catches a quieter footgun: when a toolExpectations entry names an executable that looks like a typo of one of the skill's declaredExecutables (e.g. csvsum-py when the skill declares csvsum), the harness warns before any spend. A misspelled tool name never matches the transcript, so mustRun/mustSucceed/sequence fail for the wrong reason and mustNotRun passes vacuously — the assertion looks like it checks a tool but can't. This lint is conservative (it fires only when there's a specific declared name the reference is probably a typo of, so a deliberate Bash/git reference is never flagged) and, like the presence-only lint, surfaces under --dry-run — so a zero-token dry run catches the typo before you spend.

Before/after, same eval:

// Before — antipattern: also passes for a hallucinated customer
"expectations": [
  "The output mentions Acme Freight as the top customer."
]

// After — discriminating: fails a wrong or invented result
"expectations": [
  "The output identifies Acme Freight as the top customer by revenue, matching customer_name in the input file.",
  "The output does NOT invent a customer not present in the input file."
]

Cover real scenarios; aim for ≥3 evals

Anthropic's bar is at least three evaluations, drawn from real usage and past failures, tested across models. Group evals by what they exercise — the csv-summarizer example above uses three categories:

Keep regression evals (known-good behavior that should always pass) mentally separate from capability evals (harder cases). A suite that scores 100% every time is usually too easy — a healthier capability suite leaves headroom that improvements can move.

Fixtures

Put input files under evals/fixtures/ and reference them via files. Use realistic, slightly messy data — the kind that surfaces real bugs, not toy data that everything passes.

A/B instruction-lift (--baseline)

A skill's value is its lift over what the model already does without it. --baseline runs each eval twice — with the skill and without — and reports the delta on stderr and in baseline.json. If an expectation passes in both configurations it proves nothing about the skill; make the test harder, or focus the skill on where the model genuinely needs help.

Baseline delta: +2 (with skill: 3/3, without skill: 1/3).

What the control arm actually withholds — read this before trusting a delta. The skill-absent arm is denied the skill's declaration: no --plugin-dir, and --setting-sources "" suppresses user/project settings, so the agent is never told the skill exists. It is not denied capability. The executor runs with unrestricted Bash, and the harness is context isolation, not an OS sandbox — any copy of the skill still on the filesystem is reachable in principle.

vat keeps its own copies out of that arm's way: the skill-absent arm's prompt does not name the staged subject, its working directory is a per-eval workspace outside the harness root, and it gets no plugin dir. What vat cannot remove is an ambient copy you own — your repo's dist/, a build output, or an installed plugin cache. If the control arm finds one of those and runs it, the delta stops measuring the skill.

So --baseline A/Bs the skill's instructions, on the honest assumption that no ambient copy is reachable. Read the delta accordingly:

• A near-zero delta on a skill that ships an executable is suspect before it is meaningful — check baselineIntegrity first.
• The delta is not a capability measurement. It cannot tell you "the model can't do this without my tool"; it tells you "my prose changes what the model does."

baselineIntegrity in baseline.json. Every baseline run stamps this block, clean or not (its absence means the file predates the check, never "checked and clean"):

"baselineIntegrity": {
  "contaminated": true,
  "comparable": false,
  "degraded": [ { "evalId": "lookup-2", "reason": "cwd-untracked", "detail": "cd \"$SOMEVAR\"" } ],
  "skew": [ { "evalId": "lookup-3", "withTotal": 3, "withoutTotal": 0 } ],
  "controlArmFailures": [ { "evalId": "lookup-3", "detail": "[control arm (skill withheld)] Executor timed out after 300s" } ],
  "signals": [ "harness-path", "declared-executable", "skill-content" ],
  "summary": "BASELINE CONTAMINATED: the skill-absent arm reached the skill in 2 eval(s) …",
  "findings": [ { "evalId": "lookup-1", "hits": [ { "kind": "harness-path", "match": "…", "excerpt": "…" } ] } ]
}

All eight fields are required and unconditional — an absent one means the file predates the field, never "checked and clean". Three of them are easy to skip past and each disqualifies a different thing:

• degraded — the evals whose contamination scan did not run at full strength, and why. Five reasons, and they are three different shapes — do not triage them alike:

  • Fell back to flat text matching (transcript-unparsed — the transcript yielded no stream-json events at all; cwd-unknown — no arm workspace was threaded through, so no relative path can be anchored). There is nothing to walk, so the scan reverts to the blunt instrument, which both over-reports (a find that merely prints a path reads as a reach) and under-reports (a relative reach loses the leading slash a bare-name needle requires).
  • Left a forward-only hole (cwd-untracked — a cd the walker could not evaluate, e.g. cd "$SOMEVAR", cd -, bare cd; transcript-malformed — lines that failed to parse and were silently dropped). The structured walk still ran: everything resolved before the hole is validly resolved and is kept, and only what came after — and needed a cwd, or sat on a dropped line — is lost. These do not fall back; falling back for them discarded good evidence and reintroduced the flat scanner's over-reporting, where one leading cd $HOME turned a pure directory listing into a "reached the answer key" verdict.
  • Could not judge one reach (glob-unexpanded — a shell expansion stood where a directory name belongs, e.g. cat ../../vat-*/…/SKILL.md, so no literal needle could match it). The walk ran to completion and lost nothing before or after; exactly one path could not be resolved to a real location, and vat will not guess which one it was. Anything switching on the reason enum must accept this shape too: it is neither a fallback nor a hole.

  transcript-malformed is the one degradation invisible from the outside: the parser drops the corrupted line, the surviving lines still decode, and one truncated tool call therefore deletes a contamination hit under a confident verdict. Either way, a non-empty degraded means contaminated: false was written by a scan that did not really look — the difference between "checked and clean" and "checked with a hole in it", which signals cannot tell you. Empty is the only state in which a clean verdict means what it says.

• controlArmFailures — evals whose skill-withheld arm produced no grade at all, each with the spawn/grader failure that stopped it. Reported separately from skew on purpose: skew says "the two graders disagreed about the job", this says "half the experiment never ran". Non-empty is the case the exit code stays 0 for (see Exit Codes), so this is the field a --baseline CI gate has to read.

• comparable / skew — whether the two arms were graded against the same expectations at all. A run can be perfectly clean and still incomparable.

contaminated: true also prints a warning to stderr. When you see it, discard the delta — the control had the treatment. The usual cause is an ambient copy: uninstall the plugin, or run against a tree that has no built copy of the skill. The number is still printed: contamination does not make the arithmetic wrong, it makes interpreting the result as skill lift wrong, and the warning sits directly under the number that it disqualifies.

baselineDelta in baseline.json. The subtraction itself, run-level and per-eval:

"baselineDelta": {
  "with":    { "passed": 3, "total": 3 },
  "without": { "passed": 1, "total": 3 },
  "delta": 2,
  "perEval": [ { "evalId": "lookup-1", "withPassed": 3, "withTotal": 3, "withoutPassed": 1, "withoutTotal": 3, "delta": 2 } ],
  "controlArmFailures": [],
  "truncated": null
}

controlArmFailures is required here too — the same derivation as the one in baselineIntegrity, handed to a second reader rather than computed twice, so someone who opens baselineDelta, finds delta: null, and does not yet know the contamination vocabulary can learn why from the block that withheld the number.

truncated is required and is null when the whole declared suite ran — an absent key would make "this delta covers everything" indistinguishable from a file written before the field existed, which is the same argument that makes baselineIntegrity unconditional. When cost-tier fail-fast gates the run it holds gatedByTier, firstSkippedTier, totalSkipped and the skipped evalIds. It does not withhold the delta: the tiers that ran, ran on both arms, so their subtraction is legal — what truncation changes is the scope of the claim. This matters more than it sounds, because --baseline exists to measure a skill that may not be helping, so a gating treatment-arm failure is the expected state of an interesting baseline run. A tier-0 failure in an 8-eval suite prints Baseline delta: +1 (with skill: 2/2, without skill: 1/2) — byte-identical to a complete 2-eval suite that measured everything it declared. Read truncated before quoting the number.

delta: 0 and delta: null mean different things and have different remedies. 0 is a measurement — the skill lifted nothing on those expectations, which is a real finding about the skill. null is a refusal to measure: that eval's two arms were graded against a different number of expectations, so the totals have different denominators and subtracting them would not be a delta. null appears per-eval for each skewed eval and run-level if any eval is skewed; the evals behind it are listed in baselineIntegrity.skew, and comparable goes false. It is reported separately from contaminated — a run can be clean and incomparable at once.

There are two causes and they have different remedies, so check controlArmFailures before assuming either:

• The control arm died (baselineIntegrity.controlArmFailures non-empty) — its executor or grader timed out, crashed, or wrote nothing, so that eval was graded on one arm only and shows withoutTotal: 0. This is now the likelier cause, and it is the one the exit code hides. Remedy: re-run (raise --timeout/--stall if it was a watchdog), and check the stderr warning naming the eval.
• A misbehaving grader (controlArmFailures empty, skew non-empty) — both arms ran, but one grader returned a different number of expectations than the eval declares. Remedy: audit the grader prompt and the eval's expectations; re-running usually reproduces it.

What the block can and cannot see. signals lists which detectors were armed for the run, so a clean verdict can be told apart from a blind one — an empty list means nothing was looking. The harness-path signal always runs. The declared-executable signal needs the subject's executables manifest, which resolves only for a declared subject (a skill name, or a path that maps back to one) — exactly the same restriction toolExpectations carries. Test by name for the fuller check; a bare path to a built dist/ leaves that signal unarmed. It also fires only on a path that reaches OUTSIDE the arm's own workspace (absolute, ~- or $VAR-rooted, or climbing out): a control arm denied the skill writes its own analyze.py and says so, and that is the behaviour it is supposed to exhibit, not a reach. The skill-content signal covers what the others cannot — an instruction-only skill found as an ambient copy names no path and no executable, so vat matches verbatim lines of your SKILL.md body instead. Lines your eval prompts or expectations already quote are excluded (the arm read them from vat, not from the skill), so a skill whose body is entirely quoted by its own suite leaves that signal unarmed. The block raises the floor on silent contamination; it does not prove a clean run.

Writing prompts that don't defeat their own baseline. An eval prompt that names the executable or its path (run ${CLAUDE_SKILL_DIR}/scripts/tool.mjs …) hands the skill-absent arm the answer, and no harness change can fix that. Describe the task, never the mechanism — that is also what makes the WITH arm a real test of whether the skill routes the model correctly.

Tool expectations — what the skill should run

Output correctness is not the whole story: a skill can produce the right answer for the wrong reason (e.g. reasoning it out instead of running the executable it ships). toolExpectations lets an eval assert the skill's tool behavior, judged by the grader from the transcript (preferring the structured tool_use/tool_result entries):

Names are matched leniently — the grader recognizes varied launch forms of the same executable (uv run csvsum.py, python3 csvsum.py, ./csvsum, node dist/csvsum.mjs) as all "running csvsum", so you assert the tool, not an exact command string.

mustRun means invoked, not succeeded. The verdict is judged from the transcript, which records that a tool was called and how — it cannot see a script's exit code through a shell wrapper (a Bash tool_result.is_error is false even for a command that exits non-zero). So a skill that invokes a broken executable and works around the failure still satisfies mustRun. Use mustSucceed (below) when you need "ran and succeeded."

mustSucceed closes most of that gap — it asserts the tool ran and its invoking tool_result was not an error — but it is still transcript-judged, not a captured real exit code. Be honest with yourself about the limit: a skill that swallows a non-zero exit (cmd || true, catching and silently ignoring a subprocess failure) can still read as succeeded, because the grader is judging what the transcript shows, not re-executing anything. For a hard guarantee, pair mustSucceed with a discriminating output expectations entry (e.g. "the output reflects a successful csvsum run, not an error fallback"). Where it applies, prefer mustSucceed over the older workaround of asserting "ran and succeeded" purely in prose — it gives you a structured verdict in tool-eval.json instead of one entangled with the output grade:

"toolExpectations": {
  "mustRun": ["csvsum"],
  "mustSucceed": ["csvsum"],
  "mustNotRun": ["rm"]
}

Declare your executables so the grader knows their names. A mustRun: ["csvsum"] resolves against the skill's declared executables in skills.config.<skill>.executables — each entry is { path, kind, howInvoked } (kind ∈ node|python|shell|pwsh|binary). The referenced name matches the path basename with its extension stripped (scripts/csvsum.py → csvsum) or the exact path. These flow into the grader prompt as recognition hints:

skills:
  config:
    my-skill:
      executables:
        - path: scripts/csvsum.py
          kind: python
          howInvoked: uv run csvsum.py

Tool verdicts land in their own tool-eval.json channel and never leak into grading.json. They ride the WITH-arm only in a --baseline run — the skill-absent arm was never given the skill, so there is no declared tool use to hold it to. That is a statement about what the arm was handed, not a guarantee it reached no tool: if it found an ambient copy anyway, baselineIntegrity in baseline.json is what tells you. toolExpectations is only meaningful for a declared skill subject (name, or a path that maps back to a declared skill) — a plain path/source subject has no executables manifest to resolve names against.

Only committed (or files:-injected) files stage. A name-target build stages the skill via a tracked-files tree-copy, so an untracked script (a scratch probe.mjs you never git add-ed) silently won't be in the harness — a mustRun against it then fails because the file is absent, not because it ran. Commit test scripts, or inject non-artifact files through a files: entry (the same mechanism skills use to bundle a built CLI).

Cost tiers — fail fast before you spend

Give an eval a numeric tier to order the run by cost. VAT runs ascending tiers (cheapest first), bounded-parallel within a tier, and gates between them: once a cheaper tier fails a gating expectation, the higher (more expensive) tiers are SKIPPED — never graded, never counted as passed — and the run is a fail-fast eval failure (exit 4). Put cheap, foundational checks (does it invoke at all? does it recognize the input?) in tier 0, and expensive end-to-end cases in higher tiers, so a broken foundation stops the run before it burns tokens on the hard cases. The summary names the skipped tier and eval count. Omit tier and everything runs in tier 0 (no gating).

How grading works (so you can trust the verdict)

For each eval a separate headless grader judges the executor's transcript and output files against the expectations, requiring evidence per verdict, with no partial credit and the burden of proof on the expectation. It grades the artifact, not the agent's self-reported success — and it is a different agent and (by default) a different model from the executor, so a skill cannot grade itself. VAT then recomputes the pass/fail counts from the per-expectation flags rather than trusting any self-reported summary (a mismatch is a loud GradingSkewError, not a silent pass). The reported verdict is composite: output expectations AND every declared tool verdict must pass, and the run fails closed (exit 4) if any of grading.json/friction.json/tool-eval.json is missing or invalid after the merge. Full contract: docs/skill-test-grading-schema.md.

A single red eval is a signal to investigate, not proof of a regression. The grader is a model judging free-form transcript evidence, and verdicts wobble — the same eval, same skill, same code, can flip pass/fail between two runs. Do not wire vat skill test as a hard 100%-pass CI gate; leave headroom (see Cover real scenarios above) and treat one failing eval in an otherwise-green suite as "read results/grading.json and the transcript," not "the build is broken." This is exactly why the design above leans the way it does: evidence-required, no-partial-credit grading with the burden of proof on the expectation, and a self-contradictory or missing grade fails closed (GradingSkewError / exit 4) instead of being silently waved through as a pass. The strictness is there so that when a FAIL shows up, it's telling you something real to look at — not so that every run should be expected to come back all-green.

Security Caveat and Required Acknowledgment

vat skill test run executes the skill's code on your machine. The headless session runs with --permission-mode bypassPermissions, so staged skill files run with your user account's full privileges — they can read/write your files (including credentials under ~/.claude, SSH keys, cloud configs), run shell commands, and make network requests, and the auth credential billing the run is reachable by that code. The harness gives context isolation, not an OS security sandbox, so only run skills you trust. Before spawning the executor, the command prints this warning and requires the --i-understand-this-runs-skill-code flag:

# Required — preflight exits 2 without it
vat skill test run ./dist/skills/my-skill/ --i-understand-this-runs-skill-code

--allow-unverified-skill-source skips the integrity check of the vendored skill-creator copy (during preflight, the harness verifies the committed skill-creator's per-file hash manifest; a missing or mutated manifest fails preflight with exit 2). Pass it only when you knowingly run against a modified or unverifiable vendored skill-creator:

--allow-unverified-skill-source

It does not bypass any URL/sha256 verification of the subject skill source — that verification is handled separately by the source resolver. Both --allow-unverified-skill-source and --i-understand-this-runs-skill-code are intentional friction — they ensure you have reviewed what runs before running it.

Auth Model

The preflight probe is always token-free (claude auth status --json — never a -p call).

--auth modes

…(truncated)