Jev Goal

jev-goal

Pre-register acceptance criteria, do the work, then let Jev (TypeSafe AI's evaluation model, via Vercel AI Gateway) grade the result from evidence the script gathers itself. Loop until every criterion passes. Cheap, fast, and the agent never grades its own summary.

Core rule: criteria are written and frozen BEFORE any work starts, and never edited afterwards. The script enforces this with a hash; the point is that you cannot bend the bar to fit what you built.

Workflow

Script: ~/.claude/skills/jev-goal/scripts/grade.mjs (Node 22+, key in scripts/.env).

1. Draft criteria at <project>/.claude/jev/<task-slug>.json (format below). Do this before reading much code and before editing anything. Do not show the criteria to the user for approval; they delegated that. Write the file with the Write tool or node -e + JSON.stringify, not a Bash heredoc: the Bash tool strips one level of backslashes from heredoc bodies, which silently corrupts regexes and sed delimiters.
2. Freeze: node ~/.claude/skills/jev-goal/scripts/grade.mjs freeze <file>. Dry-runs every evidence command once, then records a hash and the base commit. Failures and empty output are expected at this point (the work does not exist yet); the freeze is refused only when a command cannot run at all (shell syntax error, unknown command, malformed sed/grep expression). If refused, the file is not frozen: fix the command and freeze again. Read the dry-run output before moving on; a jev criterion flagged as printing nothing needs evidence that will produce text later.
3. Do the work as normal (other skills still apply: TDD, debugging, verification).
4. Grade: node ~/.claude/skills/jev-goal/scripts/grade.mjs grade <file>.
5. Loop: exit 1 means some criteria failed. The output names them. Fix, then grade again. Repeat until exit 0.
6. Report. On PASS the script prints a block between --- jev-goal confirmation and --- end confirmation ---: the task, when the criteria were frozen, the round count, and a table of every criterion with its question, check mode, and result. Paste that block verbatim into the final message, then summarise the work. The user reads it as proof of what was promised before the work started and how each promise was checked.

Exit codes: 0 pass · 1 fail, keep working · 2 lock/usage error · 3 maxRounds hit, stop and report · 4 grader unavailable (rate limit etc.), wait 60-120s with a background sleep and grade again; not a verdict.

Criteria file

{
  "task": "add rate limiting to POST /upload",
  "criteria": [
    { "id": "tests_pass",     "check": "exit0", "question": "Does the test suite pass?",            "evidence": "npm test" },
    { "id": "typecheck",      "check": "exit0", "question": "Does the type check pass?",            "evidence": "npx tsc --noEmit" },
    { "id": "scope",          "check": "exit0", "question": "Are changes limited to src/routes/ and test/?", "evidence": "test -z \"$(git diff --name-only \"$JEV_BASE\" | grep -vE '^(src/routes|test)/')\"" },
    { "id": "limit_in_route", "check": "jev",   "question": "Does the diff add a rate-limit check to the upload route handler?", "evidence": "git diff \"$JEV_BASE\" -- src/routes/" },
    { "id": "limit_tested",   "check": "jev",   "question": "Does the diff add a test that exercises the 429 response?", "evidence": "git diff \"$JEV_BASE\" -- test/" }
  ]
}

Optional keys: threshold (default 0.8), maxRounds (default 10), cwd (project root, relative to the file; default two levels up).

Two check modes. exit0 passes when every evidence command exits 0: deterministic, instant, no model call. jev (the default) sends the command output to Jev and passes at P(true) >= threshold. Anything a script can decide gets exit0: tests, type check, lint, "file exists", "no changes outside these paths". Reserve jev for reading a diff or output for intent: "does the diff add a test that exercises X", "does the migration make the column non-null". When any exit0 criterion fails, the Jev questions are skipped that round (shown as WAIT) so you fix concrete failures first and spend no gateway call.

Writing good criteria:

• 3 to 8 criteria. Each is one atomic, observable question that a reader could answer from the evidence output alone. Jev does not reason; it reads.
• Every criterion names the shell command(s) that prove it. The script runs them from the project root in bash. You never write the evidence yourself.
• Diffs use "$JEV_BASE", the commit recorded at freeze time, so committing during the work does not empty the diff.
• Always include the project's real test command and its type/lint check if it has one, as exit0. Add one jev criterion for the behaviour the task asked for, one for the test that proves it, and one exit0 scope criterion guarding against unrelated changes.
• A scope criterion names code and test paths only. .claude/jev/ is git-ignored (globally on this machine), so the criteria file never appears in git status; listing it as an expected file makes the criterion fail.
• Phrase questions so true means done. Avoid negatives that Jev might invert.

What Jev cannot grade

Jev reads evidence; it has no taste and does not reason. For open-ended goals ("make it look good", "make the docs clear", "clean this up") the criteria you can write are proxies: fonts loaded, a palette defined, a section present, a function shorter than N lines. A competent first draft clears proxies on round one, so grading will not push back on quality. Treat the criteria as the floor, and do the judgement check yourself: open the page in the browser, read the docs as a newcomer would, review the diff. Then say in the report what the confirmation block covers and what it does not. Do not stretch a jev question into "is this well designed?"; Jev will answer from whatever text it sees and the number will mean nothing.

The Stop hook (goal gate + nudge)

A Stop hook (scripts/stop-hook.mjs, registered in ~/.claude/settings.json) backs this skill:

• Goal gate. Criteria you froze in this session (the hook finds your grade.mjs freeze command in the transcript) hold the session until they pass, hit maxRounds, or the gate has blocked three times. Editing a file after a PASS reopens the goal; grade again. A PASS in the current turn must be followed by the confirmation block, token line included, in the final message. If the criteria cannot be met, say why in one sentence and stop; the gate yields after three blocks and Claude Code itself stops honouring blocks after eight. To abandon a goal, delete its criteria file (or its .lock); the gate treats a missing file as released and stops blocking.
• Nudge. With no goal active, after a turn that used a mutating tool, the hook asks Jev whether the final message leaves the latest request unfinished (no question to the user, no external blocker, no refusal, but promised or missing work). If so it blocks once, or twice when real progress followed the first nudge. A turn with no tool calls, or only read-style calls (Read, Grep, Glob, web fetches), is never nudged; a turn that ran Bash, a subagent, or a skill counts as work even if it only investigated. Two more cases are never nudged: a reply that only answers another Stop hook's prompt, and a turn that handed work to a background subagent, a background Bash command, or a Monitor (waiting on it is not stopping short).
• Modes. JEV_STOP_HOOK is off, shadow (log only, no blocking), or enforce. Set off in a project's .claude/settings.json env to exclude that repository. Headless claude -p and SDK runs are detected from the transcript (entrypoint: sdk-cli) and ignored; --safe-mode disables hooks entirely as well.
• Data. When the nudge runs, the redacted latest prompt, the redacted preceding assistant reply, the redacted final message, and tool names with file paths or the first word of a shell command go to Vercel AI Gateway and TypeSafe AI. The hook never sends full commands, raw tool outputs, file contents, or earlier prompts. Anything you quote in your final message does go (redacted), so keep secrets out of the reply text. Redaction is a floor, not a guarantee.
• Log and tuning. Every evaluation appends to ~/.claude/jev-stop-hook/log.jsonl. node scripts/stop-hook-report.mjs prints it; --label <n> good|bad records your verdict. Run in shadow for about a week, label, adjust the thresholds at the top of stop-hook.mjs, then switch to enforce. The hook fails open on any grader error, including the free-tier rate limit of roughly eight calls per five minutes.

Rationalizations that mean STOP

Common mistakes

• A jev criterion whose evidence prints nothing on success (grep -q). Jev needs text to read: use grep -n pattern file, or make it an exit0 criterion instead.
• Evidence that depends on the shell cwd. Commands run from the project root; use paths relative to it.
• Criteria about quality of judgement ("is the design clean?"). Jev grades observable facts. Keep judgement calls for a human or a reasoning reviewer.
• Grading from a dirty tree that includes unrelated changes. $JEV_BASE diffs show everything since freeze.
• A frozen evidence command that turns out to be broken anyway (the dry run cannot catch everything). Do not grind rounds against it. Leave that file as a recorded FAIL, create <slug>-2.json with identical criteria and the fixed command, freeze that, and say so in the report.
• Backslashes in evidence regexes. Prefer [.] for a literal dot and # as the sed delimiter when the pattern contains /, so the command survives every quoting layer.