Code Review Eval

Code Review Eval

Run bramble code-review with multiple backend/model configs against the same branch, then compare their findings side-by-side. Each config runs three turns so the eval also characterizes the resume path that /pr-polish uses across rounds.

Configs to evaluate

Three-turn flow per config

For each backend/model pair, run three turns in sequence so the eval characterizes both the initial review and the resume paths:

A turn-2/turn-3 envelope's resume_status field tells us whether the backend actually resumed ("ok") or silently degraded to a cold start ("fallback"). The status is finalized only after the backend's ReadyEvent confirms the session id, so ok is trustworthy. Cursor and codex are expected to always report ok; gemini may legitimately report fallback depending on session lifetime.

Step 1: Build and identify target

bazel build //bramble:bramble

Identify the branch to review. If an argument is given, git checkout that branch first. Otherwise use the current branch. Get the diff summary:

git diff $(git merge-base origin/main HEAD)..HEAD --stat

Step 2a: Run each config — turn 1 (fresh)

Use the Monitor tool for each config. --envelope-file writes the final ResultEnvelope to a file; stdout carries plain-text progress lines for Monitor to stream. Codex defaults to --read-only. Cursor and Gemini have no read-only mode, so they run with default permissions.

Create a fresh $LOG_DIR under /tmp/code-review-eval-{timestamp}/. For each config:

ENVELOPE_FILE="$LOG_DIR/{NAME}-envelope-r1.json"
BRAMBLE_RUN_TAG="code-review-eval:$(git branch --show-current):{NAME}:r1" \
WORK_DIR=$(pwd) bazel-bin/bramble/bramble_/bramble code-review \
  {FLAGS} --verbose --timeout 10m --envelope-file "$ENVELOPE_FILE" \
  2>"$LOG_DIR/{NAME}-stderr-r1.txt"

BRAMBLE_RUN_TAG mirrors the convention used by /pr-polish (bramble_ops.py launch_env): it propagates into ~/.bramble/logs/code-review/ so eval runs are greppable later. The tag format is code-review-eval:{branch}:{name}:r{turn}.

Arm all three Monitors in the same turn so configs run in parallel. Set Monitor timeout_ms=600000. After all Monitors complete, extract each turn's session_id into shell variables before arming turn 2:

SESSION_CODEX=$(jq -r '.session_id // empty'   "$LOG_DIR/codex-5.4-mini-envelope-r1.json"  2>/dev/null || true)
SESSION_CURSOR=$(jq -r '.session_id // empty'  "$LOG_DIR/cursor-composer2-envelope-r1.json" 2>/dev/null || true)
SESSION_GEMINI=$(jq -r '.session_id // empty'  "$LOG_DIR/gemini-3.1-flash-lite-preview-envelope-r1.json" 2>/dev/null || true)

If any SESSION_* is empty (no envelope, malformed envelope, or backend never emitted a session id), record resume_status: missing for that backend in the comparison table and skip its turn 2 and turn 3. Don't run a fresh review and call it resumed — that contaminates the eval signal.

Note: codex only reports shell command tool calls on stdout; file reads are internal to the codex SDK and not surfaced. Gemini reports tool calls via ACP.

Step 2b: Re-run each config — turn 2 (resumed, follow-up prompt)

First, defensively rebuild bramble — the bazel-bin/bramble/bramble_/bramble symlink can be invalidated between turns by unrelated bazel activity in the same workspace, and the resulting "No such file or directory" error is silent in Monitor stdout:

bazel build //bramble:bramble  # cheap when up-to-date

Then arm three Monitors in the same turn, mirroring Step 2a but with two changes:

• add --resume-session-id "$SESSION_<NAME>" to each command
• write to $LOG_DIR/{NAME}-envelope-r2.json and {NAME}-stderr-r2.txt

Skip a backend whose $SESSION_<NAME> is empty.

Do not pass --resume-prompt-style. The CLI defaults to follow-up when a resume id is set, which matches the path /pr-polish uses — the production caller this eval most needs to characterize. Canonical command (codex; cursor and gemini mirror it with backend/model swapped):

Prompt shape note (2026-05-06, updated round 10): the follow-up prompt was rewritten to drop the redundant persona and full JSON output spec re-paste (the resumed session already saw them in turn 1) and to widen the model's review scope — earlier shapes used a strict "(1)(2)(3) only" framing that biased the model toward ratifying the prior verdict (cursor turn 2 in the round-2 eval returned 0 issues with summary "HEAD is unchanged since the earlier pass"). The new shape leads with "Re-review the full diff with fresh eyes — including code you previously accepted" and rewards finding new issues over confirming the prior verdict.

Round-8 codex+cursor consensus refined the shape: the skip-test-execution suffix and scope clauses (test-quality + cross- service) are now conditionally appended when opts carries them, so a silent resume fallback (resume_status="fallback") reading the prompt cold still gets the same scope guidance a real fresh review would. The persona / full output spec / goal re-statement remain dropped.

Compare turn-2 finding counts before/after this date to baseline the bias-guard effect: a non-biased follow-up should not systematically produce LOWER turn-2 counts than turn-1 across all backends.

ENVELOPE_FILE="$LOG_DIR/codex-5.4-mini-envelope-r2.json"
BRAMBLE_RUN_TAG="code-review-eval:$(git branch --show-current):codex-5.4-mini:r2" \
WORK_DIR=$(pwd) bazel-bin/bramble/bramble_/bramble code-review \
  --backend codex --model gpt-5.4-mini --effort medium \
  --resume-session-id "$SESSION_CODEX" \
  --verbose --timeout 10m --envelope-file "$ENVELOPE_FILE" \
  2>"$LOG_DIR/codex-5.4-mini-stderr-r2.txt"

Step 2c: Re-run each config — turn 3 (resumed, fresh prompt)

Same as 2b, but also pass --resume-prompt-style fresh. Write to $LOG_DIR/{NAME}-envelope-r3.json and {NAME}-stderr-r3.txt. Same skip rule (empty session id → resume_status: missing). Rebuild bramble first for the same invalidation reason as 2b:

bazel build //bramble:bramble

ENVELOPE_FILE="$LOG_DIR/codex-5.4-mini-envelope-r3.json"
BRAMBLE_RUN_TAG="code-review-eval:$(git branch --show-current):codex-5.4-mini:r3" \
WORK_DIR=$(pwd) bazel-bin/bramble/bramble_/bramble code-review \
  --backend codex --model gpt-5.4-mini --effort medium \
  --resume-session-id "$SESSION_CODEX" --resume-prompt-style fresh \
  --verbose --timeout 10m --envelope-file "$ENVELOPE_FILE" \
  2>"$LOG_DIR/codex-5.4-mini-stderr-r3.txt"

Resume status taxonomy

Cursor or codex returning fallback is surprising — they're expected to always support resume. Flag prominently in Notes if it happens. Gemini reporting fallback is less surprising but still worth noting. unverified on any backend is a stronger signal than fallback: it means the backend never finished startup, not that the session was unavailable.

Step 3: Compare findings

After all turns complete, read each envelope and extract findings. The envelope shape is {status, backend, model, session_id, resume_status, review: {verdict, summary, issues: [...]}, schema_version, duration_ms, input_tokens, output_tokens}. Findings live at .review.issues[], not .issues[]. For each (config, turn) list:

• Issues found (.review.issues[].file/.line/.severity/.message)
• Verdict (.review.verdict: accepted / rejected)
• Confidence per issue (.review.issues[].confidence)
• Wall clock time (.duration_ms); token counts (.input_tokens / .output_tokens, often 0 — only codex reports these reliably)
• Whether the always-emit envelope guard fired (check stderr for "envelope guard" lines)
• For turns 2 and 3: .resume_status from the envelope (or local taxonomy value)

A non-zero monitor exit code with status: "error" (or no envelope at all) means the backend failed terminally — record per the taxonomy below.

Then produce two tables.

Comparison of turn-1 findings (the canonical review):

Identify:

• Consensus findings: flagged by all three configs (high confidence these are real)
• Majority findings: flagged by two of three configs (likely real)
• Unique findings: only one config caught it (investigate — real issue or FP?)
• False positives: findings that are clearly not issues
• Disagreements: findings where configs differ on severity or applicability

Follow-up review (turn 2 and turn 3) findings delta — for each config, compare turn N against turn 1 on (file, line, severity, message) keys:

Then a short prose section per backend listing notable deltas — does the resumed follow-up meaningfully refine, contradict, or just restate? Does the explicit fresh prompt on a resumed session behave differently from a brand-new fresh run?

Check .claude/skills/code-review-eval/references/known-blind-spots.md for previously identified blind spots and note if any recur.

Step 4: Log results

Append to .claude/skills/code-review-eval/data/eval-runs.log:

## {DATE} — {BRANCH}

Diff: {N} files, {+/-} lines

| Config | Turn | Resume | Findings | FPs | Verdict | Confidence | Time | Tokens (in/out) |
|--------|------|--------|----------|-----|---------|------------|------|-----------------|
| cursor-composer2 | 1 | — | N | N | ... | ... | ...s | — |
| cursor-composer2 | 2 | ok | N | N | ... | ... | ...s | — |
| cursor-composer2 | 3 | ok | N | N | ... | ... | ...s | — |
| codex-5.4-mini | 1 | — | N | N | ... | ... | ...s | N/N |
| codex-5.4-mini | 2 | ok | N | N | ... | ... | ...s | N/N |
| codex-5.4-mini | 3 | ok | N | N | ... | ... | ...s | N/N |
| gemini-3.1-flash-lite-preview | 1 | — | N | N | ... | ... | ...s | — |
| gemini-3.1-flash-lite-preview | 2 | fallback | N | N | ... | ... | ...s | — |
| gemini-3.1-flash-lite-preview | 3 | missing | — | — | ... | ... | ...s | — |

Resume incidents: gemini reported "fallback" on turn 2 (cold-restarted instead of
resuming session id <id>); turn 3 was skipped because the prior session id was empty.

Consensus (all 3, turn 1): ...
Majority (2 of 3, turn 1): ...
Unique to cursor (turn 1): ...
Unique to codex-5.4-mini (turn 1): ...
Unique to gemini-3.1-flash-lite-preview (turn 1): ...
Disagreements: ...

Follow-up deltas:
- cursor-composer2: turn 2 +N new, -N dropped, N severity changes; turn 3 ...
- codex-5.4-mini: ...
- gemini-3.1-flash-lite-preview: ...

Only emit the "Resume incidents" line when at least one backend reported fallback, missing, missing-envelope, or error. Otherwise omit it — the table alone tells the story.

Schema notes:

• The Tokens (in/out) column was added when Gemini support was introduced (2026-04-21). Historical rows without this column remain valid — treat missing as —.
• The Turn and Resume columns and the Follow-up Deltas block were added 2026-05-06 to capture three-turn resume behavior. Historical rows have one row per config (implicitly turn 1) and no resume column; treat missing as —.

Final Summary

## Code Review Eval Summary

| Config | Turn | Findings | FPs | Verdict | Time |
|--------|------|----------|-----|---------|------|
| cursor-composer2 | 1 | | | | |
| codex-5.4-mini | 1 | | | | |
| gemini-3.1-flash-lite-preview | 1 | | | | |

Resume health: cursor=<status>, codex=<status>, gemini=<status>

Best config: ...
Recommendation: ...

Resume health summarizes turn-2 status per backend (the path that actually matters for /pr-polish). If turn 3 disagrees with turn 2 in an interesting way (e.g. turn 2 ok, turn 3 fallback), call that out in Recommendation.

Key Files

Eval dataset

This skill compares backends on a live branch. To score a reviewer against a ground-truth dataset of past PRs, use the /code-review-replay skill instead — it owns the dataset pipeline.

The dataset scripts (harvest.py, collect.py, replay.py) now live under .claude/skills/code-review-replay/scripts/. /code-review-replay collect builds a per-PR record from /pr-polish history and freezes a judged ground truth into it; /code-review-replay <repo>-<pr> then scores any reviewer config mechanically against that frozen ground truth. The dataset is stored outside the repo at ~/.bramble/code-review-eval/dataset/ — it is derived from real private PRs and must never be committed.

See .claude/skills/code-review-replay/scripts/README.md for the full schema and the /code-review-replay skill for the two-mode workflow.

Source: bazelment/yoloswe — distributed by TomeVault.