Verifying review findings
A model that reviews code for defects also invents them: it misreads a guard
ten lines up, a loop bound, or a lock the caller holds. This skill splits the
review into two roles, as the open-source reviewer bluesky-social/roast does.
A finder (you, the code-review skill, any reviewer) writes each finding
with a verbatim quote of the code it is about. A verifier from a different
model family then reads that code and confirms or refutes the claim.
The verifier is Jev, a decision model from TypeSafe. It answers typed questions
(yes/no, multiple choice, ordered scale) with a probability for every answer in
about 0.3 s. Jev is not a Claude model, so when Claude finds and Jev verifies,
the verifier is less likely to share the finder's blind spots. Below,
P(x) means the probability Jev gives to answer x.
scripts/verify.py does the following for each finding:
- Grounding. The quote must occur in the file, compared with runs of
whitespace collapsed. A finding with no quote, or a quote that is not in
the file, is dropped before Jev sees it.
- The window. The verifier sees a window of the file: the top-level
block (function, class, statement) that contains the quote, with at least
12 lines on each side, capped at 160 lines. Lines are numbered and the
quoted ones are marked
>. With --diff, the file's diff hunks go along
too.
- Three questions to Jev, one call per finding, in parallel:
- verdict:
confirmed, refuted or insufficient_context
- severity if the defect is real, on the scale
info, low, medium,
high, critical
- when the finding proposes a fix, whether the fix works
- Thresholds. P(refuted) of at least 0.7 drops the finding
(
--refute-at). P(confirmed) of at least 0.7 confirms it (--confirm-at).
Anything else is uncertain. Only a refutation removes a finding; a
verifier that is unsure keeps it.
When NOT to use this skill
| Situation |
Use instead |
| Nobody has looked for defects yet |
code-review (or any reviewer) first; this skill only judges findings that exist |
| Checking that a README or docstring still matches the code |
verifying-claims |
| An adversarial read of a whole deliverable (prose, analysis, a PR's design) |
challenging |
| A routing, triage or yes/no question that is not about a code finding |
deciding-with-confidence, or the Muninn decide tool directly |
| Building a deterministic check that must be able to go red |
gating |
| The claim is about runtime behaviour the code cannot show (performance, a flaky test, production data) |
run it; a verifier reading code can say insufficient_context at best |
Stop the run when most findings come back uncertain with reason
needs_more_context: the evidence for those claims lives outside their
windows (in a caller, another file, a config). Verify those findings by
reading the code yourself.
Procedure
Throughout, V=/mnt/skills/user/verifying-review-findings/scripts/verify.py.
Write the findings as a JSON list. file (relative to the repository
root) and code_quote are required. Copy the quote character for character
from the file, one to six lines; a paraphrase is dropped.
[{"id": "1", "file": "src/util.py", "line": 6,
"description": "Off-by-one: the loop skips the last element.",
"code_quote": "for i in range(len(items) - 1):",
"fix": "for i in range(len(items)):", "severity": "high"}]
The other fields are optional. id defaults to the item's position.
line picks between several matches of the same quote. description is
the claim Jev judges. fix adds the fix question. severity is the
finder's own estimate and is passed to Jev as a claim. Findings in the
ReportFindings shape (summary and failure_scenario instead of
description) are accepted too. When you are the finder, emit this shape
directly.
Pick the path. --transport auto, the default, uses the first of
jev and gateway whose credentials are set.
| Path |
Needs |
How |
Jev direct (jev) |
TYPESAFE_API_KEY, a TypeSafe API key |
run (step 3a) |
Jev via Cloudflare Workers AI (gateway) |
CF_ACCOUNT_ID and CF_API_TOKEN, optionally CF_GATEWAY_ID |
run; pass --transport gateway when both paths have credentials |
Muninn decide tool, which calls Jev |
the Muninn MCP connector in the session |
emit, then apply (step 3b) |
Claude Haiku via deciding-with-confidence |
that skill installed next to this one |
run --transport decider; the verifier is then also Claude |
Run it.
a. With credentials in the environment:
python3 $V run findings.json --repo . --format md # default output is JSON
git diff main... > /tmp/d.diff && python3 $V run findings.json --diff /tmp/d.diff
python3 $V run findings.json --ref HEAD~1 # read files at a git ref
b. Through the Muninn decide tool:
python3 $V emit findings.json --out batch.json
emit saves the grounded findings to batch.json and prints the Jev
requests, one per surviving finding, each as {"id", "state", "questions"}. Call decide once per request, passing its state and
questions unchanged; the calls are independent, so send them together.
Save the replies in answers.json as an object that maps each finding's
id to the full text decide returned, then:
python3 $V apply batch.json answers.json --format md
Read the report. The top level has complete (false when any finding
could not be verified), counts per status, and gate. Each finding keeps
its input fields and adds:
status: confirmed, uncertain, dropped, or unverified (Jev was
not reached or replied in an unexpected shape)
reason: why it is not confirmed. One of missing_code_quote,
ungrounded_code_quote, file_not_found, refuted_by_verifier,
split_verdict, needs_more_context (insufficient_context was the
largest verdict), no_answer, or verifier_error: ...
verdict: the three verdict probabilities
severity: probabilities per level, and expected, the level nearest
the probability-weighted average
fix_ok: the probability that the fix works
claimed_severity: the finder's own severity from the input
fingerprint: a hash of the file and the whitespace-collapsed quote. Use
it to match a finding across runs, never to suppress one: two different
claims about the same lines share a fingerprint.
Act on it.
- Report
confirmed findings as findings.
- Report
uncertain ones separately and read their code yourself; the
verifier's mistakes cluster there.
- Leave
dropped findings out, but give their count and reasons so a
reader can recover one that was refuted wrongly.
- When a confirmed finding has
fix_ok under 0.5, rewrite the fix before
proposing it.
Gate a commit or CI step with --fail-on LEVEL (any severity level).
Exit codes:
| Code |
Meaning |
| 0 |
every finding verified, and no gate tripped |
| 1 |
a confirmed finding has P(severity at least LEVEL) of at least --severity-at (default 0.5) |
| 2 |
bad input, or no transport available |
| 3 |
some finding is unverified: the run is incomplete, not clean |
A pre-commit hook that should not block on an outage can let the commit
through on 3, as long as the hook prints the report so the gap is visible:
python3 $V run staged.json --fail-on high --format md; rc=$?
[ $rc -eq 3 ] && echo "verify: incomplete run, commit allowed" >&2 && rc=0
exit $rc
Earned exceptions
| Default |
Change it when |
Keep it when |
--refute-at 0.7 |
a missed defect costs more than reading a false one (security review, release gate): raise it toward 0.9 so fewer findings are dropped |
the review is advisory, and noise is what makes people stop reading |
| The gate counts only confirmed findings |
the gate guards something irreversible: lower --confirm-at so more findings count as confirmed |
uncertain findings get a human look before merge |
| One window per finding |
the claim depends on another file (a caller, a type, config) |
the evidence is in the same top-level block, which covers most of what reviewers flag |
Common failure modes
verify: no Jev transport, exit 2 → neither TYPESAFE_API_KEY nor the
Cloudflare pair is set. Use the emit/apply path, or --transport decider.
- Every finding
dropped with ungrounded_code_quote → the finder quoted
from memory or reformatted the code, or --repo points at the wrong
checkout. Run grep -F with one quote against the file to tell which.
file_not_found for files that exist → file is resolved against
--repo (default: the current directory). A path that leaves the
repository is treated as missing on purpose.
unverified with verifier_error: HTTP 4xx → check the error text.
When it names the input length, the window was too long for Jev's default
context: lower MAX_LINES (the 160-line cap, a constant at the top of
verify.py). The report says complete: false until those findings are
re-run.
unverified with no_answer after apply → answers.json has no
entry for that finding's id. Ids are strings: "1", not 1.
- A high
severity.expected on a dropped finding → severity answers "how
bad if real". Read it only on findings that survive.
Verification
python3 -m pytest /mnt/skills/user/verifying-review-findings/tests -q # offline, no network
python3 $V eval /mnt/skills/user/verifying-review-findings/assets/eval.jsonl
eval runs 36 labelled findings (16 real defects, 16 false findings, 4
quotes not in the file) and prints the counts. Grounding is deterministic,
so ungrounded_dropped must be 4/4. Jev's part gave real_kept 16/16 and
false_dropped 11/16 with jev-1.14.0 on 2026-11-07. A lower count means the
Jev version or the window settings changed; compare per item with
--out results.jsonl.
A bad success is a report with complete: true in which every finding is
confirmed near 1.0, including ones you know are false: Jev is judging the
description without the code. Check that state.code in the requests emit
prints holds the numbered window.
The eval's per-item results, the threshold sweep and what it does not cover
are in references/method.md.
Diagnosed failures
- 2026-11-07, eval item
ts-04: a finding claimed a race condition in
synchronous single-threaded JavaScript. Jev confirmed it at 0.89, the only
false finding confirmed. No threshold catches a confident error like this.
Read the code behind any concurrency claim about single-threaded code yourself.
- 2026-11-07, eval items
py-05 and sh-03: the evidence that refutes
each claim was in the same file but in a different function, outside the
window. Neither was dropped. sh-03 came back uncertain with
needs_more_context (P(insufficient_context) 0.82). py-05 came back
uncertain with split_verdict (P(confirmed) 0.55, P(insufficient_context)
0.40). Both outcomes are correct: the finding stays in the report, flagged
for a human to read.
1---2name: verifying-review-findings3description: "Checks each finding from a code review against the code it cites and keeps only the ones that hold. Drops findings whose quoted code is not in the file, then asks TypeSafe's Jev decision model whether the code shown really has the claimed defect, returning P(confirmed), P(refuted), a severity distribution and whether the proposed fix works. Use for \"verify these review findings\", \"filter false positives from the review\", \"is this bug real\", \"double-check the reviewer's claims\", \"get a second opinion on these findings\", a pre-commit or CI gate that blocks only on confirmed high-severity findings, or a finder-then-verifier review with a different model as the verifier."4metadata:5 source: https://skillmd.com/skills/oaustegard/verifying-review-findings6 version: 0.1.17---89# Verifying review findings1011A model that reviews code for defects also invents them: it misreads a guard12ten lines up, a loop bound, or a lock the caller holds. This skill splits the13review into two roles, as the open-source reviewer bluesky-social/roast does.14A **finder** (you, the `code-review` skill, any reviewer) writes each finding15with a verbatim quote of the code it is about. A **verifier** from a different16model family then reads that code and confirms or refutes the claim.1718The verifier is Jev, a decision model from TypeSafe. It answers typed questions19(yes/no, multiple choice, ordered scale) with a probability for every answer in20about 0.3 s. Jev is not a Claude model, so when Claude finds and Jev verifies,21the verifier is less likely to share the finder's blind spots. Below,22P(x) means the probability Jev gives to answer x.2324`scripts/verify.py` does the following for each finding:25261. **Grounding.** The quote must occur in the file, compared with runs of27 whitespace collapsed. A finding with no quote, or a quote that is not in28 the file, is dropped before Jev sees it.292. **The window.** The verifier sees a *window* of the file: the top-level30 block (function, class, statement) that contains the quote, with at least31 12 lines on each side, capped at 160 lines. Lines are numbered and the32 quoted ones are marked `>`. With `--diff`, the file's diff hunks go along33 too.343. **Three questions to Jev**, one call per finding, in parallel:35 - verdict: `confirmed`, `refuted` or `insufficient_context`36 - severity if the defect is real, on the scale `info`, `low`, `medium`,37 `high`, `critical`38 - when the finding proposes a fix, whether the fix works394. **Thresholds.** P(refuted) of at least 0.7 drops the finding40 (`--refute-at`). P(confirmed) of at least 0.7 confirms it (`--confirm-at`).41 Anything else is `uncertain`. Only a refutation removes a finding; a42 verifier that is unsure keeps it.4344## When NOT to use this skill4546| Situation | Use instead |47|---|---|48| Nobody has looked for defects yet | `code-review` (or any reviewer) first; this skill only judges findings that exist |49| Checking that a README or docstring still matches the code | `verifying-claims` |50| An adversarial read of a whole deliverable (prose, analysis, a PR's design) | `challenging` |51| A routing, triage or yes/no question that is not about a code finding | `deciding-with-confidence`, or the Muninn `decide` tool directly |52| Building a deterministic check that must be able to go red | `gating` |53| The claim is about runtime behaviour the code cannot show (performance, a flaky test, production data) | run it; a verifier reading code can say `insufficient_context` at best |5455Stop the run when most findings come back `uncertain` with reason56`needs_more_context`: the evidence for those claims lives outside their57windows (in a caller, another file, a config). Verify those findings by58reading the code yourself.5960## Procedure6162Throughout, `V=/mnt/skills/user/verifying-review-findings/scripts/verify.py`.63641. **Write the findings as a JSON list.** `file` (relative to the repository65 root) and `code_quote` are required. Copy the quote character for character66 from the file, one to six lines; a paraphrase is dropped.6768 ```json69 [{"id": "1", "file": "src/util.py", "line": 6,70 "description": "Off-by-one: the loop skips the last element.",71 "code_quote": "for i in range(len(items) - 1):",72 "fix": "for i in range(len(items)):", "severity": "high"}]73 ```7475 The other fields are optional. `id` defaults to the item's position.76 `line` picks between several matches of the same quote. `description` is77 the claim Jev judges. `fix` adds the fix question. `severity` is the78 finder's own estimate and is passed to Jev as a claim. Findings in the79 `ReportFindings` shape (`summary` and `failure_scenario` instead of80 `description`) are accepted too. When you are the finder, emit this shape81 directly.82832. **Pick the path.** `--transport auto`, the default, uses the first of84 `jev` and `gateway` whose credentials are set.8586 | Path | Needs | How |87 |---|---|---|88 | Jev direct (`jev`) | `TYPESAFE_API_KEY`, a TypeSafe API key | `run` (step 3a) |89 | Jev via Cloudflare Workers AI (`gateway`) | `CF_ACCOUNT_ID` and `CF_API_TOKEN`, optionally `CF_GATEWAY_ID` | `run`; pass `--transport gateway` when both paths have credentials |90 | Muninn `decide` tool, which calls Jev | the Muninn MCP connector in the session | `emit`, then `apply` (step 3b) |91 | Claude Haiku via `deciding-with-confidence` | that skill installed next to this one | `run --transport decider`; the verifier is then also Claude |92933. **Run it.**9495 a. With credentials in the environment:96 ```bash97 python3 $V run findings.json --repo . --format md # default output is JSON98 git diff main... > /tmp/d.diff && python3 $V run findings.json --diff /tmp/d.diff99 python3 $V run findings.json --ref HEAD~1 # read files at a git ref100 ```101102 b. Through the Muninn `decide` tool:103 ```bash104 python3 $V emit findings.json --out batch.json105 ```106 `emit` saves the grounded findings to `batch.json` and prints the Jev107 requests, one per surviving finding, each as `{"id", "state",108 "questions"}`. Call `decide` once per request, passing its `state` and109 `questions` unchanged; the calls are independent, so send them together.110 Save the replies in `answers.json` as an object that maps each finding's111 `id` to the full text `decide` returned, then:112 ```bash113 python3 $V apply batch.json answers.json --format md114 ```1151164. **Read the report.** The top level has `complete` (false when any finding117 could not be verified), `counts` per status, and `gate`. Each finding keeps118 its input fields and adds:119 - `status`: `confirmed`, `uncertain`, `dropped`, or `unverified` (Jev was120 not reached or replied in an unexpected shape)121 - `reason`: why it is not confirmed. One of `missing_code_quote`,122 `ungrounded_code_quote`, `file_not_found`, `refuted_by_verifier`,123 `split_verdict`, `needs_more_context` (`insufficient_context` was the124 largest verdict), `no_answer`, or `verifier_error: ...`125 - `verdict`: the three verdict probabilities126 - `severity`: `probabilities` per level, and `expected`, the level nearest127 the probability-weighted average128 - `fix_ok`: the probability that the fix works129 - `claimed_severity`: the finder's own `severity` from the input130 - `fingerprint`: a hash of the file and the whitespace-collapsed quote. Use131 it to match a finding across runs, never to suppress one: two different132 claims about the same lines share a fingerprint.1331345. **Act on it.**135 - Report `confirmed` findings as findings.136 - Report `uncertain` ones separately and read their code yourself; the137 verifier's mistakes cluster there.138 - Leave `dropped` findings out, but give their count and reasons so a139 reader can recover one that was refuted wrongly.140 - When a confirmed finding has `fix_ok` under 0.5, rewrite the fix before141 proposing it.1421436. **Gate a commit or CI step** with `--fail-on LEVEL` (any severity level).144 Exit codes:145146 | Code | Meaning |147 |---|---|148 | 0 | every finding verified, and no gate tripped |149 | 1 | a confirmed finding has P(severity at least LEVEL) of at least `--severity-at` (default 0.5) |150 | 2 | bad input, or no transport available |151 | 3 | some finding is `unverified`: the run is incomplete, not clean |152153 A pre-commit hook that should not block on an outage can let the commit154 through on 3, as long as the hook prints the report so the gap is visible:155 ```bash156 python3 $V run staged.json --fail-on high --format md; rc=$?157 [ $rc -eq 3 ] && echo "verify: incomplete run, commit allowed" >&2 && rc=0158 exit $rc159 ```160161## Earned exceptions162163| Default | Change it when | Keep it when |164|---|---|---|165| `--refute-at 0.7` | a missed defect costs more than reading a false one (security review, release gate): raise it toward 0.9 so fewer findings are dropped | the review is advisory, and noise is what makes people stop reading |166| The gate counts only confirmed findings | the gate guards something irreversible: lower `--confirm-at` so more findings count as confirmed | `uncertain` findings get a human look before merge |167| One window per finding | the claim depends on another file (a caller, a type, config) | the evidence is in the same top-level block, which covers most of what reviewers flag |168169## Common failure modes170171- **`verify: no Jev transport`, exit 2** → neither `TYPESAFE_API_KEY` nor the172 Cloudflare pair is set. Use the `emit`/`apply` path, or `--transport decider`.173- **Every finding `dropped` with `ungrounded_code_quote`** → the finder quoted174 from memory or reformatted the code, or `--repo` points at the wrong175 checkout. Run `grep -F` with one quote against the file to tell which.176- **`file_not_found` for files that exist** → `file` is resolved against177 `--repo` (default: the current directory). A path that leaves the178 repository is treated as missing on purpose.179- **`unverified` with `verifier_error: HTTP 4xx`** → check the error text.180 When it names the input length, the window was too long for Jev's default181 context: lower `MAX_LINES` (the 160-line cap, a constant at the top of182 `verify.py`). The report says `complete: false` until those findings are183 re-run.184- **`unverified` with `no_answer` after `apply`** → `answers.json` has no185 entry for that finding's `id`. Ids are strings: `"1"`, not `1`.186- **A high `severity.expected` on a dropped finding** → severity answers "how187 bad if real". Read it only on findings that survive.188189## Verification190191```bash192python3 -m pytest /mnt/skills/user/verifying-review-findings/tests -q # offline, no network193python3 $V eval /mnt/skills/user/verifying-review-findings/assets/eval.jsonl194```195196`eval` runs 36 labelled findings (16 real defects, 16 false findings, 4197quotes not in the file) and prints the counts. Grounding is deterministic,198so `ungrounded_dropped` must be 4/4. Jev's part gave `real_kept` 16/16 and199`false_dropped` 11/16 with jev-1.14.0 on 2026-11-07. A lower count means the200Jev version or the window settings changed; compare per item with201`--out results.jsonl`.202203A **bad success** is a report with `complete: true` in which every finding is204`confirmed` near 1.0, including ones you know are false: Jev is judging the205description without the code. Check that `state.code` in the requests `emit`206prints holds the numbered window.207208The eval's per-item results, the threshold sweep and what it does not cover209are in `references/method.md`.210211## Diagnosed failures212213- **2026-11-07, eval item `ts-04`:** a finding claimed a race condition in214 synchronous single-threaded JavaScript. Jev confirmed it at 0.89, the only215 false finding confirmed. No threshold catches a confident error like this.216 Read the code behind any concurrency claim about single-threaded code yourself.217- **2026-11-07, eval items `py-05` and `sh-03`:** the evidence that refutes218 each claim was in the same file but in a different function, outside the219 window. Neither was dropped. `sh-03` came back `uncertain` with220 `needs_more_context` (P(insufficient_context) 0.82). `py-05` came back221 `uncertain` with `split_verdict` (P(confirmed) 0.55, P(insufficient_context)222 0.40). Both outcomes are correct: the finding stays in the report, flagged223 for a human to read.