/judge (bro)
All mechanics live in the bro CLI. This skill is policy only.
Prereq: bro on PATH or npx -y @broject/bro@0.
What it is
A calibrated decision judge for agent loops — typed questions
(choice/score/noul) over a compact state, typed answers with
confidence back, sub-second, priced in fractions of a cent. The primary
backend is systemone — TypeSafe's System One API (POST {judge.baseUrl}/v1/systemone, Authorization: Bearer $<judge.apiKeyEnv> — the var named by judge.apiKeyEnv,
TYPESAFE_API_KEY by default; TYPESAFE_BASE_URL redirects the
default base only, never a configured host);
judge.fallback names an escalation connector (llm-judge — any
OpenAI-compatible chat endpoint) re-asked on unanswered questions and
answers under judge.confidence.
| Command |
What it does |
bro self judge decide --state <file|-> --questions <file> |
One decide() over the resolved chain — prints answers, decidedBy, model, latency, usage. --connector <name> pins the primary, --json prints the raw DecideResult |
bro self judge stats [--since <iso>] [--json] [--replay] |
Scores the verdict journal — agreement matrix (action vs recorded outcome, overall) + agreement rate per decider, blocks_correctness proxy-scored separately, calibration buckets, p50/p95 latency, mean/total cost per provider+model, and the dogfood thresholds (≥85% agreement, p50 <1s, mean <$0.01). --replay scores dogfood verdicts instead of live ones |
bro self judge replay [--pr <n>[,<n>…]…] [--merged-since <iso>] [--limit <n>] [--connector <name>] [--json] |
Dogfood: re-judges archived review threads from merged PRs on the live chain, journals each verdict as replay: true with the outcome inferred from the record (act disposition → defer bead → resolved with an outdated anchor; a push alone is not evidence — threads whose outcome can't be recovered drop out; an unresolved thread still classifies when a disposition or defer bead exists), then prints the replay-scoped stats report. Idempotent — a replayed subject is never re-judged |
Config (bro.config.json judge section): mode (off|shadow — v1
has no acting mode), model (pin in production), baseUrl,
apiKeyEnv (env var NAME, never the value), confidence (0.6),
fallback, timeoutMs (bounds the whole chained call),
maxDecisionsPerRun (50 — fresh decide() calls per invocation),
llm ({baseUrl, model, apiKeyEnv}).
Shadow mode — what mode: shadow does
Every verdict a bro command's decide() returns in shadow mode is journaled to
<git-common>/bro/judge/verdicts.jsonl (append-only, shared across
linked worktrees, nothing lands in git): act/drive thread annotation
as kind: 'act-thread' rows, bro self judge decide smoke calls as
kind: 'judge-decide' (kept out of the triage agreement set), plus
kind: 'act-disposition' rows where bro review act resolve/reply or an act
plan applies a verdict. Journaling needs a git repo — outside one
there is no common dir, the append is a silent no-op, and verdicts are
not journaled.
bro review act threads and the drive fixer prompt render a judge: line
beside each unresolved thread — blocks_correctness, severity,
action — under the same dedup key (threadId, commentSha,
headSha): a subject with a recorded verdict re-reads it for free on
repeat polls; a moved subject — or one with no verdict yet — pays for
a fresh decide(), bounded by maxDecisionsPerRun per invocation
(shared across a whole drive pass).
Policy
- Shadow is a boundary, not a mood. Verdicts annotate, never act:
thread resolution still needs
bro review act resolve/reply, the exit gate
still reads deterministic state, fixer spawns stay deterministic.
The judge earns advisory weight by measured dogfood agreement
(bro self judge replay over archived threads, bro self judge stats --replay
for the report), not enthusiasm. Never wire a verdict into a gate
decision.
- Fail-open, always. A wedged backend throws
JudgeUnavailable —
consumers treat "no verdict" as "no annotation", never a gate input.
- One call, many questions. Batch a subject's questions into one
decide() — a question-per-call loop is a cost bug.
decidedBy is honest. Escalated answers say llm-judge;
bro self judge stats scores each backend on its own record.
- Keys ride env vars only.
apiKeyEnv names the variable; config
never holds a key value.
1---2name: judge3description: "Use when judging calibrated decisions for agent loops — act triage annotation, fallback escalation. Thin wrapper over the bro CLI: `bro self judge decide` smokes the connector; the systemone → llm-judge chain lives in packages/judge. Requires `bro` (npx -y @broject/bro@0); the default systemone backend needs a TypeSafe API key, llm-judge needs its own `judge.llm` config."4metadata:5 source: https://skillmd.com/skills/theplenkov/judge6---78# /judge (bro)910**All mechanics live in the `bro` CLI.** This skill is policy only.1112Prereq: `bro` on PATH or `npx -y @broject/bro@0`.1314## What it is1516A calibrated decision judge for agent loops — typed questions17(`choice`/`score`/`noul`) over a compact state, typed answers with18confidence back, sub-second, priced in fractions of a cent. The primary19backend is **systemone** — TypeSafe's System One API (`POST20{judge.baseUrl}/v1/systemone`, `Authorization: Bearer21$<judge.apiKeyEnv>` — the var named by `judge.apiKeyEnv`,22`TYPESAFE_API_KEY` by default; `TYPESAFE_BASE_URL` redirects the23default base only, never a configured host);24`judge.fallback` names an escalation connector (**llm-judge** — any25OpenAI-compatible chat endpoint) re-asked on unanswered questions and26answers under `judge.confidence`.2728| Command | What it does |29| ------- | ------------ |30| `bro self judge decide --state <file\|-> --questions <file>` | One decide() over the resolved chain — prints answers, decidedBy, model, latency, usage. `--connector <name>` pins the primary, `--json` prints the raw DecideResult |31| `bro self judge stats [--since <iso>] [--json] [--replay]` | Scores the verdict journal — agreement matrix (`action` vs recorded outcome, overall) + agreement rate per decider, `blocks_correctness` proxy-scored separately, calibration buckets, p50/p95 latency, mean/total cost per provider+model, and the dogfood thresholds (≥85% agreement, p50 <1s, mean <$0.01). `--replay` scores dogfood verdicts instead of live ones |32| `bro self judge replay [--pr <n>[,<n>…]…] [--merged-since <iso>] [--limit <n>] [--connector <name>] [--json]` | Dogfood: re-judges archived review threads from merged PRs on the live chain, journals each verdict as `replay: true` with the outcome inferred from the record (act disposition → defer bead → resolved with an outdated anchor; a push alone is not evidence — threads whose outcome can't be recovered drop out; an unresolved thread still classifies when a disposition or defer bead exists), then prints the replay-scoped stats report. Idempotent — a replayed subject is never re-judged |3334Config (`bro.config.json` `judge` section): `mode` (off|shadow — v135has no acting mode), `model` (pin in production), `baseUrl`,36`apiKeyEnv` (env var NAME, never the value), `confidence` (0.6),37`fallback`, `timeoutMs` (bounds the whole chained call),38`maxDecisionsPerRun` (50 — fresh decide() calls per invocation),39`llm` ({baseUrl, model, apiKeyEnv}).4041## Shadow mode — what `mode: shadow` does4243Every verdict a bro command's decide() returns in shadow mode is journaled to44`<git-common>/bro/judge/verdicts.jsonl` (append-only, shared across45linked worktrees, nothing lands in git): act/drive thread annotation46as `kind: 'act-thread'` rows, `bro self judge decide` smoke calls as47`kind: 'judge-decide'` (kept out of the triage agreement set), plus48`kind: 'act-disposition'` rows where `bro review act resolve/reply` or an act49plan applies a verdict. Journaling needs a git repo — outside one50there is no common dir, the append is a silent no-op, and verdicts are51not journaled.5253`bro review act threads` and the drive fixer prompt render a `judge:` line54beside each unresolved thread — `blocks_correctness`, `severity`,55`action` — under the same dedup key (`threadId`, `commentSha`,56`headSha`): a subject with a recorded verdict re-reads it for free on57repeat polls; a moved subject — or one with no verdict yet — pays for58a fresh decide(), bounded by `maxDecisionsPerRun` per invocation59(shared across a whole drive pass).6061## Policy6263- **Shadow is a boundary, not a mood.** Verdicts annotate, never act:64 thread resolution still needs `bro review act resolve/reply`, the exit gate65 still reads deterministic state, fixer spawns stay deterministic.66 The judge earns advisory weight by measured dogfood agreement67 (`bro self judge replay` over archived threads, `bro self judge stats --replay`68 for the report), not enthusiasm. Never wire a verdict into a gate69 decision.70- **Fail-open, always.** A wedged backend throws `JudgeUnavailable` —71 consumers treat "no verdict" as "no annotation", never a gate input.72- **One call, many questions.** Batch a subject's questions into one73 `decide()` — a question-per-call loop is a cost bug.74- **`decidedBy` is honest.** Escalated answers say `llm-judge`;75 `bro self judge stats` scores each backend on its own record.76- **Keys ride env vars only.** `apiKeyEnv` names the variable; config77 never holds a key value.