Tiered Rubric Loop (tiered-rubric-loop)
A fast, loosely coupled way to drop a combined deterministic + semantic verification loop into any repository as the verification engine of an agentic harness.
It automatically reuses any Spec-Driven Development (SDD) (docs/SPEC.md or SPEC.md) and Test-Driven Development (TDD) artefacts already in the repo, or bootstraps a lightweight B1–Bn binary rubric checklist inline if none exist.
Instead of waking an expensive reasoning subagent on every single pass, or trusting a zero-thinking classifier blindly on deep security and concurrency invariants, each assertion flows through three escalating stages:
- Tier 1 — Deterministic Checks (fast): Runs local linters, typecheckers, and unit tests (
make check, codespell, ruff, pyright, pytest) first. Any failure stops the pass immediately so you never waste API calls on broken code.
- Tier 1.5 — Fast Semantic Scoring (fast model, no output tokens): Evaluates open binary (
yes/no) rubric criteria (B1–Bn) in a single prefill pass using the Gemini Decisions API (gemini-decision-1 via scripts/eval_rubric.py). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent.
- Tier 2 — Stateless Semantic Critique: Spawns a fresh, read-only reviewer subagent only when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants (
[arch]) that need deeper multi-file reasoning.
Table of Contents
How the 3-Stage Funnel Works
| Tier |
Mechanism |
Latency & Cost |
What It Owns |
| Tier 1 (Deterministic) |
Local linters, typecheckers, unit tests (make check, codespell, ruff, pyright, pytest) |
Seconds, $0 |
Syntax, static types, formatting, and deterministic test execution. Any failure short-circuits immediately before Tier 1.5 or Tier 2 runs. |
| Tier 1.5 (Calibrated Probabilistic) |
gemini-decision-1 via client.decisions.create (v1beta) in scripts/eval_rubric.py |
~150–250 ms, $0.10 / 1M input tokens, 0 output / thinking tokens |
Evaluates up to 1,000 closed-ended noul (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (PASS), rejects obvious omissions immediately (FAIL_FAST), and flags uncertain items for L2. |
| Tier 2 (Stateless Semantic Critic) |
Fresh read-only flash subagent with zero prior coding chain-of-thought |
10–40 s, standard model tokens |
Diagnoses grey-zone escalations and performs one final L2 sign-off on [arch] items after Tier 1.5 is green. |
Key safety rule: Because gemini-decision-1 runs in a single prefill forward pass with zero thinking tokens and zero file-reading tools, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is terminal for mechanical/editorial checks, fail-fast on clear omissions, and provisional for [arch] assertions.
Plain-Markdown B1–Bn Contract & Tagging
The sole shared contract is a Markdown table or bullet list of numbered binary assertions (B1–Bn) in docs/SPEC.md, SPEC.md, or LOOPS.md (with optional --spec <path> override):
| ID | Falsifiable Condition | Verification Target |
| :--- | :--- | :--- |
| `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |
| `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |
| `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |
Other supported rubric formats and tags:
- Bullet assertions:
- **B4**: CLI help panel renders in English (UK)
- Atomic sub-IDs:
B4a, B11b
- Numbered items:
1. `B5 [docs]`: ...
- Task checkboxes:
- [ ] **B6 [concurrency]**: ... or - [x] **B6**: ...
- Extended tags:
[mech], [docs], [ui], [visual], [perf], [concurrency] (unrecognised tags warn on stderr and default safely to [mech] instead of silently dropping the assertion)
| Tag |
Meaning |
Tier 1.5 Handling |
[det] (or deterministic Verification Target such as make check, pytest tests/..., codespell -s, ruff check .) |
Already proven by Tier 1 shell exit codes |
Skipped by Tier 1.5 by default (--include-det overrides) so the model never guesses whether pytest passed from a text diff. |
(untagged, [mech], [docs], [ui], [visual], [multimodal]) |
Mechanical, editorial, doc-sync, schema, or UI convention check |
Terminal PASS when P(yes) >= 0.90, confidence >= 0.80, and no evidence was truncated; FAIL_FAST when <= 0.20; otherwise ESCALATE_L2_FLASH. |
[arch] (also [security], [logic], [concurrency], [perf]) |
Concurrency, security, state invariant, performance, or test-realism check |
Provisional pass (PROVISIONAL_ARCH_PASS) at Tier 1.5; gated for one final Tier 2 (Model: "flash") review once all items clear Tier 1.5. |
Repo-relative file paths and local images named in Verification Target or inline backticks are automatically resolved and attached (blocking .env, secret env files, *.enc, and paths outside --repo-root; pass --no-auto-target-files to disable).
Writing High-Signal B1–Bn Assertions
- Keep assertions atomic: Test one falsifiable invariant per
Bi rather than bundling three functions or eight files into one question—single-invariant assertions give sharp P(yes) separation and trigger FAIL_FAST accurately on regressions.
- Name concrete paths in
Verification Target: Prefer src/auth.py, tests/unit/test_auth.py over generic Doc review / Code review so eval_rubric.py auto-attaches untouched context files even in narrow working-tree diffs.
- Separate canonical registries from summary copy: Don't assert identical title formatting across both machine registries (
navigation.yaml) and summarised overview pages (README.md, spotlight cards) unless exact parity is truly required.
Loose Coupling & Companion Skills Matrix
tiered-rubric-loop requires zero hard dependencies on other skills and degrades cleanly when optional companions or API keys are absent:
| Companion Skill / Dependency |
When Present |
Standalone Fallback (When Absent) |
maintaining-core-documentation / spec-driven-development |
Reads B1–Bn directly from docs/SPEC.md or SPEC.md. |
Reads any local SPEC.md / LOOPS.md, or asks you to agree a compact B1–Bn list inline before coding. |
test-driven-development |
Delegates the Act step to full Red-Green-Refactor execution. |
Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions. |
loopy |
Supplies the outer Observe -> Choose -> Act -> Verify -> Record -> Stop governance and LOOPS.md receipts. |
Runs the identical bounded loop directly with the receipt format below. |
local-code-review |
Stateless Tier 2 read-only subagent runs local-code-review on escalated/[arch] items. |
Uses the built-in Stateless Tier 2 Critic Prompt below. |
GEMINI_API_KEY / GOOGLE_API_KEY |
Enables ~200 ms Tier 1.5 screening via eval_rubric.py. |
When eval_rubric.py exits 3 (missing/invalid key), skips Tier 1.5 cleanly and executes Tier 1 -> Tier 2 directly while noting how to enable Tier 1.5. |
Bounded Loop Workflow
Track every pass around Observe -> Choose -> Act -> Verify -> Record -> Stop:
- Observe: Read the specification (
docs/SPEC.md, SPEC.md, or LOOPS.md), git status --short, and current failing tests/checks. Confirm B1–Bn exist (draft and confirm them first if missing).
- Choose: Pick one highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).
- Act (TDD): Write or update the failing test first (Red), confirm it fails, then implement the minimal clean change (Green) and refactor without weakening tests.
- Verify (3-Stage Funnel):
- Tier 1: Run deterministic checks (
make check or repo lint + test commands). If Tier 1 fails, stop verification immediately and return to Choose/Act—do not spend tokens on Tier 1.5 or Tier 2.
- Tier 1.5: Run
eval_rubric.py --json over the spec, diff, and touched files (plus --image / --pdf for diagram/UI specs).
- Tier 2: Escalate only grey-zone items immediately, or run a single final
[arch] critic pass once Tier 1.5 reports zero FAIL_FAST and zero grey-zone items.
- Record: Append pass outcome,
ratchet_score (mean P(yes)), and remaining failures to LOOPS.md (or session progress notes).
- Repeat or Stop: Enforce the Circuit Breaker Trio before starting another pass.
Running Tier 1.5 (eval_rubric.py)
Resolve scripts/eval_rubric.py relative to where tiered-rubric-loop is installed:
- Global Antigravity path:
~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py
- Project-local path:
.agents/skills/tiered-rubric-loop/scripts/eval_rubric.py
In sandboxed agent runtimes that block outbound HTTPS by default, invoke uv run with unsandboxed network permission (BypassSandbox: true in Antigravity) for live API calls, or pass --offline for sandboxed --dry-run checks.
# 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json
# 2. Branch or merged PR gating: evaluate committed changes against base ref / range
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
--git-range origin/main...HEAD \
--only B3,B4 \
--json
# 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check
uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \
--spec docs/SPEC.md \
--file src/auth.py \
--image docs/images/architecture.png \
--json
# 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)
uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --json
For large PRs (> 8 files or > 100 KB), shard evaluations by assertion group via --only B3,B4 so files stay within --max-file-chars (24,000) and --max-evidence-chars (180,000). If any file or combined payload is truncated, eval_rubric.py populates truncated_sources and downgrades mechanical PASS to ESCALATE_L2_FLASH unless --allow-truncated-pass is explicitly set.
Exit Codes
| Code |
Meaning |
Loop Action |
0 |
Tier 1.5 clear (all evaluated items are PASS or PROVISIONAL_ARCH_PASS) |
If final_l2_arch_ids is non-empty, run one Tier 2 (Model: "flash") critic pass on those IDs; otherwise exit loop as Success. |
1 |
One or more FAIL_FAST or ESCALATE_L2_* items (or API call error) |
Address FAIL_FAST items directly; escalate grey-zone items (ESCALATE_L2_*) only when root cause is non-obvious. |
2 |
Usage error, missing spec, or no B1–Bn assertions parsed |
Fix spec path or rubric formatting before continuing. |
3 |
Missing/invalid GEMINI_API_KEY or GOOGLE_API_KEY |
Degrade gracefully to Tier 1 -> Tier 2 and inform the user once. |
Threshold Routing & Stateless Tier 2 Critic
Default calibrated thresholds (overridable via --pass-threshold, --fail-threshold, --min-confidence):
PASS (P(yes) >= 0.90, confidence >= 0.80, mechanical): Terminal pass. No L2 model runs.
PROVISIONAL_ARCH_PASS (P(yes) >= 0.90, confidence >= 0.80, [arch]): Held until all other assertions pass, then verified once by a Tier 2 (Model: "flash") critic.
FAIL_FAST (P(yes) <= 0.20, confidence >= 0.80): Returned directly to the coding agent—do not burn an L2 turn repeating an obvious omission.
ESCALATE_L2_FLASH (0.20 < P(yes) < 0.90 or confidence < 0.80): Escalate only those specific Bi IDs to a fresh read-only subagent.
Subagent Routing Across Coding Agents
Always spawn Tier 2 as a fresh, stateless, read-only subagent that never sees the coding agent's prior chain-of-thought or failed attempts. When spawning, use a fast or flash agent. For example, in Antigravity, call invoke_subagent with TypeName: "research" and Model: "flash".
Built-In Stateless Tier 2 Critic Prompt
Pass this exact prompt (filling in only the target Bi IDs, spec path, and changed files):
You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the
coding session and must judge solely from the specification and actual files on disk.
Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].
Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.
Rules:
1. Cite exact file paths, line numbers, and observed code/test behaviour.
2. Check whether tests verify real behaviour rather than mocking away logic.
3. For each assertion ID, return PASS or FAIL with a concise root-cause note.
4. Do not invent fixes or assume uninspected files pass.
Circuit Breaker Trio & Continuous Ratchet
Stop the loop whenever any of these three conditions triggers:
- 100% Pass: Tier 1 passes, Tier 1.5 reports
exit_code == 0, and any final_l2_arch_ids clear their final Tier 2 (Model: "flash") review (Success).
- Pass Budget Exhausted: Maximum 4–5 passes per slice reached (
Exhausted—escalate remaining blockers to the user).
- 2 Consecutive Non-Improving Passes: Neither failing assertion count decreases nor
ratchet_score (mean P(yes) across evaluated B1–Bn items) improves across two consecutive passes (No progress—escalate to the user rather than spinning).
Tracking the continuous ratchet_score avoids premature stalls when a pass genuinely moves an assertion from e.g. P(yes) = 0.25 to 0.82 while still short of 0.90, and catches silent regressions immediately.
Loop Receipt Format (LOOPS.md)
Record completed runs (or append per-pass telemetry) concisely:
## Tiered Rubric Loop Receipt
- **Target**: [Slice / Feature / PR]
- **Spec**: [docs/SPEC.md] (`B1`–`Bn`)
- **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`
- **Tier 1**: `make check` -> PASS
- **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)
- **Tier 2 (`Model: "flash"`)**: Verified `[arch]` IDs `B3` -> PASS
- **Remaining / Escalation**: None (or exact failing `Bi` + file:line)
1---2name: tiered-rubric-loop3description: |4 Drops a combined deterministic and semantic rubric verification loop into any5 project, acting as the loop component of an agentic harness. Automatically6 leverages existing Spec-Driven Development (SDD) and Test-Driven Development7 (TDD) artefacts - or creates lightweight ones inline - routing each check across8 fast Tier 1 deterministic gates, Tier 1.5 calibrated scoring via the Gemini9 Decisions API (fast and 0 output tokens), and a Tier 2 stateless semantic 10 critique only for 1.5 escalations or complex `[arch]` invariants. 11 Use when running iterative implementation/refinement loops, gating12 PRs or complex slices against `B1`–`Bn` assertions (`SPEC.md` / `LOOPS.md`),13 or scoring diffs and multimodal diagrams.14metadata:15 source: https://skillmd.com/skills/derailed-dash/tiered-rubric-loop16 author: Darren "Dazbo" Lester17 repository: https://github.com/derailed-dash/dazbo-agent-skills18 cli:19 - uv20---2122# Tiered Rubric Loop (`tiered-rubric-loop`)2324A fast, loosely coupled way to drop a combined **deterministic + semantic verification loop** into any repository as the verification engine of an agentic harness.2526It automatically reuses any **Spec-Driven Development (SDD)** (`docs/SPEC.md` or `SPEC.md`) and **Test-Driven Development (TDD)** artefacts already in the repo, or bootstraps a lightweight `B1–Bn` binary rubric checklist inline if none exist.2728Instead of waking an expensive reasoning subagent on every single pass, or trusting a zero-thinking classifier blindly on deep security and concurrency invariants, each assertion flows through three escalating stages:29301. **Tier 1 — Deterministic Checks (fast)**: Runs local linters, typecheckers, and unit tests (`make check`, `codespell`, `ruff`, `pyright`, `pytest`) first. Any failure stops the pass immediately so you never waste API calls on broken code.312. **Tier 1.5 — Fast Semantic Scoring (fast model, no output tokens)**: Evaluates open binary (`yes`/`no`) rubric criteria (`B1–Bn`) in a single prefill pass using the **Gemini Decisions API** (`gemini-decision-1` via [`scripts/eval_rubric.py`](scripts/eval_rubric.py)). Straightforward editorial and doc-sync checks pass immediately, while obvious omissions fail fast without waking a subagent.323. **Tier 2 — Stateless Semantic Critique**: Spawns a fresh, read-only reviewer subagent *only* when Tier 1.5 is uncertain ("grey zone"), or for complex architectural and security invariants (`[arch]`) that need deeper multi-file reasoning.3334## Table of Contents3536- [How the 3-Stage Funnel Works](#how-the-3-stage-funnel-works)37- [Plain-Markdown `B1`–`Bn` Contract & Tagging](#plain-markdown-b1bn-contract--tagging)38- [Loose Coupling & Companion Skills Matrix](#loose-coupling--companion-skills-matrix)39- [Bounded Loop Workflow](#bounded-loop-workflow)40- [Running Tier 1.5 (`eval_rubric.py`)](#running-tier-15-eval_rubricpy)41- [Threshold Routing & Stateless Tier 2 Critic](#threshold-routing--stateless-tier-2-critic)42- [Circuit Breaker Trio & Continuous Ratchet](#circuit-breaker-trio--continuous-ratchet)43- [Loop Receipt Format (`LOOPS.md`)](#loop-receipt-format-loopsmd)4445---4647## How the 3-Stage Funnel Works4849| Tier | Mechanism | Latency & Cost | What It Owns |50| :--- | :--- | :--- | :--- |51| **Tier 1 (Deterministic)** | Local linters, typecheckers, unit tests (`make check`, `codespell`, `ruff`, `pyright`, `pytest`) | Seconds, **`$0`** | Syntax, static types, formatting, and deterministic test execution. Any failure **short-circuits immediately** before Tier 1.5 or Tier 2 runs. |52| **Tier 1.5 (Calibrated Probabilistic)** | `gemini-decision-1` via `client.decisions.create` (`v1beta`) in [`scripts/eval_rubric.py`](scripts/eval_rubric.py) | **~150–250 ms**, **`$0.10` / 1M input tokens**, **0 output / thinking tokens** | Evaluates up to 1,000 closed-ended `noul` (yes/no) assertions over a single shared diff/spec/media payload. Terminates clear mechanical passes (`PASS`), rejects obvious omissions immediately (`FAIL_FAST`), and flags uncertain items for L2. |53| **Tier 2 (Stateless Semantic Critic)** | Fresh read-only `flash` subagent with zero prior coding chain-of-thought | 10–40 s, standard model tokens | Diagnoses grey-zone escalations and performs **one final L2 sign-off** on `[arch]` items after Tier 1.5 is green. |5455**Key safety rule**: Because `gemini-decision-1` runs in a single prefill forward pass with **zero thinking tokens** and **zero file-reading tools**, it can produce false positives on subtle logic, race conditions, or auth flaws. Therefore, Tier 1.5 is **terminal** for mechanical/editorial checks, **fail-fast** on clear omissions, and **provisional** for `[arch]` assertions.5657---5859## Plain-Markdown `B1–Bn` Contract & Tagging6061The sole shared contract is a Markdown table or bullet list of numbered binary assertions (`B1–Bn`) in `docs/SPEC.md`, `SPEC.md`, or `LOOPS.md` (with optional `--spec <path>` override):6263```markdown64| ID | Falsifiable Condition | Verification Target |65| :--- | :--- | :--- |66| `B1 [det]` | Static gates (`codespell`, `ruff`, `pyright`) and offline tests exit `0` | `make check` |67| `B2` | `.env.template` and `README.md` document every newly added variable and flag | Doc review |68| `B3 [arch]` | Auth middleware rejects expired tokens and never retries `4xx` responses | `tests/unit/test_auth.py` |69```7071Other supported rubric formats and tags:7273- **Bullet assertions**: `- **B4**: CLI help panel renders in English (UK)`74- **Atomic sub-IDs**: `B4a`, `B11b`75- **Numbered items**: ``1. `B5 [docs]`: ...``76- **Task checkboxes**: `- [ ] **B6 [concurrency]**: ...` or `- [x] **B6**: ...`77- **Extended tags**: `[mech]`, `[docs]`, `[ui]`, `[visual]`, `[perf]`, `[concurrency]` (unrecognised tags warn on `stderr` and default safely to `[mech]` instead of silently dropping the assertion)7879| Tag | Meaning | Tier 1.5 Handling |80| :--- | :--- | :--- |81| `[det]` (or deterministic Verification Target such as `make check`, `pytest tests/...`, `codespell -s`, `ruff check .`) | Already proven by Tier 1 shell exit codes | **Skipped by Tier 1.5** by default (`--include-det` overrides) so the model never guesses whether `pytest` passed from a text diff. |82| *(untagged, `[mech]`, `[docs]`, `[ui]`, `[visual]`, `[multimodal]`)* | Mechanical, editorial, doc-sync, schema, or UI convention check | **Terminal `PASS`** when `P(yes) >= 0.90`, `confidence >= 0.80`, and **no evidence was truncated**; `FAIL_FAST` when `<= 0.20`; otherwise `ESCALATE_L2_FLASH`. |83| `[arch]` (also `[security]`, `[logic]`, `[concurrency]`, `[perf]`) | Concurrency, security, state invariant, performance, or test-realism check | **Provisional pass (`PROVISIONAL_ARCH_PASS`)** at Tier 1.5; gated for **one final Tier 2 (`Model: "flash"`) review** once all items clear Tier 1.5. |8485Repo-relative file paths and local images named in `Verification Target` or inline backticks are **automatically resolved and attached** (blocking `.env`, secret env files, `*.enc`, and paths outside `--repo-root`; pass `--no-auto-target-files` to disable).8687### Writing High-Signal `B1–Bn` Assertions8889- **Keep assertions atomic**: Test **one falsifiable invariant per `Bi`** rather than bundling three functions or eight files into one question—single-invariant assertions give sharp `P(yes)` separation and trigger `FAIL_FAST` accurately on regressions.90- **Name concrete paths in `Verification Target`**: Prefer `src/auth.py, tests/unit/test_auth.py` over generic `Doc review` / `Code review` so `eval_rubric.py` auto-attaches untouched context files even in narrow working-tree diffs.91- **Separate canonical registries from summary copy**: Don't assert identical title formatting across both machine registries (`navigation.yaml`) and summarised overview pages (`README.md`, spotlight cards) unless exact parity is truly required.9293---9495## Loose Coupling & Companion Skills Matrix9697`tiered-rubric-loop` requires **zero hard dependencies** on other skills and degrades cleanly when optional companions or API keys are absent:9899| Companion Skill / Dependency | When Present | Standalone Fallback (When Absent) |100| :--- | :--- | :--- |101| `maintaining-core-documentation` / `spec-driven-development` | Reads `B1`–`Bn` directly from `docs/SPEC.md` or `SPEC.md`. | Reads any local `SPEC.md` / `LOOPS.md`, or asks you to agree a compact `B1`–`Bn` list inline before coding. |102| `test-driven-development` | Delegates the **Act** step to full Red-Green-Refactor execution. | Enforces the core TDD rule inline: write/update a failing test first, change one semantic behaviour per pass, never weaken assertions. |103| `loopy` | Supplies the outer `Observe -> Choose -> Act -> Verify -> Record -> Stop` governance and `LOOPS.md` receipts. | Runs the identical bounded loop directly with the receipt format below. |104| `local-code-review` | Stateless Tier 2 read-only subagent runs `local-code-review` on escalated/`[arch]` items. | Uses the built-in **Stateless Tier 2 Critic Prompt** below. |105| `GEMINI_API_KEY` / `GOOGLE_API_KEY` | Enables ~200 ms Tier 1.5 screening via `eval_rubric.py`. | When `eval_rubric.py` exits `3` (missing/invalid key), skips Tier 1.5 cleanly and executes `Tier 1 -> Tier 2` directly while noting how to enable Tier 1.5. |106107---108109## Bounded Loop Workflow110111Track every pass around **`Observe -> Choose -> Act -> Verify -> Record -> Stop`**:1121131. **Observe**: Read the specification (`docs/SPEC.md`, `SPEC.md`, or `LOOPS.md`), `git status --short`, and current failing tests/checks. Confirm `B1`–`Bn` exist (draft and confirm them first if missing).1142. **Choose**: Pick **one** highest-value unverified semantic behaviour or failing assertion for this pass (mechanical lint/spelling fixes may be batched).1153. **Act (TDD)**: Write or update the failing test first (**Red**), confirm it fails, then implement the minimal clean change (**Green**) and refactor without weakening tests.1164. **Verify (3-Stage Funnel)**:117 - **Tier 1**: Run deterministic checks (`make check` or repo lint + test commands). If Tier 1 fails, **stop verification immediately** and return to **Choose/Act**—do not spend tokens on Tier 1.5 or Tier 2.118 - **Tier 1.5**: Run `eval_rubric.py --json` over the spec, diff, and touched files (plus `--image` / `--pdf` for diagram/UI specs).119 - **Tier 2**: Escalate only grey-zone items immediately, or run a single final `[arch]` critic pass once Tier 1.5 reports zero `FAIL_FAST` and zero grey-zone items.1205. **Record**: Append pass outcome, `ratchet_score` (mean `P(yes)`), and remaining failures to `LOOPS.md` (or session progress notes).1216. **Repeat or Stop**: Enforce the **Circuit Breaker Trio** before starting another pass.122123---124125## Running Tier 1.5 (`eval_rubric.py`)126127Resolve `scripts/eval_rubric.py` relative to where `tiered-rubric-loop` is installed:128- Global Antigravity path: `~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py`129- Project-local path: `.agents/skills/tiered-rubric-loop/scripts/eval_rubric.py`130131In sandboxed agent runtimes that block outbound HTTPS by default, invoke `uv run` with unsandboxed network permission (`BypassSandbox: true` in Antigravity) for live API calls, or pass `--offline` for sandboxed `--dry-run` checks.132133```bash134# 1. Working-tree loop: auto-discover SPEC.md / docs/SPEC.md / LOOPS.md135uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --json136137# 2. Branch or merged PR gating: evaluate committed changes against base ref / range138uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --base main --json139uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \140 --git-range origin/main...HEAD \141 --only B3,B4 \142 --json143144# 3. Explicit spec, scoped files, and multimodal diagram/UI screenshot check145uv run ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py \146 --spec docs/SPEC.md \147 --file src/auth.py \148 --image docs/images/architecture.png \149 --json150151# 4. Offline dry-run inside sandbox (verifies rubric, targets, and truncation zero-API)152uv run --offline ~/.gemini/config/skills/tiered-rubric-loop/scripts/eval_rubric.py --dry-run --json153```154155For large PRs (`> 8` files or `> 100 KB`), shard evaluations by assertion group via `--only B3,B4` so files stay within `--max-file-chars` (`24,000`) and `--max-evidence-chars` (`180,000`). If any file or combined payload is truncated, `eval_rubric.py` populates `truncated_sources` and **downgrades mechanical `PASS` to `ESCALATE_L2_FLASH`** unless `--allow-truncated-pass` is explicitly set.156157### Exit Codes158159| Code | Meaning | Loop Action |160| :--- | :--- | :--- |161| `0` | Tier 1.5 clear (all evaluated items are `PASS` or `PROVISIONAL_ARCH_PASS`) | If `final_l2_arch_ids` is non-empty, run **one** Tier 2 (`Model: "flash"`) critic pass on those IDs; otherwise exit loop as **`Success`**. |162| `1` | One or more `FAIL_FAST` or `ESCALATE_L2_*` items (or API call error) | Address `FAIL_FAST` items directly; escalate grey-zone items (`ESCALATE_L2_*`) only when root cause is non-obvious. |163| `2` | Usage error, missing spec, or no `B1`–`Bn` assertions parsed | Fix spec path or rubric formatting before continuing. |164| `3` | Missing/invalid `GEMINI_API_KEY` or `GOOGLE_API_KEY` | Degrade gracefully to **`Tier 1 -> Tier 2`** and inform the user once. |165166---167168## Threshold Routing & Stateless Tier 2 Critic169170Default calibrated thresholds (overridable via `--pass-threshold`, `--fail-threshold`, `--min-confidence`):171- **`PASS`** (`P(yes) >= 0.90`, `confidence >= 0.80`, mechanical): **Terminal pass.** No L2 model runs.172- **`PROVISIONAL_ARCH_PASS`** (`P(yes) >= 0.90`, `confidence >= 0.80`, `[arch]`): Held until all other assertions pass, then verified once by a Tier 2 (`Model: "flash"`) critic.173- **`FAIL_FAST`** (`P(yes) <= 0.20`, `confidence >= 0.80`): Returned directly to the coding agent—do **not** burn an L2 turn repeating an obvious omission.174- **`ESCALATE_L2_FLASH`** (`0.20 < P(yes) < 0.90` or `confidence < 0.80`): Escalate **only those specific `Bi` IDs** to a fresh read-only subagent.175176### Subagent Routing Across Coding Agents177178Always spawn Tier 2 as a **fresh, stateless, read-only subagent** that never sees the coding agent's prior chain-of-thought or failed attempts. When spawning, use a fast or flash agent. For example, in **Antigravity**, call `invoke_subagent` with `TypeName: "research"` and `Model: "flash"`.179180### Built-In Stateless Tier 2 Critic Prompt181182Pass this exact prompt (filling in only the target `Bi` IDs, spec path, and changed files):183184```text185You are a stateless, read-only Tier 2 semantic critic. You have NOT seen the186coding session and must judge solely from the specification and actual files on disk.187188Evaluate ONLY these rubric assertions: [INSERT B_i IDs AND CONDITIONS].189Read: [SPEC.md / docs/SPEC.md] and inspect the changed/untracked files + git diff.190191Rules:1921. Cite exact file paths, line numbers, and observed code/test behaviour.1932. Check whether tests verify real behaviour rather than mocking away logic.1943. For each assertion ID, return PASS or FAIL with a concise root-cause note.1954. Do not invent fixes or assume uninspected files pass.196```197198---199200## Circuit Breaker Trio & Continuous Ratchet201202Stop the loop whenever **any** of these three conditions triggers:2032041. **100% Pass**: Tier 1 passes, Tier 1.5 reports `exit_code == 0`, and any `final_l2_arch_ids` clear their final Tier 2 (`Model: "flash"`) review (**`Success`**).2052. **Pass Budget Exhausted**: Maximum **4–5 passes** per slice reached (**`Exhausted`**—escalate remaining blockers to the user).2063. **2 Consecutive Non-Improving Passes**: Neither failing assertion count decreases **nor** `ratchet_score` (mean `P(yes)` across evaluated `B1`–`Bn` items) improves across two consecutive passes (**`No progress`**—escalate to the user rather than spinning).207208Tracking the continuous `ratchet_score` avoids premature stalls when a pass genuinely moves an assertion from e.g. `P(yes) = 0.25` to `0.82` while still short of `0.90`, and catches silent regressions immediately.209210---211212## Loop Receipt Format (`LOOPS.md`)213214Record completed runs (or append per-pass telemetry) concisely:215216```markdown217## Tiered Rubric Loop Receipt218219- **Target**: [Slice / Feature / PR]220- **Spec**: [docs/SPEC.md] (`B1`–`Bn`)221- **Passes**: [N / 5] | **Outcome**: `Success` | `Exhausted` | `No progress` | `Blocked`222- **Tier 1**: `make check` -> PASS223- **Tier 1.5 Ratchet (`gemini-decision-1`)**: `0.58 -> 0.94 -> 0.98` (skipped `[det]`: `B1`)224- **Tier 2 (`Model: "flash"`)**: Verified `[arch]` IDs `B3` -> PASS225- **Remaining / Escalation**: None (or exact failing `Bi` + file:line)226```