Auto Iteration Loop

Auto Iteration Loop: Adversarial Review with Bounded Back-Edges

Autonomously iterate over /auto-verify's per-claim verdicts: review → implement fixes (variant-integrity, main-experiment-script, or claim-stage rewrite) → re-review, until the external reviewer gives a positive assessment with all FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS claims resolved, or the iteration budget is exhausted.

Context: $ARGUMENTS

Constants

• MAX_ITERATIONS = 6 — Hard upper bound on total back-edge actions across the whole loop. Each of the following consumes 1 iteration, regardless of which Phase A→E cycle they fall in:

  • ① variant-only fix → re-run /auto-verify <claim-id> — resume: true
  • ② main-experiment-script / EXPERIMENT_PLAN.md fix → re-run /auto-experiment (optionally followed by /auto-verify chain)
  • ③ claim-stage re-entry → rewrite claim (full /auto-claim invocation OR lightweight in-loop rewrite) → re-run downstream stages

  Pure reviewer cycles that change nothing on disk (no new experiments, no claim changes) do not consume iterations. PASS-claim consistency checks and deferred-claim recording do not consume iterations. New claims produced by ③ inherit the same budget — they do not get a fresh MAX_ITERATIONS of their own. Override via MAX_ITERATIONS: arg from agents/iteration.md.

  Resource-Fidelity invariant (all back-edges). If refine-logs/FINAL_PROPOSAL.md / refine-logs/EXPERIMENT_PLAN.md carry a resource_fidelity: strict marker (the reproduction combination — behavior-source:given + mechanism:given), every back-edge MUST preserve it: when editing EXPERIMENT_PLAN.md (type ②) or any claim rewrite (type ③), keep the top-of-file resource_fidelity: line verbatim — never drop, comment out, or change it. Dropping the marker silently disables the harness, re-opening the door to smaller-model swaps and data subsetting the harness exists to prevent. For type-③ full re-entry, detect the original run's axes from idea-stage/IDEA_REPORT.md's **Behavior-source**: / **Mechanism**: headers (or directly from the resource_fidelity: strict marker) and pass both behavior_source and mechanism into the queued auto-claim call so a reproduction-combo re-entry stays given+given (the new run re-stamps the resource_fidelity: strict marker automatically — it is marker-driven, not a flag).

• MAX_CLAIM_REENTRIES = 2 — Sub-budget within MAX_ITERATIONS for action type ③ (claim-stage re-entry). Prevents the failure mode where the reviewer keeps asking "rewrite the claim" without anyone actually fixing the experiments behind it. When claim_reentries_consumed >= MAX_CLAIM_REENTRIES, the loop refuses further ③ actions even if iterations remain — at that point the only legal back-edges are ① and ②, and if those are also exhausted the loop terminates and writes the final report.

• TARGET_SCORE = 6 — Stop when reviewer score >= this AND verdict matches a POSITIVE_VERDICT_TERMS entry AND no claim is still in FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS (three-dimensional STOP rule — see Phase B).

• POSITIVE_VERDICT_TERMS = [ready, almost] — Case-insensitive substrings that count the verdict as positive. Phase B canonicalizes the reviewer's free-form verdict to one of {ready, almost, not ready} before this check, so any reviewer-supplied synonym (e.g., accept, sufficient, good enough) must be mapped to ready / almost by Phase B's parser — see Phase B's STOP rule for the canonical check.

• REVIEW_DOC = review-stage/AUTO_REVIEW.md — Per-iteration append-only audit log (fall back to ./AUTO_REVIEW.md for legacy projects). This is the raw audit trail; the narrative report is FINAL_REPORT_DOC below.

• REVIEWER_MEMORY_DOC = review-stage/REVIEWER_MEMORY.md — Reviewer's persistent suspicion log across iterations. Append-only: each iteration's Phase B.5 adds a section under ## Iteration N, never deletes prior iterations (audit trail). Phase A of iteration 2+ prepends the full file to the reviewer prompt so the reviewer can check whether prior-iteration suspicions were genuinely addressed or sidestepped. Counters the failure mode where Claude curates a comfortable subset of context for the reviewer each iteration.

• FINAL_REPORT_DOC = review-stage/AUTO_ITERATION_FINAL_REPORT.md — Narrative final report written once in Termination (after the loop ends for any reason). Organized per-claim by their original /auto-verify category (PASS / FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS / DEFERRED), with the per-claim iteration journey, experiment/script diffs, and any claim rewrites laid out inside each claim's section. Distinct from REVIEW_DOC (which is the chronological audit log). See "Final Report" section below for the full template.

• GPU_ID = auto — GPU device(s) to pin every Phase-C /run-experiment dispatch to. auto lets the launcher / environment decide (no extra export). A single id (0) or comma-list (4,5,6,7) causes this skill to pass CUDA_VISIBLE_DEVICES=<GPU_ID> as the first positional arg to every /run-experiment invocation (the run-experiment skill exports it internally; do not treat as a shell prefix), and record the effective CUDA_VISIBLE_DEVICES in each run's run.sh. Forwarded from agents/iteration.md's gpu_id: arg (which the orchestrator forwards from /auto's GPU_ID).

• RESUME = false — When true, read review-stage/REVIEW_STATE.json and pick up from iterations_consumed + 1 instead of starting fresh. Append new iterations to the existing AUTO_REVIEW.md rather than overwriting it. Iteration budget and claim-reentry sub-budget are inherited across resume — they are never reset. If REVIEW_STATE.json already shows status: completed (positive verdict, iterations exhausted, or claim-reentry budget exhausted), return immediately without running another iteration. If status: awaiting_upstream, the orchestrator (see auto/SKILL.md) is expected to run the pending upstream calls first, then resume this loop. Default false = always start fresh and overwrite prior review state. Resume never deletes pre-existing review state. Schema fields read here: iterations_consumed, claim_reentries_consumed, status, last_verdict, last_score — see "State Persistence" below.

Reviewer LLM Configuration (mandatory, read first)

This skill calls an external LLM reviewer. Never hardcode a model name and never read the reviewer model from task.md / project READMEs / source comments. Project-level files may list available API keys for unrelated purposes (e.g., LLM-as-judge inside experiment code); those are not the reviewer config.

Resolve LLM_MODEL, LLM_BASE_URL, LLM_API_KEY strictly in this priority order before any reviewer call:

1. Project MCP config — ${PROJECT_ROOT}/.mcp.json, field mcpServers["llm-chat"].env.{LLM_MODEL,LLM_BASE_URL,LLM_API_KEY}.
2. User MCP config — ~/.claude/settings.json, same field.
3. Shell environment — $LLM_MODEL, $LLM_BASE_URL, $LLM_API_KEY.

Pre-flight check (run before Phase A of every iteration, mandatory)

LLM_MODEL_SRC=""
if [ -f .mcp.json ] && jq -e '.mcpServers["llm-chat"].env.LLM_MODEL' .mcp.json >/dev/null 2>&1 ; then
  export LLM_MODEL=$(jq -r '.mcpServers["llm-chat"].env.LLM_MODEL' .mcp.json)
  export LLM_BASE_URL=$(jq -r '.mcpServers["llm-chat"].env.LLM_BASE_URL' .mcp.json)
  export LLM_API_KEY=$(jq -r '.mcpServers["llm-chat"].env.LLM_API_KEY' .mcp.json)
  LLM_MODEL_SRC="project .mcp.json"
elif [ -f ~/.claude/settings.json ] && jq -e '.mcpServers["llm-chat"].env.LLM_MODEL' ~/.claude/settings.json >/dev/null 2>&1 ; then
  export LLM_MODEL=$(jq -r '.mcpServers["llm-chat"].env.LLM_MODEL' ~/.claude/settings.json)
  export LLM_BASE_URL=$(jq -r '.mcpServers["llm-chat"].env.LLM_BASE_URL' ~/.claude/settings.json)
  export LLM_API_KEY=$(jq -r '.mcpServers["llm-chat"].env.LLM_API_KEY' ~/.claude/settings.json)
  LLM_MODEL_SRC="user ~/.claude/settings.json"
elif [ -n "$LLM_MODEL" ] && [ -n "$LLM_BASE_URL" ] && [ -n "$LLM_API_KEY" ] ; then
  LLM_MODEL_SRC="shell env"
fi
echo "[reviewer-config] LLM_MODEL=$LLM_MODEL  LLM_BASE_URL=$LLM_BASE_URL  source=$LLM_MODEL_SRC"

Hard-fail rule: If LLM_MODEL is empty after this resolution (none of the three sources provides it), the skill MUST abort with:

"Reviewer model not configured. Add mcpServers.llm-chat.env.{LLM_MODEL,LLM_BASE_URL,LLM_API_KEY} to .mcp.json (project) or ~/.claude/settings.json (user)."

Do not guess a default. Do not fall back to a model name read from task.md or any other project file.

The earlier ## LLM Configuration example block in older versions of this skill (which duplicated the same llm-chat MCP setup) has been removed — the authoritative config resolution is the priority order above. For a worked .mcp.json example, see mcp-servers/llm-chat/ in the project repo.

API Call Method

Primary: MCP Tool

mcp__llm-chat__chat:
  prompt: |
    [Review prompt content]
  model: "${LLM_MODEL}"   # resolved per "Reviewer LLM Configuration" priority order above — never hardcode
  system: "You are a senior ML reviewer..."

Fallback: curl

curl -s "${LLM_BASE_URL}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LLM_API_KEY}" \
  -d '{
    "model": "${LLM_MODEL}",
    "messages": [
      {"role": "system", "content": "You are a senior ML reviewer..."},
      {"role": "user", "content": "[review prompt]"}
    ],
    "max_tokens": 4096
  }'

State Persistence (Compact Recovery)

Persist state to review-stage/REVIEW_STATE.json after each iteration (i.e., every Phase E):

{
  "iterations_consumed": 2,
  "claim_reentries_consumed": 1,
  "status": "in_progress",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "consecutive_noop_count": 0,
  "iteration_breakdown": [
    {"i": 1, "type": "variant_fix",       "target_claims": ["C1"], "produced_claims": []},
    {"i": 2, "type": "claim_reentry",     "target_claims": ["C2"], "produced_claims": ["C2_v2"]}
  ],
  "claim_stage_reentries": ["C2"],
  "pending_upstream_calls": [],
  "runs_total": 5,
  "gpu_hours_total": 12.4,
  "thread_id": "th_01ABC123def456",
  "timestamp": "2026-03-15T10:00:00"
}

• iterations_consumed is the cumulative count of back-edge actions (types ①/②/③) since iteration 1 — what's compared against MAX_ITERATIONS.
• claim_reentries_consumed is the cumulative count of type ③ actions only — what's compared against MAX_CLAIM_REENTRIES.
• iteration_breakdown records each iteration's action type, the claim(s) it targeted, and any new claim ids produced by claim-stage re-entry (so the final report can attribute new claims to the right ancestor).
• claim_stage_reentries is the flat list of claim ids that triggered ③ — surfaced to agents/iteration.md so the orchestrator can record which original claims got rewritten in the Ledger's journey_summary.iteration line and open_items[].
• pending_upstream_calls is non-empty only when status = awaiting_upstream: a list of {skill, args} calls the orchestrator must execute before resuming this loop.
• runs_total / gpu_hours_total are cumulative across all iterations; they monotonically increase and are the canonical budget source agents/iteration.md reports back to the orchestrator.

Field consumer map (so future maintainers know which fields are external API vs internal state):

Write this file at the end of every Phase E (after documenting the iteration).

On completion, set "status": "completed" and continue to Termination (which generates FINAL_REPORT_DOC).

On back-edge handoff to orchestrator (Phase C decided a fix requires upstream skills the iteration loop cannot run inline — e.g., full /auto-claim re-invocation), set "status": "awaiting_upstream", populate pending_upstream_calls, and return. The orchestrator runs them, then resumes this skill with RESUME=true.

Workflow

Upstream artifacts pre-flight (mandatory, run first)

This skill iterates over existing artifacts produced by upstream stages — it does not generate them from scratch. $ARGUMENTS (the optional research-direction string) is treated as reviewer context only, never as a starting point.

Before doing anything else, verify the project root contains at least one of:

• idea-stage/IDEA_REPORT.md
• refine-logs/FINAL_PROPOSAL.md
• refine-logs/EXPERIMENT_RESULTS.md
• verify/VERIFY_REPORT.md

If none of these exist, abort with:

[pre-flight] no upstream artifacts found at idea-stage/ refine-logs/ verify/.
            /auto-iteration-loop reviews existing work — it does not start from a topic.
            Run /auto (or at minimum /auto-claim then /auto-experiment) first.
            Aborting.

Do not invoke the reviewer, do not write review-stage/AUTO_REVIEW.md. Abort cleanly.

Build five disjoint claim buckets from verify/VERIFY_REPORT.md

If verify/VERIFY_REPORT.md is present, read it once at the top of every iteration alongside verify/INTEGRITY_AUDIT.md and build five disjoint claim buckets that drive Phase A's reviewer prompt. The five states mirror /auto-verify's terminal states exactly (see auto-verify/SKILL.md Phase 11 column glossary):

• verify_passed — claim ids whose state is PASS. Main-experiment verdict is robust under swaps. Routing: brief consistency check only (see below). No experiments fired.
• verify_failed — claim ids whose state is FAIL (robustness < ROBUSTNESS_THRESHOLD over enough eligible variants; the main experiment was fragile). Routing: two-phase (variant-integrity fix → optional claim-stage re-entry).
• verify_inconclusive — claim ids whose state is INCONCLUSIVE (Phase 2 main-experiment-integrity FAIL; variants never ran, or were policy-skipped under audit-only mode — semantics identical). Routing: fix the main experiment only (EXPERIMENT_PLAN.md step or main-experiment script).
• verify_zero_eligible_variants — claim ids whose state is ZERO_ELIGIBLE_VARIANTS (Phase 2 the main experiment was clean, but every Phase 9 variant FAILed integrity). Routing: fix variant scripts only (do not touch the main experiment).
• verify_integrity_only — claim ids whose state is INTEGRITY_ONLY (Phase 2 pass/warn, Stage 2 intentionally skipped). Two producers, distinguished per-claim by stage2_skip_reason in ROBUSTNESS.md: swap_variants_false (global audit-only mode) or max_verify_claims_cap (this claim was admitted by Stage 1 but not the top-K picked at Phase 3 step 0). Routing: no back-edge action — record in Open Items with per-stage2_skip_reason upgrade suggestion. See per-bucket contract below.

Plus the orthogonal (legacy):

• deferred_claims — Legacy bucket, always empty under the current architecture. Verify no longer cuts claims before audit; claims that would have been cut by MAX_VERIFY_CLAIMS in the old design now land in verify_integrity_only with stage2_skip_reason: max_verify_claims_cap. Retained as an empty placeholder for backward compatibility with older VERIFY_REPORT.md files. New runs never populate it.

The five state buckets are disjoint by construction (a claim has exactly one /auto-verify state); the legacy deferred_claims bucket is empty in new runs, so no overlap concerns arise. Under SWAP_VARIANTS=false, only two buckets can be non-empty: verify_inconclusive and verify_integrity_only (with stage2_skip_reason: swap_variants_false) — verify_passed / verify_failed / verify_zero_eligible_variants all require Phase 9 to have run and are always empty.

Per-bucket routing contract

verify_passed — brief consistency check. For each claim id in verify_passed, the reviewer does narrative + numeric consistency only:

• Does the claim wording match the supporting numbers in refine-logs/EXPERIMENT_RESULTS.md and verify/<claim_dir>/ROBUSTNESS.md?
• Are there caveats (main-experiment-WARN tag) that need to surface in the final paper but not in the iteration loop's actions?

Record one line per claim under the iteration's ### Verify-Passed Claims (brief audit) section in AUTO_REVIEW.md. Does NOT consume an iteration — the reviewer's consistency check is part of the same Phase A reviewer call.

verify_failed — two-phase routing. For each claim id in verify_failed:

• Phase 1 — Variant integrity check. Read this claim's verify/<claim_dir>/variant_audit/{EXPERIMENT,MECHANISM}_AUDIT.md and each surviving variant's verdict.json.integrity_breakdown. If any surviving variant had a WARN/FAIL that could have biased the consistency check, that's the lead — fix that variant's script (deletion + targeted re-run is fine; full sweep is not required) and re-invoke /auto-verify <claim-id> — resume: true. Consumes 1 iteration (type ①).

• Phase 2 — Claim-stage re-entry. If Phase 1 was clean (or Phase 1 was already executed in a prior iteration and the claim still came back FAIL with clean variants), the divergence is real and the claim itself needs work. Choose one of:

  • Lightweight in-loop rewrite (preferred when the reviewer has a concrete narrower claim in mind): write the new claim text directly into a claim_rewrite: block in AUTO_REVIEW.md, then chain /auto-experiment + /auto-verify targeted to the rewritten claim only. No /auto-claim invocation.
  • Full claim-stage re-entry: set status = awaiting_upstream and queue pending_upstream_calls = [{skill: "auto-claim", args: {focus: "<claim-id> — verify-failed context: ...", behavior_source: "<inherit original>", mechanism: "<inherit original>"}}, {skill: "auto-experiment", args: {target_claims: ["<new-id>"]}}, {skill: "auto-verify", args: {target_claims: ["<new-id>"]}}]. Carry the original run's behavior_source + mechanism into the auto-claim args so a reproduction-combo (given+given) re-entry re-stamps resource_fidelity: strict automatically (marker-driven — per the invariant above). The orchestrator runs them and resumes this loop with the new claim's /auto-verify result.

  Either form consumes 1 iteration (type ③) AND 1 claim-reentry sub-budget. The new claim id (when produced) is recorded in iteration_breakdown[i].produced_claims so the final report attributes it to the original claim's section, not a fresh top-level entry.

  Crucially: when claim_reentries_consumed >= MAX_CLAIM_REENTRIES, Phase 2 is not allowed even if the reviewer requests it — record the request, decline it in the iteration's writeup, and either return to Phase 1 (if there are still dirty variants left to fix) or terminate with the claim still in FAIL state (it gets surfaced in the final report's Open Items section).

  The reviewer may NOT choose "narrow scope" or "pivot" as standalone local actions — those are subsumed by Phase 2's claim rewrite.

verify_inconclusive — main-experiment-only fix. For each claim id in verify_inconclusive:

• Read this claim's verify/<claim_dir>/main_experiment_audit/{EXPERIMENT,MECHANISM}_AUDIT.md to identify which sub-audit FAILed (the ROBUSTNESS.md stub's inconclusive_reason field summarizes this).
• Fix the failing main-experiment surface: edit the relevant step in refine-logs/EXPERIMENT_PLAN.md (preserving the top-of-file resource_fidelity: marker verbatim — see the Resource-Fidelity invariant above), modify the main-experiment script (experiments/<name>/), then re-invoke /auto-experiment (with target_claims: [<id>] so only this claim's runs re-execute, if supported by the upstream skill) followed by /auto-verify. Consumes 1 iteration (type ②).
• The reviewer is not allowed to "narrow scope", "pivot", or rewrite the claim for an INCONCLUSIVE — the main-experiment methodology is what's broken, not the claim's framing. To rewrite the claim, the claim must first become FAIL (i.e., the main experiment must be fixed so a robustness verdict is computable).
• Per-claim ② cap (mirror of MAX_CLAIM_REENTRIES for ③). A stubborn INCONCLUSIVE main experiment must not monopolise the shared MAX_ITERATIONS budget and starve other claims. Count this claim's prior type-② actions in iteration_breakdown (no separate counter needed). If it has already had 2 ② attempts and is still INCONCLUSIVE, do not attempt a 3rd: record the reviewer's request, decline it in the iteration writeup, and leave the claim INCONCLUSIVE in the final report's Open Items (requires manual main-experiment repair). This frees the remaining budget for other claims; if no other claim has an actionable fix, terminate. (The cap is on iteration-loop ② attempts — at most 2 — not on the original original /auto-experiment run.)

verify_zero_eligible_variants — variant-only fix. For each claim id in verify_zero_eligible_variants:

• Read this claim's verify/<claim_dir>/variant_audit/{EXPERIMENT,MECHANISM}_AUDIT.md and the zero_eligible_reason field in ROBUSTNESS.md.
• Fix each failing variant's script (variant eval.py, mechanism harness, etc.), delete the failed variant directories under verify/<claim_dir>/variants/, then re-invoke /auto-verify <claim-id> — resume: true standalone (Phase 1's argument parser and Phase 2's main-experiment audit will skip on resume; only variant phases re-execute). Consumes 1 iteration (type ①, same as FAIL Phase 1).
• Do NOT invoke /auto-experiment — the the main experiment already passed Phase 2; rerunning it wastes compute and risks main-experiment drift.

verify_integrity_only — no-action-with-upgrade-suggestion contract. For each claim id in verify_integrity_only the reviewer must NOT propose any back-edge action — main-experiment integrity was validated at Phase 2 (main_experiment_integrity ∈ {pass, warn} in ROBUSTNESS.md) and the swap stress test was intentionally skipped. Read the per-claim stage2_skip_reason field to pick the right upgrade string:

• stage2_skip_reason: swap_variants_false (global audit-only mode) → upgrade suggestion: to stress-test under method/dataset/model swaps: /auto-verify <id> — swap-variants: true, resume: true (Phase 2 audits reused via RESUME; only Stages 2–3 run)
• stage2_skip_reason: max_verify_claims_cap (Stage 1 admitted but not top-K picked) → upgrade suggestion: to stress-test this admitted claim: /auto-verify <id> — resume: true (single-claim mode; Phase 2 audit reused via RESUME; only Stages 2–3 run for this claim)

Three requirements: (a) record each claim under ### Open Items — Unverified Under Swaps in AUTO_REVIEW.md with its stage2_skip_reason and the matching upgrade suggestion; (b) when main_experiment_integrity: warn, surface warn_source on the same line so the paper text can carry the caveat; (c) exclude from score-weighting reasoning — neither a positive nor a negative signal, just unfinished. The iteration agent's final return message surfaces this list to the orchestrator so it lands in CLAIMS_LEDGER.md's open_items[]. Does NOT consume an iteration.

deferred_claims — legacy no-op. New verify runs never populate this bucket (cap-cut claims land in verify_integrity_only with stage2_skip_reason: max_verify_claims_cap). On a legacy VERIFY_REPORT.md that still carries a ## Deferred Claims section, treat each id under the same no-action contract as verify_integrity_only (record under ### Open Items — Unverified Under Swaps with requires standalone /auto-verify <id> before this claim can enter the iteration loop; exclude from score-weighting). Does NOT consume an iteration.

Phase A reviewer prompt — explicit bucket list

When constructing Phase A's reviewer prompt, include all five state buckets explicitly so the reviewer can address them separately. Empty buckets are stated as none, not omitted (the reviewer needs to know "no claims FAILed" is different from "I forgot to tell you"):

verify_passed:                  [<comma-separated claim ids, or "none">]   # brief consistency check only, no experiment action
verify_failed:                  [<comma-separated claim ids, or "none">]   # two-phase routing — see contract
verify_inconclusive:            [<comma-separated claim ids, or "none">]   # main-experiment-only fix
verify_zero_eligible_variants:  [<comma-separated claim ids, or "none">]   # variant-only fix
verify_integrity_only:          [<comma-separated claim ids, or "none">]   # NO back-edge; record in Open Items with stage2_skip_reason-dispatched upgrade suggestion

deferred_claims is a legacy empty bucket in new runs and is not part of the reviewer prompt.

Flexibility note on upstream back-edges

When this loop triggers a back-edge (variant re-run, main-experiment re-run, or claim-stage re-entry), the iteration agent may choose to run a lightweight subset rather than the full upstream skill:

• For /auto-experiment: only the runs corresponding to the target claim need re-execute. If target_claims is honored by the upstream skill, pass it; if not, the lightweight path is to manually re-run the specific experiment scripts and update the corresponding runs/iteration_round_<N>/<run-id>/ directory.
• For /auto-verify: pass — resume: true so completed phases skip. (Iteration-loop invocations are always single-claim — /auto-verify <claim-id> — so the claim scoping is already implicit.)
• For /auto-claim: lightweight path is to write the rewritten claim text directly into AUTO_REVIEW.md and bypass the /auto-claim call entirely.

Write results back, every time. A lightweight path skips the owning skill, so the skill's own supersede-in-place never fires — the iteration agent inherits it. Whenever a re-run changes a claim's numbers or verdict, update that stage's documents in place before calling anything downstream: refine-logs/ for a main-experiment re-run, verify/ for a verify re-run, review-stage/ for this loop's own record. Keep a failed pre-registered test on the record as a failure; never relabel it.

Each lightweight or full variant of these actions consumes the same iteration count (and the same claim-reentry sub-budget when applicable). The agent is given the choice for compute efficiency, not for budget gaming.

Initialization

0. Run the Pre-flight check in the "Reviewer LLM Configuration" section above. Abort with the hard-fail message if LLM_MODEL cannot be resolved. Log the resolved values and source.
1. Check review-stage/REVIEW_STATE.json for recovery (fall back to ./REVIEW_STATE.json if not found — legacy path)
2. Resume gate (only when RESUME = true). Read fields from the schema documented under "State Persistence" above (iterations_consumed, claim_reentries_consumed, status, last_verdict, last_score, timestamp):
   • If REVIEW_STATE.json is missing or empty → start fresh (iterations_consumed = 0, claim_reentries_consumed = 0), log [resume] no prior state — starting fresh.
   • Staleness check (first, before any other branch — but exempt awaiting_upstream): parse timestamp as ISO 8601. Skip this check entirely when status == "awaiting_upstream" — that is a deliberate, live handoff waiting on orchestrator-run upstream calls (a type-③ claim re-entry's /auto-claim → /auto-experiment → /auto-verify, which routinely exceeds 24h on GPU experiments). Ageing it out would discard the just-produced upstream results and reset the inherited budget counters, orphaning live work; the long elapsed time is expected and the upstream calls produced fresh results, so the drift rationale does not apply. For any other status, if now - timestamp > 24h (86400 seconds), the prior state is too old to safely resume — log [resume] state older than 24h (timestamp=<ts>, age=<H>h) — treating as stale, ignoring REVIEW_STATE.json and starting fresh, archive the stale file as REVIEW_STATE.<ts>.stale.json (so the audit trail survives — don't delete it), and proceed as if RESUME=false. Rationale: beyond a day the codebase, results files, and reviewer model versions have likely drifted enough that "picking up from iteration N" would conflate stale context with fresh work — a concern that applies to an interrupted in_progress run, not to a deliberate awaiting_upstream handoff. If the timestamp field is absent (legacy state files), skip the staleness check and continue to the branches below — the caller accepts the risk.
   • If status == "completed" AND last_verdict matches a POSITIVE_VERDICT_TERMS entry AND last_score >= TARGET_SCORE → log [resume] prior run already satisfied target (score=<last_score>, verdict=<last_verdict>) — returning without invoking reviewer and return immediately with the existing state. (Termination should already have produced FINAL_REPORT_DOC in this case; if missing, re-run Termination's report-assembly step before returning.)
   • Else if status == "completed" AND iterations_consumed >= MAX_ITERATIONS → log [resume] prior run exhausted MAX_ITERATIONS (iterations_consumed=<n>) — returning without invoking reviewer and return.
   • Else if status == "completed" AND claim_reentries_consumed >= MAX_CLAIM_REENTRIES AND the only remaining unresolved claims would require action type ③ → log [resume] prior run exhausted MAX_CLAIM_REENTRIES — returning without invoking reviewer and return.
   • Else if status == "awaiting_upstream" → the orchestrator should have run pending_upstream_calls before resuming. Verify they completed (all expected output artifacts present on disk per the upstream skills' own contracts). If they did, clear pending_upstream_calls, flip status to "in_progress", and continue into Phase A of the next iteration with iterations_consumed and claim_reentries_consumed already incremented (Phase C of the prior iteration incremented them when it queued the upstream calls). If they did not, abort with [resume] awaiting_upstream but pending_upstream_calls outputs missing — orchestrator did not complete upstream work; aborting. Do not silently retry — the orchestrator failure must be visible.
   • Else if status == "in_progress" AND iterations_consumed >= MAX_ITERATIONS → no iterations left; flip status to "completed", log [resume] in-progress at MAX_ITERATIONS — closing out without new iteration, run Termination, and return.
   • Otherwise (status == "in_progress" AND iterations_consumed < MAX_ITERATIONS) → the next iteration's display number is iterations_consumed + 1 (this is a working value for logs / Phase E heading only — it is never persisted; the only persistent counter is iterations_consumed, which Phase C increments at its end after a real ①/②/③ action). Open AUTO_REVIEW.md in append mode and continue. Log [resume] picking up from iteration <iterations_consumed+1>/<MAX_ITERATIONS> (claim-reentry sub-budget: <claim_reentries_consumed>/<MAX_CLAIM_REENTRIES>).
3. Read project context and prior reviews
4. Initialize iteration counter (skip if step 2 already set it)

Loop (up to MAX_ITERATIONS)

Each cycle below is one Phase A→E iteration. The iteration counter (iterations_consumed) is incremented at the end of Phase C, only if Phase C actually queued or executed a back-edge action of type ① / ② / ③. A pure-review iteration whose only action is type ⓪ (narrative-only) or has no action at all does not consume budget. Stall guard against ⓪-only spin. Because ⓪ is free (it does not consume budget and runs no experiment), a reviewer that keeps proposing narrative-only fixes for a still-unresolved weakness would loop forever — burning a reviewer call each round while neither the STOP rule (score below target) nor the budget exit (iterations_consumed frozen) ever fires. The consecutive_noop_count field in REVIEW_STATE.json closes this: at Phase E, increment it when this iteration was ⓪-only (or had no action) and last_score + last_verdict are unchanged from the prior persisted state; reset it to 0 whenever a real ①/②/③ action ran or the score/verdict changed. When it reaches 2 (two consecutive non-converging no-op iterations), the loop is stalled on the reviewer side — terminate it (see Phase B's STOP conditions). The common legitimate case for a ⓪-only iteration is the final one that confirms a positive verdict after a real ①/②/③ in the prior iteration — that single ⓪ is fine; only an unchanged-score repeat trips the guard.

Phase A: Review

Iteration 1 uses the prompt shape below (no Reviewer Memory yet — file is created at the end of iteration 1 by Phase B.5). The budget line at the top of the prompt reads 0/<MAX> used for both budgets on iteration 1; that is correct and expected, not an error. Iteration 2+ prepends the contents of REVIEWER_MEMORY_DOC plus a "Previous Review Summary" block before the body — see "Prompt Template for Iteration 2+" near the end of this file for the full shape. The memory block is mandatory from iteration 2 onward and is what gives this loop adversarial continuity across iterations.

The reviewer prompt must include the five state buckets explicitly per the contract in "Build five disjoint claim buckets" above. The reviewer is told which routing options are available per bucket and which are forbidden (e.g., no claim rewrite for INCONCLUSIVE; no action for INTEGRITY_ONLY).

If MCP available:

mcp__llm-chat__chat:
  system: "You are a senior ML reviewer (NeurIPS/ICML level)."
  prompt: |
    [Iteration N/MAX_ITERATIONS of autonomous review loop]
    [Iteration budget: <iterations_consumed>/<MAX_ITERATIONS> used; claim-reentry sub-budget: <claim_reentries_consumed>/<MAX_CLAIM_REENTRIES> used]

    [Full research context: claims, methods, results, known weaknesses]
    [Changes since last iteration, if any]

    ## Per-claim verify state (from /auto-verify)
    verify_passed:                  [<ids or "none">]   # brief consistency check only
    verify_failed:                  [<ids or "none">]   # two-phase: variant-integrity check, then optional claim rewrite
    verify_inconclusive:            [<ids or "none">]   # main-experiment-only fix; you may NOT rewrite the claim
    verify_zero_eligible_variants:  [<ids or "none">]   # variant-only fix; do NOT touch the main experiment
    verify_integrity_only:          [<ids or "none">]   # NO back-edge action; record in Open Items with stage2_skip_reason-dispatched upgrade suggestion

    Tasks:
    1. Score this work 1-10 for a top venue.
    2. For each FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS claim, name the MINIMUM
       fix and tag it with the routing type (①/②/③). If you want to use ③ (claim rewrite)
       on a FAIL claim, also provide the proposed new claim text — verify that the
       claim-reentry sub-budget has remaining capacity before recommending ③.
       Do NOT propose fixes or ①/②/③ actions for INTEGRITY_ONLY claims — those are
       swap-test-skipped outputs (audit passed) with no robustness data to fix; iteration
       records them as unfinished stress-tests, not as findings that need repair.
    3. For each PASS claim, do a brief narrative + numeric consistency check.
       Flag any caveat (main-experiment-WARN) that should surface in the final paper.
    4. State clearly: is this READY for submission? Yes/No/Almost. READY requires all
       FAIL / INCONCLUSIVE / ZERO_ELIGIBLE_VARIANTS claims to be resolved.
       INTEGRITY_ONLY claims do NOT block READY (they are outside the stress-test
       scope by policy or cap) — but note their presence as a caveat in the assessment.
    5. (Iteration 1 only) Record any initial suspicions or concerns you want to
       track in future iterations — output as a `## Memory update` section so
       Phase B.5 can seed `REVIEWER_MEMORY.md` with them.

    Be brutally honest. If the work is ready, say so clearly.

Iteration 1 — save thread_id. When the MCP call returns, capture the response's thread_id (or equivalent conversation handle the MCP server uses) and persist it to REVIEW_STATE.json's thread_id field. This is what iteration 2+ uses to continue the same reviewer thread instead of pasting summaries. If the MCP server does not return a thread handle (some configurations don't), or the call used the curl fallback, write null and let iteration 2+ fall back to the paste-summary path.

Iteration 2+ — pass thread_id back when present. If REVIEW_STATE.json has a non-null thread_id, pass it as the thread_id arg on the mcp__llm-chat__chat call. The iteration-2+ prompt body becomes shorter (no need to re-paste prior reviews — the thread already has them), but the Reviewer Memory prepend block is still mandatory (it carries the structured suspicion account that the thread itself doesn't make queryable). On any error indicating the thread is unknown / expired (server returns 4xx with a thread-related message), clear thread_id to null in state and retry once without it — falling back to the paste-summary template at the end of this file.

If MCP NOT available:

curl -s "${LLM_BASE_URL}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${LLM_API_KEY}" \
  -d '{
    "model": "${LLM_MODEL}",
    "messages": [
      {"role": "system", "content": "You are a senior ML reviewer (NeurIPS/ICML level)."},
      {"role": "user", "content": "[Full review prompt with all four buckets explicit]"}
    ],
    "max_tokens": 4096
  }'

Phase B: Parse Assessment

CRITICAL: Save the FULL raw response verbatim. Then ex

…(truncated)