Auto Review Loop

Auto Review Loop: Autonomous Research Improvement

🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It already loops internally (review → fix → re-review) and the reviewer carries round-to-round memory in one threadId (codex-reply). An external timer re-enters from the top each tick — fresh threadId, reviewer memory reset — firing the verdict on wall-clock time instead of on artifact change: zero new signal, full token cost. If you want to schedule something, schedule the external wait that precedes it (experiments done → then run this once). See shared-references/external-cadence.md.

Autonomously iterate: review → implement fixes → re-review, until an independent reviewer gives a policy-approved positive assessment or MAX_ROUNDS is reached.

Context: $ARGUMENTS

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10 AND verdict ∈ {"ready", "almost"} — both must hold. This matches the operative Phase-E STOP CONDITION exactly; the verdict vocabulary is {"ready", "almost", "not ready"} (a high score with a "not ready" verdict does NOT stop the loop). Earlier wording here used or and a stale verdict set ("accept"/"sufficient"/"ready for submission") — that was an internal inconsistency; the AND form is authoritative.
• REVIEW_DOC: review-stage/AUTO_REVIEW.md (cumulative log) (fall back to ./AUTO_REVIEW.md for legacy projects)
• REVIEWER_MODEL = gpt-6-astra — Default model for the Codex backend. Must be an OpenAI model (e.g., gpt-6-astra, o3, gpt-4o). Manual backend uses a model the user chooses — it must be a recognized model from a different family (OpenAI, Anthropic, Google, DeepSeek, Moonshot/Kimi, Qwen).
• REVIEWER_BACKEND — With no reviewer directive, start as auto; Step -1 runs exactly one two-call native marker/challenge probe for the first review. A bound Copilot CLI root session uses copilot-native (built-in complementary rubber-duck subagent); an unbound/non-Copilot host keeps the existing codex default. Explicit — reviewer: codex, oracle-pro, agy, or manual bypasses the probe and selects that external backend. Explicit — reviewer: copilot retains the compatibility copilot --agent drive mode and its later Codex/manual finalizer. The native path gets both actual model IDs from host session events; it never needs COPILOT_CLI or caller-provided --executor-model. See shared-references/reviewer-routing.md.
• OUTPUT_DIR = review-stage/ — All review-stage outputs go here. Create the directory if it doesn't exist.
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.
• COMPACT = false — When true, (1) read EXPERIMENT_LOG.md and findings.md instead of parsing full logs on session recovery, (2) append key findings to findings.md after each round.
• REVIEWER_DIFFICULTY = medium — Controls how adversarial the reviewer is. Three levels:
  • medium (default): Current behavior — MCP-based review, the executor controls what context the reviewer sees.
  • hard: Adds Reviewer Memory (the reviewer tracks its own suspicions across rounds) + Debate Protocol (the executor can rebut, the reviewer rules).
  • nightmare: Everything in hard + Codex exec reviewer reads the repo directly via codex exec (the executor cannot filter what the reviewer sees) + Adversarial Verification (the reviewer independently checks if code matches claims).
• RENDER_HTML = true — When true (default), auto-render review-stage/AUTO_REVIEW.md to HTML on loop termination via /render-html. Uses --no-review (the loop itself IS the cross-model review; the HTML is a structural conversion). Set false to skip, or pass — render html: false.

⚠️ Nightmare + Manual incompatibility: If REVIEWER_BACKEND = manual and REVIEWER_DIFFICULTY = nightmare, STOP with: "difficulty: nightmare requires Codex CLI / codex exec and is not compatible with --reviewer: manual. Use difficulty: hard, or switch reviewer to codex."

💡 Override: /auto-review-loop "topic" — compact: true, human checkpoint: true, difficulty: hard

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

If no --reviewer: directive was supplied: Set REVIEWER_BACKEND to auto. At Step -1 of the first round, resolve copilot_native_evidence.py using the canonical four-layer helper chain. Generate a fresh binding <run_id>_r<round>_review_<8-random-hex> and invoke marker, wait, then invoke challenge as two distinct root Bash calls. Put the literal binding and concrete resolved helper path in both calls; Copilot Bash calls do not share variables. If the challenge binds, set REVIEWER_BACKEND to copilot-native and use that same challenge for the first review. Do not issue a second activation challenge in Phase A. If it exits 3 because no current Copilot root session is bound, use codex. Explicit reviewer directives bypass this probe. If the helper is missing, native acceptance is unavailable; use Codex only if that external backend is positively available, otherwise emit REVIEW_UNAVAILABLE.

If REVIEWER_BACKEND = copilot-native: Read the challenge nonce and host-reported executor model. Invoke the host's native task tool with agent_type: rubber-duck; do not start a subprocess and do not specify a reviewer model. The prompt contains the exact standalone ARIS_REVIEW_NONCE=<nonce> line, artifact/diff paths, the output contract, and (round 2+) review-stage/REVIEWER_MEMORY.md. It contains no executor summary or fix narrative. After the task completes, invoke copilot_native_evidence.py verify to create the evidence and raw-response artifacts. The verifier must observe one successful linked rubber-duck lifecycle and known, different host-reported model families.

Pass the evidence to both review_gate.py --native-evidence and save_trace.sh --backend copilot-native --native-evidence. A qualifying native positive may stop directly; no external finalizer is needed. A native negative continues with a fresh marker/challenge/subagent next round. Every verdict-bearing native call—including a hard-mode rebuttal ruling—gets one unique <run_id, round, purpose> artifact set and exactly one challenge. Missing, same/unknown-family, malformed, stale, or mismatched evidence is never a verdict. If native complementary dispatch is unavailable, fall back only to a positively available opposite-family backend: Anthropic/Google executor → Codex; OpenAI executor → manual with a reported non-OpenAI model. Otherwise emit REVIEW_UNAVAILABLE. Full protocol: shared-references/reviewer-routing.md.

If REVIEWER_BACKEND = copilot: Require --executor-model: if not provided → emit REVIEW_UNAVAILABLE. Determine executor family from --executor-model (see reviewer-routing.md). Router picks opposite-family profile:

• executor_family=openai → profile="aris-reviewer-claude" (anthropic)
• executor_family=anthropic → profile="aris-reviewer-openai" (openai)
• executor_family=google → profile="aris-reviewer-openai" (openai, default cross)
• executor_family=unknown → REVIEW_UNAVAILABLE (fail closed). Verify the profile file exists at .github/agents/<profile>.agent.md. If missing → REVIEW_UNAVAILABLE. Read its model: field into REVIEWER_MODEL, derive reviewer_family from that model string, and verify it differs from executor_family. Pass the same value through subprocess --model; never trust a caller-supplied family label or profile-only pinning under an Auto session. Identity assurance: --executor-model is caller-declared routing input, not runtime attestation. Record executor_model_source: caller-declared, the derived family_relation, and independence_verified: unverified. A pair of different model strings must never be promoted to independently verified. Capability gate: copilot --help must advertise --model, --effort, and --allow-tool; otherwise emit REVIEW_UNAVAILABLE. Use the copilot --agent subprocess (documented Copilot CLI form) with the selected profile, --model "$REVIEWER_MODEL", --effort xhigh, and --allow-tool=read for each review call. Multi-round: each round is a fresh copilot --agent call with the same profile; reviewer memory is carried via review-stage/REVIEWER_MEMORY.md artifact. If copilot CLI is unavailable → REVIEW_UNAVAILABLE for that drive round; do not silently substitute another transport. A later positive Copilot verdict still requires the separately documented Codex/manual finalizer. See shared-references/reviewer-routing.md for the full copilot contract.

If REVIEWER_BACKEND = codex: Use mcp__codex__codex for new review threads. Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).

If REVIEWER_BACKEND = manual: Use mcp__manual_review__review for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true} Save the returned threadId. Use mcp__manual_review__review_reply for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"model_reasoning_effort": "xhigh", "executor_model": "", "require_reviewer_model": true} A verdict-bearing manual response MUST begin with Reviewer-Model: <exact-model-id>. Derive reviewer_family from that model identity. Missing, unknown, or same-family identity cannot acquit; for a mandatory escalation, emit REVIEW_UNAVAILABLE rather than guessing.

Prompt fidelity: the manual review task must be exactly the same text that Codex would receive; the transport may add only the required Reviewer-Model: response-format instruction. Review tracing applies to every backend. Native traces are populated from the revalidated host-event artifact rather than caller model declarations.

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to review-stage/REVIEW_STATE.json after each round:

{
  "run_id": "run_20260713_a1b2c3d4",
  "round": 2,
  "threadId": null,
  "reviewer_profile": "rubber-duck",
  "reviewer_backend": "copilot-native",
  "executor_model": "claude-sonnet-4.6",
  "executor_model_source": "host-session-event",
  "executor_family": "anthropic",
  "requested_reviewer_model": null,
  "reported_reviewer_model": "gpt-5.5",
  "reviewer_model_source": "host-session-event",
  "reviewer_family": "openai",
  "family_relation": "different",
  "identity_assurance": "host_event_verified",
  "independence_verified": true,
  "native_evidence_id": "cne_0123456789abcdef0123456789abcdef",
  "native_evidence_path": "review-stage/COPILOT_NATIVE_run_20260713_a1b2c3d4_ROUND_2_REVIEW.evidence.json",
  "requires_external_acquittal": false,
  "status": "in_progress",
  "difficulty": "medium",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

• run_id — Globally unique per invocation. Generated on fresh start as run_<YYYYMMDD>_<8-char-hex> (e.g., run_20260713_a1b2c3d4). Preserved across round writes. On resume, read from state file unchanged. This binds all round state, reviewer-memory appends, and acquittal receipts to one run so a stale completed state from a previous invocation cannot leak into the current run's acquittal check.

When REVIEWER_BACKEND = copilot-native, save the evidence ID/path and the host-event executor/reviewer models, derived families, and sources. Each round is a fresh rubber-duck subagent and therefore gets a fresh evidence artifact; there is no persistent child handle. When REVIEWER_BACKEND = compatibility copilot, retain reviewer_profile, requested model, caller-declared executor model, independence_verified: "unverified", and the external-finalizer obligation. For codex save its MCP threadId; for manual save threadId and the reported reviewer identity. On resume, use reviewer_backend to select the continuation mechanism and preserve requires_external_acquittal.

Write this file at the end of every Phase E (after documenting the round). Overwrite each time — only the latest round's state matters. The run_id field MUST persist unchanged across overwrites within the same run.

On completion (positive assessment or max rounds), set "status": "completed" so future invocations don't accidentally resume a finished loop.

Append-Only External-Finalizer Receipt

Whenever a Copilot path hands the verdict to an external backend—after a positive compatibility-drive review or after a pre-verdict native dispatch failure—maintain an append-only finalizer log at review-stage/ACQUITTAL_LOG.jsonl. Each line records the Codex/manual reviewer that completed that run. A successful native rubber-duck round never needs or writes this receipt; its evidence sidecar is the acceptance record. The historical filename is retained for compatibility:

{"run_id":"run_20260713_a1b2c3d4","round":3,"backend":"codex","effort":"xhigh","verdict":"ready","score":7.5,"executor_model":"claude-sonnet-4-5","executor_model_source":"caller-declared","executor_family":"anthropic","reviewer_model":"gpt-6-astra","reviewer_model_source":"requested","reviewer_family":"openai","family_relation":"different","identity_assurance":"caller_declared","independence_verified":"unverified","trace_id":"auto-review-loop/2026-07-13_run03","timestamp":"2026-07-13T14:22:00Z"}

Rules (non-negotiable):

Why this exists: REVIEW_STATE.json is overwritten each round. The log preserves evidence that a compatibility drive verdict or failed native attempt did not terminate by itself. A successful copilot-native verdict instead uses its host-event evidence sidecar.

Output Protocols

Follow these shared protocols for all output files:

• Output Versioning Protocol — write timestamped file first, then copy to fixed name
• Output Manifest Protocol — log every output to MANIFEST.md
• Output Language Protocol — respect the project's language setting

Workflow

Initialization

1. Check for review-stage/REVIEW_STATE.json (fall back to ./REVIEW_STATE.json if not found — legacy path):
   • If neither path exists: fresh start (normal case, identical to behavior before this feature existed)
     • Generate run_id: run_<YYYYMMDD>_<8-char-hex> (e.g., run_20260713_a1b2c3d4). Use date +%Y%m%d and 8 random hex characters. This run_id persists across all round writes and binds acquittal receipts to this invocation.
   • If it exists AND status is "completed": fresh start (previous loop finished normally — but its ACQUITTAL_LOG.jsonl entries are retained as an audit trail with their own run_id, and are NOT valid for the current run's stop gate)
     • Generate a new run_id for this invocation.
   • If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
     • Generate a new run_id for this invocation.
   • If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
     • Read the state file to recover run_id, round, threadId (or evidence/profile fields for Copilot backends), reviewer_backend, last_score, pending_experiments
     • Legacy backward compat: if reviewer_backend is absent from the state file, default to codex (pre-copilot-era states did not record this field). If requires_external_acquittal is absent, default it to false; a legacy default-Codex run must not inherit the stricter Copilot-finalizer state. If run_id is absent from the state file (pre-run_id era), generate a new run_id and log: "No run_id in legacy state file; assigned run_<...> for this resume."
     • Read review-stage/AUTO_REVIEW.md to restore full context of prior rounds (fall back to ./AUTO_REVIEW.md)
     • If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
     • Resume from the next round (round = saved round + 1)
     • Use reviewer_backend to determine continuation: codex-reply for codex; a fresh marker/challenge/rubber-duck/evidence cycle for copilot-native; a fresh copilot --agent subprocess with the saved profile/model for compatibility copilot; manual_review_reply for manual
     • Log: "Recovered from context compaction. Resuming at Round N."
2. Read project narrative documents, memory files, and any prior review documents. When COMPACT = true and compact files exist: read findings.md + EXPERIMENT_LOG.md instead of full review-stage/AUTO_REVIEW.md and raw logs — saves context window.
3. Read recent experiment results (check output directories, logs)
4. Identify current weaknesses and open TODOs from prior reviews
5. Initialize round counter = 1 (unless recovered from state file)
6. Create/update review-stage/AUTO_REVIEW.md with header and timestamp
7. If this is a fresh run with no explicit reviewer directive, initialize REVIEWER_BACKEND to auto. Step -1 of Round 1 performs activation and uses that same challenge for the review. Explicit reviewer directives initialize their selected backend and bypass activation. Do not use environment heuristics.

Loop (repeat up to MAX_ROUNDS)

Step -1 — Resolve the automatic backend and prepare one native challenge:

• If REVIEWER_BACKEND is auto, resolve the native helper and run one root marker call followed by one root challenge call. Use binding <run_id>_r<round>_review_<8-random-hex> and output review-stage/COPILOT_NATIVE_<run_id>_ROUND_<round>_REVIEW.challenge.json. A bound challenge sets REVIEWER_BACKEND to copilot-native and NATIVE_CHALLENGE to that path. Exit 3/unbound sets REVIEWER_BACKEND to codex. Any other failure follows the fail-closed capability rules.
• If REVIEWER_BACKEND is already copilot-native (a later round or a resumed run), create one fresh marker/challenge pair with the same run-scoped naming pattern and set NATIVE_CHALLENGE. An unbound or invalid challenge cannot be treated as a verdict or silently relabeled.
• Explicit external or compatibility backends do nothing in this step.

The challenge created here is the challenge consumed by Phase A. Do not run another marker/challenge for the same review. Run-scoped filenames are append-only audit identities; never pass --replace to reuse evidence from an older invocation.

Step 0 — Snapshot current-round state: After Step -1 resolves auto, set round_backend = <current REVIEWER_BACKEND> and round_requires_external_acquittal = <current requires_external_acquittal, default false>. These variables label the backend and obligation that actually governed the CURRENT round. If compatibility-drive escalation occurs later in Phase B.5.1 (copilot → codex/manual), the snapshots retain their pre-escalation values while the forward-looking state is updated for the NEXT round. A native dispatch failure is different because no review occurred: replace both snapshots with the external fallback values before that reviewer call, as specified in Phase A. A successful native call never sets the finalizer obligation. Phase E uses only the resulting snapshots when documenting or writing a finalizer receipt.

Phase A: Review

Route by REVIEWER_BACKEND and REVIEWER_DIFFICULTY.

If REVIEWER_BACKEND = copilot-native, execute one fresh native cycle:

1. Use NATIVE_CHALLENGE prepared by Step -1. It must be the run-scoped ..._ROUND_<round>_REVIEW.challenge.json artifact created in this round. Do not issue a second marker/challenge here.
2. Read the returned nonce. Call the host Task tool with agent_type: rubber-duck, a fresh name, and a prompt whose first line is exactly ARIS_REVIEW_NONCE=<nonce>. Supply paths to claims, methods/code, raw results, diff/current inputs, and reviewer memory—not an executor summary. Require exactly one Score: X/10 and Verdict: ready | almost | not ready plus ranked weaknesses/minimum fixes/memory update. Do not pass a model override: Copilot's complementary strategy selects it.
3. After Task completes, run python3 "<resolved-helper>" verify --challenge "$NATIVE_CHALLENGE" --output "review-stage/COPILOT_NATIVE_<run_id>_ROUND_<round>_REVIEW.evidence.json" --response-output "review-stage/COPILOT_NATIVE_<run_id>_ROUND_<round>_REVIEW.response.md" in a new root Bash call. Use only the extracted response artifact for Phase B.
4. Exit 10 (same/unknown family), incomplete lifecycle, invalid response, or unavailable complementary model is not a review. Apply the opposite-family fallback table in reviewer-routing.md; if none is positively available, emit REVIEW_UNAVAILABLE. Never emulate rubber-duck using a slash prompt, copilot --agent rubber-duck, or generic subagent. Trace a pre-evidence dispatch failure as --backend copilot-native --status error without evidence, then trace the actual fallback separately; this error trace has no authority at the stop gate. Before fallback, run copilot_native_evidence.py validate-challenge --challenge "$NATIVE_CHALLENGE" and take EXECUTOR_MODEL only from that output. Then:
   • Anthropic/Google executor + usable Codex → set both REVIEWER_BACKEND and round_backend to codex before the external call.
   • OpenAI executor + manual reviewer reporting a known non-OpenAI model → set both values to manual before the external call.
   • Set round_requires_external_acquittal=true for either fallback, clear NATIVE_EVIDENCE, and pass the validated executor model plus the fallback's resolved reviewer model to review_gate.py. This deliberately uses the stricter external-finalizer branch, which re-derives and enforces different families. If the external call does not return a usable review, emit REVIEW_UNAVAILABLE.

If REVIEWER_BACKEND = copilot, enforce opposite-family routing from the declared executor identity FIRST:

• Require --executor-model <model> parameter. If missing → REVIEW_UNAVAILABLE. Stop.
• Derive executor_family from executor_model:
  • Model names containing gpt, o1, o3, o4, chatgpt → openai
  • Model names containing claude, sonnet, opus, haiku → anthropic
  • Model names containing gemini → google
  • Anything else → unknown
• If executor_family is unknown → REVIEW_UNAVAILABLE (fail closed). Stop.
• Treat this as route selection, not attestation: persist executor_model_source: caller-declared; a derived family_relation: different remains independence_verified: unverified unless a future stable runtime signal independently proves the parent executor model.
• Router picks opposite-family profile:
  • openai → "aris-reviewer-claude" (anthropic, forced cross-family)
  • anthropic → "aris-reviewer-openai" (openai, forced cross-family)
  • google → "aris-reviewer-openai" (openai default)
• Verify the profile file exists at .github/agents/<profile>.agent.md. If missing → REVIEW_UNAVAILABLE. Stop.
• Read the profile's first frontmatter model: value, derive its family, and verify it is known and differs from executor_family. If not, fail closed.
• Verify copilot --help exposes --model, --effort, and --allow-tool. Older/unpinned CLIs are REVIEW_UNAVAILABLE.
• Adapt the Codex MCP calls below to use the copilot --agent subprocess (documented Copilot CLI form):
  • Replace mcp__codex__codex with copilot --agent "<profile>" --model "<parsed-model>" --effort xhigh --allow-tool=read --prompt "..."
  • Each round is a fresh copilot --agent call with the same profile + review-stage/REVIEWER_MEMORY.md artifact carrying round-to-round state.
  • The prompt text and Review Tracing are identical to the Codex path.
  • If copilot CLI is unavailable → REVIEW_UNAVAILABLE (no MCP fallback).
  • If REVIEWER_DIFFICULTY = nightmare, skip Copilot (nightmare requires Codex exec): emit REVIEW_UNAVAILABLE. See shared-references/reviewer-routing.md.

If REVIEWER_BACKEND ∈ {codex, manual}: use the backend-specific MCP call per the Reviewer Calling Convention above. The prompt text is the same regardless of backend.

Medium (default) — MCP Review

Send comprehensive context to the independent reviewer using the selected backend.

For codex backend:

mcp__codex__codex:
  model: gpt-6-astra
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]

    Review the work directly from its artifacts — executor notes are not
    evidence, so read the files yourself rather than trusting my framing:
    - Claims / paper draft: <path>
    - Methods / code under review: <path(s)>
    - Raw results (verbatim files, not a summary): <path(s)>
    - Changed since last round: <changed-file paths> — read the diff, not my description

    Please act as a senior ML reviewer (NeurIPS/ICML level). Start from the
    assumption that the work is broken somewhere — your job is to find where.
    Be adversarial. Trust nothing the author tells you — verify everything
    yourself.

    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix (experiment, analysis, or reframing)
    4. State clearly: is this READY for submission? Yes/No/Almost

    Be brutally honest. If, after genuinely trying to break it, the work holds
    up and is ready, say so clearly.

    === SCOPE LIMITS (these bound what you PROPOSE, never what you look for) ===
    Report anything that is actually wrong here — including a rare-looking case, if
    this repo actually produces it. Then keep the fix in scope:
    1. This is a RESEARCH-WORKFLOW tool, not a security paper. Verification is
       welcome; over-defense is not. Assume a cooperating operator on their own
       machine — a malicious local user is NOT in the threat model.
    2. Do NOT propose SHA / hash / content-fingerprint / digest-binding schemes.
       Reporting a real defect in hashing code that already exists is fine.
    3. NO speculative machinery: do not add feature flags, migration frameworks,
       compat layers, wrappers, pins, or similar mechanisms unless evidence shows
       a current repo defect they fix or an explicit existing invariant they must
       preserve. "Load-bearing", "compatibility", and "not scaffolding" are labels,
       not evidence. Point to the failing path/artifact or invariant, and check the
       proposal's factual premises, such as whether a named package version exists.
    4. NO corner-case obsession: exotic encodings, symlink races, RTL text and
       millisecond races are out of scope unless you can show the case arises here.
    5. Where a rubric or checklist is genuinely needed, do not over-mechanize
       judgement. A clear sentence a human reads beats a scored table nobody
       maintains.
    Exception: code that runs remote commands, starts a network service, or installs
    an MCP server runs on the user's machine with their credentials — trust-boundary
    findings there are in scope and the default is strict.
    Say plainly when something is correct. Do not manufacture findings.

For manual backend: use mcp__manual_review__review with the prompt text above and config: {"model_reasoning_effort": "xhigh", "executor_model": "<actual executor model>", "require_reviewer_model": true}. Save the returned threadId.

If this is round 2+, use mcp__codex__codex-reply (codex) or mcp__manual_review__review_reply (manual) with the saved threadId.

Hard — MCP Review + Reviewer Memory

Same as medium, but prepend Reviewer Memory to the prompt. Use the selected backend.

For codex backend:

mcp__codex__codex:
  model: gpt-6-astra
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]

    ## Your Reviewer Memory (persistent across rounds)
    [Paste full contents of review-stage/REVIEWER_MEMORY.md here]

    IMPORTANT: You have memory from prior rounds. Check whether your
    previous suspicions were genuinely addressed or merely sidestepped.
    The author (the executor model) controls what context you see — be skeptical
    of convenient omissions.

    Review directly from the artifacts (paths below) — read the files yourself:
    - Claims / methods / code: <path(s)>
    - Raw results: <path(s)>
    - Changed since last round: <changed-file paths> (read the raw diff)

    Please act as a senior ML reviewer (NeurIPS/ICML level).
    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix
    4. State clearly: is this READY for submission? Yes/No/Almost
    5. **Memory update**: List any new suspicions, unresolved concerns,
       or patterns you want to track in future rounds.

    Be brutally honest. Actively look for things the author might be hiding.

    === SCOPE LIMITS (these bound what you PROPOSE, never what you look for) ===
    Report anything that is actually wrong here — including a rare-looking case, if
    this repo actually produces it. Then keep the fix in scope:
    1. This is a RESEARCH-WORKFLOW tool, not a security paper. Verification is
       welcome; over-defense is not. Assume a cooperating operator on their own
       machine — a malicious local user is NOT in the threat model.
    2. Do NOT propose SHA / hash / content-fingerprint / digest-binding schemes.
       Reporting a real defect in hashing code that already exists is fine.
    3. NO speculative machinery: do not add feature flags, migration frameworks,
       compat layers, wrappers, pins, or similar mechanisms unless evidence shows
       a current repo defect they fix or an explicit existing invariant they must
       preserve. "Load-bearing", "compatibility", and "not scaffolding" are labels,
       not evidence. Point to the failing path/artifact or invariant, and check the
       proposal's factual premises, such as whether a named package version exists.
    4. NO corner-case obsession: exotic encodings, symlink races, RTL text and
       millisecond races are out of scope unless you can show the case arises here.
    5. Where a rubric or checklist is genuinely needed, do not over-mechanize
       judgement. A clear sentence a human reads beats a scored table nobody
       maintains.
    Exception: code that runs remote commands, starts a network service, or installs
    an MCP server runs on the user's machine with their credentials — trust-boundary
    findings there are in scope and the default is strict.
    Say plainly when something is correct. Do not manufacture findings.

Nightmare — Codex Exec (GPT reads repo directly)

Do NOT use MCP. Instead, let GPT access the repo autonomously via codex exec:

codex exec "$(cat <<'PROMPT'
You are an adversarial senior ML reviewer (NeurIPS/ICML level).
This is Round N/MAX_ROUNDS of an autonomous review loop.

## Your Reviewer Memory (persistent across rounds)
[Paste full contents of review-stage/REVIEWER_MEMORY.md]

## Instructions
You have FULL READ ACCESS to this repository. The author (the executor model) does NOT
control what you see — explore freely. Your job is to find problems the
author might hide or downplay.

DO THE FOLLOWING:
1. Read the experiment code, results files (JSON/CSV), and logs YOURSELF
2. Verify that reported numbers match what's actually in the output files
3. Check if evaluation metrics are computed correctly (ground truth, not model output)
4. Look for cherry-picked results, missing ablations, or suspicious hyperparameter choices
5. Read NARRATIVE_REPORT.md or review-stage/AUTO_REVIEW.md for the author's claims — then verify each against code

OUTPUT FORMAT:
- Score: X/10
- Verdict: ready / almost / not ready
- Verified claims: [which claims you independently confirmed]
- Unverified/false claims: [which claims don't match the code or results]
- Weaknesses (ranked): [with MINIMUM fix for each]
- Memory update: [new suspicions and patterns to track next round]

Be adversarial. Trust nothing the author tells you — verify everything yourself.

=== SCOPE LIMITS (these bound what you PROPOSE, never what you look for) ===
Report anything that is actually wrong here — including a rare-looking case, if
this repo actually produces it. Then keep the fix in scope:
1. This is a RESEARCH-WORKFLOW tool, not a security paper. Verification is
   welcome; over-defense is not. Assume a cooperating operator on their own
   machine — a malicious local user is NOT in the threat model.
2. Do NOT propose SHA / hash / content-fingerprint / digest-binding schemes.
   Reporting a real defect in hashing code that already exists is fine.
3. NO speculative machinery: do not add feature flags, migration frameworks,
   compat layers, wrappers, pins, or similar mechanisms unless evidence shows
   a current repo defect they fix or an explicit existing invariant they must
   preserve. "Load-bearing", "compatibility", and "not scaffolding" are labels,
   not evidence. Point to the failing path/artifact or invariant, and check the
   proposal's factual premises, such as whether a named package version exists.
4. NO corner-case obsession: exotic encodings, symlink races, RTL text and
   millisecond races are out of scope unless you can show the case arises here.
5. Where a rubric or checklist is genuinely needed, do not over-mechanize
   judgement. A clear sentence a human reads beats a scored table nobody
   maintains.
Exception: code that runs remote commands, starts a network service, or installs
an MCP server runs on the user's machine with their credentials — trust-boundary
findings there are in scope and the default is strict.
Say plainly when something is correct. Do not manufacture findings.
PROMPT
)" --skip-git-repo-check 2>&1

Key difference: In nightmare mode, GPT independently reads code, result files, and logs. Claude cannot filter or curate what GPT sees. This is the closest analog to a real hostile reviewer who reads your actual paper + supplementary materials.

Phase B: Parse Assessment

CRITICAL: Save the FULL raw response from the reviewer verbatim (store in a variable for Phase E). For copilot-native, this must be the response artifact extracted by the evidence helper, never text copied by the executor. Do NOT discard or summarize — the raw text is the primary record.

Then extract structured fields:

• Score (numeric 1-10)
• Verdict ("ready" / "almost" / "not ready")
• Action items (ranked list of fixes)

Phase B.5: Reviewer Memory Update

After parsing the assessment, append to the canonical memory artifact at review-stage/REVIEWER_MEMORY.md. Both Copilot backends depend on this file for round-to-round continuity (each native subagent or compatibility subprocess is fresh), so the update runs regardless of REVIEWER_DIFFICULTY. No project-root fallback is permitted; create review-stage/ before the first append:

# Reviewer Memory

## Round 1 — Score: X/10

### Raw Reviewer Response (verbatim)
[Paste the COMPLETE raw reviewer response here — never summarized or curated by the executor.]

### Memory Update
[Reviewer's own memory update section, if provided — verbatim.]
- **Suspicion**: [what the reviewer flagged]
- **Unresolved**: [concerns not yet addressed]
- **Patterns**: [recurring issues the reviewer noticed]

---

## Round 2 — Score: X/10

### Raw Reviewer Response (verbatim)
[Paste the COMPLETE raw reviewer response here.]

### Memory Update
- **Previous suspicions addressed?**: [yes/no for each, with reviewer's judgment]
- **New suspicions**: [...]
- **Unresolved**: [carried forward + new]

---

Rules:

• Append-only — never delete, never truncate. The file is a reviewer-owned audit trail. The executor must never summarize, curate, or edit prior rounds' content. Append the reviewer's full raw response for this round verbatim, then append a memory update section if the reviewer provided one.
• Each round's append must be the reviewer's own words — if the reviewer's response includes a "Memory update" section, copy it verbatim as a ## Round N — Memory Update subsection after the raw response.
• This file is passed back to the reviewer in the next round's Phase A — it is the reviewer's persistent memory.
• Record the file's SHA-256 hash before each reviewer call and pass it to save_trace.sh via --memory-hash. Hash the memory as supplied to the call (pre-call artifact), not the post-append version, so the trace proves which memory was in play for that invocation.
• If the score REGRESSES round-to-round, don't just write a new memory line: diff the two rounds' raw .response.md files in .aris/traces/ first and find the exact criterion that flipped (see shared-references/review-tracing.md § Debugging With Traces). The memory file is a summary; the trace is evidence.

Phase B.5.1: Stop-Evaluation Gate

STOP CONDITION — branch by round_backend (the backend that actually ran this round), never by the forward-looking REVIEWER_BACKEND. Use the executable transition table in review_gate.py; resolve it through the canonical helper chain in shared-references/integration-contract.md §2. Its JSON decision and next_backend fields are authoritative. If the helper cannot be resolved or executed, emit REVIEW_UNAVAILABLE; do not improvise a transition.

Invoke the gate once per completed round. Pass model strings only—the helper derives families internally and does not accept caller-suppl

…(truncated)