Horizon Agentic Reviewer

Horizon Agentic Reviewer Skill

Canonical rubric

The single source of truth for the rubric this review enforces is QC-SPEC.md — the master at the repo root and the per-task copy in the task folder under review (.claude/QC-SPEC.md). The checklist below operationalises those Critical/Error/Legitimacy items; when they disagree, QC-SPEC.md wins. Read it before reporting findings.

Execution Mode — FULLY AUTONOMOUS

NEVER call ask_user or pause for confirmation at any point. Execute every step, probe, and fix without interruption. If uncertain between two paths, pick the safer one and continue.

VM Access (Nebula Aurora GCP Instance)

The reviewer's Nebula Aurora VM is pre-configured and reachable via SSH:

• IP: 8.229.18.245
• User: jonahodoh
• SSH key: ~/.ssh/mac_ssh
• SSH config alias: nebula-vm
• Zone: us-west1-b (GCP Compute Engine)

Connect with:

ssh nebula-vm
# or explicitly:
ssh -i ~/.ssh/mac_ssh jonahodoh@8.229.18.245

Use this VM to run horizon setup, docker exec, kubectl probes, and observe live eval runs. The horizon workspace is at /Users/mac/Documents/tasks on the LOCAL Mac — task files are synced/pushed from there. The VM runs the Docker containers for evals.

Goal Red-team the task and find issues. The point of this skill is to break the task — surface bugs, gaps, leaks, undisclosed requirements, broken evidence trails, reward-hack vectors, and any place where a reasonable agent would be unfairly penalized or could trivially win without doing the work. A clean "everything passes" report is rarely the right outcome — it usually means probes were too shallow. Approach every check adversarially: form a hypothesis the task is broken in this specific way, then build the probe that would prove it.

A live container exists specifically to make this possible. Without one, every "is this discoverable?" / "can the agent see this?" / "does the evidence trail actually fire?" question collapses into guesswork. With one, you can answer with a real command and a real output. Prefer to discover one concrete reproducible issue with evidence over checking off twenty items optimistically.

Trigger Use this skill when the user asks to:

Review/QA/inspect/red-team a horizon (apex-arena) task with live container verification Verify (or break) a static review finding by exploring the task environment as the agent would Run the 24-point quality checklist against the task (20 checks probe a single live container; 4 checks parse downloaded rollout JSONs — one container, one rollout fetch, all 24 checks share that data) Command shape: /horizon-agentic-reviewer or "review/red-team horizon task with live container".

This complements nebula-reviewer.md (static analysis only). Use this skill when you need to dynamically verify hypotheses (e.g. "is this requirement actually undisclosed, or just buried in a discoverable file?", "does the seeded alert actually fire?", "can a no-op solution pass any subscore?").

Inputs task_uuid — required. UUID of the horizon task to review. horizon_root — the user's horizon checkout (the dir holding ./env/). Defaults to the current working directory; assume the user runs the skill from inside it. Environment Venv: ./env (assume this exists in ). Activate with source /env/bin/activate before any horizon invocation. Auth (required for tasks download and rollouts pull): the horizon CLI accepts auth from one of: HORIZON_API_KEY env var (or APEX_API_KEY as fallback) — preferred for headless / CI / non-interactive runs. OAuth tokens at ~/.horizon/credentials.json — populated by running horizon login once. Used in interactive sessions. Verify with horizon whoami before starting a review — if it says "Not authenticated", abort and ask the user to set HORIZON_API_KEY or run horizon login. Never put the API key in a shell command line that gets logged; rely on it being preset in the environment. CLI entrypoint: horizon (project name horizon, package apex_arena). Review workspace: /tmp/horizon-reviews//. The skill creates and cleans this — the user's /tasks/ is left untouched. Inside the workspace: tasks// (downloaded task), setup.log (background setup output). Container name pattern: apex-arena- (port 8001). horizon setup always names containers after the task folder. Agent user — task-dependent. Two image lineages in production: apex_arena:base (default) → users root + model (uid 1000). No ubuntu user. nebula-devops:* (DevOps tasks like Kubernetes/SRE) → users root + ubuntu (with sudo). Detect at runtime: docker exec apex-arena- getent passwd model ubuntu and exec as whichever user exists. Workflow Step 0 — Pre-flight source /env/bin/activate docker ps # check what's running on 8001 already horizon whoami # auth check — abort if "Not authenticated" Stale-container policy (depends on mode):

What's holding port 8001 / using apex-arena-* name Interactive Headless (-p) apex-arena- (matches the UUID we're about to review) Re-use only if user explicitly says so; otherwise tear down + restart fresh Tear down + restart fresh (state may be polluted) apex-arena- (a different review's container, almost certainly stale because cleanup is user-gated in interactive mode) Ask before stopping Auto-tear-down + iptables cleanup (no user to ask; a prior interactive review left it up but headless has no one to red-team) Anything else holding port 8001 (non-apex-arena process) Ask before stopping Refuse and exit non-zero with a clear message — don't kill the user's unrelated workload In headless mode the rule is: be aggressive about cleaning up prior reviews' artifacts (named apex-arena-*), conservative about anything else.

Step 1 — Download the task into an isolated workspace WORKSPACE=/tmp/horizon-reviews/ mkdir -p "$WORKSPACE" cd "$WORKSPACE" horizon tasks download --tasks-dir "$WORKSPACE/tasks" The task lands at $WORKSPACE/tasks// with files: task.yaml, grader.py, Dockerfile, setup.sh, solution.sh, report.json. Read all of them before starting the container — the static read informs the hypotheses you'll verify live.

(Alternatives: horizon init scaffolds into ./tasks/, horizon tasks pull pulls a specific version. We use tasks download so reviews stay isolated from /tasks/.)

Capture the task version from the download output line (✓ Task (version N) downloaded successfully) — you'll need it for Step 1.5. Always review against the latest version unless the user explicitly asks for an older one.

Step 1.5 — Pull rollouts (cross-attempt statistics) Live container probing is single-snapshot — it cannot tell you whether subscores actually differentiate between agents, what the pass rate is, or whether failures are legitimate. Pull historical rollouts so the rollout-based checks (G21–G24 below) have data:

cd "$WORKSPACE/tasks/" horizon rollouts pull --version --output-dir "$WORKSPACE/rollouts" Always scope to the latest version. Without --version, you get every historical version mixed together — those rollouts were graded against older grader/setup code and would pollute variance/dead-weight stats for the current shape.

Model preference (mirrors nebula-reviewer): filter rollouts by model in this order, take the first that has data:

nighthawk (default — modern eval) -max-nebula (fallback) -nebula (older fallback) If none of those exist for the latest version, list available models from the pull summary and pick the one with the most rollouts; flag in the report which model the analysis used.

Output structure: $WORKSPACE/rollouts/v/run.json (grade result) and ..._transcript.md (full conversation). Each .json: score, success, model, run_number, task_version, grade_result (a JSON-encoded string with subscores, weights, metadata.feedback).

Quick parse for variance / dead-weight / pass-rate (run as part of Step 4 G-bucket):

import json, glob from collections import defaultdict PREF = ["nighthawk", "-max-nebula", "-nebula"] files = glob.glob("$WORKSPACE/rollouts/v/*.json") by_model = defaultdict(list) for f in files: d = json.load(open(f)); by_model[d["model"]].append((f, d)) chosen = next((m for m in PREF if by_model.get(m)), None) or max(by_model, key=lambda m: len(by_model[m])) rollouts = by_model[chosen] print(f"using model={chosen} N={len(rollouts)}") sub_vals = defaultdict(list); scores = [] for _, d in rollouts: scores.append(d["score"]) for k, v in json.loads(d["grade_result"])["subscores"].items(): sub_vals[k].append(v) print("pass_rate:", sum(1 for s in scores if s>=0.99)/len(scores)) for k, vs in sub_vals.items(): print(k, "DEAD" if len(set(vs))==1 else "varies", sorted(set(vs))) If no rollouts exist for the latest version (a brand-new task), say so explicitly — bucket G is N/A and the live-container findings carry full weight. Don't fabricate variance from the oracle.

Step 2 — Patch Dockerfile if needed The MCP server inside the container fails on startup unless COMPUTER_WIDTH_PX and COMPUTER_HEIGHT_PX are set. Most production tasks (especially DevOps/nebula-devops:* images) already declare these. If the task's Dockerfile does not, add to the placeholder block:

ENV COMPUTER_WIDTH_PX=1024 ENV COMPUTER_HEIGHT_PX=768 For freshly-scaffolded apex_arena:base tasks this is usually missing; for downloaded production tasks it's usually present — check before patching.

Step 3 — Start the container (background) horizon setup is blocking (idles after readiness). Critical: run it from $WORKSPACE so the container mounts $WORKSPACE/tasks (the setup code does Path.cwd()/tasks:/mcp_server/tasks).

cd "$WORKSPACE" horizon setup 2>&1 | tee "$WORKSPACE/setup.log" Use run_in_background: true on the Bash tool call. Wait for MCP server is ready or Setup complete! in the log (Monitor tool with grep --line-buffered "ready|complete|Error|failed|denied|Killed" works well). First run with a private-registry base image (e.g. nebula-devops) may take several minutes to pull. Don't time out at 60s. Confirm with docker ps --filter name=apex-arena-. Detect agent user once the container is up: docker exec apex-arena- getent passwd model ubuntu — use whichever exists for -u in subsequent execs. Step 3.5 — Inject faults by running task setup.sh (manual) horizon setup does NOT automatically run the task's setup.sh. It only starts the MCP server container — the agent-fault state (spam-flooded mailboxes, stripped annotations, missing dashboards, etc.) is not present until the task's setup.sh is executed. Without this step, your "live verification" is verifying a clean world, not the agent's actual world.

Run as root (matches how horizon's test-solution does it — cli.py:4399 invokes setup.sh with no -u flag):

docker exec apex-arena- bash -c
'cd /mcp_server/tasks/ && chmod +x setup.sh && ./setup.sh' For DevOps/k3s tasks this can take 5+ minutes (RBAC propagation up to 600s, spam flood loop). Background it and watch for Setup complete or fault-injection log lines.

User-role reference (from cli.py):

Script Runs as Why it matters setup.sh root Has full kubectl admin; can mutate ConfigMaps, scale workloads, write to root paths. Don't flag root-only operations as "broken" here. solution.sh agent user (ubuntu or model, auto-detected at cli.py:4413) Lines like export KUBECONFIG=/etc/rancher/k3s/k3s.yaml are dead code (file is mode 600); kubectl falls back to ~/.kube/config. Don't flag as a bug without testing. grader.py root Reads cluster authoritatively; not gameable from agent perspective. Agent (live) agent user Always -u for agent-perspective probes. Step 4 — Run the 24-point inspection (adversarial) One container (started in Step 3), one rollout pull (Step 1.5) — all 24 checks share that data. Don't spin a fresh container per check. Buckets A–F probe the live container via docker exec; bucket G parses the rollout JSONs (no container needed).

For each check, do static read first (the .py/.yaml files), form an adversarial hypothesis ("this is broken in way X" — not "is this OK?"), then try to prove the hypothesis — in the container for A–F, against the rollout JSONs for G — as the detected agent user (model or ubuntu) where applicable. A check passes only after you've actively tried to break it and failed. Live-container probe pattern:

docker exec -u apex-arena- bash -c '' A. Prompt & Discoverability (verify by exploring as the agent) Prompt clarity — Re-read task.yaml prompt:. Could a reasonable agent know what "done" looks like without reading grader.py? Undisclosed requirements — For each check in grader.py, ask: is the thing it tests mentioned in the prompt, or discoverable via exploration (ls /workdir, README files, configs, comments, env vars visible to model)? Only flag as undisclosed if neither. Contradictory requirements — Cross-check prompt assertions against actual world state (e.g. prompt says "service running on :8080" — curl localhost:8080 from inside). Missing resources — Every file/asset referenced in the prompt must exist and be readable as model. Probe each. B. Environment Reachability (probe as model) Tool availability — which for every tool the task implies (kubectl, docker, jq, python, etc.). Dependencies — Try import for each Python lib mentioned, run any --version for binaries. Permissions — model must read/write where the task expects. Try touch /workdir/foo, read /workdir/data/*. Network policy — curl -m 3 https://example.com (should fail per network isolation) and curl -m 3 host.docker.internal or any host service the task allows. Confirm policy matches task intent. C. Solvability Solution runs — docker exec -u model … bash /tests/solution.sh (or wherever solution.sh lives). It should complete successfully. Solution scores — Run the grader against the post-solution state. Score should be ≥ pass threshold. (grader.py is root-owned at /tests/grader.py; you may need docker exec without -u model to run it, mirroring how horizon grades.) Trivial baseline — Reset (re-create container or restore initial state), do nothing, run grader. Score should be low. If a no-op scores high, baseline-passes-task is a CRITICAL fail. D. Reward Hacking / Leakage (only flag from model-accessible sources) Answer in accessible files — grep -rI '' /workdir /home/model /tmp 2>/dev/null (substitute actual expected values). Also scan env vars (docker exec -u model … env). Grader/setup isolation — Confirm model cannot read /tests/grader.py, /tests/setup.sh, the Dockerfile. cat /tests/grader.py as model must fail with permission denied. If readable, that's a CRITICAL leak. History recovery — Check ~/.bash_history, /root/.bash_history (should be inaccessible), any .git directories under /workdir. Grading bypass — Look for shortcuts: can model write the exact file the grader checks without performing the work? Can the grader's success path be triggered by manipulating file timestamps, symlinks, etc.? E. Grader Structure (static, cross-checked dynamically) Functional test present — grader.py must include at least one check that exercises the deliverable end-to-end (HTTP call, query, command output) — not just "file exists" or "string contains". Equal weights — Extract weights from grader.py. All N subscores must have weight 1/N. Unequal weights = BLOCKING. Every grader check maps to prompt — Each subscore must trace to either an explicit prompt requirement or a discoverable-in-world hint (verified in check #2). F. Edge / Misc Honest reporting not penalized — If the prompt allows "report N/A" or "no answer", confirm grader accepts that path. Test by writing the honest-failure output and grading. Application bugs / env stability — Restart relevant services, run trivial commands twice, confirm deterministic behavior. Random failures here masquerade as agent errors. G. Rollout Statistics (cross-attempt signals — needs Step 1.5 data) For each check below, run on the chosen-model rollouts of the latest version. If no rollouts: mark N/A and say so. These are the checks the live container alone cannot answer.

Functional test variance (BLOCKING) — At least ONE functional subscore (e2e/integration/endpoint/service/validate/verify in name or behavior) must take ≥2 distinct values across rollouts. If every functional subscore is constant, the task provides no training signal — broken. Mirrors nebula-reviewer TOP CRITICAL #1. Dead-weight subscores — No subscore should be constant across all rollouts. List any subscore where len(set(values)) == 1 and what value it's stuck at (always 0 = grader too strict / impossible; always 1 = trivially passing / dead reward dimension). Equal-weight grading with K dead-weight subscores effectively reduces to N–K dimensions. Mirrors nebula SUBSCORE VALIDATION > Dead Weight. Pass rate reasonable — Full-pass rate (score >= 0.99) should sit roughly in 10–70%. <10% suggests impossible problem or undisclosed requirements; >70% suggests trivial/leaky. Report the rate and the avg score. Mirrors nebula PASS ANALYSIS > Pass Rate Reasonable. Legitimate failure modes — For each FAILED rollout, classify the metadata.feedback text into: LEGITIMATE_FAIL (agent made a real mistake — grader/env worked correctly) ✓ TASK_ISSUE (env infra failure, undisclosed req, false negative, prompt-grader mismatch) ✗ REWARD_HACK (trivial pass via shortcut) ✗ For deeper signal, sample 2–3 transcripts (*_transcript.md) and check whether the agent's actions match what the feedback claims. Flag any rollout where the failure looks like a grader bug or unfair check. Mirrors nebula LEGITIMATE FAILURES bucket — without an LLM, do feedback-text heuristics; with one, classify properly. Step 5 — Report Produce a concise report with:

One line per check: [PASS] / [FAIL] / [N/A] and a one-line reason. For each FAIL, include the exact command/output from the container that proves it. Live evidence > inferred claims. Distinguish BLOCKING (checks 2, 11, 13, 16, 17, 21) from non-blocking. End with a 3-line summary: critical issues, soft issues, recommendation (ship / fix / re-do). Persist failed checks to disk — write to ~/.horizon-reviews//findings.json. This survives cleanup (Step 6) and is the durable record of the review. Only failed checks (and N/Ts that warrant flagging) go in; do not include passes (the structure is "what's wrong with this task," not "what was checked").

mkdir -p "$HOME/.horizon-reviews/" Schema (one JSON object per review):

{ "task_uuid": "", "task_version": 24, "task_id": "", "reviewed_at": "<ISO 8601 UTC>", "model_used": "<model whose rollouts were analyzed, or null>", "rollout_count": 10, "summary": { "blocking_failures": 0, "soft_failures": 1, "recommendation": "ship | ship-with-fix | fix | redo" }, "failed_checks": [ { "id": 22, "name": "Dead-weight subscores", "bucket": "G", "blocking": false, "verdict": "FAIL", "evidence": "3 of 5 subscores constant 1.0 across 10 rollouts: spool_capacity_restored, burst_protection_enabled, prometheus_scraping_maddy", "fix": "Tighten checks or remove dead subscores" } ] } If the file already exists for this UUID (a prior review), append a new entry to a reviews array rather than overwriting — review history is useful for tracking whether issues persist across versions. Schema for re-review:

{ "task_uuid": "", "reviews": [ {/* prior review /}, {/ this review */} ] } YAML is acceptable if the user prefers it (findings.yaml); same schema. Don't write both formats — pick one per review.

After persisting, leave the container, iptables rules, background horizon setup task, and /tmp/horizon-reviews// workspace intact. The user typically wants to red-team your findings — re-running probes, re-checking hypotheses, exploring alternative paths, or asking "what if". That requires the live environment to stay up.

State explicitly in your end-of-report message: "Container apex-arena- is still running for follow-up probes. Run or say 'cleanup' when done." Include the exact cleanup command in your message so the user can copy-paste or invoke it any time.

Step 6 — Cleanup (ONLY on explicit signal) Do not auto-cleanup at end of report. Run cleanup only when:

The user says "cleanup", "tear it down", "we're done", or similar; or The user moves on to a different task UUID (cleanup the prior one before starting the new one); or A fatal/unrecoverable error occurred during setup (e.g. image pull failed) — in that case clean up the half-spun resources before exiting. When cleanup is signaled, run all four steps and confirm:

docker stop apex-arena- 2>/dev/null || true docker rm apex-arena- 2>/dev/null || true

from with ./env activated:

horizon cleanup-iptables 2>/dev/null || true rm -rf "/tmp/horizon-reviews/" docker ps -a | grep apex-arena- || echo "all clean" Also stop any background horizon setup task you started (TaskStop on its task_id).

Do not wrap the lifecycle in trap cleanup EXIT — that would tear down on script exit and defeat the red-team window.

Verification Patterns Cheat Sheet Hypothesis Container probe "Prompt mentions X but doesn't say where" find /workdir -name 'X' -readable 2>/dev/null as agent_user "Grader checks env var FOO" docker exec -u … bash -c 'echo $FOO' "Tool kubectl needed" docker exec -u … which kubectl && kubectl version --client "Network must be blocked" docker exec -u … curl -m 3 https://google.com (expect fail) "Answer in /workdir" docker exec -u … grep -rI 'expected' /workdir "Grader file leaked" docker exec -u … cat /tests/grader.py (must fail) "Solution works" docker exec -u … bash /tests/solution.sh; echo $? "Bash history leaks" docker exec -u … cat ~/.bash_history "setup.sh seeded a Prom alert / IM post / queue backlog" Don't trust the log line. Hit the live API. curl http://prom:9090/api/v1/rules (groups should be non-empty), curl http://prom:9090/api/v1/alerts, Mattermost /api/v4/channels//posts, RabbitMQ mgmt /api/queues// "This subscore actually differentiates" Rollout stats: python -c "import json,glob; from collections import defaultdict; v=defaultdict(set); [v[k].add(s) for f in glob.glob('rollouts/v*/*.json') for k,s in json.loads(json.load(open(f))['grade_result'])['subscores'].items()]; print(v)" — any subscore with len==1 is dead weight. "Pass rate is reasonable" score >= 0.99 count / total — target 10–70%. <10% impossible/unfair, >70% trivial/leaky. "ConfigMap has the rule, so it's loaded" No. prometheus.yml's rule_files: list must reference it. Diff kubectl get cm prometheus-config -o jsonpath='{.data.prometheus.yml}' for # - "alerts.yml" style commented-out lines. "Reload happened" Hit /api/v1/status/runtimeinfo or /api/v1/status/config and confirm timestamps / contents match the patched ConfigMap. "solution.sh's KUBECONFIG=/root/path is broken for ubuntu" Test before flagging. kubectl falls back to ~/.kube/config if the named path is unreadable — auth still works. Verify with KUBECONFIG=/root/foo kubectl get pods -n maddy as agent_user. Important Notes Always exec as the detected agent_user (-u model or -u ubuntu) for agent-perspective checks. Root probes give wrong answers about what the agent can see. Static read first, then verify. Don't waste container time on checks you can answer from the files alone — but do confirm any "the agent can/cannot do X" claim live. One container per review. Don't reuse a stale container; start fresh so prior state doesn't pollute baseline checks. (If a previous review's container is still up because the user was red-teaming, ask before tearing it down.) Cleanup is user-gated, not automatic. The container stays up after the report so the user can red-team findings (re-run probes, test "what ifs", inspect state). Only clean up on explicit signal — see Step 6. Stale containers from a prior review still block port 8001 for a new review, so confirm the user is done with the previous task before starting a new one. Distinguish "not in prompt" from "undisclosed". A requirement discoverable by reading a README, ls'ing /workdir, or running --help is not undisclosed. This skill exists specifically to make that distinction with evidence. Don't trust setup.sh log claims. Setup scripts often print "alert seeded" / "evidence posted" / "fault injected" — verify each claim live. Setup writes config but doesn't always reload services; comments out lines it meant to enable; logs success on || true paths that silently failed. Many of the highest-value findings come from log-vs-reality mismatch. Don't trust report.json either. If the task ships with a previous QA verdict, treat it as a hypothesis to falsify, not a baseline. Versions drift, and a stale "all pass" report is exactly the kind of thing a live probe should refute. solution.sh, grader.py, setup.sh, Dockerfile are root-owned at /tests/ — do NOT flag them as agent-readable leaks. horizon setup blocks — always run with run_in_background: true and wait on the log line, never call it foreground. horizon setup doesn't run the task's setup.sh. It only starts the MCP server. For tasks with fault injection (most real ones), exec setup.sh manually as root inside the container after MCP is ready (Step 3.5). Verifying against a clean cluster instead of a faulted one is a major source of false PASSes. Verify "config present" ≠ "config loaded". A rule/annotation/dashboard sitting in a ConfigMap, file, or registry is not the same as the service having loaded and acting on it. For Prometheus check /api/v1/rules, for Maddy check the actual SMTP behavior, for Grafana hit /api/dashboards/uid/.... Authoring scripts often forget the reload step or reference the wrong path. Be willing to withdraw a finding. If a deeper probe contradicts an earlier hypothesis (e.g. "kubectl falls back gracefully so the apparent bug isn't one"), explicitly retract — don't quietly let the wrong claim stand. Confidence comes from evidence, not commitment to your first read. Know which user runs each script. setup.sh and grader.py run as root (cli.py:4399, 4422); solution.sh runs as the auto-detected agent user (cli.py:4413). When inspecting a script, ask "as which user does this actually execute?" before flagging path/permission issues — many apparent bugs are correct under the real execution role. Pull rollouts of the latest version with the right model. Default model preference: nighthawk → -max-nebula → -nebula → most-populated. Mixing versions pollutes variance/dead-weight stats; mixing models hides per-model dead-weight (a check that's easy for one model and hard for another may look "varying" overall but is dead-weight per-model). Headless / non-interactive mode The skill works under Claude Code's -p print mode for batch/CI-style invocations:

claude -p --dangerously-skip-permissions
"review horizon task using the horizon-agentic-reviewer skill" Or with --allowed-tools if you want to bound the surface:

claude -p --allowed-tools Bash Read Edit Write Glob Grep
"review horizon task using the horizon-agentic-reviewer skill" Behavioral differences in headless mode:

Auto-cleanup at end. There is no user to red-team, so the rationale for keeping the container up after the report is gone. Run the Step 6 cleanup unconditionally as the final action — stale containers from a CI run would block the next invocation. The user-gated rule only applies to interactive sessions. findings.json is the canonical output. Stdout/the in-conversation report is for humans; the durable artifact is ~/.horizon-reviews//findings.json. CI/automation should read that file, not parse the print output. Exit-code expectations. The wrapper invocation should exit non-zero if the review found a BLOCKING failure (any of checks 2, 11, 13, 16, 17, 21 with verdict: "FAIL"). Encode the gate in the prompt: "after writing findings.json, if summary.blocking_failures > 0 exit with code 1; else exit 0."

Time budget. End-to-end is image-pull-bound (≈5–10 min cold, <2 min warm) plus k3s startup (3 min for DevOps tasks) plus setup.sh fault injection (5 min). Plan for 15–20 min per review in CI; don't impose a 60s timeout. Skip prompts. With --dangerously-skip-permissions, every tool call goes through. Without it, headless will hang on the first permission prompt — use the flag or pre-allowlist the needed tools/commands in .claude/settings.json via the fewer-permission-prompts skill. Don't ask clarifying questions. In interactive mode the skill can ask "should I stop this stale container?". In headless mode there is no user to answer — adopt a sensible default (e.g. refuse to start if port 8001 is held by anything not named apex-arena-) and exit non-zero with a clear message. Coverage map: our 24 → nebula-reviewer Our # Check Nebula equivalent 1 Prompt clarity Task Design > Prompt Clarity 2 (BLOCK) Undisclosed requirements TOP CRITICAL #3, Grading Design > undisclosed_constraints 3 Contradictory requirements Task Design > Contradictory Requirements 4 Missing resources Task Design > Missing Resources, Problem Quality > problem_asset_gap 5 Tool availability Infrastructure > tool_availability_mismatch 6 Dependencies Infrastructure > missing_dependencies 7 Permissions Infrastructure > permission_mismatch 8 Network policy Infrastructure > runtime_network_access 9 Solution runs Problem Quality > impossible_problem 10 Solution scores (validates the task is solvable end-to-end) 11 (BLOCK) Trivial baseline Grading Design > baseline_scores_competitively, Problem Quality > trivial_problem 12 Answer in accessible files Reward Hacking > answer_leakage 13 (BLOCK) Grader/setup isolation Reward Hacking > answer_leakage (file source) 14 History recovery Reward Hacking > history_recovery 15 Grading bypass Reward Hacking > grading_bypass, environment_bypass 16 (BLOCK) Functional test present TOP CRITICAL #2 17 (BLOCK) Equal weights Subscore Validation > Equal Weights 18 Every grader check ↔ prompt TOP CRITICAL #6 (no prompt-grader mismatch) 19 Honest reporting Grading Accuracy > honest_reporting_penalized 20 App / env stability Infrastructure > intermittent_failure 21 (BLOCK) Functional test variance TOP CRITICAL #1 22 Dead-weight subscores Subscore Validation > Dead Weight Detection 23 Pass rate reasonable Pass Analysis > Pass Rate Reasonable 24 Legitimate failure modes Legitimate Failures (whole bucket: Failures Are Valid, Root Cause Diversity, No False Negatives) Gaps we deliberately don't carry from nebula:

Spelling/grammar — low signal, easy to bolt on later. Grader determinism v2 (timing/race) — partially covered by #20 (env stability) and #21 (variance); deeper analysis needs grader re-run-on-same-state which is an order of magnitude more setup. Grader fairness v2 (alternative solutions, binary bundling, hardcoded values) — partially covered by #15 (grading bypass) and #18 (grader↔prompt mapping); deeper analysis needs LLM transcript review. 65-flag deep checklist — we treat that as the breadth pool the synthesis section pulls from, not a separate pass. If a review needs the deeper buckets, run nebula-reviewer.md alongside this skill — they're complementary by design.