Codex Review

Skill: Codex review

Purpose

Have a second model (OpenAI Codex) independently re-derive the current analysis from the same data and compare it to Claude's original. Codex gets the question and the metric definitions, but never sees Claude's SQL, numbers, or conclusions — it writes its own queries and computes its own results. The skill then reconciles the two: AGREE, DISAGREE, or PARTIAL per finding. Two models agreeing from independent derivations is strong evidence the analysis is sound; a disagreement points to exactly where to look.

This pairs with /reliability (same model, run N times — tests stability). /codex-review uses a different model once — it tests correctness by independent agreement.

When to Use

• User says /codex-review, "validate with codex", "codex review", "second opinion from codex", "independently verify this", "does codex agree", "cross-check with the other model"
• After producing a finding the user is about to act on and wants a second model to confirm
• Routed here whenever multi-model validation of an analytical result is wanted

Invocation

/codex-review [finding or artifact path] — validate the most recent analysis by default, or scope to a single finding/file if given. Example: /codex-review after answering "What's our 30-day retention?"

Instructions

⛔ HARD GATE — read before anything else

This skill is worthless unless a different model (Codex) does the validation. If Codex is not ready, you (Claude) MUST NOT perform the validation yourself. Claude re-checking Claude's analysis is circular — it produces a confident "validated ✓" that means nothing and actively misleads the student.

The rule: if Step 1's preflight returns a non-empty missing list, your ONLY job this turn is to help the student set up Codex. You may not proceed to Steps 2–7, and you may not substitute any other model, your own reasoning, a re-run of the SQL, or an "approximate" check. There is no fallback that uses Claude. Setup is the task when Codex is missing — completing it is the helpful outcome, not skipping ahead to a verdict.

Step 1 — Preflight: is Codex usable? (decision matrix)

Run the deterministic check:

python3 helpers/provenance/codex_validation.py --check

It returns JSON: {"codex_cli", "plugin", "auth", "missing": [...]}. Route on missing:

• Empty missing → Codex is ready. Go to Step 2.
• "codex_cli" present → the Codex CLI isn't installed. Tell the user to run:

  npm install -g @openai/codex
  

  (Requires Node.js 18.18+.)
• "plugin" present → the Claude Code plugin isn't installed. Show these commands for the user to paste (the skill cannot run them — they're interactive):

  /plugin marketplace add openai/codex-plugin-cc
  /plugin install codex@openai-codex
  /reload-plugins
  /codex:setup
  

• "auth" present → Codex is installed but not authenticated. Tell the user to run:

  codex login
  

  (Sign in with a ChatGPT account or an API key.)

If missing is non-empty, stop after giving the setup step and end the turn with "Once that's done, re-run /codex-review and I'll have Codex check it." The next invocation re-runs --check and proceeds only when missing is empty.

Restart gate. If the student just installed the plugin, also remind them the plugin's tools aren't loaded until they run /reload-plugins — so the sequence is install → /reload-plugins → re-run /codex-review. (auth is best-effort: if --check returns auth: null with the CLI and plugin present, proceed — the live Codex run is the real gate and will surface any login error.)

Keep this simple and one-step-at-a-time: name only the first missing piece, let the student fix it, then re-run the check. Setup may take two or three turns (CLI, then plugin + reload, then login); that is the expected, correct path — not a detour from the "real" work.

Step 2 — Resolve what's being validated

Identify, for the most recent analysis (or the scoped finding):

• The question it answered.
• The metric definitions / scope / time-window Claude used — pull from the metric dictionary (metrics/index.yaml), the analysis design spec, or the analysis itself.
• The active dataset (.knowledge/active.yaml).
• Claude's original result(s) — the headline number(s), the SQL, and the conclusion.

If it's ambiguous what to validate (no recent finding, multiple candidates), ask the user which finding or artifact to check, and offer a path.

Step 3 — Write the validation brief (blind to Claude's numbers)

Create a timestamped run directory: working/codex_validation/<UTC-timestamp>-<question-slug>/.

Write brief.md in it containing ONLY what Codex needs to answer the same question the same way, independently:

• The question.
• The metric definition(s), scope, and time-window (so Codex measures the same thing).
• The active dataset id and how to reach the data: read .knowledge/active.yaml, the active dataset's .knowledge/datasets/{active}/schema.md and quirks.md; connect with from helpers.data.connection_manager import ConnectionManager (or the local DuckDB/CSV fallback in the dataset manifest's local_data if no warehouse is reachable).
• An instruction to log its queries the way the repo expects.

Do NOT put Claude's SQL, result numbers, or conclusion in brief.md. That blindness is the whole point — it's what makes Codex's derivation independent.

Separately, stash Claude's original result in claude_original.md in the same run dir (headline number(s), SQL, conclusion). This file is for the Step 5 comparison only — it is not given to Codex.

Step 4 — Run Codex independently

Dispatch the codex:codex-rescue subagent (Agent tool) with brief.md and this output contract:

Independently answer the analytics question in this brief against the active dataset. Connect to the data and write your own SQL — do not ask for or assume anyone else's queries or numbers. Use the metric definition exactly as given. Log your queries. Then report ONLY:

• headline: <the single number you'd report> (one per finding if multiple)
• sql: <the query/queries you actually ran>
• measured: <numerator, denominator, grain, window, filters>
• conclusion: <one or two sentences>

Capture Codex's full response to codex_independent.md in the run dir.

(Fallback: if the subagent's output is unreliable or unavailable, run codex exec via Bash with the same brief and output contract, and save the result to the same file.)

Step 5 — Compare (skeptical reconciliation)

Put Codex's numbers next to Claude's (claude_original.md) and assign a verdict per finding:

• AGREE — numbers match within a sensible tolerance and the conclusions align.
• DISAGREE — a material gap. Show both numbers, both SQL approaches, and the most likely cause (different filter, cohort, join grain, window). Investigate which derivation is right — do not average them.
• PARTIAL — same direction, different magnitude, or agreement on some sub-results only.

Write a verdict.md (the human-readable comparison table) AND a verdict.json for the deterministic audit log, shaped: {"question": "<q>", "model": "codex", "findings": [{"name": "<finding>", "verdict": "AGREE|DISAGREE|PARTIAL"}, ...]}.

Step 6 — Record the run (tracked + auditable)

Append the run to the audit log:

python3 helpers/provenance/codex_validation.py --log <that run directory>

It reads verdict.json, counts the verdicts deterministically, and appends one line to .knowledge/codex-review/log.jsonl. The run dir now holds the full provenance: brief.md, claude_original.md, codex_independent.md, verdict.md, verdict.json.

Step 7 — Report (short, on-screen)

Frame it as independent multi-model validation, then show the comparison:

• Headline: e.g. "Claude found 38% 30-day retention; Codex independently derived 38% from its own query — AGREE." Or: "Codex got 31% vs Claude's 38% — DISAGREE: Codex filtered to activated users only; Claude counted all signups."
• The per-finding Finding | Claude | Codex | Verdict | Why table from verdict.md.
• Where it was saved (the run dir + .knowledge/codex-review/log.jsonl).

Then the honest framing:

• All AGREE — "A second model independently reproduced this from its own queries. That's strong evidence the result is sound — not proof, but two independent derivations agreeing."
• Any DISAGREE / PARTIAL — "The two models diverge here. That's the check earning its keep: one of these derivations is wrong, or the metric is under-defined. Resolve the gap before acting on the number."

On any DISAGREE, offer to re-run the relevant analysis step, define the metric via /metric-spec, or log the lesson via /log-correction.

Rules

1. No Codex, no validation — and no Claude fallback. If preflight's missing is non-empty, stop at setup. Never validate with Claude, another model, your own reasoning, or a re-run of the SQL. A Claude-checks-Claude result is circular and must never be presented as a validation. This is the one rule that cannot be bent. (See the Hard Gate above.)
2. Codex must be blind to Claude's numbers. Never include Claude's SQL, result numbers, or conclusions in brief.md. If you can't keep them out, the run isn't independent — say so rather than presenting a false validation.
3. Counting is deterministic. Verdict tallies come from codex_validation.py --log reading verdict.json, never estimated in prose.
4. One missing piece at a time in preflight. Don't dump every install step at once — name the first gap, let the user fix it, re-check.
5. Respect the restart gate. After a plugin install, halt until /reload-plugins.
6. Same definitions, independent derivation. Codex answers the same question with the same metric definition — only the SQL and numbers are its own.

Edge Cases

• Codex not installed (the common student case) → help the student install/log in, then stop; validation happens on the next run once --check is clean.
• No recent analysis to validate → ask the user what to check; offer a path or finding.
• Codex can't reach the warehouse → the brief should hand it the local DuckDB/CSV fallback (manifest.local_data) so it can still derive independently.
• Codex defines the metric differently anyway → flag it: the disagreement may be definitional, not an error. Surface both definitions and recommend /metric-spec.
• auth: null from preflight → proceed; the live run surfaces any real login error.

Notes

• The plugin (openai/codex-plugin-cc) is just the simple, supported path to an installed + authenticated Codex CLI. The validation itself runs Codex against the data, not a code diff — the plugin's /codex:review (diff review) is a different thing and isn't used here.
• Complements /reliability: that re-runs the same model to test stability; this runs a different model once to test correctness by independent agreement.