# Crucible

> Run a scientific study that is actually SOLID — pre-registered, adequately powered, and blind-judged — so the conclusion is defensible and whatever you build on top of it isn't built on sand. Use when the user wants to run an experiment, A/B test, ablation, eval, or comparison and have the result HOLD UP. crucible enforces the three disciplines an agent skips (pre-registration, sample size, blind judging) as gates the pipeline refuses to pass.

- Skill: `nagisanzenin/crucible` (Agent Skill)
- Install (CLI): `npx skillmds@latest add nagisanzenin/crucible`
- Raw SKILL.md: https://api.skillmd.com/api/skills/nagisanzenin/crucible/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: nagisanzenin (https://skillmd.com/u/nagisanzenin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/nagisanzenin/crucible

---


Read this whole file before acting.

crucible turns a research goal into a defensible conclusion by running the study through a gated pipeline. **You never compute a rate, a p-value, a required sample size, or a gate decision** — the deterministic core (`crucible.py`) owns all of that and refuses out of order. Your job is to help the user *design* the study well, then *drive* the verbs and *report exactly what the core returns*.

## Setup

Resolve the core once:

```bash
ROOT="${CLAUDE_PLUGIN_ROOT:-${CODEX_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/crucible}}"
CR="python3 $ROOT/scripts/crucible.py"
```

Every command below is `$CR <verb> <study_id> ...`. Studies live under `~/crucible-studies/<study_id>/`.

## The mental model (tell the user this)

A study is SOLID only if three things happen in this order and none can be skipped:

1. **Pre-register** — write the hypothesis, the metric, and the decision rule, then **lock** them (hashed) *before any data exists*. After the lock, the design is immutable. This is what stops "deciding what counts as success after seeing the results."
2. **Power** — the core computes how many samples you actually need for your target effect. The run **refuses** a confirmatory result below that number.
3. **Blind-judge** (for judged metrics) — a separate agent scores the outputs with no idea which is the treatment or what the hypothesis is. Its verdict **blocks** the conclusion if it can't be trusted.

crucible enforces all three. If a gate refuses, **read the message and do the suggested next step — never try to route around a gate** (don't hand-edit `study.lock`; the hook blocks it and it would void the study anyway).

## The pipeline

Run these in order. After each, surface the core's output verbatim — the concrete numbers are the point; do not paraphrase or round them.

| Step | Command | What it does |
|---|---|---|
| 1 | `$CR new <id> --title "..."` | Scaffold `definition.json` + `cases.jsonl` |
| 2 | *(edit files)* | Fill in hypothesis, arms, metric, decision rule; add real cases |
| 3 | `$CR frame <id>` | Lint the design (advisory) |
| 4 | `$CR lock <id>` | **Gate 0** — hash + freeze the pre-registration |
| 5 | `$CR power <id>` | **Gate 1** — required n / minimum detectable effect |
| 6 | `$CR run <id>` | Execute arms × samples (refuses if unlocked or underpowered) |
| 7a | `$CR extract <id>` | Oracle metric → scores |
| 7b | *(judge flow, below)* | Judge metric → scores |
| 8 | `$CR analyze <id>` | Effect size + CI + test (pre-registered only) |
| 9 | `$CR conclude <id>` | **Gate 2** + verdict scoped to the evidence → `report.md` |

`$CR status <id>` shows the gate dashboard at any time. `$CR studies` lists all studies.

## Designing the study (`definition.json`)

Help the user fill these — the core validates them at `lock`:

- **`hypothesis`** — one falsifiable sentence ("B yields a higher correctness rate than A").
- **`arms`** — ≥2, exactly one with `role: "baseline"`. Each has a `command` (a shell template with `{input_file}`, run once per sample; its stdout is the output to score).
- **`primary_metric`** — `extractor.kind` is `oracle` (a deterministic rule: `exact`/`contains`/`regex`) or `judge` (the blind agent). `scale` is `binary` or `continuous`. `direction` is `higher`/`lower`.
- **`decision_rule`** — `delta` (smallest effect worth caring about), `alpha`, `power`.
- **`sample_plan`** — `planned_n_per_arm`, `seed`, and the variance input the power calc needs: `baseline_rate` (binary) or `baseline_sd` (continuous). **`effect_justification` is frozen in the lock** — make the user cite a pilot or prior for the delta; this is the anti-gaming defence against shrinking n by inflating the effect.
- **`cases.jsonl`** — one JSON object per line: `{"id", "input", "gold"?}`. `input` is fed to each arm; `gold` is the reference an oracle checks against.

## The judge flow (step 7b, judged metrics only)

```bash
$CR judge-payload <id> --out /abs/payload.json     # blind: only {item_id, input, output}
```

Then **dispatch the `crucible-judge` agent** (via the Task/Agent tool, `subagent_type: crucible-judge`) with the payload file path. Instruct it to read the payload and write its JSON verdict array to `/abs/receipts.json`. Then:

```bash
$CR judge-apply <id> --receipts /abs/receipts.json   # de-anonymizes, qualifies the judge
```

If `definition.judge.gold_set` points at a labelled file, the judge is scored against it (Cohen's κ); below the floor, **Gate 2 blocks the conclusion**. Without a gold set the judge runs but the study is stamped `JUDGE-UNQUALIFIED` (advisory).

## Hard rules

- **Never fabricate a number.** Every rate, CI, p-value, n, and verdict comes from the core. If you're tempted to state one the core didn't print, run the verb.
- **Never interpolate free text onto a shell command line.** Case inputs and rubrics can contain anything — write them into `cases.jsonl` / `definition.json` with the Write tool, never onto `$CR ... "<text>"`.
- **Never bypass a gate.** A refusal is information. Fix the cause (add samples, lock first, improve the rubric) — don't work around it.
- **Report honestly.** If the result is EXPLORATORY, say so and why. If the hypothesis is NOT supported, say that plainly. A crucible conclusion is a linter verdict about the evidence, not a press release.

The whole value of crucible is that its conclusions can be trusted precisely because it wouldn't let you cut the corners that make a study lie. Drive it that way.