The-Utopia-Studio Guardrail Design
What it does
Turns "how do we stop it doing something bad?" into a spec a build can enforce. The
fellow brings the failure taxonomy, the chosen autonomy level, and the cost of one
failure per mode (from eval-first-spec). This skill places each mode on a
cost-of-error × volume matrix, assigns a three-layer guardrail stack — rules in code,
a confidence threshold, human sign-off — sized to the quadrant, derives the confidence
threshold from the acceptable failure rate, and writes a sign-off trigger for every
path where a high-cost failure could ship on low confidence. The output is the filled
template.md. It refuses one guardrail applied to everything, and it
refuses a felt confidence number.
The Icarus reframe
A guardrail is not one thing you bolt on the end. It is three layers, each cheaper than the next, each sized to a single number: cost-of-error × volume. Rules in code catch the knowably-wrong for free and run on everything. A confidence threshold catches the uncertain — but only if the threshold is derived from the acceptable failure rate and calibrated against real outcomes, never a 0.8 that "looks safe". Human sign-off is the most expensive layer, so it is spent only where a failure is costly enough to be worth a person's minute — and it must be spent there, because the one thing you can never do is let a catastrophic-cost action ship on a model's confidence alone. The move that a generic prompt never makes: match the mechanism to the quadrant, which means adding a human where cost is high and removing the human where cost is low.
When to use / When NOT
Use when the eval-first spec exists (a failure taxonomy, an autonomy level, a cost-of-one-failure per mode) and the question is now what mechanism enforces it. This skill is the depth build of component 4 — validate & guardrails — in the compound-system architecture. Trigger phrases: "design the guardrails", "when does a human sign off", "how do we handle failures", "what confidence threshold", "where's the human in the loop".
| Not this skill | Use instead | Why |
|---|---|---|
| "What acceptable failure rate per mode / which autonomy level L0–L4" | eval-first-spec |
That derives the rate and picks the level from cost-of-one-failure. This skill consumes both and builds the mechanism that holds them. Run it first; it hands this the taxonomy and the rates. |
| "Design the whole AI system / the component pipeline / the effort split" | compound-system-architecture |
That names component 4 exists. This designs what is inside it. Compose: the architecture says "a guard sits here", this says "here is the guard, layer by layer, sized to cost × volume". |
| "We've held accuracy for a month — can we drop the human sign-off now" | refine-flywheel |
That promotes autonomy as an eval result, never by feel. This designs the guard at the current level; removing a layer as autonomy climbs is the flywheel's job, gated by a re-run of the eval set. |
| "Structure the pilot / price it / what's the commercial success metric" | pilot-six-term-sheet |
Commercial, not correctness. |
If no eval-first spec exists — no taxonomy, no cost-of-one-failure — do not invent them
to fill the matrix. Ask the one question that unblocks it (what does one failure cost,
and how often can it happen?) or route to eval-first-spec. A guardrail sized against
made-up costs is worse than none: it looks rigorous and guards nothing.
Method
Copy this checklist:
Guardrail Design progress:
- [ ] Step 1: Pull the inputs from eval-first-spec (taxonomy, level, cost, rate)
- [ ] Step 2: Band each mode — cost-of-error × volume → quadrant → guardrail class
- [ ] Step 3: Assign the three-layer stack per mode, in order, with a fail-mode
- [ ] Step 4: Derive the confidence threshold from the acceptable rate + calibration
- [ ] Step 5: Write a sign-off trigger for every high-cost / low-confidence path
- [ ] Step 6: Cost the stack and feed it back to cost-per-outcome
- [ ] Step 7: Run the kill line before returning
Step 1 — pull the inputs. From eval-first-spec: the failure taxonomy (the named
modes — Miss, False alarm, Confidently wrong, Wrong format, Too slow, Silent failure, plus
any product-specific), the chosen autonomy level L0–L4, the cost-of-one-failure per mode,
and the derived acceptable rate per mode. Tag each [Fact] / [Assumption]. If a mode
has no cost-of-one-failure, its guardrail cannot be sized — send it back, do not guess.
Step 2 — band and place. Give each mode a cost band and a volume band.
| Cost-of-error band | Test (loaded cost of one undetected failure) | Acceptable rate tends to |
|---|---|---|
| Catastrophic | Irreversible: safety, regulatory breach, data loss, or worth many outcomes | → 0 |
| High | Reversible but expensive: remediation, lost deal, trust; worth > one outcome | Tight |
| Low | A shrug: the user self-corrects; worth < one outcome | Loose |
| Volume band | Test (the operational line, not an arbitrary count) |
|---|---|
| Low | One reviewer can attend to every instance of this mode within its SLA without dropping the day job |
| High | Reviewing every instance is impossible or unaffordable; only the flagged subset gets a human |
There is no Medium band. A mode that feels "Medium" rounds up to High for stack-sizing, never down: if you cannot review every instance within its SLA, it is High — it gets the confidence gate + exception review, not a promise of full per-item review you cannot keep.
Place each mode on the matrix; read off its guardrail class:
| Volume Low (reviewable) | Volume High (not fully reviewable) | |
|---|---|---|
| Catastrophic | Q1 — code hard-block on the knowable-bad + human sign-off on the action; confidence never gates alone; fail-safe | Q2 — code hard-block + conservative confidence gate + human on flagged exceptions; monitor residual; fail-safe |
| High | Q3 — confidence gate + human sign-off below τ; fail-safe | Q4 — code rules + confidence gate, human on exceptions only, alert on rate breach; fail-safe on the action |
| Low | Q5 — confidence gate + log; no human; fail-open | Q6 — code rules only; accept residual; monitor aggregate; no per-item human; fail-open |
Two laws fall out of the matrix. First: a Catastrophic or High mode may never rest on model confidence alone — it carries a code hard-block, a human sign-off, or both. Second: a Low-cost mode carries no per-item human review — a sign-off you cannot afford is one you will quietly stop doing. If your grid puts the same stack in every cell, you have not designed guardrails; you have picked a habit.
Step 3 — the three-layer stack. Every output passes the layers in order, cheapest first. The matrix decides which layers are active for each mode; layer 1 always runs.
| Layer | What it is | Catches | Cost |
|---|---|---|---|
| 1. Rules in code | Deterministic checks: schema, ranges, hard limits, denylist of banned actions, idempotency, rate limit, PII redaction, heartbeat | The knowably-wrong, and Silent failure | ~free, runs on 100% |
| 2. Confidence threshold | An independent score per output; below τ → escalate or refuse | The uncertain | cheap; needs calibration |
| 3. Human sign-off | Route to a named person before the action commits | The costly-and-uncertain | expensive; ration it |
Layer 2's score must be independent of the reasoning model — a judge, or a historical pass-rate lookup — matching the architecture rule that component 4 cannot be component 3b. A model grading its own output is not a guardrail. State a fail-mode for each mode: fail-safe (block / hold / refuse on timeout or missing signal) for Catastrophic and High; fail-open (ship and log) is permitted only for Low.
Step 4 — derive the threshold. τ is the confidence score above which output auto-proceeds. It is derived, not chosen:
Set τ = the lowest score whose calibration bucket (and every bucket above it) shows a measured failure rate ≤ acceptable_rate(mode).
You need a calibration curve: bucket golden-case outputs by score, measure the actual pass
rate in each bucket. Without that curve, τ is [Hypothesis]; the safe interim is to route
the whole mode to human sign-off until the curve exists. A raw "the model says 90%" is
opinion (0.1 on the ladder) — the Confidently-wrong mode is exactly the model being sure
and wrong, so its self-report cannot be the guard against it.
Step 5 — the sign-off trigger. For every path that reaches human sign-off, fill a row: trigger (which mode × which confidence band), reviewer (one named role), SLA (how long they have), fail-mode (what happens on timeout). Every Catastrophic and High mode must have a trigger here, or the kill line fails.
Step 6 — cost the stack. flagged_volume × review_minutes × loaded_rate is a real line
in cost-per-outcome. If the guardrail blows the budget from eval-first-spec, the honest
outcomes are: tighten what gets flagged, lower the autonomy so fewer actions need guarding,
or concede the wedge is not economic. Do not silently widen τ to make review affordable —
that un-derives Step 4.
Step 7 — the kill line. Run the self-check at the foot of template.md.
Evidence standard
Money moved 1.0 → behaviour observed 0.7 → artefact shown 0.5 → verbal commitment 0.3 →
opinion 0.1. Here the ladder scores the two numbers the whole spec rests on: the
cost-of-error and the volume per mode, and the confidence calibration. A
cost-of-error argued from opinion ("that would be really bad", 0.1) cannot size a
guardrail; it needs an artefact or behaviour — a real incident, a regulation, a remediation
invoice, a measured event rate. The confidence threshold is [Fact] only when a
calibration curve backs it (behaviour, 0.7); a felt 0.8 is opinion and is flagged owed.
Tag every cost, rate, volume, and threshold [Fact] / [Assumption] / [Hypothesis]. A
guardrail spec is done when every Catastrophic/High mode has a code block or a sign-off
trigger, τ is derived not felt, and the stack's cost is written back into cost-per-outcome.
Gotchas
- The uniform stack. "Input validation + output validation + a human reviews it" applied identically to every output. It wastes sign-off on the Low-cost high-volume modes — so in practice that review silently gets skipped — and under-guards the Catastrophic case, where a schema check is not a sign-off. The matrix exists to break this habit; a grid with one class in every cell is the auto-fail.
- Uncalibrated confidence. Treating the model's self-reported certainty as the threshold. LLM self-confidence is not calibrated and is highest on the Confidently-wrong mode. A real τ needs an independent score measured against golden-case outcomes. No calibration curve → no threshold, only a number; route to human until the curve exists.
- Sign-off that fails open. The human is named as the guard, but on timeout the action ships anyway "so we don't block the user". On a Catastrophic or High mode that turns the guard into theatre. High cost fails safe — it holds — or it is not a guard.
- Rubber-stamp by over-flagging. Routing everything to a person does not make it safe; a
reviewer facing 500 approvals a shift approves by reflex, which is the L2 rubber-stamp failure
from
eval-first-spec. Over-flagging destroys the guard as surely as under-flagging. Size the flagged volume to what one reviewer can actually attend to within the SLA.
Examples
examples/sample.md — Barrier Intelligence's gas-safety alerting built as a guardrail spec: the counter-intuitive finding that for a safety product the risky auto-action is suppression, not alerting; a Miss placed Catastrophic × low-true-volume with a deterministic LEL hard-block that no confidence score may gate; a False alarm placed High × high-volume where a derived τ suppresses sensor noise but a code rule forbids auto-dismissing any soft anomaly in a manned zone; and a human sign-off, fail-safe, that a shift safety engineer must give before any potential hazard is permanently closed.
Related skills
eval-first-spec(07, sibling) — derives the acceptable failure rate per mode, the autonomy level, and the cost-of-one-failure. This skill consumes all three; it does not re-derive them. Its golden cases are the calibration set that Step 4's threshold is measured against.compound-system-architecture(07, sibling) — names component 4 (validate & guardrails) and the rule that it must be independent of the reasoning model. This skill is the depth build of that one component. The architecture says a guard sits there; this says what it is.refine-flywheel(08) — promotes autonomy as an eval result. When accuracy has been earned and a guard layer can come off, that decision is the flywheel's, gated by re-running the eval set — not this skill's, and never by feel.pilot-six-term-sheet(07, sibling) — the commercial contract. Guardrail cost lands in this skill's cost-per-outcome, which the term sheet prices against; keep the boundary.- Supersedes nothing. New skill; no prior guardrail spec exists in the pack to replace.