The-Utopia-Studio What it does
Turns a shipped product into a loop that compounds. It takes a v1 that is live behind a human sign-off gate and stands up four stations: ship (the current autonomy level and its gate), observe (instrument tweak-time — the edit distance between what the system proposed and what the human committed), learn (re-run the entire golden set from eval-first-spec on every model release and on a fixed floor cadence), and refine (move autonomy up or down strictly on the scored eval result). The artefact is the filled template.md: a cadence doc that names what runs, how often, who owns it, and the exact numbers that move autonomy. A loop that raises autonomy on a hunch is not this skill; it is the failure this skill exists to stop.
The Icarus reframe
The generic post-launch loop is build-measure-learn with an engagement dashboard: usage goes up and to the right, everyone nods, nothing about the product's trustworthiness is known. Icarus replaces the vanity metric with tweak-time — a behaviour signal (0.7 on the ladder), read from production traces, that measures how much a customer edits an output before accepting it; tweak-time falling to zero is the product earning the right to act. It replaces "ship it and move on" with a standing regression harness: a new model release is a regression risk until the eval set re-scores it, not a free upgrade, so the whole golden set re-runs before any model change reaches production. And it replaces the autonomy promotion-by-confidence with a ratchet driven by evidence in both directions — the L0–L4 level from eval-first-spec climbs one rung only when the eval re-run passes that rung's derived failure rates over N real cycles, and drops when a model release regresses them. Tweak-time nominates; the eval confirms; feelings never vote.
When to use / When NOT
Use it once a product is shipped or in a paid pilot, behind a sign-off gate, WITH an eval set and an autonomy ladder already produced by eval-first-spec. Trigger phrases: "what happens after launch", "improve the product", "set up the loop", "a new model came out — should we upgrade", "can we raise the autonomy level / drop the sign-off yet".
Hard precondition: no eval set, no flywheel. The learn station has nothing to re-run without the 20-case golden set, and the refine station has no ladder to climb without the L0–L4 levels. If those do not exist, stop and route to eval-first-spec. Do not invent an eval set here.
Do not use it for:
| Request | Belongs to |
|---|---|
| "Which metrics / what's our North Star / does retention flatten / cost-per-outcome dashboard" | metrics-that-matter (08, sibling). It picks the metric set; this builds the ship→observe→learn→refine cadence and the autonomy ratchet. This skill reads tweak-time and cost-per-outcome; that skill decides the wider scorecard. Route metric selection there. |
| "Turn these production traces into interviews / what unmet need do the logs reveal" | trace-to-interview (08, sibling). It mines the qualitative discovery question from one trace; this reads traces as a quantitative behaviour signal (tweak-time) and as the eval-regression check. Same raw feed, different question. |
| "Write the golden cases / set the autonomy level / define what working means" | eval-first-spec (07). It creates the eval set and ladder before build; this re-runs them after launch. If there is no eval set, you are here too early. |
| "Keep the opportunity tree alive / set up continuous discovery" | continuous-discovery-engine (02). That is the discovery-side loop; this is the delivery/quality-side loop. They run beside each other (dual-track) and share the trace feed; neither owns the other. |
Method
Fill in template.md. Check the precondition, then the four stations, then the compounding check and the kill line.
Station 0 — Precondition gate
Before anything, confirm three things exist. If any is missing, the flywheel cannot turn; name the gap and route.
| Precondition | If missing |
|---|---|
A golden set (the 20+ pass/fail cases from eval-first-spec) |
Route to eval-first-spec. There is nothing to re-run. Stop. |
| A chosen autonomy level L0–L4 with per-mode acceptable rates | Route to eval-first-spec. There is no ladder to climb. Stop. |
| The product is live behind a human sign-off gate | If it ships with no sign-off, that is a guardrail-design gap, not a flywheel. |
Station 1 — Ship (behind sign-off)
State the starting position, not the destination. Record the current autonomy level (usually L1 Drafts or L2 Acts-on-approval — the level eval-first-spec shipped) and the sign-off gate: every output is reviewed and committed by a named human before it acts. The flywheel's whole job is to earn the right to loosen this gate, one measured rung at a time. Autonomy starts low on purpose.
Station 2 — Observe (tweak-time)
Instrument one behaviour: tweak-time — how much the human edits the system's proposed output before accepting it. Read it from the production trace, not a survey. Track three numbers per output type:
- clean-accept rate: share of outputs committed with zero edits
[Fact, from traces] - median edit fraction: how much of the accepted output was changed
- the trend over weeks: falling, flat, or rising
Read the bands like this:
| Clean-accept rate | Trend | Reading and action |
|---|---|---|
| any | falling over weeks | The product is regressing or drifting. Freeze autonomy, run the learn station now, find the cause before anything else. |
| < 30% | flat | Still a draft tool. Hold autonomy. Cluster the edits — a systematic edit is a missing feature or a wrong default, not noise. |
| 30–70% | rising | Earning trust. Hold the level. Watch which failure modes sit behind the remaining edits. |
| > 70% | sustained over the level's N cycles | Nominates a level-up. It does not grant one. Carry the nomination to Station 4; the eval decides. |
Tweak-time is a nomination, never a promotion. Everything else (retention, cost-per-outcome, North Star) is metrics-that-matter's scorecard — reference it, do not rebuild it here.
Station 3 — Learn (re-run the WHOLE eval set)
This is the station the kill line guards. Re-run every golden case, not a sample, on any of these triggers:
| Trigger | Why it is mandatory |
|---|---|
| Any model version change (provider release, model swap) | A new model can silently regress a case that passed last week. "Obviously better" is not a measurement. Re-run before it reaches production. |
| Any change to the core prompt / system message | Same input, changed reasoning, unknown result. |
| Any change to retrieval or the corpus | The evidence the model reasons over moved. |
| A fixed floor cadence (monthly) with no change at all | Catches drift, data-distribution shift, and provider-side silent updates. |
On each re-run, log to a standing eval ledger and compare per-mode pass rates against two baselines: the last run, and eval-first-spec's derived acceptable_rate(mode).
| Date | Model / change | Miss | False alarm | Confidently wrong | ... | Δ vs last | Action |
|---|---|---|---|---|---|---|---|
| [date] | [what changed] | [rate] | [rate] | [rate] | [better/worse] | [ship / hold / demote] |
The regression rule overrides aggregate improvement: a regression on any safety-critical mode (a Miss in gas safety, a Confidently-wrong in a decision that moves money) blocks the release or demotes autonomy even if the overall pass rate went up. Net-better does not buy a safety regression.
Station 4 — Refine (promote autonomy by eval, never by feel)
Move autonomy along the eval-first-spec ladder using the scored eval, gated by tweak-time. One rung at a time; no skipping.
| Condition | Autonomy action |
|---|---|
| Tweak-time nominates (>70% clean-accept sustained N cycles) AND an eval re-run at the next level's acceptable rates passes over N real cycles, no safety-mode regression | Promote one rung |
| Eval re-run holds the current level's rates | Hold |
Eval re-run regresses below the current level's acceptable_rate (common after a model release) |
Demote / re-gate behind sign-off until a passing re-run |
| Tweak-time nominates but no eval was re-run at the next level | Hold. The nomination is 0.7 evidence for a hypothesis; the promotion needs the scored eval. Feel is not a vote. |
N is the same "gate to earn the next level" that eval-first-spec Part 3 already set (e.g. Miss ≤ 3% over 40 real shifts). Do not re-derive it; carry it forward.
Compounding check and kill line
A loop is only a flywheel if each turn lowers cost_per_outcome (from eval-first-spec Part 4) — fewer human review minutes as autonomy rises, or fewer retries as quality rises — freeing margin for the next discovery cycle. If cost-per-outcome is flat turn over turn while usage climbs, it is a loop spinning in place. Name it.
Kill line. The cadence doc ships only if (1) the learn station re-runs the whole eval set on every model release, and (2) every autonomy change is tied to a scored eval result. A loop that promotes autonomy by feel, or that upgrades a model with no re-run, is auto-fail. Report the gap; do not dress a vanity loop as a flywheel.
Evidence standard
This skill weights behaviour and money over opinion. The ladder: money moved 1.0 → behaviour observed 0.7 → artefact shown 0.5 → verbal commitment 0.3 → opinion 0.1.
| Signal in this loop | Rung | Use |
|---|---|---|
| Cost-per-outcome fell / customer expanded spend | 0.7–1.0 | Proves the flywheel compounds. |
| Tweak-time (clean-accept rate from traces) | 0.7 | Nominates an autonomy level-up. Never grants it. |
| Eval re-run pass rate on the real golden set | 0.5–0.7 | The only signal that moves autonomy. |
| "The team feels it's ready" / "users like it" | 0.1 | Cannot move autonomy. Named as opinion, parked. |
An autonomy promotion must rest on the scored eval (≥ 0.5). No promotion is ever justified by the 0.1 "feels reliable now." Tag every rate [Fact] (measured on real traces or the real golden set) or [Assumption] (projected until a real re-run confirms it).
Gotchas
Promoting on tweak-time alone. A 90% clean-accept rate feels like proof, but it is a proxy. Until the golden set is re-scored at the higher level's acceptable rates, the level is not earned. Tweak-time nominates; the eval confirms. Skipping the eval is promoting by feel with a number attached.
Treating a model upgrade as a free win. Skipping the re-run because the new model is "obviously better" is the exact failure the kill line names. A silent regression on one mode can outweigh every aggregate gain, and you will not see it without re-running all 20 cases. Newer is unknown, not better, until scored.
Tweak-time survivorship at higher autonomy. You only observe edits on outputs a human reviewed. At L3+ the un-reviewed outputs generate no tweak-time signal — so the metric blinds you exactly where cost-of-error is highest. Above L2, supplement tweak-time with the flagged-exception review rate and periodic audits; do not trust the clean-accept rate alone.
Clean-accept gamed by emptier output. If the model learns to hedge — vaguer, safer outputs a human cannot disagree with — tweak-time falls for the wrong reason. Guard: the golden set's must-refuse and adversarial bands must still pass. A rising clean-accept rate with a falling golden pass rate is drift, not trust.
Vanity loop. Usage up, edits still 100%, is a product people are forced to use, not one they trust. If tweak-time and cost-per-outcome are flat, "engagement is growing" is noise.
Examples
examples/sample.md — the post-launch flywheel for Mentix's shift-handover machine-risk digest, picking up exactly where eval-first-spec shipped it at L1: tweak-time instrumented on the digests (clean-accept climbing 41% → 78%), a model-release re-run that regresses the Miss rate and correctly holds autonomy despite a higher aggregate score, and an L1→L2 promotion granted only after the re-scored eval clears the derived gate — with cost-per-outcome falling $3.55 → $2.90/shift as the compounding proof.
Related skills
eval-first-spec (07) — creates the golden set and the L0–L4 ladder that this skill re-runs and climbs. Upstream, once, before build. If it has not run, this skill has no inputs. They are two ends of one asset: that skill writes the eval, this one keeps it alive.
metrics-that-matter (08, sibling) — owns the metric scorecard (AARRR, the retention curve that must flatten, the North Star, cost-per-outcome to the cent). This skill consumes cost-per-outcome and tweak-time; it does not choose the scorecard. Reference it for retention and North Star; do not restate the M9 retention pack.
trace-to-interview (08, sibling) — turns one production trace into a discovery interview. Same raw feed as this skill's observe station, opposite question: it asks "what unmet need does this trace reveal", this asks "is the product earning autonomy". Route the qualitative discovery question there.
continuous-discovery-engine (02) — the discovery-side loop (the living opportunity tree). This is the delivery-side loop. They run dual-track and share the trace feed; keep the boundary.
Supersedes: none. New skill; no prior post-launch flywheel exists in the pack to replace.