Measuring a model, not your environment
An environment exists to measure a model. Every point a task takes away
is a claim about the model, and the claim is only honest if the model is
what took it away.
This skill is the index. Load the one that matches your phase.
| phase |
skill |
| generating a world, its records, its documents |
building-simulated-worlds |
| keeping a multi-hour recording alive without corrupting it |
running-recorded-simulations |
| checking a task idea against what the world actually holds |
validating-task-premises |
| turning a surviving premise into a graded task |
authoring-graded-tasks |
| proving the task is fair before spending a rollout |
gating-environment-quality |
| reading trials and classifying what went wrong |
analyzing-rollout-failures |
| the score is out of band and you need to move it |
iterating-task-difficulty |
The one rule everything else serves
Five things can take a point away. Only the last may ship.
| class |
what it means |
what it demands |
| E — environment |
the fact is not reachable through the served surface, or the record contradicts itself with no stated precedence |
postmortem; the environment is wrong |
| D — data |
the world's content is incoherent — a document whose body contradicts its registered name, an entry booked against the wrong parent |
postmortem; the world is wrong |
| H — harness |
the agent never got to answer: incompatible tool bridge, rate limit, setup timeout, abandoned delegation |
not a score at all; fix or discard |
| T — task |
ambiguous instruction, wrong oracle, wrong grader, unfair tolerance |
fix and re-measure |
| M — model |
reachable, unambiguous, oracle independently confirmed, and it got it wrong anyway |
the only one that may ship |
These are not equally likely. On a mature environment T is the most
common and the easiest to mistake for M, because a task defect and a
model failure both present as a number below 1.0.
The decisive question
A defect blocks every trial. Difficulty just makes most of them miss.
Before calling any miss M, ask whether any trial of any model found
that row. If one did, the row is findable and the rule is applicable —
whatever the others did is theirs. If none did across many trials,
suspect yourself first.
The loop
- Record a world. Deterministic, fails loudly, coherence green. A
recording measured in hours needs a supervisor that resumes rather
than restarts, and a freeze on anything that changes what is being
written — see
running-recorded-simulations.
- Rebuild derived state wholesale and reconcile counts against the
source of truth.
- Measure the premise before cutting the task. Does the pattern
occur at all, is it spread rather than piled at one edge of the
window, can the agent reach it through a tool, and does the rule
discriminate? Premises written from a reasonable picture of the domain
fail often enough that this pays for itself immediately, and a starved
premise produces rows nobody can find — which reads as model failure.
See
validating-task-premises.
- Cut a task. Brief, stated rule, declared rule-kind, structural
floors met.
- Gate it. Falsify each gate once — break the thing deliberately and
confirm the gate fails.
- Measure at k≥3, and k=9 wherever completion is unreliable. Read
every trial log before recording a number.
- Classify every miss. Anything not M is a defect: fix it and return
to step 5.
- When the numbers move, re-derive the queue from the blocking
condition rather than extending the previous plan. Long asynchronous
work drifts silently: nothing errors, and the runs keep measuring
something you have stopped needing.
- Write down what generalizes — as guidance, not as a run record.
The laws are written down; read them before designing
A repository practising this method should carry a docs/LAWS.md: one
numbered entry per measured law, each with the number behind it and the
decision it would have changed. A candidate becomes a law when it has both;
without them it is a note and belongs in a skill.
Read it before designing a task, changing a grader, or believing a score.
Add to it whenever something is paid for. When a law is DISPROVED, strike
it and record what replaced it rather than deleting it — the reasoning that
made it plausible will recur, and a struck law is the cheapest way to stop
it recurring twice.
Where findings go
A finding that generalizes is not a run record. If a measurement changes
how the next environment should be built, it belongs in the guidance
tree where the next build will read it. Filing it under a dated run
folder means re-deriving it later at full cost.
Record falsified hypotheses with their measurements, including the ones
that make a headline result smaller. Correct earlier conclusions in
place rather than appending a softening note. And date difficulty
claims separately from defect claims: a defect is permanent once found,
while a difficulty claim is only true of the tiers that were measured.
1---2name: measuring-model-limits3description: Use when building or fixing an RL environment, eval task, or agent benchmark - the entry point that routes to world-building, task-authoring, gating, rollout analysis, and difficulty iteration. Enforces the rule that only a model failure may ship.4---56# Measuring a model, not your environment78An environment exists to measure a model. Every point a task takes away9is a claim about the model, and the claim is only honest if the model is10what took it away.1112This skill is the index. Load the one that matches your phase.1314| phase | skill |15|---|---|16| generating a world, its records, its documents | `building-simulated-worlds` |17| keeping a multi-hour recording alive without corrupting it | `running-recorded-simulations` |18| checking a task idea against what the world actually holds | `validating-task-premises` |19| turning a surviving premise into a graded task | `authoring-graded-tasks` |20| proving the task is fair before spending a rollout | `gating-environment-quality` |21| reading trials and classifying what went wrong | `analyzing-rollout-failures` |22| the score is out of band and you need to move it | `iterating-task-difficulty` |2324## The one rule everything else serves2526Five things can take a point away. Only the last may ship.2728| class | what it means | what it demands |29|---|---|---|30| **E — environment** | the fact is not reachable through the served surface, or the record contradicts itself with no stated precedence | postmortem; the environment is wrong |31| **D — data** | the world's content is incoherent — a document whose body contradicts its registered name, an entry booked against the wrong parent | postmortem; the world is wrong |32| **H — harness** | the agent never got to answer: incompatible tool bridge, rate limit, setup timeout, abandoned delegation | not a score at all; fix or discard |33| **T — task** | ambiguous instruction, wrong oracle, wrong grader, unfair tolerance | fix and re-measure |34| **M — model** | reachable, unambiguous, oracle independently confirmed, and it got it wrong anyway | **the only one that may ship** |3536These are not equally likely. On a mature environment **T is the most37common and the easiest to mistake for M**, because a task defect and a38model failure both present as a number below 1.0.3940## The decisive question4142> **A defect blocks every trial. Difficulty just makes most of them miss.**4344Before calling any miss M, ask whether *any* trial of *any* model found45that row. If one did, the row is findable and the rule is applicable —46whatever the others did is theirs. If none did across many trials,47suspect yourself first.4849## The loop50511. **Record a world.** Deterministic, fails loudly, coherence green. A52 recording measured in hours needs a supervisor that resumes rather53 than restarts, and a freeze on anything that changes what is being54 written — see `running-recorded-simulations`.552. **Rebuild derived state wholesale** and reconcile counts against the56 source of truth.573. **Measure the premise before cutting the task.** Does the pattern58 occur at all, is it spread rather than piled at one edge of the59 window, can the agent reach it through a tool, and does the rule60 discriminate? Premises written from a reasonable picture of the domain61 fail often enough that this pays for itself immediately, and a starved62 premise produces rows nobody can find — which reads as model failure.63 See `validating-task-premises`.644. **Cut a task.** Brief, stated rule, declared rule-kind, structural65 floors met.665. **Gate it.** Falsify each gate once — break the thing deliberately and67 confirm the gate fails.686. **Measure at k≥3, and k=9 wherever completion is unreliable.** Read69 every trial log before recording a number.707. **Classify every miss.** Anything not M is a defect: fix it and return71 to step 5.728. **When the numbers move, re-derive the queue from the blocking73 condition** rather than extending the previous plan. Long asynchronous74 work drifts silently: nothing errors, and the runs keep measuring75 something you have stopped needing.769. **Write down what generalizes** — as guidance, not as a run record.7778## The laws are written down; read them before designing7980A repository practising this method should carry a `docs/LAWS.md`: one81numbered entry per measured law, each with the number behind it and the82decision it would have changed. A candidate becomes a law when it has both;83without them it is a note and belongs in a skill.8485Read it before designing a task, changing a grader, or believing a score.86Add to it whenever something is paid for. When a law is DISPROVED, strike87it and record what replaced it rather than deleting it — the reasoning that88made it plausible will recur, and a struck law is the cheapest way to stop89it recurring twice.9091## Where findings go9293A finding that generalizes is not a run record. If a measurement changes94how the *next* environment should be built, it belongs in the guidance95tree where the next build will read it. Filing it under a dated run96folder means re-deriving it later at full cost.9798Record falsified hypotheses with their measurements, including the ones99that make a headline result smaller. Correct earlier conclusions **in100place** rather than appending a softening note. And date difficulty101claims separately from defect claims: a defect is permanent once found,102while a difficulty claim is only true of the tiers that were measured.
punitarani