# Measuring Model Limits

> Use when building or fixing an RL environment, eval task, or agent benchmark - the entry point that routes to world-building, task-authoring, gating, rollout analysis, and difficulty iteration. Enforces the rule that only a model failure may ship.

- Skill: `punitarani/measuring-model-limits` (Agent Skill)
- Install (CLI): `npx skillmds@latest add punitarani/measuring-model-limits`
- Raw SKILL.md: https://api.skillmd.com/api/skills/punitarani/measuring-model-limits/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: punitarani (https://skillmd.com/u/punitarani)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/punitarani/measuring-model-limits

---


# Measuring a model, not your environment

An environment exists to measure a model. Every point a task takes away
is a claim about the model, and the claim is only honest if the model is
what took it away.

This skill is the index. Load the one that matches your phase.

| phase | skill |
|---|---|
| generating a world, its records, its documents | `building-simulated-worlds` |
| keeping a multi-hour recording alive without corrupting it | `running-recorded-simulations` |
| checking a task idea against what the world actually holds | `validating-task-premises` |
| turning a surviving premise into a graded task | `authoring-graded-tasks` |
| proving the task is fair before spending a rollout | `gating-environment-quality` |
| reading trials and classifying what went wrong | `analyzing-rollout-failures` |
| the score is out of band and you need to move it | `iterating-task-difficulty` |

## The one rule everything else serves

Five things can take a point away. Only the last may ship.

| class | what it means | what it demands |
|---|---|---|
| **E — environment** | the fact is not reachable through the served surface, or the record contradicts itself with no stated precedence | postmortem; the environment is wrong |
| **D — data** | the world's content is incoherent — a document whose body contradicts its registered name, an entry booked against the wrong parent | postmortem; the world is wrong |
| **H — harness** | the agent never got to answer: incompatible tool bridge, rate limit, setup timeout, abandoned delegation | not a score at all; fix or discard |
| **T — task** | ambiguous instruction, wrong oracle, wrong grader, unfair tolerance | fix and re-measure |
| **M — model** | reachable, unambiguous, oracle independently confirmed, and it got it wrong anyway | **the only one that may ship** |

These are not equally likely. On a mature environment **T is the most
common and the easiest to mistake for M**, because a task defect and a
model failure both present as a number below 1.0.

## The decisive question

> **A defect blocks every trial. Difficulty just makes most of them miss.**

Before calling any miss M, ask whether *any* trial of *any* model found
that row. If one did, the row is findable and the rule is applicable —
whatever the others did is theirs. If none did across many trials,
suspect yourself first.

## The loop

1. **Record a world.** Deterministic, fails loudly, coherence green. A
   recording measured in hours needs a supervisor that resumes rather
   than restarts, and a freeze on anything that changes what is being
   written — see `running-recorded-simulations`.
2. **Rebuild derived state wholesale** and reconcile counts against the
   source of truth.
3. **Measure the premise before cutting the task.** Does the pattern
   occur at all, is it spread rather than piled at one edge of the
   window, can the agent reach it through a tool, and does the rule
   discriminate? Premises written from a reasonable picture of the domain
   fail often enough that this pays for itself immediately, and a starved
   premise produces rows nobody can find — which reads as model failure.
   See `validating-task-premises`.
4. **Cut a task.** Brief, stated rule, declared rule-kind, structural
   floors met.
5. **Gate it.** Falsify each gate once — break the thing deliberately and
   confirm the gate fails.
6. **Measure at k≥3, and k=9 wherever completion is unreliable.** Read
   every trial log before recording a number.
7. **Classify every miss.** Anything not M is a defect: fix it and return
   to step 5.
8. **When the numbers move, re-derive the queue from the blocking
   condition** rather than extending the previous plan. Long asynchronous
   work drifts silently: nothing errors, and the runs keep measuring
   something you have stopped needing.
9. **Write down what generalizes** — as guidance, not as a run record.

## The laws are written down; read them before designing

A repository practising this method should carry a `docs/LAWS.md`: one
numbered entry per measured law, each with the number behind it and the
decision it would have changed. A candidate becomes a law when it has both;
without them it is a note and belongs in a skill.

Read it before designing a task, changing a grader, or believing a score.
Add to it whenever something is paid for. When a law is DISPROVED, strike
it and record what replaced it rather than deleting it — the reasoning that
made it plausible will recur, and a struck law is the cheapest way to stop
it recurring twice.

## Where findings go

A finding that generalizes is not a run record. If a measurement changes
how the *next* environment should be built, it belongs in the guidance
tree where the next build will read it. Filing it under a dated run
folder means re-deriving it later at full cost.

Record falsified hypotheses with their measurements, including the ones
that make a headline result smaller. Correct earlier conclusions **in
place** rather than appending a softening note. And date difficulty
claims separately from defect claims: a defect is permanent once found,
while a difficulty claim is only true of the tiers that were measured.