# Iterating Task Difficulty > Use when an eval task scores at ceiling or out of its target band and you need to move it - covers which difficulty levers are measured to do nothing, the coverage-versus-rule distinction, and which levers are forbidden. Load before changing a task to change its score. - Skill: `punitarani/iterating-task-difficulty` (Agent Skill) - Install (CLI): `npx skillmds@latest add punitarani/iterating-task-difficulty` - Raw SKILL.md: https://api.skillmd.com/api/skills/punitarani/iterating-task-difficulty/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Productivity - Author: punitarani (https://skillmd.com/u/punitarani) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/punitarani/iterating-task-difficulty --- # Moving a score honestly ## The theorem to plan around > Expert-solvable ⇒ a rule exists ⇒ a frontier model applies it. This is a definition, not pessimism. "Expert-solvable" means a competent professional could produce the answer from a stated rule. A frontier model applies stated rules at roughly 93% per row and does not degrade with row count. The only ways below that ceiling are to remove the rule — which destroys expert-solvability — or to grade all-or-nothing, which converts the measurement into a coin flip. **Difficulty targets should name a capability tier, not a number in the abstract.** ## A floor is the first thing to move, before any lever A task cannot be hard for a model if doing nothing already scores in the band. Measure that before designing difficulty at all: across one shipped dataset's fifteen keyed tasks, an **empty register with correct scalars scored a median 0.405**, and "report every candidate" reached **0.990** on one of them. Every band judgement made there had been made without a floor, because the tool that computes floors returned an empty dict for that dataset's task shape and an empty result reads as *no floors exist* rather than *this function cannot see them*. The consequence for a three-model mean: on the three tasks that dataset called in-band, **of six non-frontier scores, three sit inside the range a no-comprehension dump produces, two sit 0.01–0.06 above it, and one sits below** — with the frontier model at 1.000 on all three. The band was one model at ceiling averaged with two scoring where dumping already scores. So the order is: floor first, then difficulty. A lever applied to a task whose floor is 0.9 buys nothing. ## Levers measured to do nothing Each was built, measured against a frontier model, and came back at ceiling. Do not spend a build on them again without a reason this list does not cover. | lever | how it was tested | result | |---|---|---| | volume | contested rows doubled at equal judgment depth | 0.84 → **0.81** | | depth | four independent judgments per row, traps on 19 of 30 | 0.84 → **0.87** (wrong way) | | width | 1,300 entries, 27 pages, ~200 rows | ceiling | | coverage | rows 189 → 507, corpus 328 → 1,547 messages | score went **up** | | correlated error | a task built for it | 1.000 in 26 shell commands | | lexical near-miss | 171 near-miss temptations | 1.000 | | semantic synonym | 70 synonyms excluded by rule | 1.000 | | chained derivation | three dependent steps per row | **403 of 403** correct | | office files | 19 workbooks, 61 sheets, no index, nothing queryable | 1.000 | The mechanism predicts the next failed lever: > **Deterministic gradeability implies programmatic solvability.** The oracle is a program, so a task with a deterministic answer key is by construction reducible to a program — and an agent with a shell will write one. Scale is then a *cost*, not a difficulty. What all those levers share: the agent computes each row locally and mechanically from text already pulled onto disk. Against a written script, per-row rules are free however long the chain, and independent errors average out instead of compounding. ## The distinction that does work Two kinds of difficulty, behaving oppositely under bounding. **Coverage difficulty** — did the agent enumerate the corpus? Bimodal: 1.000 or ~0.3, decided by whether it finished. It is *luck*, it produces unstable means, and it **disappears** the moment the corpus is small enough to finish. **Rule difficulty** — did the agent apply the stated rule to text it has already read? A **rate**. One model read 1,574 of 1,585 messages — 99.3% coverage — and still found 48 of 110 rows, catching 23 of 82 occurrences of a single common word. **A rate does not care how much text there is**, so it survives bounding. This is why "make the corpus small enough that every tier finishes" and "keep it hard" are not in conflict, once you know which kind you have. ### Why the distinction stays hidden A frontier model has neither problem, so coverage and rule application look identical when a model does both perfectly. **Any difficulty conclusion drawn from a single frontier model is drawn from a sample that could not have shown otherwise.** Measure at least one tier that fails, or you are measuring your own ceiling. ## Confusability beats length — and the operative variable is off-sense share The hardest rule measured was the *shortest*: two spellings of one common word, sitting a hair from its own inflections, missed at 70% by a model that had read the text. A seven-form list of distinctive date patterns scored far higher for every tier. **Fewer forms, more temptation.** **Corrected by measurement.** The natural reading of that result — pick the family with the densest *excluded* near-misses, the inflections a careless matcher would over-admit — is wrong, and it is wrong in a way that wastes builds. A machine matching a word boundary is never confused by a neighbouring inflection; only a human or a model reading for *meaning* is. What actually predicts the miss is the **off-sense share of the admitted form**: how often the required word appears in the corpus meaning something other than the thing the register is named after. In the family that produced the hardest measured task, a majority of occurrences of the admitted word are adjectival (*the complete picture*, *the complete, dated calendar*), idiomatic, future (*I can typically complete this analysis*), or conditional (*once that call is complete*) — measured at 79% inside the graded window. Those are precisely the rows the weaker tiers dropped: a model reading for sense filters them out, and the rule says they count. So the selection metric is: **for each candidate family, hand-classify a sample of admitted-form occurrences as on-sense or off-sense, and choose the highest off-sense share.** Exclusion density is a decoy — and worse, it is easy to satisfy with a family that is dead on the actual corpus. Check both forms are alive before anything else: one family that looked ideal on paper had its second spelling appear in a single message out of 1,585, and another's in none at all. Two shapes with the same root, both measured: - **A form inside a longer phrase**, where the rule admits it and a reader hears a hedge (`within a day` inside *"within a day or two"*). - **One sentence carrying two forms that resolve to different values**, where the second reads as an explanation of the first. Every trial found the first; two of nine found the second. ## Adding a graded fact widens the gap, it does not lower the mean Grading "which rule produced this row" cost the weaker tier 0.05 and the stronger tier nothing — the stronger model had to match the form in order to emit the row at all, so naming it was free. Extra graded facts are useful for **discrimination between tiers**, and nearly useless for pulling a mean down. ## Do the band's arithmetic before designing for it A three-model mean with a frontier tier pinned at 1.000 is not a target on all three models — it is a budget on the other two. For a ceiling of 0.8: mean = (1.000 + a + b) / 3 <= 0.80 <=> a + b <= 1.40 Every task measured in band on one suite has the frontier at 1.000; what separated in from out was entirely the weaker pair's sum, and the boundary is about 0.05 wide. Three in-band tasks summed 0.73, 1.32 and 1.38; the one that missed summed 1.43. Two consequences worth internalising before a build: - **"A frontier model would score 1.000" is not, by itself, a rejection.** It is the expected case, and the design question is what the *named* weaker tiers do. Treat frontier 1.000 as a defect only when the frontier *misses* — because then the miss is far more likely to be a task defect than a capability limit. - **The margin is thin, so measure the mechanism before building.** At a 0.05-wide boundary, an unmeasured guess about difficulty is a coin flip, and the honest response to landing at 0.81 is to report it rather than to lever it. ## Joint dependency is the lever; row count is not At a per-extraction accuracy `a`, a row that turns on ONE extraction is right with probability `a`, and F1 over such rows is `a` whether there are 26 of them or 117. Adding rows adds work, not difficulty. A row that needs `k` facts is right with `a**k`. That is the whole lever, and it is the only one measured on this tree to move a frontier model off ceiling. **Measured.** A month-end snapshot design was costed at 117 rows against 26, with 67% of the added rows new-or-changed, and estimated to move the score by nothing — each snapshot row still hinged on a single statement. Discarded before it was built. What did move it: adding the chain's other END to a key already holding its last statement and its length (0.817 → 0.706), and adding a figure that needs EVERY link rather than the ends (0.852 → 0.755). **Pick facts that fail independently.** Both ends of a chain and its length are three different reads of the same chain: a reader who finds one end has none of the other two, and none is derivable from the others. Three facts that all fall out of one lookup give you `a`, not `a**3`. ## Put the hard fact in the KEY, not in a field Same fact, same task, same window: keyed (owner, meeting), date a field row_f1 1.000 keyed (owner, meeting, due) row_f1 0.179 A wrong key means the row is not matched at all — every field on it misses AND the invented row draws the extra-row penalty. In a field it degrades the score by one part in N. Key placement is the strongest dial available and it costs nothing. ## Choose the key by watching the dump floor, not by arguing Before believing a key, measure what a reader who reports every candidate without comprehending anything scores. **Measured twice in one day.** A slippage register keyed on `(owner, meeting, due, slips)` paid a dump **0.426** — inside the band, with half the target range sitting above a strategy that never read a transcript, because `slips` is 0 for half the rows and guessing zero is worth half the column. Adding `first_due` took it to **0.000**: a dump can see the statements in front of it, but it cannot know where a chain STARTED. A floor inside the band is a design defect, not a note. ## A short window has no chain to reconstruct Before concluding a supersession task is at ceiling, measure how often the thing being superseded actually moves. **Measured:** at 42 days the median person revises **once** and the strongest tier scores 1.000; at 147 days the median is 3.5. Same task, same rule, same grader. A task was nearly redesigned when the window was the whole problem — and lengthening it also dropped the dump floor from 0.372 to 0.000. ## Legitimate levers - how much of the record a task covers - how many independent facts per row - how confusable the rule is - which capability tier the target is set against ## Forbidden levers - tightening a tolerance - reweighting criteria so the same work scores less - withholding a rule, or trick wording - grading a fact the surface does not serve The test: **change what the agent must do, never what the same work is worth.** A task that scores lower without being harder is a scoring artifact, and the "calibration" is a property of your grader rather than of the world. ## When a task sits just outside the band Say so. A task that measures the same value across three independent methodological corrections is a stable measurement of something just outside the target — not a task fighting noise. Report it out rather than adjusting until it isn't. And if you do apply a lever, **declare it before seeing which way it moves the result.** ## A band can be manufactured entirely by the answer key The most expensive way to be wrong about difficulty: a task sits at 0.33-0.51 across three model families, looks perfectly calibrated, and is measuring its own defects. The tell is available before any lever is chosen. **Decompose the loss per criterion.** If a scalar reads 0.0 for every trial of every model while the counts-of-what-was-read read 1.0, the models opened the whole corpus and disagree with the key, not with each other. On one task `superseded_count` and `distinct_owners` were 0.0 in nine of nine trials while `meetings_read` and `turns_read` were exact. Then check recall against the subset of the key you can defend by reading the source. the strongest tier scored 10/10, 9/10, 10/10 there while its graded `live.f1` read 0.53-0.59 — and it was 100% correct on every evidence field of every row it matched. The reward said 0.508; the model was at ceiling. **Correct the key before choosing a lever**, even though correcting it raises the scores — that is the point. A lever applied on top of a broken key tunes the defect. And be ready for the honest outcome: once corrected, that task ran 0.838/0.838/0.838 for the frontier tier. Identical three times, one genuine row error, the rest an exact-match integer. The apparent band was gone and the real difficulty was small. **Report that rather than re-widening the window to recover a number** — the window levers under a word ceiling move supersession density by fractions, and coverage was already measured not to survive.