Iterating Task Difficulty

Moving a score honestly

The theorem to plan around

Expert-solvable ⇒ a rule exists ⇒ a frontier model applies it.

This is a definition, not pessimism. "Expert-solvable" means a competent professional could produce the answer from a stated rule. A frontier model applies stated rules at roughly 93% per row and does not degrade with row count. The only ways below that ceiling are to remove the rule — which destroys expert-solvability — or to grade all-or-nothing, which converts the measurement into a coin flip.

Difficulty targets should name a capability tier, not a number in the abstract.

A floor is the first thing to move, before any lever

A task cannot be hard for a model if doing nothing already scores in the band. Measure that before designing difficulty at all: across one shipped dataset's fifteen keyed tasks, an empty register with correct scalars scored a median 0.405, and "report every candidate" reached 0.990 on one of them. Every band judgement made there had been made without a floor, because the tool that computes floors returned an empty dict for that dataset's task shape and an empty result reads as no floors exist rather than this function cannot see them.

The consequence for a three-model mean: on the three tasks that dataset called in-band, of six non-frontier scores, three sit inside the range a no-comprehension dump produces, two sit 0.01–0.06 above it, and one sits below — with the frontier model at 1.000 on all three. The band was one model at ceiling averaged with two scoring where dumping already scores.

So the order is: floor first, then difficulty. A lever applied to a task whose floor is 0.9 buys nothing.

Levers measured to do nothing

Each was built, measured against a frontier model, and came back at ceiling. Do not spend a build on them again without a reason this list does not cover.

The mechanism predicts the next failed lever:

Deterministic gradeability implies programmatic solvability.

The oracle is a program, so a task with a deterministic answer key is by construction reducible to a program — and an agent with a shell will write one. Scale is then a cost, not a difficulty.

What all those levers share: the agent computes each row locally and mechanically from text already pulled onto disk. Against a written script, per-row rules are free however long the chain, and independent errors average out instead of compounding.

The distinction that does work

Two kinds of difficulty, behaving oppositely under bounding.

Coverage difficulty — did the agent enumerate the corpus? Bimodal: 1.000 or ~0.3, decided by whether it finished. It is luck, it produces unstable means, and it disappears the moment the corpus is small enough to finish.

Rule difficulty — did the agent apply the stated rule to text it has already read? A rate. One model read 1,574 of 1,585 messages — 99.3% coverage — and still found 48 of 110 rows, catching 23 of 82 occurrences of a single common word. A rate does not care how much text there is, so it survives bounding.

This is why "make the corpus small enough that every tier finishes" and "keep it hard" are not in conflict, once you know which kind you have.

Why the distinction stays hidden

A frontier model has neither problem, so coverage and rule application look identical when a model does both perfectly. Any difficulty conclusion drawn from a single frontier model is drawn from a sample that could not have shown otherwise. Measure at least one tier that fails, or you are measuring your own ceiling.

Confusability beats length — and the operative variable is off-sense share

The hardest rule measured was the shortest: two spellings of one common word, sitting a hair from its own inflections, missed at 70% by a model that had read the text. A seven-form list of distinctive date patterns scored far higher for every tier. Fewer forms, more temptation.

Corrected by measurement. The natural reading of that result — pick the family with the densest excluded near-misses, the inflections a careless matcher would over-admit — is wrong, and it is wrong in a way that wastes builds. A machine matching a word boundary is never confused by a neighbouring inflection; only a human or a model reading for meaning is.

What actually predicts the miss is the off-sense share of the admitted form: how often the required word appears in the corpus meaning something other than the thing the register is named after. In the family that produced the hardest measured task, a majority of occurrences of the admitted word are adjectival (the complete picture, the complete, dated calendar), idiomatic, future (I can typically complete this analysis), or conditional (once that call is complete) — measured at 79% inside the graded window. Those are precisely the rows the weaker tiers dropped: a model reading for sense filters them out, and the rule says they count.

So the selection metric is: for each candidate family, hand-classify a sample of admitted-form occurrences as on-sense or off-sense, and choose the highest off-sense share. Exclusion density is a decoy — and worse, it is easy to satisfy with a family that is dead on the actual corpus. Check both forms are alive before anything else: one family that looked ideal on paper had its second spelling appear in a single message out of 1,585, and another's in none at all.

Two shapes with the same root, both measured:

• A form inside a longer phrase, where the rule admits it and a reader hears a hedge (within a day inside "within a day or two").
• One sentence carrying two forms that resolve to different values, where the second reads as an explanation of the first. Every trial found the first; two of nine found the second.

Adding a graded fact widens the gap, it does not lower the mean

Grading "which rule produced this row" cost the weaker tier 0.05 and the stronger tier nothing — the stronger model had to match the form in order to emit the row at all, so naming it was free. Extra graded facts are useful for discrimination between tiers, and nearly useless for pulling a mean down.

Do the band's arithmetic before designing for it

A three-model mean with a frontier tier pinned at 1.000 is not a target on all three models — it is a budget on the other two. For a ceiling of 0.8:

mean = (1.000 + a + b) / 3 <= 0.80   <=>   a + b <= 1.40

Every task measured in band on one suite has the frontier at 1.000; what separated in from out was entirely the weaker pair's sum, and the boundary is about 0.05 wide. Three in-band tasks summed 0.73, 1.32 and 1.38; the one that missed summed 1.43.

Two consequences worth internalising before a build:

• "A frontier model would score 1.000" is not, by itself, a rejection. It is the expected case, and the design question is what the named weaker tiers do. Treat frontier 1.000 as a defect only when the frontier misses — because then the miss is far more likely to be a task defect than a capability limit.
• The margin is thin, so measure the mechanism before building. At a 0.05-wide boundary, an unmeasured guess about difficulty is a coin flip, and the honest response to landing at 0.81 is to report it rather than to lever it.

Joint dependency is the lever; row count is not

At a per-extraction accuracy a, a row that turns on ONE extraction is right with probability a, and F1 over such rows is a whether there are 26 of them or 117. Adding rows adds work, not difficulty.

A row that needs k facts is right with a**k. That is the whole lever, and it is the only one measured on this tree to move a frontier model off ceiling.

Measured. A month-end snapshot design was costed at 117 rows against 26, with 67% of the added rows new-or-changed, and estimated to move the score by nothing — each snapshot row still hinged on a single statement. Discarded before it was built. What did move it: adding the chain's other END to a key already holding its last statement and its length (0.817 → 0.706), and adding a figure that needs EVERY link rather than the ends (0.852 → 0.755).

Pick facts that fail independently. Both ends of a chain and its length are three different reads of the same chain: a reader who finds one end has none of the other two, and none is derivable from the others. Three facts that all fall out of one lookup give you a, not a**3.

Put the hard fact in the KEY, not in a field

Same fact, same task, same window:

keyed (owner, meeting), date a field    row_f1 1.000
keyed (owner, meeting, due)             row_f1 0.179

A wrong key means the row is not matched at all — every field on it misses AND the invented row draws the extra-row penalty. In a field it degrades the score by one part in N. Key placement is the strongest dial available and it costs nothing.

Choose the key by watching the dump floor, not by arguing

Before believing a key, measure what a reader who reports every candidate without comprehending anything scores.

Measured twice in one day. A slippage register keyed on (owner, meeting, due, slips) paid a dump 0.426 — inside the band, with half the target range sitting above a strategy that never read a transcript, because slips is 0 for half the rows and guessing zero is worth half the column. Adding first_due took it to 0.000: a dump can see the statements in front of it, but it cannot know where a chain STARTED.

A floor inside the band is a design defect, not a note.

A short window has no chain to reconstruct

Before concluding a supersession task is at ceiling, measure how often the thing being superseded actually moves.

Measured: at 42 days the median person revises once and the strongest tier scores 1.000; at 147 days the median is 3.5. Same task, same rule, same grader. A task was nearly redesigned when the window was the whole problem — and lengthening it also dropped the dump floor from 0.372 to 0.000.

Legitimate levers

• how much of the record a task covers
• how many independent facts per row
• how confusable the rule is
• which capability tier the target is set against

Forbidden levers

• tightening a tolerance
• reweighting criteria so the same work scores less
• withholding a rule, or trick wording
• grading a fact the surface does not serve

The test: change what the agent must do, never what the same work is worth. A task that scores lower without being harder is a scoring artifact, and the "calibration" is a property of your grader rather than of the world.

When a task sits just outside the band

Say so. A task that measures the same value across three independent methodological corrections is a stable measurement of something just outside the target — not a task fighting noise.

Report it out rather than adjusting until it isn't. And if you do apply a lever, declare it before seeing which way it moves the result.

A band can be manufactured entirely by the answer key

The most expensive way to be wrong about difficulty: a task sits at 0.33-0.51 across three model families, looks perfectly calibrated, and is measuring its own defects.

The tell is available before any lever is chosen. Decompose the loss per criterion. If a scalar reads 0.0 for every trial of every model while the counts-of-what-was-read read 1.0, the models opened the whole corpus and disagree with the key, not with each other. On one task superseded_count and distinct_owners were 0.0 in nine of nine trials while meetings_read and turns_read were exact.

Then check recall against the subset of the key you can defend by reading the source. the strongest tier scored 10/10, 9/10, 10/10 there while its graded live.f1 read 0.53-0.59 — and it was 100% correct on every evidence field of every row it matched. The reward said 0.508; the model was at ceiling.

Correct the key before choosing a lever, even though correcting it raises the scores — that is the point. A lever applied on top of a broken key tunes the defect.

And be ready for the honest outcome: once corrected, that task ran 0.838/0.838/0.838 for the frontier tier. Identical three times, one genuine row error, the rest an exact-match integer. The apparent band was gone and the real difficulty was small. Report that rather than re-widening the window to recover a number — the window levers under a word ceiling move supersession density by fractions, and coverage was already measured not to survive.