Validating Task Premises

Measure the premise before you write the task

Every task starts as a guess about what a world contains. The guess feels like knowledge because it is a reasonable picture of how the domain works — people set deadlines, meetings concern projects, revisions describe edits. Worlds generated by models are only sometimes that picture.

On one build, eight premises were written from that kind of confident picture and then measured. Three died, one had to be inverted, and one lost a graded field. Every death was cheap, because it happened before the oracle, the grader and the rollout were built on top of it.

The failure this prevents is the worst kind: a task starved by its own premise produces rows nobody can find, and a model that finds none of them looks like a model that failed.

Six checks, cheapest first

Run them in order and stop as soon as one kills the premise.

1. Liveness — does the pattern occur at all?

Count the raw occurrences of every token, form, or relation the rule depends on. Not the ones you would write; the ones the world wrote.

A date-handling task needed interval phrasings (within N days) and calendar dates in one message. Both were zero across hundreds of messages. Another needed each meeting to name the matter it concerned. Matching the matter list against every transcribed meeting title gave zero of ninety-three — every meeting was a standing one, a docket call or a practice huddle, and standing meetings are named after the recurrence rather than the work.

Re-run later on the same world, still recording, it gave 11 of 157, and the difference is worth more than either number. Eight of the eleven are one standing meeting that happens to carry a client's name in its title; three are genuinely matter-specific, scheduled ad hoc by someone who needed a meeting about a thing. The structural finding held. The absolute count did not, because rare events accumulate.

Write the window into the claim. A corpus measurement is true of the corpus you measured, and a world that is still growing will invalidate any count stated as though it were a property. "Zero" became "2% of meetings, ad hoc" without anything changing about the task. Neither task was hard. Both were impossible.

Expect a heavy skew. In one corpus a single deadline form carried 74% of all occurrences and three of seven forms carried none. A rule listing all seven scores three sevenths of its own vocabulary against nothing, and the four that survive are not evenly matched either — one of them is three quarters of every hit.

2. Read the rows, not the count

A count can agree with your premise while the rows refute it. This is the check people skip because the number already looks right.

A task needed one sentence carrying two different dates. The detector found eighteen. Reading them: fourteen were compound spellings of a single deadline (by tomorrow EOD — two form-words, one date), and the rest were multi-item lists sharing one due date. The true count was approximately zero.

A detector that confirms what you already believe has told you nothing until you read what it matched.

3. Distribution over time — is the rate an artifact?

A rate computed over a window can be dominated by what happens at one edge of it, and a total will never show you. This is one line:

collections.Counter(day_of(row) for row in signal)

A scheduling-conflict task measured 54 real conflicts against 240 near-miss decoys — a healthy-looking 4.4:1 trap ratio. Grouped by date, 47 of the 54 fell on day one, where the world seeds far more events than it creates on any later day. Outside that burst the world produced one conflict in seventeen days. The ratio looked good precisely because the artifact inflated the numerator.

Both edges do this, and they differ in what they cost you:

• Start-of-record artifacts (seeding bursts, everything created at once) poison the signal. Usually fatal.
• End-of-record artifacts are excludable, and often improve the rule. Questions asked on the final recorded day were 100% unanswered — nobody had a chance to reply. Fixing it meant a stated response window and a register that closes several units before the record does. The defect became a design constraint.

4. Reachability — can the agent actually get it?

Three different things, and each can be true while the next is false:

1. the fact is in the world log
2. it is in the state served to the agent
3. a tool the agent can call returns it

A task was designed on meeting transcripts, and built as far as a working projection before anyone checked layer 2: the transcripts were in the log and no served surface carried them. Not hard — impossible, silently.

Layer 3 bites too. A column present in a served table but never returned by any tool is unreachable in practice. And if your surfaces mirror real products, adding a tool to fix this is its own defect: an agent trained against an invented tool learns a call that fails in the real product. Put the data where the real product would put it.

Check the solver too. A reference solver reading the world log rather than the served state computes an oracle the agent provably cannot reproduce.

5. Degeneracy — does every graded field discriminate?

Compute the distribution of each graded field across the real rows. A field that is constant grades nothing while looking like work.

One register carried a cross-surface boolean — had the author logged time that day. Measured: true for 98 of 98 rows, because everyone who touches a document logs time that day. It read like a reconciliation check and was a free point.

6. Admission rate — is the rule worth applying?

What fraction of candidates does the rule admit?

One rule admitted 98 of 119 revisions. Inverted — register the ones declaring no change — it admitted 22, and over-admitting now destroys precision while skimming destroys recall.

Read the schema; never guess a field name

A query against a field that does not exist returns empty, and empty looks exactly like a finding. Twice in one session a guessed name produced a clean, plausible, completely false zero: narrative and hours did not exist where the real fields were note and minutes; an event tag was guessed and reported "0 time entries" for the most common record in the world.

Print the schema first, then assert the fields you need exist, so a rename fails loudly instead of quietly:

fields = set(rows[0])
for need in ("note", "minutes", "billable"):
    assert need in fields, f"{need} absent; have {sorted(fields)}"

The same discipline applies to any check that returns nothing. A check that reports no problem needs proof it is capable of reporting one — break the input deliberately and confirm it complains.

Retire rather than widen

When a premise fails, the tempting repair is to widen the rule until the corpus fills it: admit two weeks alongside within N days, count compound spellings as two dates, drop the boundary that excludes the decoys.

Widening trades a rule the world does not write for a rule nobody stated. The register then measures the author's vocabulary rather than the model, and it will score in band for entirely the wrong reason.

Retire the task and take the mechanism from something you measured. Write the numbers into the task file where the next person will make the same decision, not into a report they will not read.

The helpers

scripts/premise_checks.py implements the generic half of the above — liveness, concentration-by-bucket, field degeneracy, and admission-rate banding — over plain lists and dicts, so it works against any world:

from premise_checks import liveness, concentration, degeneracy, admission

concentration(rows, key=lambda r: r["date"])  # -> flags a 47-of-54 day
degeneracy(rows, fields=["author", "logged"])  # -> flags the 98/98 column

Related: authoring-graded-tasks for turning a surviving premise into a brief, iterating-task-difficulty for moving a task already in flight.