Judgment Or Code

Judgment or code

Decide per OUTPUT, and write its acceptance rule down the way the operator would state the goal, never in the phrasing that makes your preferred tool fit.

The failure this prevents is not choosing wrong once. It is choosing by habit: reaching for a regex because regexes are cheap and testable, or reaching for a model because models feel smart, and then writing the acceptance rule backwards from the tool you already picked.

The three failures this was written after

Read these first. Each rule below is one of them generalised.

• 203 vacuous outreach hooks. A cheap model was asked to "be specific" about companies whose input records differed only in trivia. The model was not the problem; the grounding was. Every hook passed the check "mentions the company", and every hook was garbage.
• 48 of 50 "confirmed" profile URLs were wrong. Confirmation rested on a name-only string match against the claim's own subject. N fields scraped from one source is one signal, not corroboration.
• A 200-word name blocklist still misfiring on real names. Every miss produced another entry. A list that keeps growing to chase an open boundary is judgment mechanised, and it never converges.

The nine

1. Code owns the checkable; checkable never certifies quality. Exact match, schema, enum, arithmetic, state, or a finite table the business maintains as policy: code. But passing a mechanical check never makes creative output good. "mentions the city" passes, and is garbage.

2. An LLM judge owns every open meaning boundary. Semantic failure (vacuous, wrong entity, off topic) in an open-ended category goes to a judge model with a written rubric plus real failing examples. Closed policy registries are code plus normalisation. Try the cheaper exits first: normalise the input, fix the source data, drop the unreliable feature.

3. Grounding before model tier, and the difference must be usable. Two subjects must receive inputs that differ in material a writer could actually build the intended output from. Differentiated trivia is still starvation. Fix what is fetched before touching the prompt or the model.

4. Judge creative batches against sampled siblings, plus the class definition. Could whoever the output lands on tell it was made for them? Is it welcome, rather than merely identifying? Does it share a skeleton with its siblings? Per-row judging of a batch product is a blind spot: each row passes, and the batch is obviously machine-made.

5. Real-world claims need evidence that could disagree. N fields scraped from the claim's own subject are one signal. Confirmed requires independent corroboration or a first-party authoritative source; otherwise the state is unconfirmed or conflicting: first-class, visible, and never rounded up downstream.

6. Verdicts are categorical, with the evidence tied to the decision. Code thresholding a model's own uncalibrated score is judgment laundering. A genuinely calibrated probability is evidence; the default is a category plus a quoted reason.

7. Hard invariants stay mechanical. Spend, schemas, dedup, state machines, send and publish gates: code, enforced once, no repair-loop stacking. A gate boolean may carry an upstream judged verdict; the gate itself never improvises.

8. A human gate is sized to human attention, and thinning is defined. The machine judge rejects or flags; the human reads everything flagged plus a stated sample of the approved; machine- approved output ships only for a class the human approved as a class. Bulk approve is not review.

9. Spend on judgment where wrongness is expensive. Output that reaches an outside person or gates money gets judged whatever the calls cost. Internal low-stakes fields do not buy a jury. Cheap calls are no licence to add LLM passes where code computes the one right answer.

Tripwires: any one means stop and re-route

• An open-category word list that needed another entry.
• A model told to "be specific" on class-shared input.
• "Confirmed" derived from single-provenance evidence.
• An uncalibrated self-score thresholded by code.
• An LLM inserted where code already computes the answer.
• A review step that has quietly become bulk approve.

The short version

Code owns the checkable. A judge owns every open meaning boundary. Passing a mechanical check never certifies that creative output is good.