Baseline Before Model

Run the boring thing first

On an address-resolution task, a deterministic directory lookup with an alias table and edit distance scored 96.7%. The language models given the same items and no reference data scored 65.0% and 53.3%. The lookup ran in 2.8ms at zero cost against roughly 2,000ms and real spend.

That result is not an argument about model quality. It is an argument about architecture. A prompt-only model is the wrong tool for a lookup, and no leaderboard will ever tell you that, because "should this be a model at all" is not a property of any model on it.

The refusal

Do not recommend a model for a task with no measured non-model baseline. If nobody has built the boring version, the honest output is "unknown, and here is the baseline to build first".

Candidate baselines, in order of how often they win

1. Lookup against a table or directory you already own. Extraction and normalisation tasks are frequently this in disguise.
2. Rules or a decision table, when the logic is genuinely small and stable.
3. Classical retrieval (BM25) before any embedding pipeline.
4. A small local model before a frontier API model.
5. The existing process, measured. Often nobody has ever scored the humans.

Procedure

1. Write the baseline behind the same interface the model would use, so the whole evaluation stack runs on it unchanged.
2. Score it on the same items, with the same grader.
3. State the caveat that travels with the number. If the baseline holds reference data the models were not given, that is lookup versus no-lookup, not lookup versus retrieval. Say it plainly. The honest claim is stronger than the sloppy one.
4. Add the missing arm where it matters: the model with the reference data in context. That is the comparison that decides the architecture.
5. Report cost and latency alongside accuracy. This is usually where the baseline wins by two orders of magnitude.

Output contract

• baseline description and what data it holds
• baseline accuracy, cost, latency
• model accuracy, cost, latency, on identical items
• the caveat, stated in the same breath as the numbers
• whether the model-with-reference-data arm was run, and if not, that it is missing

What this prevents

Six weeks of prompt engineering on a task a hash map solves better, and a vendor conversation about which lab to buy from when the answer was to buy from neither.