Verify By Failure

Verify by failure

A green test proves almost nothing. A watchdog that never fires looks identical to a watchdog that works. The loop below exists because both of those cost me a production outage that lasted 46 days.

Use it for any task of three or more steps.

Step 0. Plan with visible progress — before the first action

Three steps or more means a plan first, work second. The person you report to should be able to see movement without reading your prose.

If TodoWrite is available, keep the plan there. One item in_progress at a time.

If it isn't (it is not connected in every session), draw the progress in text. A compact block at the top of any reply where something moved:

▓▓▓▓▓▓▒░░░  step 3 of 5 · 60%
✓ Measured the queue — 132 developers with no dossier
✓ Research: Capstone Asset, LKF Properties
▸ Writing dossiers, verifying against the matcher     ← now
  Refreshing the open-questions list
  Writing the log entry

Rules for the block:

• Ten characters exactly: ▓ done, ▒ in flight, ░ ahead.
• Always both the step count and the percentage. One answers "how much is left", the other answers "where am I".
• Phrase items by outcome ("dossiers written and matched"), never by action ("search the web").
• Found new work along the way? Add the item and recompute — don't hide it.
• Keep it short. It replaces narrating your progress; it doesn't accompany it.

For long-running work spanning many sessions, show two levels:

▓▓▓▓▓▓▓▓▒░  step 4 of 5 · 80%   ·   dossiers overall: 70 / 198 (33%)

A ready-made panel for this lives in pulse/ — a single self-contained HTML file you can publish as an artifact and keep on screen while the agent works.

Step 1. Measure first, fix second

The most expensive mistake is fixing what isn't broken while missing what is.

Reproduce the behaviour the way production runs it, not the way you picture it.

I once counted three finished records that "weren't reaching users because the names didn't match" and was about to patch aliases. Simulating the real matcher showed it also resolves by slug and id — all three matched. The alarm was false and the fix would have been damage. The difference between reading the code and executing its logic against live data.

In practice:

• Count on production data, not on a hunch. A number in a report beats an opinion.
• If the code already exists, copy its function into your measurement — its normalizer, its field list — rather than writing "roughly the same thing". My first audit reported 100% of projects missing a field; re-running it with the function's own field list gave 98%. Different lists, different answers.
• Measure the scale of what you found. "Found a broken project" and "found 4 of 361" are different tasks.

Step 2. Verify by failure, not by success

A positive test goes green almost always — including on completely dead code.

I wrote a watchdog for a hung service using Cyrillic variable names. Bash does not accept them. The loop threw, execution fell straight through to the restart line, and the watchdog began restarting a live service blindly every five minutes — while returning zero, so it looked healthy. My positive test printed "all good". It was caught only when I froze the workers and watched whether it recovered them.

In practice:

• Test a safety net by breaking the thing it guards.
• Reproduce that failure, not a similar one. The service hung with its port still open, so the test was SIGSTOP, not systemctl stop.
• Before declaring a component dead, check all of its inputs. I declared a provider down based on one key of three; the other two were alive and the diagnosis went sideways for half a day.

Step 3. Never invent data

In real-estate — or medicine, or finance — a made-up number is a legal risk, not a styling choice.

• No yield figure on record? Write "none". Don't substitute something plausible.
• Sources disagree (16 villas on the site, 11 in our database)? Put both in the fact itself, flagged "verify". A broker who quotes one of them to a client gets caught.
• Don't know the right value? Don't fix it. A 408 m² penthouse mislabelled "studio" stayed as-is, listed precisely in the findings. Guessing the type is not an option.
• Derive success flags from the result, never from the intent. A media server returned captionsAi: use_ai — an echo of the request. When the AI failed and a template was used instead, statistics still showed a healthy feature. The breakage became invisible exactly where seeing it mattered most.

Step 4. Second pair of boots: look through the END USER's eyes

You build for the operator, but the person who pays is at the other end of the product, and half of all findings live where a developer never goes.

Always walk both surfaces:

• The operator — the tools, the dashboard, the admin. What they see on a phone, not in a mockup.
• The end user — the shared link, opened the way a customer opens it: on a phone, without a login, possibly tomorrow, possibly by their spouse instead of them.

What tends to break quietly on the user's side:

• Absurdities on the storefront: "studio — $4.2M", a "Project figures" slide with no figures, an "interior" slide showing an exterior. Only eyes catch these.
• Promises you can't keep: guaranteed returns, dates from stale data.
• Dead states: a blank screen instead of "no data", a spinner with no end.
• The second-visitor scenario: the link opened a day later, by a different person, on a different device. That one permanently broke an AI chat once.

Check it the way a human does: open it, poke it, look. A screenshot and a DOM read are evidence. "It should work according to the code" is not.

Step 5. Stress-test at the end of EVERY batch

Batch closed means run the checks. Not "when there's time" — always, or the breakage surfaces in front of a user instead of in front of you.

The minimum:

1. The end-to-end battery for your core feature. Read past "0 errors" to the things that actually hurt: cross-tenant leaks, empty answers, p90 latency.
2. The liveness watchdog — model providers, core API, external services, billing.
3. Your eyes — open what you changed, in both surfaces (Step 4).

If the batch touched data, add the relevant audit for that data.

Compare against the previous run. "51% substantive" means nothing on its own. It means something as "was 53%, now 51% — within noise" or "halved, investigating". A number with no prior measurement is not a result.

Step 6. Periodically: take a meta position

Every few batches, step above the task. Not "what am I doing" but "am I on the right thing at all".

Ask these out loud in the report when the answer isn't obvious:

• Am I fixing the symptom or the cause? A 500k price floor was hiding broken prices. The right fix was the prices, not a higher floor.
• Is this worth the work? A system prompt could be restructured to cut 30% of input tokens — but the entire monthly LLM bill was $1.43, and most of it was my own testing. The measurement cancelled the task, and that is a good outcome.
• What are we not measuring? Token counters accumulated for a month and nobody ever read them. The most expensive holes are where there is no instrument at all.
• Whose call is this really? Product forks and money are the owner's, not yours.

Do the same for the outside world, regularly rather than by accident:

• Same name ≠ same company. Two "Ocean" companies, two "Dusit" companies — different entities. One brand can ship through different legal entities per project. A company may not be findable by its registered name at all — search by the product instead.
• Guaranteed returns are claims, not facts. "6% guaranteed", "up to 40% growth" — record them as statements with a direct note on what to ask. Whoever repeats them owns the consequences.
• Verifiability beats volume. An award with a public registry beats "award winning". A public company beats a private one — you can open the filings.

Write meta-observations into the log as their own line. They outlive the task.

Step 7. Write the log — before the report, not after

Keep a running log the next session will actually read.

Record not "what I did" but what is now known:

• breakage → root cause → why nobody noticed sooner;
• finding → measurement (a number!) → what was decided about it;
• an error in your own reasoning → how it was caught.

Traps get their own line — they repeat. Real ones from this loop's history:

• \b word boundaries in JS regex don't work with non-Latin alphabets;
• timeout doesn't exist on macOS by default — port checks fail silently;
• bash variable names must be Latin — Cyrillic ones throw and fall through;
• pkill -f "pattern" inside a script matches the script's own command line;
• a gthread worker keeps sending heartbeats while all its threads are wedged, so gunicorn's --timeout never catches that hang.

Step 8. Report: an "Executive summary" block, not a diary

The reader reads to make a decision. Everything that doesn't serve a decision is noise.

Any reply that closes a step ends with this block:

## Executive summary

**Done.** One or two lines: what works now and what proves it — a number, a link, a run.
Not "fixed the video service" but "a reel renders in 67 seconds, verified end to end".

**Found along the way.** With scale: "4 projects out of 361", not "found a problem".
If you didn't measure the scale, say you didn't.

**Needs you.** Only what you cannot do: money, access, a product decision.
Nothing? Write "nothing".

**Next.** One next step and an honest pace: "132 to go, that's several more sessions" —
not a cheerful "almost done".

Keep the heading verbatim — it's how the reader finds the summary without reading the rest. No closed step means no block: just the progress bar.

Don't write: a retelling of your steps, a list of files touched, "I decided to use this approach", apologies, or anything the reader already knows.

What stops the loop and asks

Work autonomously, but don't decide these:

• Money. Topping up a provider, paying for hosting, buying a domain.
• Credentials. Never type passwords or tokens into a prompt. Need machine access? Ask for your key to be authorized — don't ask for a password.
• Product forks. "Redesign the template" versus "backfill the data for 352 records" are two different months of work. Your job is the measurement and the options.
• Irreversible actions. Deleting data, rotating keys, changing a live schema — show exactly what and how much, then do it.

Everything else — deploys, code, scripts, database writes, research, audits — do it and report the result.