Use The Tool For Its Purpose

Use the tool for its purpose (to find where the bugs actually are)

Purpose

An LLM asked to "find bugs in this repo" will read code and produce plausible smells: a maintainer's least favorite kind of contribution, because every one has to be disproved by hand. The alternative costs more wall-clock and almost nothing in reviewer trust: run the project for the reason it exists, on real input, at the scale a real user would. Defects that survive months of green CI are concentrated in the one workflow nobody walks end to end — and a bug found by being blocked by it arrives with a reproduction, a stack trace, and an obvious severity, none of which you had to argue for.

When to use

• You want to contribute to a repo you don't own and don't yet know, and you're deciding what.
• The project is a tool, CLI, or pipeline with an end-to-end workflow you can run.
• You have a real task that happens to need this tool — best case. Be a user first; the bug list writes itself.

When NOT to use

• The pain already came to you from a live thread or a CI failure — start from that instead: follow-the-thread (traverse it) or consolidate-a-scattered-thread (map it).
• The project isn't runnable in reasonable time/resources (huge cluster, paid service, 100GB corpus you don't have). Say so and pick a different sourcing route rather than faking a run.
• Library-only code with no workflow to walk — there is no "purpose" to exercise.

The practice (checklist)

• Run it for a real purpose, not as a demo. Have an actual question you want the tool's answer to. Purposeless runs stop at the happy path.
• Realistic N, real data. --test-mode, sample corpora and N=1 are exactly the paths that are already covered. Bugs live at N≥2 and on the full input.
• Target the step everyone skips. Rank candidate surfaces by how likely a demo is to reach them: - the gap between "it ran" and "you have a result" — aggregate, compare, summarize, report, export, publish; - the second run: leftover state, caches, a checked-out repo that now has untracked files, "combine with previous results" code; - optional/rare inputs — the operation that returns null, the empty result set, the field nobody sets.
• Finish the workflow. Fixing one blocker usually reveals the next; a run that ends in a green checkmark you didn't verify the output of is not a finished run. Done when you have the artifact you originally wanted and have read its fields — not when the command stopped raising.
• Reproduce against the REAL method, with a control that passes. Bypass only the wiring you must (constructor/config/store); never mock the code under test. The control — the same call with healthy input, succeeding — is what proves the trigger is the trigger and not your harness.
• Rule out your own setup before you call it a project bug. Re-do it the way a normal user would (a real git clone, not your staged copy).
• Check whether the "missing" thing is documented before reporting it missing. Grep the config and the docs; a hit rescopes your report from "feature absent" to the real, narrower defect — which is the one that lands.
• Separate proved from observed. Report what you reproduced. Anything you merely saw and did not root-cause goes in as prose context inside a report you did prove, not as a claim of its own.

Rationalizations

RECEIPT

MERGE-receipt — 3 defects → 6 issues + 6 PRs across two projects, found by finishing one benchmark campaign, and now all six PRs MERGED. Verified via gh 2026-08-02: solr-orbit #59 (2026-07-27), #58 (07-29) and #60 (07-30) merged by the author of the port; OSB #1097 (07-28), #1096 (07-29) and #1098 (07-30) merged upstream. All six issues closed. This shipped as a SOURCING-receipt — the method had demonstrably found real defects while nothing had merged yet — and the merges upgraded it to the stricter class without a word of the method changing. Worth keeping visible: the sourcing claim was true before the merge evidence existed, which is the whole reason that receipt class is defined.

The task was a real question: does Solr on Lucene 11 regress against Lucene 10.4? Answering it needed 5 runs per config averaged — so it needed solr-orbit aggregate, and aggregate could not run at all on main:

Why they survived: all three sit between "you ran the benchmark" and "you have a result." --test-mode never reaches them. N=1 never reaches them — nothing to aggregate. CI never reaches them — every test in the aggregator suite passes a bare Mock as the test run, and a Mock answers any attribute name, so the wrong name reads fine. 1099 green tests downstream and 1422 upstream, over a command that could not succeed once.

Each was reproduced against the real method with a passing control — e.g. calculate_weighted_average(null throughput) → TypeError: '<' not supported between instances of 'NoneType' and 'NoneType', beside a control on numeric throughput that returns cleanly, plus the bare-Mock control that reproduces what CI does and shows no error. The upstream defects were then reproduced by running OSB, not by reading it, and confirmed through the real CLI going ❌ FAILURE → ✅ SUCCESS with output checked field by field.

Two near-misses this checklist caught, both of which would have been noise: a __pycache__ failure that was my own staging (downgraded to a .gitignore nit, not reported), and "the workloads repo is undocumented" (it is documented — rescoped to the narrow parity gap that became #57/#1095).

The upstream half of this receipt is the sibling skill's: port-the-report-upstream.

Lifecycle

• Signals it worked: the report arrives with a stack trace and a control, and a maintainer engages with severity rather than asking "how did you hit this?"; your own blocked task is what dates the bug.
• What to log on a misfire: the surface you exercised, whether the finding survived the setup check, and whether the "missing" thing turned out documented. Record it in LEDGER.md.
• Death criterion: if a run of this method across a whole project yields only findings that don't survive the control/setup checks, the method is producing noise in that project — retire it there, not everywhere.
• Relates to: upstream propagation is port-the-report-upstream (do it after this one lands the downstream evidence). Sibling to follow-the-thread — that one starts from someone else's pain, this one from your own. What you don't report is offer-dont-grab's discipline applied to findings.