Point The Cheap Probes At Young Code

Purpose

Picking a listed issue puts you in a queue. In a fast house the queue moves in hours — measured: three well-specified DataFusion issues, all three had an open PR within a day, one of them filed by the reporter alongside their own fix. A defect you generate yourself has no queue.

These four probes generate them. They need no understanding of the domain, they run before you have read a line of the architecture, and each produces a reproduction — which is the artifact maintainers are short of, not opinions.

When to use

• First contact with a repo, straight after the door has been priced.
• The tracker looks rich but every item is assigned or has a PR.
• A help wanted thread asks for reproductions in so many words.
• Before believing any suite is clean because CI is green.

When NOT to use

• The house prohibits the class of contribution you would make — price that first.
• You already have a specific, unclaimed, well-specified defect. Then go do that.
• The repo is very old, very large and fuzz-integrated and you have limited time: expect negatives (see the receipt). Spend the hour on a younger target.

The practice (checklist)

• 0 — The gate, and time it. One named test, alone, from a cold clone. If this is expensive the other probes are too.

go test -v -run '^TestSomethingReal$' ./pkg/...      # 16s measured, cold clone, no build config
cargo test -p some-crate --lib some_test             # 189s cold, then sub-second
./runtest --single unit/x --only "name"              # bespoke harness — read it first
./gradlew :mod:test --tests X                        # ⚠️ toolchain-gated; wrong JDK = never runs

⚠️ Get a real test name from go test -list '.*' / cargo test -- --list. Inventing one produces a green run over nothing.

• ⛔ 1 — Assert on the selector's yield, in every probe. Five harnesses, all of which treat "filter matched nothing" as success, ranked by how honestly they say so:

⇒ Ask this of any new runner before trusting a sweep: what does it print when it selects nothing? A word-splitting bug once fed 27 fragments as test names and all 27 reported green.

• 2 — Order dependence. Control arm first.

go test -count=1 ./pkg/...                                  # CONTROL: must be green
for s in 1 42 7 999; do go test -count=1 -shuffle=$s ./pkg/...; done

⭐ Green in declared order and red under some seeds is the signature. ⚠️ Also grep CI for shuffle — if it is absent, this class cannot be caught upstream, which is why it survives. (Confirm the grep with a control string you know is in those files.)

• 3 — Standalone runnability: does any test pass only because a sibling ran first?

go test -list '.*' ./pkg | grep '^Test' > names.txt
while IFS= read -r t; do
  o=$(go test -count=1 -timeout 180s -v -run "^${t}\$" ./pkg 2>&1)
  y=$(printf '%s\n' "$o" | grep -c '^=== RUN')
  [ "$y" -eq 0 ] && { echo "ZEROYIELD $t"; continue; }     # ⭐ not a pass — a no-op
  echo "$o" | grep -q '^ok' || echo "FAILALONE $t"
done < names.txt

⭐ Report zeroyield next to the failure count. failalone=0, zeroyield=0 over 481 tests is a real zero; failalone=0 alone is indistinguishable from 481 filters matching nothing. ⚠️ timeout is GNU coreutils and absent on macOS — use the runner's own -timeout. Mine silently never ran and reported all 481 as zero-yield.

• 4 — Races and fuzzing, where the language gives them free.

go test -race -count=1 -run 'SuspectFamily' ./pkg/...     # count "WARNING: DATA RACE"
go test -run '^$' -fuzz '^FuzzX$' -fuzztime 90s ./pkg/    # ⭐ assert on "execs: N"

⭐ A crasher input is the reproduction, written to testdata/ for you. ⚠️ execs: 0 means the fuzzer never ran; 3M execs with 0 crashes is a result.

• 5 — Run one probe at a time, and never edit what a running sweep is measuring. I ran two harnesses in one checkout and voided 24 runs with fake failures; then rewrote a sweep's script and deleted its output file while it was still running, voiding it again. Also kill orphaned test processes from a stopped sweep before starting the next — they contend for CPU and manufacture timing failures.

• 6 — Before building a fix, pre-flight the finding. Search issues and PRs for the symbol, the test name and the env var; and check whether the upstream this repo forked from carries the same code. A defect in a post-fork feature is a different (and better) proposition than one shared with the parent.

Rationalizations

RECEIPT

⭐ The probes are asymmetric in yield, and the asymmetry is the point.

The OpenTofu finding, in full, because it is the shape to expect: TestTemplateFile green in declared order, red under seeds 1 and 42. The assertion text named the cause — invalid value for TF_TEMPLATE_RECURSION_DEPTH: strconv.Atoi: parsing "apple". A sibling test used os.Setenv with a deliberately invalid value and never restored it; t.Setenv does, and the repo already used t.Setenv 117 times. One token, both arms proved (green: 23 packages × 5 seeds; red: fix stashed, seeds 1 and 42 fail again), gofmt and vet clean, pre-flight clean, and the env var has 0 hits in the repo this one forked from.

⛔ It was never offered — the house's PR template requires certifying that no AI coding assistant was used. ⇒ Price the door's policy before running any of this, or you will do excellent work you cannot give away. That is a separate skill: price-the-door-before-you-enter §5b.

Lifecycle

Written 2026-08-14 from the Prometheus/OpenTofu pair. Extend it when a probe finds something on a sixth harness, or when a target contradicts the age asymmetry — a young repo that yields nothing, or an old one that yields, is the interesting case and should be recorded here with its numbers.