# Point The Cheap Probes At Young Code

> Use when you need to FIND work in an unfamiliar repo rather than pick it off a tracker — a new house, or one whose issue backlog is claimed faster than you can read it. Four probes cost minutes each and need no domain knowledge: does any test fail when run alone, does the suite survive a shuffled order, does the race detector fire, does a fuzz target crash. Each one asserts on the SELECTOR'S YIELD, never on the exit code, because every runner treats "matched nothing" as success. And the yield of these probes is a function of the codebase's age and hardening, not of your effort: four probes over a thirteen-year-old repo with OSS-Fuzz integration found nothing in an hour, and the same shuffle probe found a real defect on a three-year-old fork in one run. Trigger terms: new repo, where do I start, find a bug, no good first issues, backlog is claimed, flaky, test isolation, order dependent, -shuffle, -race, go test, cargo test, runtest.

- Skill: `serhiy-bzhezytskyy/point-the-cheap-probes-at-young-code` (Agent Skill)
- Install (CLI): `npx skillmds@latest add serhiy-bzhezytskyy/point-the-cheap-probes-at-young-code`
- Raw SKILL.md: https://api.skillmd.com/api/skills/serhiy-bzhezytskyy/point-the-cheap-probes-at-young-code/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: serhiy-bzhezytskyy (https://skillmd.com/u/serhiy-bzhezytskyy)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/serhiy-bzhezytskyy/point-the-cheap-probes-at-young-code

---


## Purpose

Picking a listed issue puts you in a queue. In a fast house the queue moves in hours — measured:
three well-specified DataFusion issues, all three had an open PR within a day, one of them filed by
the reporter alongside their own fix. **A defect you generate yourself has no queue.**

These four probes generate them. They need no understanding of the domain, they run before you have
read a line of the architecture, and each produces a *reproduction* — which is the artifact
maintainers are short of, not opinions.

## When to use

- First contact with a repo, straight after the door has been priced.
- The tracker looks rich but every item is assigned or has a PR.
- A `help wanted` thread asks for reproductions in so many words.
- Before believing any suite is clean because CI is green.

## When NOT to use

- The house prohibits the class of contribution you would make — price that first.
- You already have a specific, unclaimed, well-specified defect. Then go do that.
- The repo is very old, very large and fuzz-integrated **and** you have limited time: expect
  negatives (see the receipt). Spend the hour on a younger target.

## The practice (checklist)

- [ ] **0 — The gate, and time it.** One named test, alone, from a cold clone. If this is expensive the
other probes are too.

```bash
go test -v -run '^TestSomethingReal$' ./pkg/...      # 16s measured, cold clone, no build config
cargo test -p some-crate --lib some_test             # 189s cold, then sub-second
./runtest --single unit/x --only "name"              # bespoke harness — read it first
./gradlew :mod:test --tests X                        # ⚠️ toolchain-gated; wrong JDK = never runs
```
⚠️ Get a **real** test name from `go test -list '.*'` / `cargo test -- --list`. Inventing one
produces a green run over nothing.

- [ ] **⛔ 1 — Assert on the selector's yield, in every probe.** Five harnesses, all of which treat
"filter matched nothing" as success, ranked by how honestly they say so:

| harness | on a filter matching nothing | the guard |
|---|---|---|
| ⭐⭐ `cargo test` | `0 passed; … 548 filtered out` | compare `filtered out` to the suite total — a number |
| ⭐ `go test -run` | `no tests to run` · `[no tests to run]` | count `=== RUN` lines, or grep the string |
| Gradle `--tests` | silence | `actionable tasks` count; XML mtime |
| JMH | `No matching benchmarks` | grep the string |
| ⛔ Tcl `--only` | **nothing at all** | you must build your own `[ok]: <name>` assertion |

⇒ Ask this of any new runner **before** trusting a sweep: *what does it print when it selects
nothing?* A word-splitting bug once fed 27 fragments as test names and all 27 reported green.

- [ ] **2 — Order dependence. Control arm first.**

```bash
go test -count=1 ./pkg/...                                  # CONTROL: must be green
for s in 1 42 7 999; do go test -count=1 -shuffle=$s ./pkg/...; done
```
⭐ Green in declared order and red under some seeds is the signature. ⚠️ **Also grep CI for
`shuffle`** — if it is absent, this class *cannot* be caught upstream, which is why it survives.
(Confirm the grep with a control string you know is in those files.)

- [ ] **3 — Standalone runnability: does any test pass only because a sibling ran first?**

```bash
go test -list '.*' ./pkg | grep '^Test' > names.txt
while IFS= read -r t; do
  o=$(go test -count=1 -timeout 180s -v -run "^${t}\$" ./pkg 2>&1)
  y=$(printf '%s\n' "$o" | grep -c '^=== RUN')
  [ "$y" -eq 0 ] && { echo "ZEROYIELD $t"; continue; }     # ⭐ not a pass — a no-op
  echo "$o" | grep -q '^ok' || echo "FAILALONE $t"
done < names.txt
```
⭐ **Report `zeroyield` next to the failure count.** `failalone=0, zeroyield=0` over 481 tests is a
real zero; `failalone=0` alone is indistinguishable from 481 filters matching nothing.
⚠️ `timeout` is GNU coreutils and **absent on macOS** — use the runner's own `-timeout`. Mine
silently never ran and reported all 481 as zero-yield.

- [ ] **4 — Races and fuzzing, where the language gives them free.**

```bash
go test -race -count=1 -run 'SuspectFamily' ./pkg/...     # count "WARNING: DATA RACE"
go test -run '^$' -fuzz '^FuzzX$' -fuzztime 90s ./pkg/    # ⭐ assert on "execs: N"
```
⭐ A crasher input **is** the reproduction, written to `testdata/` for you. ⚠️ `execs: 0` means the
fuzzer never ran; 3M execs with 0 crashes is a result.

- [ ] **5 — Run one probe at a time, and never edit what a running sweep is measuring.** I ran two
harnesses in one checkout and voided 24 runs with fake failures; then rewrote a sweep's script and
deleted its output file **while it was still running**, voiding it again. Also kill orphaned test
processes from a stopped sweep before starting the next — they contend for CPU and manufacture
timing failures.

- [ ] **6 — Before building a fix, pre-flight the finding.** Search issues *and* PRs for the symbol, the
test name and the env var; and check whether the upstream this repo forked from carries the same
code. A defect in a post-fork feature is a different (and better) proposition than one shared with
the parent.

## Rationalizations

| what you will tell yourself | what to do |
|---|---|
| *"CI is green, the suite is clean."* | CI runs one order. Grep it for `shuffle`; if absent, that class is invisible there. |
| *"Zero findings — this repo is solid."* | Only if every probe reported its yield. Otherwise you measured your harness. |
| *"The sweep found nothing, so the probe is useless."* | It is a property of the target's age. Point it at younger code before discarding it. |
| *"I'll bundle both leaks I found."* | Bundle only what reproduces. A non-reproducing change beside a reproducing one invites a scope objection and slows both. |
| *"exit=0, so it passed."* | `$?` after a pipe is the pipe's status. Capture first: `o=$(cmd 2>&1); rc=$?`. |

## RECEIPT

⭐ **The probes are asymmetric in yield, and the asymmetry is the point.**

| target | age | probes | result |
|---|---|---|---|
| `prometheus/prometheus` @ `1add2a0` | 13 yrs, OSS-Fuzz | shuffle (3 seeds, 30 pkgs) · **481** standalone `./tsdb` tests · `-race` on 133 tests · 8 fuzz targets × 90 s | ⛔ **four clean negatives** — `failalone=0 zeroyield=0`, 0 races, **24.4 M execs, 0 crashes** |
| ⭐ `opentofu/opentofu` @ `67b7b48` | 3 yrs, fork | the same shuffle probe, **one run** | ⭐⭐ **a real defect** |

The OpenTofu finding, in full, because it is the shape to expect: `TestTemplateFile` green in
declared order, red under seeds `1` and `42`. The assertion text named the cause —
`invalid value for TF_TEMPLATE_RECURSION_DEPTH: strconv.Atoi: parsing "apple"`. A sibling test used
`os.Setenv` with a deliberately invalid value and never restored it; `t.Setenv` does, and the repo
already used `t.Setenv` **117** times. One token, both arms proved (green: 23 packages × 5 seeds;
red: fix stashed, seeds 1 and 42 fail again), `gofmt` and `vet` clean, pre-flight clean, and the
env var has **0** hits in the repo this one forked from.

⛔ **It was never offered** — the house's PR template requires certifying that no AI coding
assistant was used. ⇒ Price the door's *policy* before running any of this, or you will do
excellent work you cannot give away. That is a separate skill:
`price-the-door-before-you-enter` §5b.

## Lifecycle

Written 2026-08-14 from the Prometheus/OpenTofu pair. Extend it when a probe finds something on a
sixth harness, or when a target contradicts the age asymmetry — a young repo that yields nothing, or
an old one that yields, is the interesting case and should be recorded here with its numbers.