Real World Gauntlet

Real-World Validation Gauntlet

Unit tests with self-made fixtures prove the code matches your imagination, not that it works. This skill runs a gauntlet: real third-party inputs, evidence captured, failures fixed, then re-run. Every rule below was paid for by a real bug (see references/bug-case-studies.md).

The validation ladder

Know which rung you are on, and never claim a higher one:

1. L1 unit tests — logic correct. Input is yours. Weakest evidence.
2. L2 real-world gauntlet (this skill) — third-party inputs test compatibility, false-positive rate, and platform behavior.
3. L3 dogfooding — you depend on the tool daily; telemetry accumulates.
4. L4 external users — strangers install it and file issues. The only terminal proof. Everything before it is your own judgment.

Designing the gauntlet

Pick 2–4 attacks per tool, each using an input you do not control:

Hard rules:

• Never simulate the real thing when the real thing is reachable. npx the real package, gh the real PR, clone the real repo. Mocks hide exactly the bugs you are hunting.
• Measure false positives explicitly. Run against inputs known-good in reality (popular repos, official servers). Every finding on a known-good input is a false positive — fix the tool, do not whitelist the input.
• Capture evidence per attack: exact command, exit code, key output. If you cannot re-run it, it did not happen.
• Fix-then-rerun: a gauntlet finding is only closed after the fixed tool re-passes the SAME real-world attack, plus a regression test derived from it.

Platform-only bugs: the debug-branch technique

When something fails on CI (Linux) but not locally (Windows), do not guess. Create a throwaway branch that adds diagnostics at the failure point (print the rendered command, child stderr, env vars), push it, let the real runner execute it, read the log, then delete the branch. Ten minutes of ground truth beats an hour of local theorizing. Real example: a sh: 3: Replace: not found line in CI logs revealed a multi-line prompt being executed as shell commands (""path"" double-quoting) — invisible on Windows where cmd.exe forgives it.

Common real-world-only failure classes

1. Normalized echoes — your detector strips the payload you sent, but the real system returns it transformed (Windows /→\, case changes, wrapper text). Strip on normalized forms, not exact strings.
2. Shell quoting asymmetry — Windows cmd.exe forgives nested quotes; POSIX sh executes the tail of a split command. Test prompt-carrying command lines on Linux always.
3. Vocabulary vs data-flow — a scanner that fires on words ("env vars") will flag debugging lessons in popular repos. Gate on actual data flow (file paths, $VAR expansions, network sinks).
4. Runner-injected environment — CI sets GITHUB_EVENT_PATH/GITHUB_TOKEN in every step; tests asserting "no environment" flip code paths. Strip the runner's injected variables in the test launcher.
5. Cold-start vs warm timings — first npx run downloads packages and blows handshake timeouts. Warm the cache, or raise the specific timeout.

Exit criteria

The gauntlet is done when: every attack has captured evidence, every finding is fixed with a regression test, known-good inputs produce zero false positives, and both OS CI matrices are green. Then say exactly what was validated and what was not — "not-inspected ≠ clean" applies to gauntlets too.

See references/bug-case-studies.md for the five bugs this method caught: each has symptom → investigation → root cause → fix → generalized lesson.