# Real World Gauntlet

> Validates a tool against real-world third-party inputs instead of self-made fixtures: official servers, real repositories, real PRs, real version drift, and cross-platform CI. Use when unit tests are green but you need evidence the tool works outside the lab — before publishing, after a major refactor, or whenever a security/precision claim needs to hold on data you did not invent. Catches the class of bugs fixtures structurally cannot contain.

- Skill: `zhengqiuyang/real-world-gauntlet` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add zhengqiuyang/real-world-gauntlet`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zhengqiuyang/real-world-gauntlet/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Security
- License: MIT
- Author: zhengqiuyang (https://skillmd.com/u/zhengqiuyang)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/zhengqiuyang/real-world-gauntlet

---


# Real-World Validation Gauntlet

Unit tests with self-made fixtures prove the code matches your imagination, not
that it works. This skill runs a **gauntlet**: real third-party inputs, evidence
captured, failures fixed, then re-run. Every rule below was paid for by a real
bug (see `references/bug-case-studies.md`).

## The validation ladder

Know which rung you are on, and never claim a higher one:

1. **L1 unit tests** — logic correct. Input is yours. Weakest evidence.
2. **L2 real-world gauntlet** (this skill) — third-party inputs test
   compatibility, false-positive rate, and platform behavior.
3. **L3 dogfooding** — you depend on the tool daily; telemetry accumulates.
4. **L4 external users** — strangers install it and file issues. The only
   terminal proof. Everything before it is your own judgment.

## Designing the gauntlet

Pick 2–4 attacks per tool, each using an input you do not control:

| Tool type | Real-world attack |
|---|---|
| CLI that parses files | Run against 2–3 popular real repositories (one huge, one quirky) |
| Client of a protocol/server | Run against the **official reference server**, not a mock |
| Version-sensitive tool | Exercise two **real published versions** (old vs new) |
| CI/gate tool | Run against **real public PRs** via the platform API |
| Scheduler/runner | Run a **real useful job** with real side effects, twice (idempotence) |
| Anything cross-platform | Push to **both Linux and Windows CI** before trusting local runs |

Hard rules:

- **Never simulate the real thing when the real thing is reachable.** `npx` the
  real package, `gh` the real PR, clone the real repo. Mocks hide exactly the
  bugs you are hunting.
- **Measure false positives explicitly.** Run against inputs known-good in
  reality (popular repos, official servers). Every finding on a known-good
  input is a false positive — fix the tool, do not whitelist the input.
- **Capture evidence per attack**: exact command, exit code, key output. If
  you cannot re-run it, it did not happen.
- **Fix-then-rerun**: a gauntlet finding is only closed after the fixed tool
  re-passes the SAME real-world attack, plus a regression test derived from it.

## Platform-only bugs: the debug-branch technique

When something fails on CI (Linux) but not locally (Windows), do not guess.
Create a throwaway branch that adds diagnostics at the failure point (print
the rendered command, child stderr, env vars), push it, let the real runner
execute it, read the log, then delete the branch. Ten minutes of ground truth
beats an hour of local theorizing. Real example: a `sh: 3: Replace: not found`
line in CI logs revealed a multi-line prompt being executed as shell commands
(`""path""` double-quoting) — invisible on Windows where `cmd.exe` forgives it.

## Common real-world-only failure classes

1. **Normalized echoes** — your detector strips the payload you sent, but the
   real system returns it transformed (Windows `/`→`\`, case changes, wrapper
   text). Strip on normalized forms, not exact strings.
2. **Shell quoting asymmetry** — Windows `cmd.exe` forgives nested quotes;
   POSIX `sh` executes the tail of a split command. Test prompt-carrying
   command lines on Linux always.
3. **Vocabulary vs data-flow** — a scanner that fires on words ("env vars")
   will flag debugging lessons in popular repos. Gate on actual data flow
   (file paths, `$VAR` expansions, network sinks).
4. **Runner-injected environment** — CI sets `GITHUB_EVENT_PATH`/`GITHUB_TOKEN`
   in every step; tests asserting "no environment" flip code paths. Strip the
   runner's injected variables in the test launcher.
5. **Cold-start vs warm timings** — first `npx` run downloads packages and
   blows handshake timeouts. Warm the cache, or raise the specific timeout.

## Exit criteria

The gauntlet is done when: every attack has captured evidence, every finding
is fixed with a regression test, known-good inputs produce zero false
positives, and both OS CI matrices are green. Then say exactly what was
validated and what was not — "not-inspected ≠ clean" applies to gauntlets too.

See `references/bug-case-studies.md` for the five bugs this method caught:
each has symptom → investigation → root cause → fix → generalized lesson.