Dogfood The Shipped Artifact

dogfood-the-shipped-artifact

The in-process test runner (CliRunner, TestClient, direct module invocation) bypasses the entry-point layer that ships to users. A bug in the front-door routing layer is invisible to 100%-test-pass CI. This skill closes that gap.

The core failure mode

Real receipt (tensor-grep v1.15.0, 2026-06-26): tg search PATTERN PATH --rank was tested via CliRunner — every test passed. The shipped binary crashed with rg: unrecognized flag --rank because the bootstrap front-door (tensor_grep.cli.bootstrap:main_entry) forwarded ALL plain searches to ripgrep before the Typer app ran, and --rank/--bm25 were missing from its allow-list (_TG_ONLY_SEARCH_FLAGS). CliRunner invoked the Typer app directly and NEVER touched bootstrap. Required patch: v1.15.1, same day.

The gap: [project.scripts] tg = tensor_grep.cli.bootstrap:main_entry is the real user path. CliRunner calls the Typer app, skipping main_entry entirely. These are different code paths.

When to invoke

• After any pip / npm / cargo publish.
• After adding a new flag or subcommand to a CLI.
• After changing entry-point / front-door routing logic.
• Before cutting a release tag when changes touch the CLI dispatch layer.
• When "all tests pass" but a flag/feature was added that the suite covers via CliRunner/TestClient only.

The dogfood protocol

1 — Identify the real entry point

pyproject.toml [project.scripts] (Python) / package.json bin (Node) / Cargo.toml [[bin]] (Rust). This is the only production-faithful invocation. Everything else is a test approximation.

2 — Build a clean-env smoke harness (Docker preferred; fresh venv acceptable for quick checks)

# scripts/dogfood/Dockerfile — installs the PUBLISHED wheel, runs the REAL binary
FROM python:3.12-slim
ARG TG_VERSION
RUN pip install --no-cache-dir "tensor-grep==${TG_VERSION}"
RUN tg --version                          # fails the build early on a packaging regression
COPY dogfood_features.py /dogfood_features.py
ENTRYPOINT ["python", "/dogfood_features.py"]

# scripts/dogfood/dogfood_features.py — invoke via subprocess (the REAL binary), never import
import subprocess, sys, json
def run(args, want_exit=0, forbid=None):
    r = subprocess.run(args, capture_output=True, text=True)
    out = r.stdout + r.stderr
    if r.returncode != want_exit or (forbid and forbid in out):
        print(f"FAIL: {' '.join(args)} (exit {r.returncode})\n{out[:200]}"); sys.exit(1)
    return out
# Cover every distinct routing path — not just happy-path --help
run(["tg", "search", "def", "."])
run(["tg", "search", "def", ".", "--rank"], forbid="unrecognized flag")   # bypasses ripgrep
json.loads(run(["tg", "search", "def", ".", "--json"]))
run(["tg", "orient", "."])
assert json.loads(run(["tg", "orient", ".", "--json"]))["routing_reason"] == "orient"
print("ALL DOGFOOD CHECKS PASSED")

docker build --build-arg TG_VERSION=<VERSION> -t tg-dogfood scripts/dogfood
docker run --rm tg-dogfood

3 — Scope the feature matrix to every distinct routing path

Any flag/subcommand that lives in the front-door routing layer (before the app object receives control) CANNOT be exercised by CliRunner/TestClient.

4 — Wire into release verification (not a one-off)

Run dogfood against the published version after each release. Post-publish CI: wait ~90s for PyPI's JSON API to propagate (the simple index lags), then pip install ==version in a clean job and run the harness. A patch-fix cadence: (a) reproduce via the real binary in Docker, (b) add a dogfood test that catches it, (c) fix the routing layer, (d) re-run dogfood in Docker before tagging.

The CliRunner / TestClient bypass table

Hard rules

1. Never declare a CLI release good on CliRunner/TestClient alone — they bypass the entry-point layer.
2. Every new flag that changes routing requires a dogfood test, not just a CliRunner test.
3. Docker is the default dogfood env; a fresh venv is OK only for quick iteration (Docker also catches missing runtime deps the dev env provides transitively).
4. Run dogfood against the PUBLISHED version, not a local/editable build (different wheel layout, possibly missing files).
5. Distinguish install-smoke (does it install + --version/--help?) from feature-smoke (does each command's distinct routing path work?) — both are needed; install-smoke alone would not have caught the --rank crash.

Extend to generated / detached code (v1.17.5 receipt)

The dogfood principle extends beyond the entry-point layer: any generated or detached code (helper scripts that run in a subprocess, self-upgrade helpers injected into a temp path, post-install scripts) must be EXECUTED to verify behavior — reading or reviewing the source is insufficient.

Real receipt (v1.17.5, 2026-06-28): Windows self-upgrade helpers are generated strings injected into a detached process. The checksum-verification gate (sha256 must match before os.replace) was verified by actually compiling the helper code with compile() and asserting the gate behavior (checksum mismatch → exception, no replacement), not by code review alone. Pattern: compile(src, "<helper>", "exec") + exec() in a controlled test, assert that the guard fires BEFORE the side-effecting operation.

More broadly: "trust but verify subagents" — re-run claimed validations. A subagent that says "the test passes" may have tested a synthetic payload shape rather than real production output. Always verify the assertion tests the REAL behavior.

Anti-patterns

Sibling skills

• gh-release-ship-loop — the containing release loop; this skill extends its LIVE-VERIFY phase (presence-check → install-and-exercise-the-real-binary).
• verify-completion-signal — the exit-code-vs-artifact discipline; relevant when the dogfood subprocess exits 0.
• Same "an agent/test's confident pass is a hypothesis" philosophy applies to subagent output, not just the test harness.

History

• 2026-06-26: created from the tensor-grep v1.15.0 → v1.15.1 receipt. tg search --rank tested only via CliRunner; the bootstrap front-door forwarded it to ripgrep which rejected it; 100% of tests passed and the crash shipped. Harness now lives at scripts/dogfood/ and covers every feature routing path against the installed binary.
• 2026-06-28 (v1.17.5): extended to generated/detached code — checksum-gated self-upgrade helper required compile()+exec() behavioral test, not code review. Subagent trust-but-verify pattern added. TG_VERSION example updated to <VERSION> placeholder.