# Dogfood The Shipped Artifact

> Use AFTER publishing any CLI, tool, or service package — and BEFORE declaring a release good. Triggers on: 'release shipped', 'published to PyPI/npm', 'v1.X.Y is live', 'the tests all pass', 'CliRunner passes', 'TestClient passes'. Never trust an in-process test runner as proof the REAL shipped binary works: CliRunner (Click/Typer) and FastAPI TestClient invoke the app object DIRECTLY, bypassing the entry-point / front-door bootstrap layer. Install the PUBLISHED artifact in a clean env (Docker or fresh venv) and invoke the REAL binary across every feature flag combination before declaring the release good.

- Skill: `oimiragieo/dogfood-the-shipped-artifact` (Agent Skill)
- Install (CLI): `npx skillmds@latest add oimiragieo/dogfood-the-shipped-artifact`
- Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/dogfood-the-shipped-artifact/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: oimiragieo (https://skillmd.com/u/oimiragieo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/oimiragieo/dogfood-the-shipped-artifact

---


# dogfood-the-shipped-artifact

The in-process test runner (CliRunner, TestClient, direct module invocation) bypasses the entry-point layer that ships to users. A bug in the front-door routing layer is invisible to 100%-test-pass CI. This skill closes that gap.

## The core failure mode

Real receipt (tensor-grep v1.15.0, 2026-06-26): `tg search PATTERN PATH --rank` was tested via `CliRunner` — every test passed. The shipped binary crashed with `rg: unrecognized flag --rank` because the bootstrap front-door (`tensor_grep.cli.bootstrap:main_entry`) forwarded ALL plain searches to ripgrep *before* the Typer app ran, and `--rank`/`--bm25` were missing from its allow-list (`_TG_ONLY_SEARCH_FLAGS`). `CliRunner` invoked the Typer app directly and NEVER touched bootstrap. Required patch: v1.15.1, same day.

**The gap:** `[project.scripts] tg = tensor_grep.cli.bootstrap:main_entry` is the real user path. `CliRunner` calls the Typer app, skipping `main_entry` entirely. These are different code paths.

## When to invoke

- After any pip / npm / cargo publish.
- After adding a new flag or subcommand to a CLI.
- After changing entry-point / front-door routing logic.
- Before cutting a release tag when changes touch the CLI dispatch layer.
- When "all tests pass" but a flag/feature was added that the suite covers via CliRunner/TestClient only.

## The dogfood protocol

### 1 — Identify the real entry point

`pyproject.toml` `[project.scripts]` (Python) / `package.json` `bin` (Node) / `Cargo.toml` `[[bin]]` (Rust). This is the only production-faithful invocation. Everything else is a test approximation.

### 2 — Build a clean-env smoke harness (Docker preferred; fresh venv acceptable for quick checks)

```dockerfile
# scripts/dogfood/Dockerfile — installs the PUBLISHED wheel, runs the REAL binary
FROM python:3.12-slim
ARG TG_VERSION
RUN pip install --no-cache-dir "tensor-grep==${TG_VERSION}"
RUN tg --version                          # fails the build early on a packaging regression
COPY dogfood_features.py /dogfood_features.py
ENTRYPOINT ["python", "/dogfood_features.py"]
```

```python
# scripts/dogfood/dogfood_features.py — invoke via subprocess (the REAL binary), never import
import subprocess, sys, json
def run(args, want_exit=0, forbid=None):
    r = subprocess.run(args, capture_output=True, text=True)
    out = r.stdout + r.stderr
    if r.returncode != want_exit or (forbid and forbid in out):
        print(f"FAIL: {' '.join(args)} (exit {r.returncode})\n{out[:200]}"); sys.exit(1)
    return out
# Cover every distinct routing path — not just happy-path --help
run(["tg", "search", "def", "."])
run(["tg", "search", "def", ".", "--rank"], forbid="unrecognized flag")   # bypasses ripgrep
json.loads(run(["tg", "search", "def", ".", "--json"]))
run(["tg", "orient", "."])
assert json.loads(run(["tg", "orient", ".", "--json"]))["routing_reason"] == "orient"
print("ALL DOGFOOD CHECKS PASSED")
```

```bash
docker build --build-arg TG_VERSION=<VERSION> -t tg-dogfood scripts/dogfood
docker run --rm tg-dogfood
```

### 3 — Scope the feature matrix to every distinct routing path

Any flag/subcommand that lives in the front-door routing layer (before the app object receives control) CANNOT be exercised by CliRunner/TestClient.

| Feature | Routing path | CliRunner reaches it? | Dogfood reaches it? |
|---|---|---|---|
| `tg search PATTERN PATH` | bootstrap → ripgrep passthrough | No (bypasses bootstrap) | YES |
| `tg search PATTERN PATH --rank` | bootstrap → detect `--rank` → Python handler | No | YES |
| `tg orient PATH` | bootstrap → dispatch to orient command | No | YES |
| `tg search --help` | Typer app help | Yes | YES |

### 4 — Wire into release verification (not a one-off)

Run dogfood against the **published** version after each release. Post-publish CI: wait ~90s for PyPI's JSON API to propagate (the simple index lags), then `pip install ==version` in a clean job and run the harness. A patch-fix cadence: (a) reproduce via the real binary in Docker, (b) add a dogfood test that catches it, (c) fix the routing layer, (d) re-run dogfood in Docker before tagging.

## The CliRunner / TestClient bypass table

| Test harness | What it invokes | What it bypasses |
|---|---|---|
| `CliRunner.invoke(app, ...)` | Typer/Click app object | entry-point script, bootstrap front-door, any routing before `app()` |
| `FastAPI TestClient(app)` | ASGI app object | server, startup hooks (unless `with TestClient(app):`), real port binding |
| `app.main()` in tests | `main()` function | entry-point registration, script-name detection |
| real binary via subprocess | the full `[project.scripts]` path | nothing — this IS the user path |

## Hard rules

1. **Never declare a CLI release good on CliRunner/TestClient alone** — they bypass the entry-point layer.
2. **Every new flag that changes routing requires a dogfood test**, not just a CliRunner test.
3. **Docker is the default dogfood env**; a fresh venv is OK only for quick iteration (Docker also catches missing runtime deps the dev env provides transitively).
4. **Run dogfood against the PUBLISHED version**, not a local/editable build (different wheel layout, possibly missing files).
5. **Distinguish install-smoke (does it install + `--version`/`--help`?) from feature-smoke (does each command's distinct routing path work?)** — both are needed; install-smoke alone would not have caught the `--rank` crash.

## Extend to generated / detached code (v1.17.5 receipt)

The dogfood principle extends beyond the entry-point layer: any **generated or detached code** (helper scripts that run in a subprocess, self-upgrade helpers injected into a temp path, post-install scripts) must be EXECUTED to verify behavior — reading or reviewing the source is insufficient.

**Real receipt (v1.17.5, 2026-06-28):** Windows self-upgrade helpers are generated strings injected into a detached process. The checksum-verification gate (`sha256` must match before `os.replace`) was verified by actually compiling the helper code with `compile()` and asserting the gate behavior (checksum mismatch → exception, no replacement), not by code review alone. Pattern: `compile(src, "<helper>", "exec")` + `exec()` in a controlled test, assert that the guard fires BEFORE the side-effecting operation.

More broadly: "trust but verify subagents" — re-run claimed validations. A subagent that says "the test passes" may have tested a synthetic payload shape rather than real production output. Always verify the assertion tests the REAL behavior.

## Anti-patterns

| Anti-pattern | Risk |
|---|---|
| "All N tests pass via CliRunner — ship it" | entry-point routing bugs invisible to CliRunner slip through |
| "I tested locally with the editable install" | editable installs resolve deps from the dev env; the published wheel may be missing one |
| "The Docker smoke just runs `--help`" | `--help` invokes the app directly; only full feature invocations exercise front-door routing |
| "The version is on PyPI so it works" | version presence ≠ feature correctness; the entry point could be registered to a renamed symbol |
| "I reviewed the generated helper code — it looks right" | generated/detached code must be EXECUTED (compile + exec in a test) to confirm the guard fires; reading is insufficient |
| "The subagent says the test passes" | re-run claimed validations; a subagent may have tested a synthetic shape, not the real emitted payload |

## Sibling skills

- `gh-release-ship-loop` — the containing release loop; this skill extends its LIVE-VERIFY phase (presence-check → install-and-exercise-the-real-binary).
- `verify-completion-signal` — the exit-code-vs-artifact discipline; relevant when the dogfood subprocess exits 0.
- Same "an agent/test's confident pass is a hypothesis" philosophy applies to subagent output, not just the test harness.

## History

- 2026-06-26: created from the tensor-grep v1.15.0 → v1.15.1 receipt. `tg search --rank` tested only via CliRunner; the bootstrap front-door forwarded it to ripgrep which rejected it; 100% of tests passed and the crash shipped. Harness now lives at `scripts/dogfood/` and covers every feature routing path against the installed binary.
- 2026-06-28 (v1.17.5): extended to generated/detached code — checksum-gated self-upgrade helper required compile()+exec() behavioral test, not code review. Subagent trust-but-verify pattern added. TG_VERSION example updated to `<VERSION>` placeholder.