Source Snapshot
The core move: separate retrieval from consumption. Retrieve once,
deterministically, into a pinned artifact with provenance. Then agents and the LLM read
the artifact, never the live source. Same input → same artifact → same downstream
behavior. Live WebFetch/MCP on every run is the non-determinism you're removing.
When to snapshot vs. fetch live
| Situation | Do |
|---|---|
| Fact gates a decision and is reused (docs, registry data, schemas, versions) | Snapshot to a pinned artifact, commit it |
| One-off exploration, throwaway lookup | Fetch live (WebFetch, MCP, c7search) — no artifact |
| Value changes the build output if it drifts (IaC generation, pinned deps) | Snapshot + pin a version; refresh on a cadence, never live-per-run |
If you find yourself fetching the same URL/endpoint across runs, that's the signal to snapshot it.
Choose the extractor by content type
- Article / blog / prose → main-content extractor (Defuddle, or Mozilla Readability)
→ Markdown. Strips nav/ads/chrome; the cleaned result is small and stable. Defuddle
is often not a PATH binary — the npm package is
defuddle(the olddefuddle-cliis deprecated, "merged into defuddle"; sameparse … --mdinterface). The producer resolves a runner automatically (installeddefuddlebinary →pnpm dlx defuddle→bunx→npx defuddle) and never pins@latest, so a cached package is reused instead of re-downloaded. If no prose extractor is available, it falls back to markitdown. Don't hand-write annpx …@latestcall — that forces a reinstall and is blocked outright in pnpm-enforced environments. - Why not just markitdown for an article? markitdown can fetch a URL — but it does a
faithful full-page HTML→Markdown conversion (it only drops
<script>/<style>; nav, sidebar, and footer chrome come through). For a prose article you want the boilerplate gone, which is exactly what a main-content extractor does. That's the only reason defuddle is preferred here — not any inability of markitdown to read web pages. - Reference docs, API pages, tables, specs → the
markdown-converterskill (markitdown). Structure — headings, tables, links — is the part you'll cite, so preserve it rather than flattening to prose. - APIs / registries / anything with a schema → request structured output (JSON/YAML)
and store it as data, not prose. A
registry-snapshot.jsonthe verifier checks by key beats a page it matches by string. Prefer the source's cached/offline mode (e.g. a cached provider schema) over a live call where one exists.
The procedure
- Fetch the source (record the exact URL/endpoint + the time, from
args— the clock is unavailable mid-run, so pass timestamps in). - Extract / normalize with the right extractor for the type (above).
- Write a provenance header, then the content, to a stable path. Name the file so the same source maps to the same path (slug or content hash).
- Commit the artifact. Review changes with
git diff— a snapshot diff is how you see upstream drift.
Provenance header (every snapshot)
For Markdown artifacts, a YAML front-matter block:
---
source_url: https://...
retrieved_at: 2026-06-11T00:00:00Z # passed in; do not invent
extractor: markitdown 0.x | defuddle | readability
content_sha256: <hash of the normalized body>
pinned_version: "1.2.3" # for versioned sources (registry, package)
---
For JSON artifacts, the same fields under a top-level "_provenance" key. Provenance is
what makes a snapshot auditable and a citation trustworthy.
Pinning & refresh
- Pin a version for versioned sources (a registry module, a package). The snapshot is tied to that version; bumping it is a deliberate, reviewable change.
- Refresh on a cadence, not per-run — a scheduled job or a
make refresh-*target that re-snapshots and commits. The diff is the changelog. - Never silently fall back to a live fetch when a snapshot is stale; surface staleness so the caller decides.
Producer: scripts/snapshot.py
A stdlib-only helper that makes the routing above resilient — it detects which extractors are actually installed, picks one by content type, falls back when the preferred is missing, and fails cleanly (exit 1, structured error) when none can handle the type — it never crashes or fabricates.
Run it by its absolute path — it lives in this skill's base directory (announced when the
skill loads, usually ~/.claude/skills/source-snapshot); a bare snapshot.py won't resolve
from the repo you're working in. It's stdlib-only, so uv or python3 both work. Define a
srcsnap function at the start of each command block (a function passes arguments correctly
under bash and zsh — a plain $VAR doesn't word-split in zsh — and shell state doesn't persist
between tool calls, so re-declare it per block):
# uv is the preferred runner; resolve it even when it's not on a bare PATH (non-interactive
# shells often drop ~/.local/bin or the Homebrew bin -> `uv: command not found`):
UV="$(command -v uv || ls "$HOME/.local/bin/uv" "$HOME/.cargo/bin/uv" /opt/homebrew/bin/uv /usr/local/bin/uv 2>/dev/null | head -1)"
srcsnap() { "$UV" run python "$HOME/.claude/skills/source-snapshot/scripts/snapshot.py" "$@"; }
# uv not installed? It's stdlib-only, so plain python3 works too:
# srcsnap() { python3 "$HOME/.claude/skills/source-snapshot/scripts/snapshot.py" "$@"; }
# Decide what would be used, without fetching (deterministic):
srcsnap --format json plan --content-type prose
# Force availability for tests/CI (or to preview a leaner box):
srcsnap --have markitdown plan --content-type prose # -> markitdown (fell back)
srcsnap --have none plan --content-type prose # -> exit 1, clean error
# Extract + write a provenance-stamped artifact:
srcsnap run https://example.com/post --content-type prose --out snapshots/post.md
srcsnap run report.pdf --content-type doc --out snapshots/report.md
Preference: prose → defuddle > readability > markitdown; doc → markitdown.
markitdown is the most reliable fallback (works via uvx 'markitdown[all]' even with no
local install). For defuddle the producer auto-resolves the runner and the verified CLI
shape (defuddle parse {src} --md); you only need an env override for a custom install or
a different prose tool ({src} = source slot):
# Usually unnecessary — the producer already finds defuddle. Override only for a custom path:
export SNAPSHOT_DEFUDDLE_CMD="defuddle parse {src} --md"
export SNAPSHOT_READABILITY_CMD="readable {src}"
Tip — kill the startup friction for good: install defuddle once so it's a PATH binary
and never re-resolves: pnpm add -g defuddle (gives the defuddle command; use the
defuddle package, not the deprecated defuddle-cli). The producer then uses it directly;
no dlx/npx, no per-run download.
Resilience is covered by evals/ (the no-markitdown / no-defuddle matrix) — run
cd evals && uv run python grade.py.
How this feeds the fleet
Snapshots are fact-verifier's tier-1 sources: a committed registry-snapshot.json
or cached doc lets it cite snapshot.json:line deterministically and offline, instead of
hitting the network and hoping. Build the retriever once; reuse the artifact across many
agent runs. See docs/agent-fleet-architecture.md.
Anti-patterns
- Re-fetching the same source live on every run (non-deterministic, slow, rate-limited).
- Storing raw HTML or a whole noisy page when you needed one section.
- Snapshots with no provenance — you can't tell what they are or when they drifted.
- Flattening structured data (tables, schemas) to prose, then trying to match it by string.
- Inventing a
retrieved_ator a CLI invocation — pass timestamps in; verify commands.
