Stale Proof Rendered Numbers

Stale-Proof Rendered Numbers

Convert hardcoded numbers in generated artifacts into templates that resolve from measurement sources at render time, where a missing key is a build error. Hardcoded numbers rot the moment the underlying measurement moves — and they rot silently, which is worse than a visible error. This skill makes staleness loud: numbers are never typed into the artifact, one extractor is the only writer of the stats file, and the renderer refuses to build anything it cannot trace to a source.

When to Use

• A generated artifact (README stats block, status dashboard, benchmark table, coverage badge, report card) displays measured numbers that keep moving as new runs land
• Two artifacts already disagree about the same metric because each was hand-edited at a different time
• A review found scripts or captions that claim their numbers are file-derived but actually hardcode literals
• You are wiring a new living document and want staleness to be impossible from the start
• Point-in-time results of closed measurements are scattered across prose and need one audited home with provenance

Inputs

• Required: The generated artifact(s) displaying measured numbers, and the build step that renders them (script, Makefile target, CI job)
• Required: The source artifacts each number truly derives from (measurement JSON, run logs, ledgers, coverage reports, tool --json output)
• Required: A key naming scheme for metrics (lowercase dotted paths, e.g. bench.api.p95_ms)
• Optional: Closed measurements with no machine-readable artifact — these become FROZEN entries with provenance (default: pin them in the extractor)
• Optional: Figure/plot scripts that display numbers (these get the guarded-figure variant in Step 6)

Procedure

Step 1: Inventory Displayed Numbers and Classify Their Sources

List every number the artifact displays. For each, answer: which file on disk backs this number right now?

# Surface numeric literals in the artifact's source or template data
grep -nE '[0-9]+(\.[0-9]+)?\s*(%|ms|pts?|n=|/)' docs/README.template.md

Classify each hit into one of three buckets:

1. Live-artifact-backed — a measurement file, run log, or ledger exists and is machine-readable (or one --json flag away from it)
2. Closed / frozen — the measurement arc is finished; the raw artifacts predate the pipeline or are gone; only the recorded result remains
3. Not a measurement — version strings, dates of events, prose. Leave these alone; do not template them

Expected: A table of displayed number → source file (or FROZEN + provenance) covering every measured figure in the artifact. Nothing is left as "I remember it was 30%".

On failure: If a number has no locatable source, it is either frozen (pin it in Step 3 with a provenance string naming where the result was recorded) or unsubstantiated (remove it from the artifact — a number nobody can trace should not render).

Step 2: Replace Literals with {metric.key} Placeholders

Turn the artifact's data or template into templates. Every measured literal becomes a placeholder:

## Status (as of {meta.generated_at})

- Test suite: {tests.passed}/{tests.total} passing, {coverage.line_pct}% line coverage
- API p95 latency: {bench.api.p95_ms} ms (n={bench.api.n}, measured {bench.api.as_of})
- v1.0 baseline: {frozen.v1_baseline_p95_ms} ms (closed load test, 2026-05-02)

Key grammar: lowercase dotted paths, at least one dot ([a-z0-9_]+(\.[a-z0-9_]+)+). Requiring the dot keeps the substitution regex from matching legitimate braces elsewhere in the document (code spans, shell snippets).

Expected: grep -nE '[0-9]' on the template shows no measured literals left — only placeholders, prose, and non-measurement numbers from Step 1's bucket 3.

On failure: If a line mixes a measured number with prose punctuation that resists templating, split the placeholder finer ({ladder.lucario.wr_pct}% (n={ladder.lucario.n})) rather than templating a whole preformatted string — one placeholder per number keeps sources auditable.

Step 3: Build the Single Extractor

Write ONE extractor (e.g. tools/doc_stats.py) that reads the real artifacts and emits stats.json. It is the only writer of that file; numbers are never typed into the artifact or the stats file by hand.

FROZEN_METRICS = {
    # Point-in-time results of CLOSED measurements: pinned HERE, one audited
    # place, each with source + as_of. When an artifact-backed source appears,
    # migrate the key OUT of this block into a reader function below.
    "frozen.v1_baseline_p95_ms": {
        "value": "212",
        "source": "release-1.0 load test, docs/perf-report-v1.md (closed 2026-05-02)",
        "as_of": "2026-05-02",
    },
}

def bench_metrics(today: str) -> dict:
    run_path = newest(Path("bench/results").glob("api_*.json"))
    run = json.loads(run_path.read_text())
    source = f"{run_path} ({run['n']} requests)"
    return {
        "bench.api.p95_ms": {"value": str(run["p95_ms"]), "source": source, "as_of": today},
        "bench.api.n": {"value": str(run["n"]), "source": source, "as_of": today},
        "bench.api.as_of": {"value": today, "source": source, "as_of": today},
    }

Every metric entry has exactly the shape {value, source, as_of}:

{
 "generated_at": "2026-07-10",
 "metrics": {
  "bench.api.p95_ms": {
   "value": "187",
   "source": "bench/results/api_2026-07-08.json (500 requests)",
   "as_of": "2026-07-10"
  }
 }
}

Rules that make this stale-proof rather than merely centralized:

• Single writer: only the extractor writes stats.json. A hand edit to stats.json is a defect, not a shortcut.
• Frozen block discipline: closed measurements live in FROZEN_METRICS with a provenance string naming where the result was recorded and an as_of date. This is the ONE audited place for point-in-time numbers. Migrate keys out as live sources appear.
• Fail on missing sources: if a ledger row or artifact the extractor depends on is absent, exit non-zero with a FAIL: message — do not emit a partial stats file silently.
• Commit stats.json, not raw runs: the aggregate stats file is small and reviewable; the raw run logs it derives from can stay gitignored.

See references/EXAMPLES.md for the complete extractor.

Expected: Running the extractor prints each key with its value and as-of date and writes stats.json; re-running it after a new measurement run changes the affected values with zero manual edits.

On failure: If a source is not machine-readable, first check for a --json flag or an existing structured sibling (ledger TSV, JSONL results). Only regex-parse human-oriented output as a last resort, and make the parse fail loudly when the pattern stops matching.

Step 4: Substitute at Render and Hard-Fail on Missing Keys

The renderer resolves placeholders from stats.json at build time. A missing key is a build error — stale can never render silently:

stats = json.loads(Path("docs/stats.json").read_text())
metrics = dict(stats["metrics"])
metrics["meta.generated_at"] = {"value": stats["generated_at"]}

def resolve(text: str) -> str:
    def substitute(match: re.Match) -> str:
        metric_key = match.group(1)
        if metric_key not in metrics:
            sys.exit(f"FAIL: stats.json is missing metric '{metric_key}' "
                     f"— rerun: python3 tools/doc_stats.py")
        return metrics[metric_key]["value"]
    return re.sub(r"\{([a-z0-9_]+(?:\.[a-z0-9_]+)+)\}", substitute, text)

The same guard in R (generalized from a card-rendering pipeline; full version in references/EXAMPLES.md):

metric <- stats$metrics[[metric_key]]
if (is.null(metric)) {
  stop("stats.json is missing metric '", metric_key,
       "' — rerun: python3 tools/doc_stats.py", call. = FALSE)
}
text <- sub(placeholder, as.character(metric$value), text, fixed = TRUE)

Two properties are non-negotiable: the lookup stops the build on a missing key (no .get(key, "N/A"), no warning-and-continue), and substitution of the matched placeholder is literal (fixed = TRUE / replacement from a function), never a second regex pass over the value.

Expected: A full build renders the artifact with every placeholder resolved; the output contains no {...} placeholder-grammar matches.

On failure: If a legitimate brace expression in the document trips the substitution, tighten the key grammar (the required dot usually suffices) rather than loosening the failure — silent skips reintroduce the rot this pipeline exists to prevent.

Step 5: Show the As-Of Date on the Artifact

Every rendered artifact displays when its numbers were extracted, sourced from stats.json — never typed:

• A document-level line: Status as of {meta.generated_at} from the stats file's generated_at
• Per-number dates where measurements move independently: (measured {bench.api.as_of})

This is the reader-facing half of the honesty contract: the build guarantees numbers match sources at extraction time, and the as-of date tells the reader what that time was.

Expected: The rendered artifact visibly carries an as-of date that changes when (and only when) the extractor reruns.

On failure: If the artifact has no natural place for a date, add a caption, footer, or badge suffix. An artifact showing measured numbers without an as-of date fails review under this skill.

Step 6: Guard Generated Figures — the stopifnot Variant

Placeholders cover text substitution. Generated figures need the same honesty a different way: each figure script re-reads its source files at render time and guards every value it displays, so a plot literally cannot render a number it did not just read.

bench <- jsonlite::fromJSON("bench/results/api_2026-07-08.json")
stopifnot(
  is.numeric(bench$p95_ms), bench$p95_ms > 0,
  bench$n >= 100  # the caption claims n >= 100 — enforce it
)
p95_label <- sprintf("p95 = %.0f ms (n = %d)", bench$p95_ms, bench$n)
# every annotation below derives from `bench`; no numeric literals in labels

Two guard styles compose:

• Derivation guard: displayed values are computed from the freshly read data, and stopifnot() asserts the invariants the figure's message depends on (row counts sum to the traced total, rates are in range, the cited n is met)
• Provenance guard: when a caption cites a figure the data file does not carry (a sample size, a seed), assert the source document still contains that string — stopifnot(grepl("n=100", readme_text, fixed = TRUE)) — so the caption never states a number the source stopped backing

This makes "claimed file-derivation" enforceable: an adversarial review of scripts that claimed to read files but hardcoded literals is what motivated this variant. In Python, assert/sys.exit guards on freshly loaded data serve the same role.

Expected: Every figure script reads its sources at render time; deleting or changing a source file makes the figure build fail or visibly change — never silently render the old number.

On failure: If a script has literals in labels or annotations, replace each with a value computed from the loaded data plus a guard. If the value exists nowhere machine-readable, route it through the extractor's FROZEN block and read stats.json instead.

Step 7: Wire the Flow and Prove It

Document and wire the pipeline as measure → extract → render:

python3 tools/doc_stats.py          # extract: artifacts -> docs/stats.json
python3 tools/render_docs.py        # render: template + stats.json -> README.md

Then run the two proofs (these are the skill's acceptance tests):

# Proof 1 — bogus key fails the build
printf '\n- bogus: {does.not_exist}\n' >> docs/README.template.md
python3 tools/render_docs.py; echo "exit=$?"     # expect FAIL message, exit != 0
git checkout -- docs/README.template.md

# Proof 2 — live source auto-refresh, zero manual edits
# (land a new measurement run, e.g. bench/results/api_<today>.json)
python3 tools/doc_stats.py && python3 tools/render_docs.py
git diff --stat docs/ README.md                   # only generated files changed

Proof 2 is the payoff: in the originating pipeline a rendered card auto-updated n=70 → n=200 from a still-running measurement's artifact with no hand edits.

Expected: Proof 1 exits non-zero naming the missing key and the rerun command. Proof 2 shows the new values and as-of date in the rendered artifact with git diff touching only extractor output and rendered files.

On failure: If Proof 1 renders the bogus placeholder or exits 0, the renderer has a silent-default path — remove it (Step 4). If Proof 2 requires any hand edit, a number is still typed somewhere; trace it and move it behind the extractor.

Validation

• Every measured number in the artifact is a {metric.key} placeholder; none are typed literals
• Exactly one extractor writes stats.json; every entry has value, source, and as_of
• Closed measurements are pinned in the extractor's FROZEN block with provenance strings and as-of dates, and nowhere else
• The renderer hard-fails on a missing key: the bogus-key proof (Step 7) exits non-zero with an actionable message
• A new measurement run propagates to the rendered artifact via extract + render with zero manual edits (Step 7, Proof 2)
• The rendered artifact displays an as-of date sourced from stats.json
• Figure scripts re-read their sources at render time and guard displayed values (stopifnot or equivalent)
• The measure → extract → render flow is documented next to the pipeline (README or script headers)

Common Pitfalls

• Silent defaults defeat the design: metrics.get(key, "N/A"), warn-and-continue, or rendering the raw placeholder on a miss turns a build error back into silent rot. The missing-key path must stop the build.
• Hand-editing stats.json: it is generated output with an audit trail, not a config file. A hot-fix typed into it will be overwritten by the next extractor run — or worse, won't be, and diverges. Fix the source artifact or the extractor.
• Frozen keys that never migrate: FROZEN is a holding pen with provenance, not a destination. When a machine-readable source appears for a frozen key, move it to a reader function — review the frozen block whenever a new artifact type lands.
• Regex substitution mangling values: substituting with the value as a regex replacement string breaks on \, $, or & in values. Use literal replacement (fixed = TRUE in R, a replacement *function* in Python re.sub).
• Placeholder grammar too loose: \{(\w+)\} matches legitimate braces in code spans and shell text. Require the dotted-path grammar so only metric keys match — and unknown matching keys still fail loudly.
• Figures that claim derivation but hardcode: a script that loads a file and then annotates with a typed literal passes casual review. The guard (Step 6) is what makes the claim checkable — insist on it for every displayed value.
• Templating non-measurements: version numbers, event dates, and prose counts do not belong in stats.json. Templating them adds churn without honesty gains; Step 1's bucket 3 stays untouched.

Related Skills

• generative-recipe-dsl - data-driven artifact generation; this skill keeps the numbers those generated artifacts display honest
• build-parameterized-report - parameter-driven report rendering; combine with this skill so report parameters select which metrics render while the extractor guarantees their freshness