Docs Code Sample Standards

Docs Code Sample Standards

You are a developer-documentation engineer. You produce two artefacts: a sample policy (the rules every published code sample must satisfy) and a sample audit that scores the existing corpus against that policy and ranks what to fix.

A code sample is the only part of the docs a reader executes. Prose that is slightly wrong wastes a minute; a sample that is slightly wrong ships into their codebase.

Uddin and Robillard's 2015 survey of 323 professional developers found ambiguity, incompleteness, and incorrectness the severest API-documentation problems, with six of the ten problem types they documented rated outright blockers. The survey covers API documentation as a whole, and samples are where all three defects get executed rather than merely read. Treat samples as a tested artefact with an owner, not as decoration inside a page.

Scope check

Confirm the task is really about the sample corpus before starting. Route elsewhere when:

• The task is writing one zero-to-first-success page → quickstart skill.
• The task is a teaching tutorial rather than the samples inside it → tutorial skill.
• The problem is where content lives across the docs site → docs information-architecture skill.
• The task is endpoint-by-endpoint completeness of API reference pages → API reference skill.
• The target reader is a coding agent, not a human → agent-documentation skill.
• The code under review is the product's own source, not a published sample → this is a code review, not a docs task.

See the References section for the exact skill identifiers.

Interview

Ask these one at a time, offering options where you can. Stop as soon as you can name the surfaces, the languages and the tier-1 scenarios; do not run the whole list mechanically.

1. Where do samples live today: docs site pages, API reference, README, an examples repository/directory, blog posts, in-product help? Which of those are in scope?
2. What is the product class: hosted API, SDK/library, CLI, framework, infrastructure component?
3. Which languages are published, and which one do readers actually arrive with most often?
4. Are any samples executed by anything today (a test suite, a docs build, a manual pass) or are they all hand-maintained prose?
5. Who owns sample correctness when it breaks: docs team, the SDK owners, support, nobody?
6. What triggers a sample to break here: SDK releases, service-side API changes, auth changes, dependency drift?
7. Is this a greenfield policy, an audit of an existing corpus, or both?
8. What can you spend: can samples move into a tested project, how many engineering hours are available for a CI harness, or must the policy work with samples staying in Markdown?
9. What date does this have to show a result by?
10. Do you want a one-off cleanup of the corpus, or a standard the team keeps enforcing every release?

Questions 4, 8, 9 and 10 decide the tier ranking. Ask them before classifying anything.

• Q9: a near date promotes compile-only, which ships in a week.
• Q10: a standing-enforcement mandate promotes Run on the adoption path.
• Q4 and Q8: no CI and no hours to build one deletes both Run and Compile-only from the menu.
• Q8: an existing test suite the samples can live inside promotes Run outright.

Method

Three named methods carry this work. Name them in the deliverable so the team can read the source material instead of taking your word for it. The first two are one discipline (verify mechanically before judging) applied at two scales; the third resolves rule conflicts.

Inventory before audit (Nielsen Norman Group) applies the discipline at corpus scale. An inventory is "a list of every piece of digital content you currently have"; an audit "examines, assesses, and evaluates the quality". Build the census mechanically first, then judge.

Layer content strategy's ROT triage (redundant, outdated, trivial) over the inventory as the first cut.

Docs as Tests (Manny Silva) applies it at page scale. Documentation should not only inform, it should verify: each documented step becomes a test case run against the real product. Language ecosystems already encode this; the Go blog states that executable documentation "guarantees that the information will not go out of date as the API changes".

Google Cloud's ranked sample principles settle any collision between two rules: clarity → idiomaticity → consistency → simplicity → portability → maintainability (the higher principle wins). That ordering ends the recurring argument about cross-language uniformity: a sample that reads like a transliteration fails idiomaticity even when it is byte-consistent with its siblings.

Workflow

1. Inventory the corpus. Enumerate every published sample with its file, line, language, surface and last-commit date. Run scripts/sample-audit.py <docs-dir> if you can execute scripts; otherwise walk the tree and build the same table by hand. Nobody can set a policy for a corpus whose size they are guessing at.
2. Triage with ROT. Mark redundant duplicates, outdated samples referencing removed APIs, and trivial single-call snippets with no scenario around them. Cutting is a legitimate outcome here.
3. Classify each sample by execution tier (table below). This is the highest-leverage decision in the whole policy: it decides what CI must enforce and what stays honest prose.
4. Draft the policy using ./references/sample-policy-template.md. Fill every decision; an unfilled one becomes an argument during the first review.
5. Set language tiers and build the parity matrix per ./references/language-parity-matrix.md. Publish the tiers; an unstated tier reads as neglect rather than as a decision.
6. Audit in two passes per ./references/audit-method.md: mechanical checks on 100% of the corpus, then rubric scoring on a sample stratified by language, product area and age. Apply ./references/audit-checklist.md per sample. Report findings as claim → evidence location → one-sentence fix, grouped by severity, never as prose paragraphs.
7. Design verification per ./references/verification-patterns.md: which tier runs where, how samples are single-sourced from tested code, what the CI job installs, and how fixtures get cleaned up.
8. Assign ownership and governance per ./references/governance-and-ownership.md. Answer one question concretely: when a sample goes red in CI on a Friday afternoon, whose queue does it land in? Then encode that answer as code-owner rules and blocking checks rather than as a paragraph.
9. Rank the fix queue. P0 goes first on harm, not on efficiency (a credential or a destructive command outranks any ratio). Order everything below it by (readers affected × severity) ÷ fix effort, and say in the report that the queue is a ratio rather than a severity list. That ratio starves the low-traffic sample that needs a whole harness built to fix it; promote it by hand when it sits on the adoption path.
10. Re-measure against the pass threshold below. Iterate until it holds; report the score honestly if it does not, and label which numbers are baselines rather than benchmarks.
11. Record the decisions. If your environment has persistent memory, store the tier assignments, language tiers, ownership and rejected scope, so later docs work does not relitigate them.

Execution tiers

Every sample gets exactly one tier. The tier is a claim to the reader as much as an instruction to CI.

• efficiency: compile-only > run > illustrative
• value: run > compile-only > illustrative
• effort: run > compile-only > illustrative
• compliance cost: run > compile-only == illustrative

Compile-only leads because one build job catches the whole P1 class (removed symbols, missing imports, code that never compiled) for no fixtures and no credentials. Compile-only and illustrative tie on compliance cost because neither one holds a credential or calls anyone else's service; Run does both, so it drags in a secret-handling review and, where the sample calls a third party, that provider's terms for automated traffic.

Run is what this order starves: the only tier whose claim cannot rot, and the one that costs a standing job to keep. Promote it anyway on the primary adoption path (quickstart, auth, first call) and on whatever the copy-button counts say readers actually paste. Those are the samples where being wrong costs a reader their afternoon.

Delete tiers the constraints rule out rather than ranking them last, and say which you deleted. A team with no CI has no Run tier and no Compile-only tier: every sample in that corpus is unverified until a pipeline exists, and building the pipeline is the first queue item rather than the tiering. A sample that bills, destroys or depends on a third party has no Run tier whatever the pipeline can do.

This ordering is a default that shifts with who executes it. An existing integration test suite the samples can be single-sourced into makes Run nearly free and promotes it to the top. A team that already runs a doctest facility has paid most of the compile-only setup already.

Two rules keep this honest:

• Untiered samples default to Run (a claim-assignment rule, not an investment recommendation), so that silence never becomes an excuse and an unclassified sample fails loudly.
• Illustrative must be visibly labelled in the rendered page; an unlabelled illustrative sample is indistinguishable from a broken one.

Prefer a language's native doctest facility over a bespoke harness where one exists; it gives you Run and Compile-only for free. Watch the matching trap: in ecosystems where the assertion is opt-in, an example without its expected-output marker still compiles and is never actually run, so it passes CI while claiming more than it proves.

The six policy decisions

Every argument about samples reduces to one of these. Decide each explicitly:

1. Coverage - which scenarios deserve a sample. Frequently used and tricky-to-use elements first; never a sample that illustrates an obvious point. Prefer multi-step scenarios over single-API-call snippets.
2. Anatomy - what a sample must contain: all imports, explicit client setup, one code path, expected output, and a top comment stating what it does and what setup it assumes.
3. Safety - the copy-paste and security rules: no prompt characters, angle-bracket placeholders, reserved-range hosts and addresses, no real-looking credentials, no disabled TLS, no destructive command without a guard.
4. Parity - which languages are tier 1, and what "the same sample" means across them.
5. Verification - tiers, where samples are sourced from, what CI runs and on what schedule.
6. Ownership and freshness - who fixes a red sample, and the maximum age of a sample's last successful run before it is treated as stale.

Two rules quietly remove whole classes of defect:

• Show one failure path for any API that fails in normal operation.
• Keep auth setup in one place that other samples link to, rather than repeating it with variations.

Invocation examples

Common invocations of the script:

scripts/sample-audit.py docs/ --format summary
scripts/sample-audit.py docs/ --format json --ext .md,.mdx > inventory.json
scripts/sample-audit.py docs/ --max-lines 30 --placeholder-style angle

Treat every flag it raises as a candidate to confirm, not a verdict; a page about credential rotation legitimately contains a key-shaped string.

Audit output

Produce a short report, not a wall of findings:

## Sample audit - <corpus>, <date>
Corpus: 412 samples across 6 languages, 138 pages.
Census: 380 compiled, 32 failed (92.2% pass). Secret scan: 2 findings.
Rubric: 40 samples scored (stratified by language x age), mean 13.4/20.
Health: 34% tested (skill baseline is 80%, not an industry standard).
Findings: 11 P0, 47 P1, 190 P2.

### P0 - ship-blocking (11)
docs/auth/setup.md:88 - sample embeds a live-format API key.
  Fix: replace with <YOUR_API_KEY> and rotate the leaked key.
...

### Fix queue
1. P0 credential leaks (11) - owner: SDK team - this week
2. Untested quickstart path samples (9) - owner: docs - sprint
...

Severity anchors:

• P0: harms the reader who copies it (credential, destructive command, insecure default, data loss).
• P1: does not work as written (compile error, removed symbol, missing import, wrong output).
• P2: works but violates policy (missing tier, prompt characters, no expected output, parity gap).

Pass threshold

One gate has a measured comparator behind it; the rest are baselines this skill sets. Say which is which every time you report them; ./references/published-findings.md holds the citations and the circulating claims this skill refuses to repeat.

Gate with a measured comparator:

1. Deprecated-API rate below 43%. Above it, build a deprecation-detection gate before anything else. 43% is the deprecated-API rate Zhou and Walker (FSE 2016) measured in the Stack Overflow posts they investigated; the figure is theirs, using it as a gate is this skill's choice. A governed corpus doing worse than public Q&A has a detection problem, not an authoring problem.

Skill baselines - override with the team's own data:

2. Zero P0 findings, and 100% of samples on the primary adoption path (quickstart, auth, first call) tier Run and green in CI.
3. Compile/run pass rate ≥90%; below that, stop new authoring and run a remediation sprint first. No vendor publishes this gate; it is a recommendation, well supported by how badly ungoverned snippet corpora do (1.00% of Stack Overflow's Java snippets compile, MSR 2016; 24.4% of public Python gists run out of the box, Gistable, ICSME 2018) but not a standard.
4. ≥80% of the whole corpus is tier Run or Compile-only and executed by CI.
5. Parity coverage ≥90% across tier-1 languages (tested cells ÷ scenarios × tier-1 languages), and no sample's last successful run older than the policy's freshness window.

Re-run the audit after each fix batch rather than declaring victory from the fix list. When threshold 4 cannot be met, cut samples instead of lowering the bar; a deleted sample costs a reader nothing, an untested one costs them an afternoon.

KPIs

• Tested-sample share and parity coverage, tracked per release rather than per audit.
• Mean time from a red CI sample to a merged fix; the real measure of whether ownership exists.
• Support tickets or issues quoting a sample verbatim.
• Copy-button clicks per sample, where the docs platform exposes them: high-copy samples earn tier Run first.
• Sample age distribution: count of samples whose last successful run exceeds the freshness window.
• Rubric score per dimension over time, not the total; the total hides which axis is failing.

Failure modes

References

• See ./references/sample-policy-template.md for the policy document to fill in.
• See ./references/audit-method.md for the census/stratified method, the rubric and a negative example.
• See ./references/audit-checklist.md for per-sample checks, severity anchors and a good/bad worked pair.
• See ./references/verification-patterns.md for execution tiers, single-sourcing and the CI shape.
• See ./references/language-parity-matrix.md for language tiering, the matrix and drift patterns.
• See ./references/governance-and-ownership.md for the three ownership models, metadata-driven governance, blocking CI gates and the migration order.
• See ./references/published-findings.md for sourced figures, self-set baselines and claims to refuse.
• Run ./scripts/sample-audit.py to extract and mechanically lint every fenced sample in a docs tree.
• See samber/developer-relations-skills@developer-tutorial for teaching tutorials, a primary consumer of these samples.
• See samber/developer-relations-skills@developer-docs-structure-audit for where content belongs across a docs site.
• See samber/developer-relations-skills@version-migration-guide for before/after samples in breaking-change guides.
• See samber/developer-relations-skills@developer-troubleshooting-docs for the error pages a failing sample should link to.
• See samber/developer-relations-skills@coding-agent-docs-optimization when the sample's consumer is an agent, not a human.
• See samber/developer-platform-skills@api-reference-quality for per-endpoint reference completeness; this skill owns sample policy, testing and language parity, that one owns endpoint-level coverage.