Interface Facts

Interface Facts

A model's knowledge of an external interface is a snapshot of whenever it was trained, and vendors ship. So an external interface fact is unverified until it is fetched. The cost is asymmetric: a wrong fact about our own code fails a gate, a wrong fact about someone else's interface gets committed as prose and is believed for months.

Covers CLI flags and output formats, API fields and endpoints, model ids, context windows, token accounting, pricing, rate limits, config schemas, and version-gated behaviour.

The order of evidence

1. Our own code that already touches the interface. If an adapter, parser or hook reads the thing, that code is the evidence — it was written against a real probe and it is under test.
2. The installed binary, interrogated. --help, --version, a --dry-run, an invalid flag whose error names the valid set. This outranks documentation when the question is what this version accepts: the docs describe a release, the binary on the path is the one the command will run.
3. The vendor's current documentation, fetched now.
4. The vendor's changelog, which dates the change.
5. The vendor's source or issue tracker, where a documented feature that does not work gets contradicted.
6. Third-party writing — weakest, never sufficient alone.

A landing page or feature grid is marketing. A README is a claim.

Prefer our own code over our own prose

This is the rule that gets skipped and the one that has cost the most. A design document's summary of an interface is a claim about a claim. Read the implementation before quoting the document, and read the section a summary cites before trusting the summary.

Measured instance: a design note carried "copilot has no counts" in a components table while section 7.5 immediately below documented the extraction mechanism and runner.py implemented it — mapping copilot's session.shutdown modelMetrics onto our usage split, with summation semantics verified against 15 local stores. The stale summary was read, it matched a prior assumption, and it was repeated to the owner as fact.

Pin the version and the date

Record what you verified against; the answer has a shelf life. This repo already does it where it does it at all — copilot 1.0.77, codex 0.146.0, "re-probed 1.0.75". An unpinned capability claim cannot be re-checked, only re-argued.

How to fetch

Scope the search to the vendor so third-party summaries cannot outrank the source, then fetch the specific page and ask a narrow question — exact flag names, exact field names, whether a machine-readable output mode exists — rather than "tell me about X". Use browser tools only when a page needs JavaScript to render.

Machine-readable routes, reached only after the binary

Run the installed binary first. This table makes rung 3 cheap, and cheap is how it goes wrong. Recorded incident, 2026-08-19 on this repo's own toolchain before it was updated: uv was installed at 0.11.28 while 0.12.5 was released, and uv's changelog gives 0.12.0 "Define build systems by default with uv init" — so the current documentation described a uv init the installed binary did not have. uv init --help on 0.11.28 already listed both --package and --no-package, so even a flag probe missed it and only running the command found the default. A route below answers "what does this release document", never "what will this command do".

Every row was fetched, not recalled, on 2026-08-19. Follow redirects, and read an unexpected hop or a 404 as route rot rather than as absence of the fact: docs.claude.com/llms.txt 301s to platform.claude.com/llms.txt, and developers.openai.com/codex/llms.txt 308s to learn.chatgpt.com/docs/llms.txt while the OpenAI index still advertises the old path. Probe the row before you quote it.

GitHub Docs names its Article, Article Body, Page List and Search APIs the preferred route for automated tools, and the Search API requires an undocumented client_name parameter: the example GitHub prints in its own llms.txt, /api/search/v1?query=actions&language=en&version=free-pro-team@latest, answers 400 Missing required parameter 'client_name' for external requests, and the same URL with &client_name=<something naming us> answers 200. /api/article and /api/article/body answered 200 without it — send it anyway rather than depending on that asymmetry holding.

Three refusals go with the table. Never cache or commit fetched documentation: a cache lands in a consumer tree and the upstream licence is not ours to redistribute. Never cite an aggregator, mirror or docs-summarising site as the source of an interface fact — only the vendor's own host counts. Never record a fact a route gave you without the version it describes and the date you fetched it.

Distinguish "the capability exists" from "we can reach it"

Separate questions; conflating them produces a confident wrong answer in either direction. Copilot reports per-model token usage interactively via /usage, and every plan moved to token-based billing on 2026-06-01 — yet the docs show no flag or JSON mode that emits it, and the mechanism our adapter uses is out of band: the dispatch supplies the session UUID with --session-id, which makes the store path knowable before the store exists. "It is documented" does not mean "our dispatch can read it", and "not on stdout" does not mean unmeterable.

State the claim at the precision the evidence supports

A correct-but-vague note invites a future reader to "fix" it. Copilot genuinely has no config-file deny — but a config file does exist, ~/.copilot/permissions-config.json, which stores approvals. Written as "has no config file", the next reader finds that file, reads the note as stale, and removes the invocation-time injection it justifies.

Record it where the next reader will hit it

A fact established and not written down gets re-derived or re-guessed. Put it on the bead (basicly tracker write -- comments add <id> "verified <date> against <url>: ..."), and correct the document that carried the stale version rather than adding the new claim beside it — two claims that disagree cost more than one that is wrong.

When a repo document and the vendor disagree

The vendor wins on what the interface does; the repo wins on what we decided. Correct the document in place and name what refuted it — a fresher number pasted over an older one with no provenance is the same unchecked declaration one generation on. Where a value we depend on can be checked against data we already record, wire the comparison as a gate: runner.window_violations exists because a declared context window of 200000 sat against a recorded occupancy of 223221 for months, mechanically detectable the whole time.