Add Observability

add-observability — design the event stream, then wire it

Logging is a contract, not a scattering of log.info calls. Ad-hoc logging produces a stream nobody can query and everybody is afraid to delete. Decide the shape first, write it into the project's architecture doc, then implement against it.

Rule 0 — never log payloads

Content does not go into logs. At any level. By design. Prompts, request and response bodies, document text, source code, personal data — none of it. Log names and metrics: the input's identifier, the output's identifier, sizes, counts, durations, outcome.

Why this is a rule and not a level:

• Logs are built to leave the machine — into a central, searchable, retained index that far more people can read than can read the source data.
• A "log payloads only at DEBUG" switch is a switch someone flips during an incident and forgets. There should be no switch.
• The content already exists somewhere authoritative (the DB, the artifact on disk, the request store). The log's job is to tell you which one to go read.

If you cannot debug from names + sizes + durations, add a field, not the payload — e.g. prompt_chars instead of the prompt, result_count instead of the results.

Procedure

1. Read the project's architecture doc and find where a cross-cutting concern is documented (a ## Storage-style section). You are adding a sibling ## Observability section.

2. Choose the sink, and justify it. Default: stderr. Check whether stdout is already spoken for — an MCP server, a CLI that pipes structured output, anything whose stdout is a protocol. If so, stderr is a correctness constraint, not a preference: one stray line corrupts the channel. Say that in the doc so nobody "tidies" it later.

3. Choose the format from the destination, not from taste. If the logs will ever reach a log stack (Elasticsearch/Kibana, Loki, Splunk), the answer is JSON Lines — one self-contained event per line, parseable with no custom grok pattern. Offer a text renderer for local terminal use; it renders the same events and is a display choice only.

4. Write the event vocabulary as a table — event.action → emitter → fields. An event is a noun-ish area.thing (enrich.note, tool.search, embed.batch), not a sentence. This table is the artifact that stops the stream drifting into free-form prose.

5. Fix the field names, and treat them as an interface. Stable and flat. Align with ECS where it costs nothing: @timestamp (ISO-8601 UTC), log.level, event.action, event.duration. Renaming a field later breaks every saved query and dashboard built on it — so decide once, in the doc.

6. Add correlation. A run_id (or request/trace id) minted per process or per request, on every event, so one invocation groups into a trace. Add the tenant-ish dimension too (repo, service, customer) — several of them will share one index.

7. Keep records single-line. An exception becomes error.type + error.message on one line. A multi-line traceback gets split by the shipper into unrelated documents, and you lose the very failure you were chasing.

8. The app writes a stream — it does not talk to the log stack. No Elasticsearch client, no network calls on the logging path. A collector (Filebeat/Promtail/vector) ships it. This keeps the service runnable offline and independent of whether a log stack exists yet.

9. Log the invisible failure modes. Every pipeline has at least one failure that raises nothing — a truncated prompt, a silently dropped batch, a fallback that quietly returns empty. Find it and emit a warning with the measurement that reveals it (e.g. prompt size vs the context window). This is usually the highest-value event in the whole stream.

10. Levels change volume, never content. INFO = start/finish, counts, warnings. DEBUG = per-item events with timings. Payload logging is not a level (see Rule 0). Level and format come from env — a new env var, so follow the project's change-propagation map: config + env example + README + architecture doc, same PR.

11. Wire it at the boundaries, not everywhere: every outbound call (model, HTTP, DB batch), every per-item loop in a long pass, every tool/endpoint entry point, and the run itself. Use a timing context manager so event.duration and outcome=ok|error are automatic rather than remembered.

12. Test the payload rule. Write a test that puts a recognizable identifier (a class name, a secret-looking token) through the logging path and asserts it does not appear in the serialized events, while its size does. The rule then survives the next person who wants to "just temporarily log the request".

Sanity check before you finish

• Can you answer these from the logs alone, with no payloads? What was slowest? Which items failed and where? Did anything approach a limit? How did this run compare to the last one? If not, add fields.
• Would a run that crashed halfway be distinguishable from one still running?
• Is every field name one you are willing to keep for years?

Triggering contract

Should fire: "add logging to the new service" · "нужно логирование, потом в Kibana" · "we can't tell what this job did — add observability". Should NOT fire: "why is this test failing" (debugging, not designing a log stream) · "add a metrics dashboard" (visualization of an existing stream).