Observability-First Debugging
Overview
Do not jump from symptoms to fixes when the failure surface is still wide.
Core principle: expand observability until the problem is bounded, then fix the bounded problem.
When to Use
Use this skill when:
- the bug reproduces but the cause is unclear
- multiple layers could be responsible
- a test fails without enough local evidence
- the problem appears flaky, timing-sensitive, or environment-dependent
- a user asks to add logs, instrument code, or investigate before fixing
Do not lead with broad instrumentation when the root cause is already obvious from current evidence.
Rule
Before making speculative code changes, add enough targeted observability to answer the next unknown.
Instrumentation should reduce uncertainty, not just increase log volume.
Workflow
- Define the symptom precisely.
- State the current unknowns.
- Choose the narrowest boundary that can separate likely causes.
- Add targeted instrumentation at that boundary.
- Reproduce and inspect the new evidence.
- Repeat only until the failure surface is bounded.
- Apply the fix to the bounded cause.
- Remove temporary instrumentation and keep only durable observability that has ongoing value.
Where to Instrument
Prefer adding evidence at decision points and boundaries such as:
- request entry and exit
- state transitions
- condition branches with surprising outcomes
- external API or database calls
- retry loops, timeout paths, and exception handlers
- async queue handoffs, worker boundaries, and concurrency joins
- data parsing, normalization, and schema validation points
What Good Instrumentation Looks Like
Good instrumentation is:
- targeted to a concrete hypothesis or unknown
- cheap enough to run during reproduction
- reversible if it is only for local diagnosis
- structured enough to correlate inputs, state, and outputs
- specific about IDs, branches, counts, timing, and error context
Include identifiers that let you stitch a single failing path together across layers.
What to Avoid
- adding logs everywhere without a question they answer
- changing behavior while instrumenting unless necessary for safety
- logging secrets, tokens, passwords, or sensitive payloads
- leaving noisy temporary logs in hot paths after the issue is understood
- declaring a fix before the new evidence isolates the cause
Escalation Pattern
Use the smallest step that can answer the next question:
- existing logs and repro output
- additional local logs or assertions
- structured metrics, counters, or timing
- trace-style correlation across components
- invasive instrumentation only if lighter steps do not bound the issue
Output Expectations
When using this skill, report:
- the symptom being investigated
- the unknown being narrowed next
- the instrumentation added
- what the new evidence showed
- whether the issue is now bounded
- which instrumentation was kept or removed
1---2name: observability-first-debugging3description: Use when debugging bugs, incidents, flaky tests, regressions, production issues, or unclear system behavior where the root cause is not yet bounded. Applies when Codex should increase observability first by adding targeted logs, metrics, traces, or assertions before proposing a fix. Triggers include requests to debug, investigate, instrument, add logs, narrow down, reproduce, or "wrap the problem with logs".4---56# Observability-First Debugging78## Overview910Do not jump from symptoms to fixes when the failure surface is still wide.1112**Core principle:** expand observability until the problem is bounded, then fix the bounded problem.1314## When to Use1516Use this skill when:1718- the bug reproduces but the cause is unclear19- multiple layers could be responsible20- a test fails without enough local evidence21- the problem appears flaky, timing-sensitive, or environment-dependent22- a user asks to add logs, instrument code, or investigate before fixing2324Do not lead with broad instrumentation when the root cause is already obvious from current evidence.2526## Rule2728Before making speculative code changes, add enough targeted observability to answer the next unknown.2930Instrumentation should reduce uncertainty, not just increase log volume.3132## Workflow33341. Define the symptom precisely.352. State the current unknowns.363. Choose the narrowest boundary that can separate likely causes.374. Add targeted instrumentation at that boundary.385. Reproduce and inspect the new evidence.396. Repeat only until the failure surface is bounded.407. Apply the fix to the bounded cause.418. Remove temporary instrumentation and keep only durable observability that has ongoing value.4243## Where to Instrument4445Prefer adding evidence at decision points and boundaries such as:4647- request entry and exit48- state transitions49- condition branches with surprising outcomes50- external API or database calls51- retry loops, timeout paths, and exception handlers52- async queue handoffs, worker boundaries, and concurrency joins53- data parsing, normalization, and schema validation points5455## What Good Instrumentation Looks Like5657Good instrumentation is:5859- targeted to a concrete hypothesis or unknown60- cheap enough to run during reproduction61- reversible if it is only for local diagnosis62- structured enough to correlate inputs, state, and outputs63- specific about IDs, branches, counts, timing, and error context6465Include identifiers that let you stitch a single failing path together across layers.6667## What to Avoid6869- adding logs everywhere without a question they answer70- changing behavior while instrumenting unless necessary for safety71- logging secrets, tokens, passwords, or sensitive payloads72- leaving noisy temporary logs in hot paths after the issue is understood73- declaring a fix before the new evidence isolates the cause7475## Escalation Pattern7677Use the smallest step that can answer the next question:78791. existing logs and repro output802. additional local logs or assertions813. structured metrics, counters, or timing824. trace-style correlation across components835. invasive instrumentation only if lighter steps do not bound the issue8485## Output Expectations8687When using this skill, report:8889- the symptom being investigated90- the unknown being narrowed next91- the instrumentation added92- what the new evidence showed93- whether the issue is now bounded94- which instrumentation was kept or removed
qoli