QA Observability

QA Observability

Use telemetry as a QA signal and a debugging substrate. Treat logs, metrics, traces, and profiles as evidence for test outcomes, release readiness, and production regressions.

Core references live in data/sources.json. Prefer primary docs and re-check volatile external facts before recommending versions, pricing, or vendor features.

Quick Start (Default)

If key context is missing, ask for: critical user journeys, service/dependency inventory, environments (local/staging/prod), current telemetry stack, and current SLO/SLA commitments.

1. Establish the minimum bar: correlation IDs, structured logs, traces, and golden metrics (latency, traffic, errors, saturation).
2. Verify propagation: confirm traceparent and your request ID flow across boundaries end-to-end.
3. Make failures diagnosable: every integration or E2E failure should capture a trace link or trace ID plus correlated logs, and critical degraded paths should expose structured error metadata such as rate-limit codes, retry hints, and state-transition markers.
4. Define SLIs/SLOs and an error budget policy; wire multi-window burn-rate alerts.
5. Produce artifacts: a readiness checklist, an SLO definition, and alert rules using assets/checklists/template-observability-readiness-checklist.md, assets/monitoring/slo/slo-definition.yaml, and assets/monitoring/slo/prometheus-alert-rules.yaml.

Default QA stance

• Treat telemetry as acceptance criteria, especially for integration and E2E flows.
• Require correlation: request ID plus trace ID across service boundaries.
• For critical journeys, make auth redirects, rate limits, and state-sync lag diagnosable with structured codes or attributes instead of opaque text-only errors.
• Prefer SLO-based release gates and burn-rate alerts over raw infrastructure thresholds.
• Treat sampling, cardinality, retention, and cost as quality constraints.
• Redact PII and secrets by default in logs, spans, and attributes.
• Treat logs and profiles as ecosystem-dependent in OpenTelemetry: confirm language and backend support before promising a vendor-neutral implementation.
• The OTel Span Events API (Span.AddEvent, Span.RecordException) is being deprecated in favour of log-based events (announced March 2026). Write new event instrumentation via the Logs API; existing span event data remains functional during the gradual transition.

Expert Judgment (what a checklist misses)

• Zero observability, first hour: Do not start with a dashboard. Instrument, in order: (1) one structured log line per request with request_id, trace_id, route, status, duration_ms; (2) the four golden metrics (latency histogram, traffic counter, error counter, saturation gauge) on the single busiest entry point; (3) one end-to-end trace for the single highest-revenue or highest-incident-rate user journey. Skip dashboards, SLOs, and alerting until these three exist — they are the substrate everything else reads from, and building alerting on top of nothing produces false confidence, not safety.
• Logs vs. traces, the actual decision rule: Reach for a trace when the question is "where in this one request did time or an error go" — traces are cheap to read only when propagation already works end-to-end. Reach for a log when the question is "did this business event happen, and what were its exact values" — logs are the durable record of state transitions (payment captured, order cancelled, flag evaluated) that a trace's short retention window will not have next month. When both would answer the question, prefer the trace for latency/causality debugging and the log for audit, compliance, or anything a support engineer needs six weeks from now. Metrics answer "how often" and "how much" cheaply at scale; never reconstruct a rate or percentile by scanning logs or traces if a metric could have carried it.
• Sampling, the trade a checklist glosses over: Head sampling is a bet that the sample is representative — it is not, for errors, once error rate is below the sampling ratio (see references/sampling-strategies.md for the survivor-bias math). Tail sampling fixes that at the cost of collector-side buffering, memory, and the operational complexity of routing all spans of one trace to the same collector instance. Default recommendation: head-sample low-value traffic at a low, stable ratio; tail-sample (or always keep) errors and traces above a latency threshold; never derive SLIs from sampled trace data — derive them from metrics, which are not sampled.
• Alert fatigue is a design failure, not a tuning problem: if a service has more than 3-5 paging alerts, the fix is consolidation onto SLO burn rate (see references/alerting-strategies.md), not better thresholds on the existing alerts. An on-call engineer who receives more than ~2 pages per 12-hour shift will start ignoring pages before the quarter ends, regardless of how correct any individual alert is.
• Cost control is a first-class design constraint, not a later optimization pass: cardinality (unique label/attribute combinations) is the dominant cost driver in metrics and the dominant query-latency driver in high-cardinality log/trace backends — a single unbounded label (user ID, order ID, raw URL path) can turn a $200/month Prometheus instance into a $20k/month one. Budget cardinality and retention per signal before instrumenting, not after the bill arrives; treat a proposed new label as a design review item, not a one-line PR. Hold that budget as a cost tradeoff, not a prohibition: high cardinality and high dimensionality are the properties that make a system observable in the first place — they are what lets you slice an anomaly by the dimension nobody predicted — so budget them deliberately rather than reflexively stripping dimensions, and know which debugging questions each cut makes unanswerable (see references/core-analysis-loop.md).
• Observability tells you where to debug, not what the bug is: the scope boundary is worth stating plainly — "Observability is not for debugging your code logic. Observability is for figuring out where in your systems to find the code you need to debug" (Majors, Fong-Jones & Miranda, Observability Engineering, Early Release ch. 10; final ed. ch. 11). Observability "operates on the order of systems, not on the order of functions"; instrumenting at the line level to debug code logic would emit enough output to swamp most observability systems, at an estimated cost of "1X-10X as much as your system itself" (the book's estimate). The practical rule: use observability to narrow down which component, host, instance type, build ID, or user population the problem lives in — then copy the context out and switch to an actual debugger for the code logic. The two are a telescope and a microscope, not competitors.

Workflow

1. Establish the baseline: logs, metrics, traces, correlation, and at least one diagnosable critical journey.
2. Instrument with OpenTelemetry: auto-instrument first, then add manual spans for business workflow boundaries.
3. Verify context propagation across HTTP, queues, and RPC boundaries.
4. Define SLIs/SLOs, error budgets, and burn-rate alerts. For low-traffic services, prefer event-count or window sizing guidance from SRE workbook material.
5. Make failures diagnosable: attach trace links, key logs, and relevant metrics to failed tests.
6. Add performance evidence only after telemetry is trustworthy: profiling, load tests, exemplars, and baselines.

Quick reference

ASCII Flow

Observability QA request
  -> Identify critical journeys, services, dependencies, and environments
  -> Establish correlation IDs, structured logs, traces, and golden metrics
  -> Verify propagation across HTTP, queues, jobs, and RPC boundaries
  -> Define SLIs, SLOs, error budgets, and burn-rate alerts
  -> Attach trace/log/metric evidence to test and release failures
  -> Use profiles and load evidence only after telemetry is trustworthy

Navigation

Open these guides when needed:

Implementation guides:

• references/core-observability-patterns.md
• references/opentelemetry-best-practices.md
• references/distributed-tracing-patterns.md
• references/slo-design-guide.md
• references/performance-profiling-guide.md
• references/observability-maturity-model.md
• references/anti-patterns-best-practices.md
• references/alerting-strategies.md
• references/dashboard-design-patterns.md
• references/log-aggregation-patterns.md
• references/methods-red-use-golden.md
• references/sampling-strategies.md
• references/tools-ebpf-apm.md
• references/core-analysis-loop.md
• references/information-theory-applied.md — Information-theory applied recipes for observability: alert-noise audit, sampling budget, KL drift detection.
• references/control-theory-applied.md — Control-theory applied recipes for observability: anti-flap thresholds, burn-rate damping, Kalman health score.
• references/queueing-theory-applied.md — Queueing applied recipes for observability: Little's-Law saturation SLOs, USL release-regression, end-to-end latency budgets.
• references/reliability-theory-applied.md — Reliability primitives (MTBF/MTTR, availability, FMEA, error budgets) applied to observability and SLO design.

Templates:

• assets/checklists/template-observability-readiness-checklist.md
• assets/observability/template-logging-setup.md
• assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md
• assets/opentelemetry/python/opentelemetry-python-setup.md
• assets/monitoring/slo/slo-definition.yaml
• assets/monitoring/slo/prometheus-alert-rules.yaml
• assets/monitoring/grafana/grafana-dashboard-slo.json
• assets/monitoring/grafana/template-grafana-dashboard-observability.json
• assets/load-testing/load-testing-k6.js
• assets/load-testing/template-load-test-artillery.yaml
• assets/performance/frontend/template-lighthouse-ci.json
• assets/performance/backend/template-nodejs-profiling-config.js

Curated sources:

• data/sources.json

Do / Avoid

Do

• Start with correlation IDs, structured logs, and traces as the minimum bar
• Use SLO-based burn-rate alerts over raw infrastructure thresholds
• Redact PII and secrets by default in logs, spans, and attributes
• Use OpenTelemetry semantic conventions for standard protocol attributes
• Add manual spans around business workflow boundaries, not route-level duplicates

Avoid

• Logging secrets or high-cardinality IDs as metric labels
• Alerting on raw infrastructure metrics without SLO context
• Inventing custom attribute names when semantic conventions exist
• Adding instrumentation without a sampling and cardinality strategy
• Trusting auto-instrumentation alone for business workflow visibility

Scripts

Stdlib-only Python CLI tools. No pip dependencies — run with Python 3.9+.

Quick start:

# Score observability maturity (0–100) across 6 signal dimensions
python scripts/observability_scorer.py maturity \
  --input data/sample-observability-profile.json

# Calculate SLO error budget burn rates and status flags
python scripts/observability_scorer.py slo \
  --input data/sample-slo-data.json

# Full readiness report (maturity + SLO) written to a Markdown file
python scripts/observability_scorer.py report \
  --input data/sample-observability-profile.json \
  --slos  data/sample-slo-data.json \
  --output report.md

See scripts/README.md for full CLI reference and input schema.

Data

Sample input files for the scripts.

Related Skills

Tool selection notes

• Default to OpenTelemetry + OTLP + Collector where possible.
• Prefer SLO-based burn-rate alerting over alerting on raw infrastructure metrics.
• Use semantic conventions for standard protocol attributes; avoid inventing parallel names for HTTP, DB, messaging, or RPC spans.
• When auto-instrumentation already creates server spans, add manual spans around business workflow boundaries instead of duplicating route spans.
• When asked to pick vendors or tools, start from data/sources.json and validate time-sensitive claims with current docs or releases.

Fact-Checking

• Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
• Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
• Prefer primary sources; report source links and dates for volatile information.
• If web access is unavailable, state the limitation and mark guidance as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.