QA Observability and Performance Engineering
Use telemetry (logs, metrics, traces, profiles) as a QA signal and a debugging substrate.
Core references (see data/sources.json): OpenTelemetry, W3C Trace Context, and SLO practices (Google SRE).
Quick Start (Default)
If key context is missing, ask for: critical user journeys, service/dependency inventory, environments (local/staging/prod), current telemetry stack, and current SLO/SLA commitments (if any).
- Establish the minimum bar: correlation IDs + structured logs + traces + golden metrics (latency, traffic, errors, saturation).
- Verify propagation: confirm
traceparent (and your request ID) flow across boundaries end-to-end.
- Make failures diagnosable: every test failure captures a trace link (or trace ID) plus the correlated logs.
- Define SLIs/SLOs and error budget policy; wire burn-rate alerts (prefer multi-window burn rates).
- Produce artifacts: a readiness checklist plus an SLO definition and alert rules (use
assets/checklists/template-observability-readiness-checklist.md and assets/monitoring/slo/*).
Default QA stance
- Treat telemetry as part of acceptance criteria (especially for integration/E2E tests).
- Require correlation: request_id + trace_id (traceparent) across boundaries.
- Prefer SLO-based release gating and burn-rate alerting over raw infra thresholds.
- Budget overhead: sampling, cardinality, retention, and cost are quality constraints.
- Redact PII/secrets by default (logs and attributes).
Core workflows
- Establish the minimum bar (logs + metrics + traces + correlation).
- Instrument with OpenTelemetry (auto-instrument first, then add manual spans for key paths).
- Verify context propagation across service boundaries (traceparent in/out).
- Define SLIs/SLOs and error budget policy; wire burn-rate alerts.
- Make failures diagnosable: capture a trace link + key logs on every test failure.
- Profile and load test only after telemetry is reliable; validate against baselines.
Quick reference
| Task |
Recommended default |
Notes |
| Tracing |
OpenTelemetry + Jaeger/Tempo |
Prefer OTLP exporters via Collector when possible |
| Metrics |
Prometheus + Grafana |
Use histograms for latency; watch cardinality |
| Logging |
Structured JSON + correlation IDs |
Never log secrets/PII; redact aggressively |
| Reliability gates |
SLOs + error budgets + burn-rate alerts |
Gate releases on sustained burn/regressions |
| Performance |
Profiling + load tests + budgets |
Add continuous profiling for intermittent issues |
| Zero-code visibility |
eBPF (OpenTelemetry zero-code) + continuous profiling (Parca/Pyroscope) |
Use when code changes are not feasible |
Navigation
Open these guides when needed:
| If the user needs... |
Read |
Also use |
| A minimal, production-ready baseline |
references/core-observability-patterns.md |
assets/checklists/template-observability-readiness-checklist.md |
| Node/Python instrumentation setup |
references/opentelemetry-best-practices.md |
assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md, assets/opentelemetry/python/opentelemetry-python-setup.md |
| Working trace propagation across services |
references/distributed-tracing-patterns.md |
assets/checklists/template-observability-readiness-checklist.md |
| SLOs, burn-rate alerts, and release gates |
references/slo-design-guide.md |
assets/monitoring/slo/slo-definition.yaml, assets/monitoring/slo/prometheus-alert-rules.yaml |
| Profiling/load testing with evidence |
references/performance-profiling-guide.md |
assets/load-testing/load-testing-k6.js, assets/load-testing/template-load-test-artillery.yaml |
| A maturity model and roadmap |
references/observability-maturity-model.md |
assets/checklists/template-observability-readiness-checklist.md |
| What to avoid and how to fix it |
references/anti-patterns-best-practices.md |
assets/checklists/template-observability-readiness-checklist.md |
| Alert design and fatigue reduction |
references/alerting-strategies.md |
assets/monitoring/slo/prometheus-alert-rules.yaml |
| Dashboard hierarchy and layout |
references/dashboard-design-patterns.md |
assets/monitoring/grafana/template-grafana-dashboard-observability.json |
| Structured logging and cost control |
references/log-aggregation-patterns.md |
assets/observability/template-logging-setup.md |
Implementation guides (deep dives):
references/core-observability-patterns.mdreferences/opentelemetry-best-practices.mdreferences/distributed-tracing-patterns.mdreferences/slo-design-guide.mdreferences/performance-profiling-guide.mdreferences/observability-maturity-model.mdreferences/anti-patterns-best-practices.mdreferences/alerting-strategies.mdreferences/dashboard-design-patterns.mdreferences/log-aggregation-patterns.md
Templates (copy/paste):
assets/checklists/template-observability-readiness-checklist.mdassets/opentelemetry/nodejs/opentelemetry-nodejs-setup.mdassets/opentelemetry/python/opentelemetry-python-setup.mdassets/monitoring/slo/slo-definition.yamlassets/monitoring/slo/prometheus-alert-rules.yamlassets/monitoring/grafana/grafana-dashboard-slo.jsonassets/monitoring/grafana/template-grafana-dashboard-observability.jsonassets/load-testing/load-testing-k6.jsassets/load-testing/template-load-test-artillery.yamlassets/performance/frontend/template-lighthouse-ci.jsonassets/performance/backend/template-nodejs-profiling-config.js
Curated sources:
Scope boundaries (handoffs)
- Pure infrastructure monitoring (Kubernetes, Docker, CI/CD):
../ops-devops-platform/SKILL.md
- Database query optimization (SQL tuning, indexing):
../data-sql-optimization/SKILL.md
- Application-level debugging (stack traces, breakpoints):
../qa-debugging/SKILL.md
- Test strategy design (coverage, test pyramids):
../qa-testing-strategy/SKILL.md
- Resilience patterns (retries, circuit breakers):
../qa-resilience/SKILL.md
- Architecture decisions (microservices, event-driven):
../software-architecture-design/SKILL.md
Tool selection notes (2026)
- Default to OpenTelemetry + OTLP + Collector where possible.
- Prefer burn-rate alerting against SLOs over alerting on raw infra metrics.
- Treat sampling, cardinality, and retention as part of quality (not an afterthought).
- When asked to pick vendors/tools, start from
data/sources.json and validate time-sensitive claims with current docs/releases if the environment allows it.
Fact-Checking
- Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
- Prefer primary sources; report source links and dates for volatile information.
- If web access is unavailable, state the limitation and mark guidance as unverified.
Converted and distributed by TomeVault — claim your Tome and manage your conversions.
1---2name: qa-observability3description: Implement OpenTelemetry logs/metrics/traces, SLI/SLO gates, burn-rate alerts, and APM integrations. Use when adding or validating observability. Use when this capability is needed.4---56# QA Observability and Performance Engineering78Use telemetry (logs, metrics, traces, profiles) as a QA signal and a debugging substrate.910Core references (see `data/sources.json`): OpenTelemetry, W3C Trace Context, and SLO practices (Google SRE).1112## Quick Start (Default)1314If key context is missing, ask for: critical user journeys, service/dependency inventory, environments (local/staging/prod), current telemetry stack, and current SLO/SLA commitments (if any).15161. Establish the minimum bar: correlation IDs + structured logs + traces + golden metrics (latency, traffic, errors, saturation).172. Verify propagation: confirm `traceparent` (and your request ID) flow across boundaries end-to-end.183. Make failures diagnosable: every test failure captures a trace link (or trace ID) plus the correlated logs.194. Define SLIs/SLOs and error budget policy; wire burn-rate alerts (prefer multi-window burn rates).205. Produce artifacts: a readiness checklist plus an SLO definition and alert rules (use `assets/checklists/template-observability-readiness-checklist.md` and `assets/monitoring/slo/*`).2122## Default QA stance2324- Treat telemetry as part of acceptance criteria (especially for integration/E2E tests).25- Require correlation: request_id + trace_id (traceparent) across boundaries.26- Prefer SLO-based release gating and burn-rate alerting over raw infra thresholds.27- Budget overhead: sampling, cardinality, retention, and cost are quality constraints.28- Redact PII/secrets by default (logs and attributes).2930## Core workflows31321. Establish the minimum bar (logs + metrics + traces + correlation).332. Instrument with OpenTelemetry (auto-instrument first, then add manual spans for key paths).343. Verify context propagation across service boundaries (traceparent in/out).354. Define SLIs/SLOs and error budget policy; wire burn-rate alerts.365. Make failures diagnosable: capture a trace link + key logs on every test failure.376. Profile and load test only after telemetry is reliable; validate against baselines.3839## Quick reference4041| Task | Recommended default | Notes |42|------|---------------------|-------|43| Tracing | OpenTelemetry + Jaeger/Tempo | Prefer OTLP exporters via Collector when possible |44| Metrics | Prometheus + Grafana | Use histograms for latency; watch cardinality |45| Logging | Structured JSON + correlation IDs | Never log secrets/PII; redact aggressively |46| Reliability gates | SLOs + error budgets + burn-rate alerts | Gate releases on sustained burn/regressions |47| Performance | Profiling + load tests + budgets | Add continuous profiling for intermittent issues |48| Zero-code visibility | eBPF (OpenTelemetry zero-code) + continuous profiling (Parca/Pyroscope) | Use when code changes are not feasible |4950## Navigation5152Open these guides when needed:5354| If the user needs... | Read | Also use |55|---|---|---|56| A minimal, production-ready baseline | `references/core-observability-patterns.md` | `assets/checklists/template-observability-readiness-checklist.md` |57| Node/Python instrumentation setup | `references/opentelemetry-best-practices.md` | `assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md`, `assets/opentelemetry/python/opentelemetry-python-setup.md` |58| Working trace propagation across services | `references/distributed-tracing-patterns.md` | `assets/checklists/template-observability-readiness-checklist.md` |59| SLOs, burn-rate alerts, and release gates | `references/slo-design-guide.md` | `assets/monitoring/slo/slo-definition.yaml`, `assets/monitoring/slo/prometheus-alert-rules.yaml` |60| Profiling/load testing with evidence | `references/performance-profiling-guide.md` | `assets/load-testing/load-testing-k6.js`, `assets/load-testing/template-load-test-artillery.yaml` |61| A maturity model and roadmap | `references/observability-maturity-model.md` | `assets/checklists/template-observability-readiness-checklist.md` |62| What to avoid and how to fix it | `references/anti-patterns-best-practices.md` | `assets/checklists/template-observability-readiness-checklist.md` |63| Alert design and fatigue reduction | `references/alerting-strategies.md` | `assets/monitoring/slo/prometheus-alert-rules.yaml` |64| Dashboard hierarchy and layout | `references/dashboard-design-patterns.md` | `assets/monitoring/grafana/template-grafana-dashboard-observability.json` |65| Structured logging and cost control | `references/log-aggregation-patterns.md` | `assets/observability/template-logging-setup.md` |6667Implementation guides (deep dives):68- `references/core-observability-patterns.md`69- `references/opentelemetry-best-practices.md`70- `references/distributed-tracing-patterns.md`71- `references/slo-design-guide.md`72- `references/performance-profiling-guide.md`73- `references/observability-maturity-model.md`74- `references/anti-patterns-best-practices.md`75- `references/alerting-strategies.md`76- `references/dashboard-design-patterns.md`77- `references/log-aggregation-patterns.md`7879Templates (copy/paste):80- `assets/checklists/template-observability-readiness-checklist.md`81- `assets/opentelemetry/nodejs/opentelemetry-nodejs-setup.md`82- `assets/opentelemetry/python/opentelemetry-python-setup.md`83- `assets/monitoring/slo/slo-definition.yaml`84- `assets/monitoring/slo/prometheus-alert-rules.yaml`85- `assets/monitoring/grafana/grafana-dashboard-slo.json`86- `assets/monitoring/grafana/template-grafana-dashboard-observability.json`87- `assets/load-testing/load-testing-k6.js`88- `assets/load-testing/template-load-test-artillery.yaml`89- `assets/performance/frontend/template-lighthouse-ci.json`90- `assets/performance/backend/template-nodejs-profiling-config.js`9192Curated sources:93- `data/sources.json`9495## Scope boundaries (handoffs)9697- Pure infrastructure monitoring (Kubernetes, Docker, CI/CD): `../ops-devops-platform/SKILL.md`98- Database query optimization (SQL tuning, indexing): `../data-sql-optimization/SKILL.md`99- Application-level debugging (stack traces, breakpoints): `../qa-debugging/SKILL.md`100- Test strategy design (coverage, test pyramids): `../qa-testing-strategy/SKILL.md`101- Resilience patterns (retries, circuit breakers): `../qa-resilience/SKILL.md`102- Architecture decisions (microservices, event-driven): `../software-architecture-design/SKILL.md`103104## Tool selection notes (2026)105106- Default to OpenTelemetry + OTLP + Collector where possible.107- Prefer burn-rate alerting against SLOs over alerting on raw infra metrics.108- Treat sampling, cardinality, and retention as part of quality (not an afterthought).109- When asked to pick vendors/tools, start from `data/sources.json` and validate time-sensitive claims with current docs/releases if the environment allows it.110111## Fact-Checking112113- Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.114- Prefer primary sources; report source links and dates for volatile information.115- If web access is unavailable, state the limitation and mark guidance as unverified.116117---118> Converted and distributed by [TomeVault](https://tomevault.io/claim/vasilyu1983) — claim your Tome and manage your conversions.119<!-- tomevault:4.0:skill_md:2026-04-11 -->
tomevault-io