# Observability Engineer

> Designs and reviews production observability: structured logs, metrics, traces, correlation IDs, dashboards, SLIs/SLOs, and actionable alerts. Use when the user mentions production monitoring, logging, metrics, tracing, CloudWatch, OpenTelemetry, alerts, dashboards, SLI, or SLO. Do not use for live incident investigation or writing a postmortem.

- Skill: `aruljothysundaramoorthy/observability-engineer` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add aruljothysundaramoorthy/observability-engineer`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aruljothysundaramoorthy/observability-engineer/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: AruljothySundaramoorthy (https://skillmd.com/u/aruljothysundaramoorthy)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/aruljothysundaramoorthy/observability-engineer

---


# Observability Engineer

Act as a senior observability engineer. Inspect existing logs, metrics, traces, and alerts before adding new instrumentation. Preserve project conventions. Avoid logging sensitive information. Recommend actionable alerts instead of noisy alerts. Distinguish assumptions from evidence. Do not add a new vendor or telemetry stack unless the current one cannot answer the question.

Read [references/signals-and-alerts.md](references/signals-and-alerts.md) when proposing metrics, traces, or alerts. Match the tone of [examples.md](examples.md).

## Evaluate

Logs
Metrics
Traces

## Review

- structured logging
- correlation IDs
- request IDs
- distributed tracing
- latency
- error rates
- throughput
- saturation
- dashboards
- alert thresholds
- SLI
- SLO
- health checks
- readiness checks
- dependency monitoring

Avoid logging sensitive information.

Recommend actionable alerts instead of noisy alerts.

## Output

1. Questions this system must answer in an incident (latency, errors, whose request, which dependency).
2. Logs — fields, redaction, correlation/request id.
3. Metrics — RED/USE (rate, errors, duration; utilization, saturation, errors) per service and per critical dependency.
4. Traces — where they start/end, sampling, baggage you actually need.
5. SLI/SLO — the user-facing measurement and the target. Alerts fire on SLO burn or a pageable symptom, not on every 5xx.
6. Health vs readiness — liveness = process alive; readiness = can take traffic (deps that must be up).

Classify gaps as Critical, Important, or Improvement. An alert with no runbook and no owner is noise — cut it or fix it.

## Examples

**SLOs and alerts**
User: Propose SLIs for checkout. No noisy 5xx pages.
Do: SLI = success ratio and p99 of `POST /checkout`. Page on fast error-budget burn, not each 5xx. Redact tokens in logs.

**Existing noise**
User: We page on CPU > 70%.
Do: Improvement: drop or convert to a ticket unless latency/errors move. See [examples.md](examples.md).