Overview
Observability is the ability to understand the internal state of a system from its external outputs. For AI systems this is especially critical: agents make decisions that are hard to interpret without detailed telemetry.
The three pillars: Logs (what happened), Traces (how long and where), Metrics (aggregate health).
When to Use
- Before deploying any new service to production
- When adding AI agent capabilities to an existing system
- When debugging production issues
- When designing multi-agent pipelines
Process
Step 1: Structured Logging
- All logs must be structured (JSON, not free text). Fields:
timestamp, level, service, traceId, message, context.
- Log levels used correctly:
ERROR: Something failed that requires immediate attentionWARN: Something unexpected happened but the system recoveredINFO: Normal significant events (requests received, jobs completed)DEBUG: Detailed diagnostic information (off in production by default)
- Never log secrets, PII, or auth tokens.
- For AI systems, log: prompt inputs (sanitized), model outputs, token counts, latency, model version.
Verify: Logs are structured JSON. No secrets in logs. AI interactions logged.
Step 2: Distributed Tracing
- Every request gets a unique
traceId generated at the entry point.
traceId is propagated through all downstream calls (HTTP headers, message queues, agent calls).- Each service/agent creates a span for its work, with: start time, end time, parent span ID.
- Use OpenTelemetry as the standard instrumentation library.
Verify: You can trace a single request across all services/agents in a single view.
Step 3: Metrics
- Define and track key metrics:
- RED metrics: Rate (requests/sec), Errors (error rate %), Duration (latency p50/p95/p99)
- AI-specific: Token usage, prompt cost, model latency, hallucination rate, retrieval precision
- Dashboards: one dashboard per service with RED metrics, one dashboard for AI system health.
Verify: RED metrics are tracked for every service. AI-specific metrics tracked for AI systems.
Step 4: Alerting
- Alerts must be actionable — every alert should have a runbook.
- Alert on symptoms (high error rate, high latency), not just causes.
- AI-specific alerts: token budget exceeded, model error rate spike, retrieval failure rate spike.
- On-call rotation: someone is responsible for every alert at all times.
Verify: Every alert has a runbook. On-call rotation defined.
Common Rationalizations (and Rebuttals)
| Excuse |
Rebuttal |
| "We'll add monitoring after launch" |
You'll be fighting fires blind. Add it before. |
| "Console.log is enough" |
In production, console.log is noise. Structured logs with context are signals. |
| "The AI model handles it internally" |
Model internals are a black box. You must observe the inputs and outputs. |
Verification
References
1---2name: observability3description: Structured logging, distributed tracing, and alerting for AI systems and traditional services. You can't fix what you can't see.4---56## Overview78Observability is the ability to understand the internal state of a system from its external outputs. For AI systems this is especially critical: agents make decisions that are hard to interpret without detailed telemetry.910The three pillars: **Logs** (what happened), **Traces** (how long and where), **Metrics** (aggregate health).1112## When to Use1314- Before deploying any new service to production15- When adding AI agent capabilities to an existing system16- When debugging production issues17- When designing multi-agent pipelines1819## Process2021### Step 1: Structured Logging22231. All logs must be **structured** (JSON, not free text). Fields: `timestamp`, `level`, `service`, `traceId`, `message`, `context`.242. Log levels used correctly:25 - `ERROR`: Something failed that requires immediate attention26 - `WARN`: Something unexpected happened but the system recovered27 - `INFO`: Normal significant events (requests received, jobs completed)28 - `DEBUG`: Detailed diagnostic information (off in production by default)293. **Never log secrets, PII, or auth tokens.**304. For AI systems, log: prompt inputs (sanitized), model outputs, token counts, latency, model version.3132**Verify:** Logs are structured JSON. No secrets in logs. AI interactions logged.3334### Step 2: Distributed Tracing35365. Every request gets a unique `traceId` generated at the entry point.376. `traceId` is propagated through all downstream calls (HTTP headers, message queues, agent calls).387. Each service/agent creates a **span** for its work, with: start time, end time, parent span ID.398. Use OpenTelemetry as the standard instrumentation library.4041**Verify:** You can trace a single request across all services/agents in a single view.4243### Step 3: Metrics44459. Define and track key metrics:46 - **RED metrics**: Rate (requests/sec), Errors (error rate %), Duration (latency p50/p95/p99)47 - **AI-specific**: Token usage, prompt cost, model latency, hallucination rate, retrieval precision4810. Dashboards: one dashboard per service with RED metrics, one dashboard for AI system health.4950**Verify:** RED metrics are tracked for every service. AI-specific metrics tracked for AI systems.5152### Step 4: Alerting535411. Alerts must be **actionable** — every alert should have a runbook.5512. Alert on symptoms (high error rate, high latency), not just causes.5613. AI-specific alerts: token budget exceeded, model error rate spike, retrieval failure rate spike.5714. On-call rotation: someone is responsible for every alert at all times.5859**Verify:** Every alert has a runbook. On-call rotation defined.6061## Common Rationalizations (and Rebuttals)6263| Excuse | Rebuttal |64|--------|----------|65| "We'll add monitoring after launch" | You'll be fighting fires blind. Add it before. |66| "Console.log is enough" | In production, console.log is noise. Structured logs with context are signals. |67| "The AI model handles it internally" | Model internals are a black box. You must observe the inputs and outputs. |6869## Verification7071- [ ] Structured JSON logging on all services72- [ ] No secrets in logs73- [ ] Distributed tracing with trace ID propagation74- [ ] RED metrics tracked for all services75- [ ] AI-specific metrics tracked (tokens, cost, latency)76- [ ] Alerts configured with runbooks7778## References7980- [production-deployment skill](../production-deployment/SKILL.md)81- [multi-agent-orchestration skill](../multi-agent-orchestration/SKILL.md)82- OpenTelemetry documentation
gabrielmoreira