# Observability

> Query Prometheus metrics, Loki logs, and Grafana dashboards for diagnostics and incident response.

- Skill: `majiayu000/observability-5` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds add majiayu000/observability-5`
- Raw SKILL.md: https://api.skillmd.com/api/skills/majiayu000/observability-5/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: majiayu000 (https://skillmd.com/u/majiayu000)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/majiayu000/observability-5

---


# Observability Tools

Query metrics, logs, and dashboards for diagnostics and incident response.

## Prometheus (Metrics)

Query metrics for performance analysis and alerting.

```
# CPU usage by pod
prometheus_query({
  query: 'rate(container_cpu_usage_seconds_total{namespace="my-service"}[5m])'
})

# Memory usage
prometheus_query({
  query: 'container_memory_usage_bytes{namespace="my-service"}'
})

# HTTP request rate
prometheus_query({
  query: 'rate(http_requests_total{namespace="my-service"}[5m])'
})

# Error rate
prometheus_query({
  query: 'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])'
})
```

## Loki (Logs)

Query logs for debugging and incident investigation.

```
# Application logs
loki_query({
  query: '{namespace="my-service", app="api"} |= "error"',
  limit: 100
})

# Structured log parsing
loki_query({
  query: '{namespace="my-service"} | json | level="error"'
})

# Time-based filtering
loki_query({
  query: '{namespace="my-service"}',
  start: "2024-01-01T00:00:00Z",
  end: "2024-01-01T01:00:00Z"
})
```

## Common Queries

| Scenario | Query Type | Example |
|----------|------------|---------|
| High latency | Prometheus | `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))` |
| Errors spike | Loki | `{app="api"} \|= "error" \| json \| count by (error_type)` |
| Memory leak | Prometheus | `container_memory_usage_bytes{pod=~"api.*"}` |
| Failed requests | Loki | `{app="api"} \| json \| status >= 500` |

## Incident Response Flow

1. **Check alerts** - What triggered?
2. **Query metrics** - Is it resource exhaustion?
3. **Query logs** - What errors are occurring?
4. **Correlate** - Match timestamps across metrics and logs
5. **Identify root cause** - Database? Network? Code bug?

## Best Practices

1. **Start broad, then narrow** - Filter down to specific pods
2. **Use time ranges** - Don't query unbounded
3. **Correlate metrics + logs** - Same time window
4. **Check dashboard first** - Grafana may have pre-built views