Observability metrics
Use this skill when
- Adding a metric, a health probe, or touching
/metrics//health behaviour.
- Changing auth posture for observability endpoints.
- Verifying that a deployment is reachable and reporting sane telemetry.
- Debugging production-like issues where the question is whether telemetry itself is healthy.
Do not use this skill when
- The core risk is secret handling or request-handling safety (use
security-basics).
- The work is wiring new deployment infrastructure rather than validating telemetry endpoints.
Inputs to gather
- The service port and any token needed to reach
/metrics in the current environment.
- Whether metrics auth is enabled in this tier.
- The specific metric name or endpoint response being validated.
First move
curl the /health and /metrics endpoints directly and read the status/body before assuming application-level issues.
Outputs
- Direct
/health and /metrics check results, including any auth requirement needed to access protected metrics in the current environment.
- New or updated metric registration and instrumentation wired at startup instead of per request.
- Verification that health output, Prometheus metric text, and sensitive-exposure constraints match the repo's observability contract.
Standard endpoints
| Endpoint |
Purpose |
Default auth |
GET /health |
Liveness/readiness check |
Public |
GET /metrics |
Prometheus-format operational metrics |
Optional — protect in production |
Verification commands
curl -i http://localhost:<PORT>/health
curl -i http://localhost:<PORT>/metrics
# If metrics auth is enabled:
curl -i -H "Authorization: Bearer $AUTH_TOKEN" http://localhost:<PORT>/metrics
/health → expect 200 OK
/metrics → expect 200 OK with Prometheus text format (# HELP, # TYPE lines)
Security
/metrics exposes error rates, latency, queue depths, and operational state — treat it as sensitive.
- Enable auth protection for
/metrics in production (env var or reverse proxy ACL).
/health can remain public — it must be reachable by load balancer health checks.
- Structured logs must not contain secrets, auth tokens, full URLs with credentials, or PII.
Adding a metric
- Define the metric using your metrics library (e.g.
prometheus.NewCounter, prometheus.NewHistogram).
- Register it during app initialisation — not per-request.
- Instrument the code path where the metric is recorded.
- Start the server and verify the metric appears in
GET /metrics output.
Troubleshooting
| Symptom |
Fix |
/health returns non-200 |
Check DB connectivity and app startup logs |
/metrics returns 401 |
Pass Authorization: Bearer <token> or check METRICS_AUTH_ENABLED setting |
| Expected metric not in output |
Confirm it was registered at startup; confirm the instrumented code path executed |
Workflow
- Hit
/health and /metrics directly and read status/body before blaming app logic.
- Confirm auth/protection expectations for metrics in the current environment.
- Review metric registration (startup-time, not per-request) and naming/labels for the change.
- Add or fix instrumentation at the right boundary; keep high-cardinality labels under control.
- Ensure
/health stays LB-safe and free of internal leakage; treat /metrics as sensitive in prod.
- Check structured logs for secret/PII scrubbing at the logging boundary.
- Re-curl endpoints and confirm Prometheus text/health output matches the repo contract.
Guardrails
- Protect
/metrics in production; treat it as sensitive operational data.
- Keep
/health reachable by load balancers and free of internal-state leakage.
- Register metrics at startup, not per request, to avoid duplicate-registration panics.
- Never let structured logs carry secrets, tokens, full credential URLs, or PII.
Validation
Run the verification commands above and confirm health status, Prometheus text format, authentication behavior, and sensitive-data constraints match the repository contract.
Reference files
- Read
references/examples.md when you need concrete user utterances, expected behaviour, or a model answer shape to mirror.
- Read
references/edge-cases.md when the request is a near miss, partially matches this skill, or the first attempt fails.
1---2name: observability-metrics3description: "Validate and operate application /health, /metrics, and logging behavior. Use when adding metrics, changing observability endpoint auth, or verifying deployment telemetry — not for general production debugging where telemetry health is not the question."4license: GNU GPL v35metadata:6 source: https://skillmd.com/skills/matt-riley/observability-metrics7 version: 1.3.2 # x-release-please-version8 owner: mattriley9 category: ops10 audience: general-coding-agent11 maturity: stable12 kind: task13---1415# Observability metrics1617## Use this skill when1819- Adding a metric, a health probe, or touching `/metrics`/`/health` behaviour.20- Changing auth posture for observability endpoints.21- Verifying that a deployment is reachable and reporting sane telemetry.22- Debugging production-like issues where the question is whether telemetry itself is healthy.2324## Do not use this skill when2526- The core risk is secret handling or request-handling safety (use `security-basics`).27- The work is wiring new deployment infrastructure rather than validating telemetry endpoints.2829## Inputs to gather3031- The service port and any token needed to reach `/metrics` in the current environment.32- Whether metrics auth is enabled in this tier.33- The specific metric name or endpoint response being validated.3435## First move3637- `curl` the `/health` and `/metrics` endpoints directly and read the status/body before assuming application-level issues.3839## Outputs4041- Direct `/health` and `/metrics` check results, including any auth requirement needed to access protected metrics in the current environment.42- New or updated metric registration and instrumentation wired at startup instead of per request.43- Verification that health output, Prometheus metric text, and sensitive-exposure constraints match the repo's observability contract.4445## Standard endpoints4647| Endpoint | Purpose | Default auth |48| -------------- | ------------------------------------- | -------------------------------- |49| `GET /health` | Liveness/readiness check | Public |50| `GET /metrics` | Prometheus-format operational metrics | Optional — protect in production |5152## Verification commands5354```bash55curl -i http://localhost:<PORT>/health56curl -i http://localhost:<PORT>/metrics5758# If metrics auth is enabled:59curl -i -H "Authorization: Bearer $AUTH_TOKEN" http://localhost:<PORT>/metrics60```6162- `/health` → expect `200 OK`63- `/metrics` → expect `200 OK` with Prometheus text format (`# HELP`, `# TYPE` lines)6465## Security6667`/metrics` exposes error rates, latency, queue depths, and operational state — treat it as sensitive.6869- Enable auth protection for `/metrics` in production (env var or reverse proxy ACL).70- `/health` can remain public — it must be reachable by load balancer health checks.71- Structured logs must not contain secrets, auth tokens, full URLs with credentials, or PII.7273## Adding a metric74751. Define the metric using your metrics library (e.g. `prometheus.NewCounter`, `prometheus.NewHistogram`).762. Register it during app initialisation — not per-request.773. Instrument the code path where the metric is recorded.784. Start the server and verify the metric appears in `GET /metrics` output.7980## Troubleshooting8182| Symptom | Fix |83| ----------------------------- | --------------------------------------------------------------------------------- |84| `/health` returns non-200 | Check DB connectivity and app startup logs |85| `/metrics` returns 401 | Pass `Authorization: Bearer <token>` or check `METRICS_AUTH_ENABLED` setting |86| Expected metric not in output | Confirm it was registered at startup; confirm the instrumented code path executed |8788## Workflow89901. Hit `/health` and `/metrics` directly and read status/body before blaming app logic.912. Confirm auth/protection expectations for metrics in the current environment.923. Review metric registration (startup-time, not per-request) and naming/labels for the change.934. Add or fix instrumentation at the right boundary; keep high-cardinality labels under control.945. Ensure `/health` stays LB-safe and free of internal leakage; treat `/metrics` as sensitive in prod.956. Check structured logs for secret/PII scrubbing at the logging boundary.967. Re-curl endpoints and confirm Prometheus text/health output matches the repo contract.9798## Guardrails99100- Protect `/metrics` in production; treat it as sensitive operational data.101- Keep `/health` reachable by load balancers and free of internal-state leakage.102- Register metrics at startup, not per request, to avoid duplicate-registration panics.103- Never let structured logs carry secrets, tokens, full credential URLs, or PII.104105## Validation106107Run the verification commands above and confirm health status, Prometheus text format, authentication behavior, and sensitive-data constraints match the repository contract.108109## Reference files110111- Read `references/examples.md` when you need concrete user utterances, expected behaviour, or a model answer shape to mirror.112- Read `references/edge-cases.md` when the request is a near miss, partially matches this skill, or the first attempt fails.