aspectrr APM Service Health
Assess APM service health using Observability APIs, ES|QL against APM indices, and Elasticsearch APIs. Use SLOs, firing alerts, ML anomalies, throughput, latency, error rate, and dependency health.
Health criteria
Synthesize health from all of the following when available:
| Signal | What to check |
|---|---|
| SLOs | Burn rate, status (healthy/degrading/violated), error budget. |
| Firing alerts | Open or recently fired alerts for the service or dependencies. |
| ML anomalies | Anomaly jobs; score and severity for latency, throughput, or error rate. |
| Throughput | Request rate; compare to baseline or previous period. |
| Latency | Avg, p95, p99; compare to SLO targets or history. |
| Error rate | Failed/total requests; spikes or sustained elevation. |
| Dependency health | Downstream latency, error rate, availability. |
| Infrastructure | CPU usage, memory; OOM and CPU throttling on pods/containers/hosts. |
| Logs | App logs filtered by service or trace ID for context and root cause. |
Using ES|QL for APM metrics
Always filter by service.name (and service.environment when relevant). Combine with a time range on @timestamp:
WHERE service.name == "my-service-name" AND service.environment == "production"
AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"
Example: Throughput and error rate
FROM traces*apm*,traces*otel*
| WHERE service.name == "api-gateway"
AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"
| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == "failure" BY BUCKET(@timestamp, 1 hour)
| EVAL error_rate = failures / request_count
| SORT @timestamp
| LIMIT 500
Workflow
- [ ] Step 1: Identify the service (and time range)
- [ ] Step 2: Check SLOs and firing alerts
- [ ] Step 3: Check ML anomalies (if configured)
- [ ] Step 4: Review throughput, latency (avg/p95/p99), error rate
- [ ] Step 5: Assess dependency health
- [ ] Step 6: Correlate with infrastructure and logs
- [ ] Step 7: Summarize health and recommend actions
Step 1: Identify the service
Confirm service name and time range. If the user has not provided the time range, assume last hour.
Step 2: Check SLOs and firing alerts
SLOs: Call the SLOs API to get SLO definitions and status for the service.
Alerts: For active APM alerts, call /api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active.
Step 3: Check ML anomalies
If ML anomaly detection is used, query ML job results for the service and time range.
Step 4: Review throughput, latency, and error rate
Use ES|QL against traces*apm*,traces*otel* or metrics*apm*,metrics*otel* for throughput, latency, and error rate.
Step 5: Assess dependency health
Obtain dependency data via ES|QL on traces or metrics. Flag slow or failing dependencies.
Step 6: Correlate with infrastructure and logs
- Infrastructure: Use resource attributes from traces (
k8s.pod.name,container.id,host.name) and query infrastructure indices for CPU and memory. - Logs: Use ES|QL or Elasticsearch on log indices with
service.nameortrace.idto explain behavior.
Step 7: Summarize and recommend
State health (healthy / degraded / unhealthy) with reasons; list concrete next steps.
Guidelines
- Use Observability APIs and ES|QL on
traces*apm*,traces*otel*/metrics*apm*,metrics*otel*. - Always use the user's time range; avoid assuming "last 1 hour" if the issue is historical.
- When SLOs exist, anchor the health summary to SLO status and burn rate.
- Add
LIMIT nto cap rows and token usage. - Prefer coarser
BUCKET(@timestamp, ...)when only trends are needed.