APM Service Health
Assess APM service health using Observability APIs,
ES|QL against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use
SLOs, firing alerts, ML anomalies, throughput, latency (avg/p95/p99), error rate, and dependency health.
Where to look
- Observability APIs (Observability APIs): Use the
SLOs API (Stack |
Serverless) to get SLO definitions, status, burn
rate, and error budget. Use the Alerting API
(Stack |
Serverless) to list and manage alerting
rules and their alerts for the service. Use APM annotations API to create or search annotations when needed.
- ES|QL and Elasticsearch: Query
traces*apm*,traces*otel* and metrics*apm*,metrics*otel* with ES|QL (see
Using ES|QL for APM metrics) for throughput, latency, error rate, and dependency-style
aggregations. Use Elasticsearch APIs (e.g. POST _query for ES|QL, or Query DSL) as documented in the Elasticsearch
repo for indices and search.
- APM Correlations: Run the apm-correlations script to get attributes that correlate with high-latency or failed
transactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to
Elasticsearch significant_terms on
traces*apm*,traces*otel*. See
APM Correlations script.
- Infrastructure: Correlate via resource attributes (e.g.
k8s.pod.name, container.id, host.name) in
traces; query infrastructure or metrics indices with ES|QL/Elasticsearch for CPU and memory. OOM and CPU
throttling directly impact APM health.
- Logs: Use ES|QL or Elasticsearch search on log indices filtered by
service.name or trace.id to explain
behavior and root cause.
- Observability Labs: Observability Labs and
APM tag for patterns and troubleshooting.
Health criteria
Synthesize health from all of the following when available:
| Signal |
What to check |
| SLOs |
Burn rate, status (healthy/degrading/violated), error budget. |
| Firing alerts |
Open or recently fired alerts for the service or dependencies. |
| ML anomalies |
Anomaly jobs; score and severity for latency, throughput, or error rate. |
| Throughput |
Request rate; compare to baseline or previous period. |
| Latency |
Avg, p95, p99; compare to SLO targets or history. |
| Error rate |
Failed/total requests; spikes or sustained elevation. |
| Dependency health |
Downstream latency, error rate, availability (ES|QL, APIs, Kibana repo). |
| Infrastructure |
CPU usage, memory; OOM and CPU throttling on pods/containers/hosts. |
| Logs |
App logs filtered by service or trace ID for context and root cause. |
Treat a service as unhealthy if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe
degradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service/trace context) to
explain why and suggest next steps.
Using ES|QL for APM metrics
When querying APM data from Elasticsearch (traces*apm*,traces*otel*, metrics*apm*,metrics*otel*), use ES|QL by
default where available.
- Availability: ES|QL is available in Elasticsearch 8.11+ (technical preview; GA in 8.14). It is always
available in
Elastic Observability Serverless Complete tier.
- Scoping to a service: Always filter by
service.name (and service.environment when relevant). Combine with a
time range on @timestamp:
WHERE service.name == "my-service-name" AND service.environment == "production"
AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"
- Example patterns: Throughput, latency, and error rate over time: see Kibana
trace_charts_definition.ts
(getThroughputChart, getLatencyChart, getErrorRateChart). Use from(index) → where(...) → stats(...) /
evaluate(...) with BUCKET(@timestamp, ...) and WHERE service.name == "<service_name>".
- Performance: Add
LIMIT n to cap rows and token usage. Prefer coarser BUCKET(@timestamp, ...) (e.g. 1 hour)
when only trends are needed; finer buckets increase work and result size.
APM Correlations script
When only a subpopulation of transactions has high latency or failures, run the apm-correlations script to list
attributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana
internal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on
traces*apm*,traces*otel*.
# Latency correlations (attributes over-represented in slow transactions)
node skills/observability/service-health/scripts/apm-correlations.js latency-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]
# Failed transaction correlations
node skills/observability/service-health/scripts/apm-correlations.js failed-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]
# Test Kibana connection
node skills/observability/service-health/scripts/apm-correlations.js test [--space <id>]
Environment: KIBANA_URL and KIBANA_API_KEY (or KIBANA_USERNAME/KIBANA_PASSWORD) for Kibana; for fallback,
ELASTICSEARCH_URL and ELASTICSEARCH_API_KEY. Use the same time range as the investigation.
Workflow
Service health progress:
- [ ] Step 1: Identify the service (and time range)
- [ ] Step 2: Check SLOs and firing alerts
- [ ] Step 3: Check ML anomalies (if configured)
- [ ] Step 4: Review throughput, latency (avg/p95/p99), error rate
- [ ] Step 5: Assess dependency health (ES|QL/APIs / Kibana repo)
- [ ] Step 6: Correlate with infrastructure and logs
- [ ] Step 7: Summarize health and recommend actions
Step 1: Identify the service
Confirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most
relevant. Use ES|QL on traces*apm*,traces*otel* or metrics*apm*,metrics*otel* (e.g.
WHERE service.name == "<name>") or Kibana repo APM routes to obtain service-level data. If the user has not provided
the time range, assume last hour.
Step 2: Check SLOs and firing alerts
SLOs: Call the SLOs API to get SLO definitions and status for the service (latency, availability),
healthy/degrading/violated, burn rate, error budget. Alerts: For active APM alerts, call
/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active.
When checking one service, include both rules where params.serviceName matches the service and rules where
params.serviceName is absent (all-services rules). Do not query .alerts* indices for active-state checks. Correlate
with SLO violations or metric changes.
Step 3: Check ML anomalies
If ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the
service and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to
narrow Steps 4–5.
Step 4: Review throughput, latency, and error rate
Use ES|QL against traces*apm*,traces*otel* or metrics*apm*,metrics*otel* for the service and time range to get
throughput (e.g. req/min), latency (avg, p95, p99), error rate (failed/total or 5xx/total). Example:
FROM traces*apm*,traces*otel* | WHERE service.name == "<service_name>" AND @timestamp >= ... AND @timestamp <= ... | STATS ....
Compare to prior period or SLO targets. See Using ES|QL for APM metrics.
Step 5: Assess dependency health
Obtain dependency and service-map data via ES|QL on traces*apm*,traces*otel*/metrics*apm*,metrics*otel* (e.g.
downstream service/span aggregations) or via APM route handlers in the Kibana repo that expose
dependency/service-map data. For the service and time range, note downstream latency and error rate; flag slow or
failing dependencies as likely causes.
Step 6: Correlate with infrastructure and logs
- APM Correlations (when only a subpopulation is affected): Run
node skills/observability/service-health/scripts/apm-correlations.js latency-correlations|failed-correlations --service-name <name> [--start ...] [--end ...]
to get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See
APM Correlations script.
- Infrastructure: Use resource attributes from traces (e.g.
k8s.pod.name, container.id, host.name) and
query infrastructure/metrics indices with ES|QL or Elasticsearch for CPU and memory. OOM and CPU
throttling directly impact APM health; correlate their time windows with APM degradation.
- Logs: Use ES|QL or Elasticsearch on log indices with
service.name == "<service_name>" or
trace.id == "<trace_id>" to explain behavior and root cause (exceptions, timeouts, restarts).
Step 7: Summarize and recommend
State health (healthy / degraded / unhealthy) with reasons; list concrete next steps.
Examples
Example: ES|QL for a specific service
Scope with WHERE service.name == "<service_name>" and time range. Throughput and error rate (1-hour buckets; LIMIT
caps rows and tokens):
FROM traces*apm*,traces*otel*
| WHERE service.name == "api-gateway"
AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"
| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == "failure" BY BUCKET(@timestamp, 1 hour)
| EVAL error_rate = failures / request_count
| SORT @timestamp
| LIMIT 500
Latency percentiles and exact field names: see Kibana trace_charts_definition.ts.
Example: "Is service X healthy?"
- Resolve service X and time range. Call SLOs API and Alerting API; run ES|QL on
traces*apm*,traces*otel*/metrics*apm*,metrics*otel* for throughput, latency, error rate; query
dependency/service-map data (ES|QL or Kibana repo).
- Evaluate SLO status (violated/degrading?), firing rules, ML anomalies, and dependency health.
- Answer: Healthy / Degraded / Unhealthy with reasons and next steps (e.g.
Observability Labs).
Example: "Why is service Y slow?"
- Service Y and slowness time range. Call SLOs API and Alerting API; run ES|QL for Y and dependencies;
query ML anomaly results.
- Compare latency (avg/p95/p99) to prior period via ES|QL; from dependency data identify high-latency or failing deps.
- Summarize (e.g. p99 up; dependency Z elevated) and recommend (investigate Z; Observability Labs for latency).
Example: Correlate service to infrastructure (OpenTelemetry)
Use resource attributes on spans/traces to get the runtimes (pods, containers, hosts) for the service. Then check
CPU and memory for those resources in the same time window as the APM issue:
- From the service’s traces or metrics, read resource attributes such as
k8s.pod.name, k8s.namespace.name,
container.id, or host.name.
- Run ES|QL or Elasticsearch search on infrastructure/metrics indices filtered by those resource values and the
incident time range. Check CPU usage and memory consumption (e.g.
system.cpu.total.norm.pct); look for
OOMKilled events, CPU throttling, or sustained high CPU/memory that align with APM latency or error spikes.
Example: Filter logs by service or trace ID
To understand behavior for a specific service or a single trace, filter logs accordingly:
- By service: Run ES|QL or Elasticsearch search on log indices with
service.name == "<service_name>" and time
range to get application logs (errors, warnings, restarts) in the service context.
- By trace ID: When investigating a specific request, take the
trace.id from the APM trace and filter logs by
trace.id == "<trace_id>" (or equivalent field in your log schema). Logs with that trace ID show the full request
path and help explain failures or latency.
Guidelines
- Use Observability APIs (SLOs API,
Alerting API) and ES|QL on
traces*apm*,traces*otel*/metrics*apm*,metrics*otel* (8.11+ or Serverless), filtering by service.name (and
service.environment when relevant). For active APM alerts, call
/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active.
When checking one service, evaluate both rule types: rules where params.serviceName matches the target service, and
rules where params.serviceName is absent (all-services rules). Treat either as applicable to the service before
declaring health. Do not query .alerts* indices when determining currently active alerts; use the Alerting API
response above as the source of truth. For APM correlations, run the apm-correlations script (see
APM Correlations script); for dependency/service-map data, use ES|QL or Kibana repo route
handlers. For Elasticsearch index and search behavior, see the Elasticsearch APIs in the Elasticsearch repo.
- Always use the user's time range; avoid assuming "last 1 hour" if the issue is historical.
- When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,
throughput, latency, error rate, and dependencies.
- When analyzing only application metrics ingested via OpenTelemetry, use the ES|QL TS (time series) command for
efficient metrics queries. The TS command is available in Elasticsearch 9.3+ and is always available in
Elastic Observability Serverless.
- Summary: one short health verdict plus bullet points for evidence and next steps.
1---2name: observability-service-health3description: >4 Assess APM service health using SLOs, alerts, ML, throughput, latency, error rate,5 and dependencies. Use when checking service status, performance, or when the user6 asks about service health.7metadata:8 source: https://skillmd.com/skills/zuoguyoupan2023/observability-service-health9 author: elastic10 version: 0.1.011---1213# APM Service Health1415Assess APM service health using [Observability APIs](https://www.elastic.co/docs/solutions/observability/apis),16**ES|QL** against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use17SLOs, firing alerts, ML anomalies, throughput, latency (avg/p95/p99), error rate, and dependency health.1819## Where to look2021- **Observability APIs** ([Observability APIs](https://www.elastic.co/docs/solutions/observability/apis)): Use the22 **SLOs API** ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-slo) |23 [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-slo)) to get SLO definitions, status, burn24 rate, and error budget. Use the **Alerting API**25 ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-alerting) |26 [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-alerting)) to list and manage alerting27 rules and their alerts for the service. Use **APM annotations API** to create or search annotations when needed.28- **ES|QL and Elasticsearch:** Query `traces*apm*,traces*otel*` and `metrics*apm*,metrics*otel*` with **ES|QL** (see29 [Using ES|QL for APM metrics](#using-esql-for-apm-metrics)) for throughput, latency, error rate, and dependency-style30 aggregations. Use Elasticsearch APIs (e.g. `POST _query` for ES|QL, or Query DSL) as documented in the Elasticsearch31 repo for indices and search.32- **APM Correlations:** Run the **apm-correlations** script to get attributes that correlate with high-latency or failed33 transactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to34 Elasticsearch significant_terms on `traces*apm*,traces*otel*`. See35 [APM Correlations script](#apm-correlations-script).36- **Infrastructure:** Correlate via **resource attributes** (e.g. `k8s.pod.name`, `container.id`, `host.name`) in37 traces; query infrastructure or metrics indices with ES|QL/Elasticsearch for CPU and memory. **OOM** and **CPU38 throttling** directly impact APM health.39- **Logs:** Use **ES|QL** or Elasticsearch search on log indices filtered by `service.name` or `trace.id` to explain40 behavior and root cause.41- **Observability Labs:** [Observability Labs](https://www.elastic.co/observability-labs) and42 [APM tag](https://www.elastic.co/observability-labs/blog/tag/apm) for patterns and troubleshooting.4344## Health criteria4546Synthesize health from all of the following when available:4748| Signal | What to check |49| --------------------- | ------------------------------------------------------------------------- |50| **SLOs** | Burn rate, status (healthy/degrading/violated), error budget. |51| **Firing alerts** | Open or recently fired alerts for the service or dependencies. |52| **ML anomalies** | Anomaly jobs; score and severity for latency, throughput, or error rate. |53| **Throughput** | Request rate; compare to baseline or previous period. |54| **Latency** | Avg, p95, p99; compare to SLO targets or history. |55| **Error rate** | Failed/total requests; spikes or sustained elevation. |56| **Dependency health** | Downstream latency, error rate, availability (ES\|QL, APIs, Kibana repo). |57| **Infrastructure** | CPU usage, memory; OOM and CPU throttling on pods/containers/hosts. |58| **Logs** | App logs filtered by service or trace ID for context and root cause. |5960Treat a service as **unhealthy** if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe61degradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service/trace context) to62explain _why_ and suggest next steps.6364## Using ES|QL for APM metrics6566When querying APM data from Elasticsearch (`traces*apm*,traces*otel*`, `metrics*apm*,metrics*otel*`), use **ES|QL by67default** where available.6869- **Availability:** ES|QL is available in **Elasticsearch 8.11+** (technical preview; GA in 8.14). It is **always70 available** in71 [Elastic Observability Serverless Complete tier](https://www.elastic.co/docs/solutions/observability/observability-serverless-feature-tiers).72- **Scoping to a service:** Always filter by `service.name` (and `service.environment` when relevant). Combine with a73 time range on `@timestamp`:7475```esql76WHERE service.name == "my-service-name" AND service.environment == "production"77 AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"78```7980- **Example patterns:** Throughput, latency, and error rate over time: see Kibana `trace_charts_definition.ts`81 (`getThroughputChart`, `getLatencyChart`, `getErrorRateChart`). Use `from(index)` → `where(...)` → `stats(...)` /82 `evaluate(...)` with `BUCKET(@timestamp, ...)` and `WHERE service.name == "<service_name>"`.83- **Performance:** Add `LIMIT n` to cap rows and token usage. Prefer coarser `BUCKET(@timestamp, ...)` (e.g. 1 hour)84 when only trends are needed; finer buckets increase work and result size.8586## APM Correlations script8788When only a **subpopulation** of transactions has high latency or failures, run the **apm-correlations** script to list89attributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana90internal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on91`traces*apm*,traces*otel*`.9293```bash94# Latency correlations (attributes over-represented in slow transactions)95node skills/observability/service-health/scripts/apm-correlations.js latency-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]9697# Failed transaction correlations98node skills/observability/service-health/scripts/apm-correlations.js failed-correlations --service-name <name> [--start <iso>] [--end <iso>] [--last-minutes 60] [--transaction-type <t>] [--transaction-name <n>] [--space <id>] [--json]99100# Test Kibana connection101node skills/observability/service-health/scripts/apm-correlations.js test [--space <id>]102```103104**Environment:** `KIBANA_URL` and `KIBANA_API_KEY` (or `KIBANA_USERNAME`/`KIBANA_PASSWORD`) for Kibana; for fallback,105`ELASTICSEARCH_URL` and `ELASTICSEARCH_API_KEY`. Use the same time range as the investigation.106107## Workflow108109```text110Service health progress:111- [ ] Step 1: Identify the service (and time range)112- [ ] Step 2: Check SLOs and firing alerts113- [ ] Step 3: Check ML anomalies (if configured)114- [ ] Step 4: Review throughput, latency (avg/p95/p99), error rate115- [ ] Step 5: Assess dependency health (ES|QL/APIs / Kibana repo)116- [ ] Step 6: Correlate with infrastructure and logs117- [ ] Step 7: Summarize health and recommend actions118```119120### Step 1: Identify the service121122Confirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most123relevant. Use ES|QL on `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` (e.g.124`WHERE service.name == "<name>"`) or Kibana repo APM routes to obtain service-level data. If the user has not provided125the time range, assume last hour.126127### Step 2: Check SLOs and firing alerts128129**SLOs:** Call the **SLOs API** to get SLO definitions and status for the service (latency, availability),130healthy/degrading/violated, burn rate, error budget. **Alerts:** For active APM alerts, call131`/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.132When checking one service, include both rules where `params.serviceName` matches the service and rules where133`params.serviceName` is absent (all-services rules). Do not query `.alerts*` indices for active-state checks. Correlate134with SLO violations or metric changes.135136### Step 3: Check ML anomalies137138If ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the139service and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to140narrow Steps 4–5.141142### Step 4: Review throughput, latency, and error rate143144Use **ES|QL** against `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` for the service and time range to get145**throughput** (e.g. req/min), **latency** (avg, p95, p99), **error rate** (failed/total or 5xx/total). Example:146`FROM traces*apm*,traces*otel* | WHERE service.name == "<service_name>" AND @timestamp >= ... AND @timestamp <= ... | STATS ...`.147Compare to prior period or SLO targets. See [Using ES|QL for APM metrics](#using-esql-for-apm-metrics).148149### Step 5: Assess dependency health150151Obtain dependency and service-map data via **ES|QL** on `traces*apm*,traces*otel*`/`metrics*apm*,metrics*otel*` (e.g.152downstream service/span aggregations) or via APM route handlers in the **Kibana repo** that expose153dependency/service-map data. For the service and time range, note downstream latency and error rate; flag slow or154failing dependencies as likely causes.155156### Step 6: Correlate with infrastructure and logs157158- **APM Correlations (when only a subpopulation is affected):** Run159 `node skills/observability/service-health/scripts/apm-correlations.js latency-correlations|failed-correlations --service-name <name> [--start ...] [--end ...]`160 to get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See161 [APM Correlations script](#apm-correlations-script).162- **Infrastructure:** Use **resource attributes** from traces (e.g. `k8s.pod.name`, `container.id`, `host.name`) and163 query infrastructure/metrics indices with **ES|QL** or Elasticsearch for **CPU** and **memory**. **OOM** and **CPU164 throttling** directly impact APM health; correlate their time windows with APM degradation.165- **Logs:** Use **ES|QL** or Elasticsearch on log indices with `service.name == "<service_name>"` or166 `trace.id == "<trace_id>"` to explain behavior and root cause (exceptions, timeouts, restarts).167168### Step 7: Summarize and recommend169170State health (**healthy** / **degraded** / **unhealthy**) with reasons; list concrete next steps.171172## Examples173174### Example: ES|QL for a specific service175176Scope with `WHERE service.name == "<service_name>"` and time range. Throughput and error rate (1-hour buckets; `LIMIT`177caps rows and tokens):178179```esql180FROM traces*apm*,traces*otel*181| WHERE service.name == "api-gateway"182 AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp <= "2025-03-01T23:59:59Z"183| STATS request_count = COUNT(*), failures = COUNT(*) WHERE event.outcome == "failure" BY BUCKET(@timestamp, 1 hour)184| EVAL error_rate = failures / request_count185| SORT @timestamp186| LIMIT 500187```188189Latency percentiles and exact field names: see Kibana `trace_charts_definition.ts`.190191### Example: "Is service X healthy?"1921931. Resolve service X and time range. Call **SLOs API** and **Alerting API**; run **ES|QL** on194 `traces*apm*,traces*otel*`/`metrics*apm*,metrics*otel*` for throughput, latency, error rate; query195 dependency/service-map data (ES|QL or Kibana repo).1962. Evaluate SLO status (violated/degrading?), firing rules, ML anomalies, and dependency health.1973. Answer: Healthy / Degraded / Unhealthy with reasons and next steps (e.g.198 [Observability Labs](https://www.elastic.co/observability-labs)).199200### Example: "Why is service Y slow?"2012021. Service Y and slowness time range. Call **SLOs API** and **Alerting API**; run **ES|QL** for Y and dependencies;203 query ML anomaly results.2042. Compare latency (avg/p95/p99) to prior period via ES|QL; from dependency data identify high-latency or failing deps.2053. Summarize (e.g. p99 up; dependency Z elevated) and recommend (investigate Z; Observability Labs for latency).206207### Example: Correlate service to infrastructure (OpenTelemetry)208209Use **resource attributes** on spans/traces to get the runtimes (pods, containers, hosts) for the service. Then check210CPU and memory for those resources in the same time window as the APM issue:211212- From the service’s traces or metrics, read resource attributes such as `k8s.pod.name`, `k8s.namespace.name`,213 `container.id`, or `host.name`.214- Run **ES|QL** or Elasticsearch search on infrastructure/metrics indices filtered by those resource values and the215 incident time range. Check **CPU usage** and **memory consumption** (e.g. `system.cpu.total.norm.pct`); look for216 **OOMKilled** events, **CPU throttling**, or sustained high CPU/memory that align with APM latency or error spikes.217218### Example: Filter logs by service or trace ID219220To understand behavior for a specific service or a single trace, filter logs accordingly:221222- **By service:** Run **ES|QL** or Elasticsearch search on log indices with `service.name == "<service_name>"` and time223 range to get application logs (errors, warnings, restarts) in the service context.224- **By trace ID:** When investigating a specific request, take the `trace.id` from the APM trace and filter logs by225 `trace.id == "<trace_id>"` (or equivalent field in your log schema). Logs with that trace ID show the full request226 path and help explain failures or latency.227228## Guidelines229230- Use **Observability APIs** ([SLOs API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-slo),231 [Alerting API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-alerting)) and **ES|QL** on232 `traces*apm*,traces*otel*`/`metrics*apm*,metrics*otel*` (8.11+ or Serverless), filtering by `service.name` (and233 `service.environment` when relevant). For active APM alerts, call234 `/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.235 When checking one service, evaluate both rule types: rules where `params.serviceName` matches the target service, and236 rules where `params.serviceName` is absent (all-services rules). Treat either as applicable to the service before237 declaring health. Do not query `.alerts*` indices when determining currently active alerts; use the Alerting API238 response above as the source of truth. For APM correlations, run the **apm-correlations** script (see239 [APM Correlations script](#apm-correlations-script)); for dependency/service-map data, use ES|QL or Kibana repo route240 handlers. For Elasticsearch index and search behavior, see the **Elasticsearch** APIs in the Elasticsearch repo.241- Always use the **user's time range**; avoid assuming "last 1 hour" if the issue is historical.242- When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,243 throughput, latency, error rate, and dependencies.244- When analyzing **only application metrics ingested via OpenTelemetry**, use the ES|QL **TS** (time series) command for245 efficient metrics queries. The TS command is available in **Elasticsearch 9.3+** and is **always available** in246 Elastic Observability Serverless.247- Summary: one short health verdict plus bullet points for evidence and next steps.