Service Reliability
Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before
the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational,
whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring
SLOs through the Kibana API.
Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a
threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics
monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong
produces alerts that are technically correct and operationally useless. For diagnosing a service that is already
degraded, and for the incident workflow itself, use the observability-sre-triage skill; for general rule lifecycle
mechanics use the kibana-alerting-rules skill.
Environment Configuration
This skill executes Elasticsearch operations through the elastic CLI. If the
elastic CLI is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,
GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document
maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API
directly.
Analysis without cluster access
The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the
evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
from that evidence and deliver the conclusion.
When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
useful without a cluster; one that only asks for setup is not.
SLO and alerting operations run against Kibana and use the kbn: prefix (for example,
POST kbn:/api/observability/slos); data validation runs against Elasticsearch with a bare path (for example,
POST /_query). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path
becomes kbn:/s/<space_id>/api/observability/slos.
Full request-body schemas for every SLI type live in references/slo-api-schemas.md, the
burn-rate rule schema in references/burn-rate-rules.md, and curated official
documentation in references/documentation.md.
Jobs to be done
- Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or
synthetics monitor
- Choose an SLI type for a given service and data shape, and validate the underlying fields before committing
- Set a target, time window, and budgeting method that are defensible against measured history
- Decide whether to group an SLO, and refuse high-cardinality grouping
- Create, verify, update, reset, disable, and delete SLOs
- Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities
- Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable
rules
- Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI
Output discipline
Applies to every response produced under this skill.
- Commit to the best-supported conclusion. Recommend one SLI design and defend it. Do not enumerate every SLI type
with equal weight and leave the choice to the user — that is not a recommendation, it is a menu.
- State the target and window you chose, and why, once. Do not restate the justification per bullet.
- Do not invent field names. Verify every field against the mapping before writing it into an indicator. If a field
cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap.
- Do not speculate past the evidence. If the measured history does not support a target, say what it does support.
- Report absence as absence. Zero events means the data is missing or the service is not emitting; it never means
the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%.
- Do not pad. No restating the question, no narrating which queries were run unless the result mattered.
- End on the finding. No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a
recommendations list, phrased as recommendations, not as questions.
Process: route the reliability concern to an instrument
Do this before designing anything. Most bad SLOs are threshold rules wearing a costume.
Name the user-visible symptom. Ask what a user or downstream consumer would notice and complain about. If the
answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a
capacity ceiling, not a reliability target. Route it to a threshold rule.
Check whether coverage already exists. List SLOs with GET kbn:/api/observability/slos, alerting rules with
GET kbn:/api/alerting/rules/_find, and anomaly jobs with GET /_ml/anomaly_detectors. Adding a second instrument
on a signal that is already covered is the most common source of duplicate pages.
Pick the instrument.
| Concern |
Instrument |
Why, and what it costs |
| A user-visible success or latency outcome you want to budget over weeks |
SLO + burn-rate rule |
Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window. |
| A hard bound with a known safe value and an immediate operator action |
Threshold or custom rule |
Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds. |
| A signal with no fixed threshold, strong seasonality, or many entities |
Anomaly detection job |
Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager. |
| Reachability of a user-facing endpoint from outside your own telemetry |
Synthetics monitor |
The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure. |
| An internal job with no consumer contract, or a service still changing daily |
None |
An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO. |
Do not skip the outage case. An SLI built from a service's own logs or traces cannot see the service disappear:
with zero events there is no total, so the ratio is undefined and the SLO neither burns nor recovers. Pair every
request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule.
This is the one place where two instruments on one signal is correct rather than duplicative.
Process: design the SLO
Locate the data and confirm the fields exist. Resolve the index pattern with GET /_resolve/index/<pattern>,
then confirm every field the indicator will reference with GET /<index>/_mapping or GET /_field_caps. OTel-native
data lives in traces-*.otel-*, metrics-*.otel-*, and logs-*.otel-*, with service.name populated on all three;
APM aggregate metrics live in metrics-service_summary.1m.otel-*, metrics-service_transaction.1m.otel-*,
metrics-transaction.1m.otel-*, and metrics-service_destination.1m.otel-*. Do not assume a status-code, duration,
or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes
cleanly and means nothing; How a missing field fails shows exactly how.
Two field placements are worth knowing because they are commonly guessed wrong: http.response.status_code is on
traces-*.otel-* and not on logs-*.otel-*, and transaction.duration.us is on traces-*.otel-* and not on the
metrics-*.1m.otel-* rollups, which carry transaction.duration.histogram and transaction.duration.summary
instead. Confirm both against the deployment in front of you rather than trusting this list.
Validate the SLI with ES|QL before creating anything. Run the good/total ratio over recent history with
POST /_query so you know the measured baseline. This query is exploratory and is written in ES|QL, not KQL:
FROM traces-*.otel-*
| WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 30 day
| STATS total = COUNT(*), good = COUNT(*) WHERE http.response.status_code < 500
| EVAL achieved = good::DOUBLE / total
Against an OTel demo cluster this returns total: 1393887, good: 1393887, achieved: 1.0. That is the measured
input to step 4, not a target to copy — a clean 30 days argues for a target below 100%, not at it.
HTTP status lives on traces, not on logs. http.response.status_code is populated on traces-*.otel-*. OTel log
records do not carry an HTTP status, so the field does not exist in logs-*.otel-* at all, and its coverage on
traces is per-service: only spans emitted by an HTTP server carry it. Filtering kind == "Server" keeps the
service's own inbound requests and drops the client spans that report the status of calls it made to others. Confirm
coverage for the specific service before building an indicator on it:
FROM traces-*.otel-*
| WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 24 hour
| STATS spans = COUNT(*), with_status = COUNT(http.response.status_code)
On the same cluster cart returns spans: 46420, with_status: 46420, while checkout — a gRPC service — returns
spans: 4887, with_status: 0. When with_status is 0 there is no HTTP status to budget. Use
sli.apm.transactionErrorRate, or build the ratio on event.outcome, which is populated on every OTel span
regardless of protocol:
FROM traces-*.otel-*
| WHERE service.name == "checkout" AND kind == "Server" AND @timestamp >= NOW() - 30 day
| STATS total = COUNT(*), good = COUNT(*) WHERE event.outcome == "success"
| EVAL achieved = good::DOUBLE / total
That returns total: 147009, good: 146996, achieved: 0.9999. Write good in the positive form (== "success"),
not as a negation of "failure" — a null outcome must not count as good.
This split is not specific to Serverless or to one demo application. The same measurement across every service on a
Stack 9.4.4 cluster, over traces-*.otel-* with kind == "Server":
| Service |
Server spans |
http.response.status_code coverage |
event.outcome coverage |
| catalog |
49,271 |
0% |
100% |
| gateway |
37,773 |
100% |
100% |
| orders |
37,754 |
100% |
100% |
| payments |
34,710 |
100% |
100% |
| recommendations |
34,692 |
0% |
100% |
| shipping |
34,531 |
100% |
100% |
Two of six services have no HTTP status at all while every one of the six has event.outcome on every span. An
availability SLO built on http.response.status_code < 500 for catalog would compute good: 0 against
total: 49271 — an achieved SLI of 0%, a fully consumed error budget, and a burn-rate rule that pages continuously
against a healthy service. Nothing in the API response or the SLO UI flags this; the numerator is simply always zero.
Run the coverage check above for the specific service every time, and prefer event.outcome when you are writing one
indicator to cover several services.
Never substitute a plausible-looking status field for a missing one — see
How a missing field fails for what that costs.
Also check that traffic is thick enough for a ratio to be meaningful. If the thinnest buckets carry only a handful of
events, a single failure swings the SLI by whole percentage points and the SLO will be noise:
FROM traces-*.otel-*
| WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 7 day
| STATS events = COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour)
| SORT events ASC
| LIMIT 10
Bound this one explicitly. Without a @timestamp predicate it buckets the entire retention of the trace data
streams, which is cheap on a demo cluster and expensive on a customer's.
Choose the SLI type from the data shape, not from preference.
| SLI type |
Use when |
sli.kql.custom |
Raw logs or documents where good and total are expressible as filters over events |
sli.metric.custom |
Pre-aggregated metric fields where good and total are equations over sums or counts |
sli.metric.timeslice |
A metric compared against a threshold per slice, such as a p95 latency ceiling |
sli.histogram.custom |
Histogram fields, using a range for good and a value count for total |
sli.apm.transactionDuration |
APM transaction latency against a millisecond threshold |
sli.apm.transactionErrorRate |
APM transaction success rate |
sli.synthetics.availability |
Synthetics monitor uptime for a user-facing endpoint |
Prefer an APM or synthetics type when it fits: they encode the service, environment, and transaction dimensions for
you and stay correct when the underlying index layout changes. Reach for sli.kql.custom when the outcome is only
visible in raw events, and for sli.metric.custom when the service already emits its own counters.
Set a target you can meet. Take the measured baseline from step 2 and set the target at or just below it, then
ratchet upward once the service earns it. A target above the measured baseline burns the entire budget on day one,
the SLO stays red permanently, and the team stops looking at it. objective.target is a decimal between 0 and 1 —
0.995, not 99.5. Sanity-check the target against the budget it implies over 30 days:
| Target |
Error budget over 30 days |
| 99% |
7h 12m |
| 99.5% |
3h 36m |
| 99.9% |
43m 12s |
| 99.95% |
21m 36s |
| 99.99% |
4m 19s |
If a single rolling deploy, a node restart, or one dependency blip costs more than the whole budget, the target is
unachievable and should be rejected, not accepted with a caveat.
Choose the time window. timeWindow.type is rolling (7d, 30d, 90d) or calendarAligned (1w, 1M).
Rolling windows move continuously, so budget recovers gradually and burn-rate alerting stays meaningful — this is the
default for anything operational. Calendar-aligned windows reset at the period boundary, which matches contractual or
monthly-reporting language but produces a budget cliff on the first of the month. Use rolling for paging, and add a
calendar-aligned SLO alongside it only when someone genuinely reports on calendar periods.
Choose the budgeting method. occurrences divides good events by total events across the whole window, so a
high-traffic hour dominates and a quiet overnight outage barely registers. timeslices chops the window into slices,
marks each slice good or bad against objective.timesliceTarget, and divides good slices by total slices, so every
period counts equally. Choose timeslices when low-traffic periods matter or when the indicator is a threshold on an
aggregate rather than a countable good/total. sli.metric.timeslice requires budgetingMethod: "timeslices" —
the pairing is not optional, and objective.timesliceTarget and objective.timesliceWindow become required.
Decide grouping deliberately. groupBy creates one independent SLO instance per unique value, each with its own
transform buckets and its own alerts. Measure the cardinality before setting it:
FROM traces-*.otel-*
| WHERE @timestamp >= NOW() - 24 hour
| STATS instances = COUNT_DISTINCT(service.name)
Keep the @timestamp bound. A cardinality aggregation with no time predicate runs over the whole retention of the
trace data streams; a recent window answers the same question at a fraction of the cost.
Cardinality is not the only check — the dimension also has to be populated. groupBy on a field that is mapped
but null produces one degenerate instance covering everything, which looks like a working grouped SLO and is not.
Confirm with COUNT(<field>) alongside COUNT(*) before setting it.
Group on stable, bounded dimensions such as service.name, service.environment, or k8s.namespace.name. Refuse
high-cardinality fields — trace.id, url.full, user.id, and churning identifiers like k8s.pod.name — and say
why rather than creating the SLO and warning afterward. If per-entity visibility is genuinely needed on a wide
dimension, filter to the entities that matter instead of grouping across all of them. Synthetics SLOs are
auto-grouped by monitor and location; do not set groupBy manually.
Create and verify. Build the body per references/slo-api-schemas.md and
POST kbn:/api/observability/slos. Then read it back with GET kbn:/api/observability/slos/{id} and confirm it is
computing before reporting success — a created SLO whose transform has not started yet returns no summary data.
KQL is an API contract here, not a style choice
ES|QL is the query language for Observability, and every exploratory query you run to validate an SLI — baselines,
cardinality checks, traffic distribution — must be ES|QL against POST /_query.
The exception is inside the SLO request body. The good, total, and filter fields of sli.kql.custom (and the
filter fields of the other indicator types, plus the --kql-query parameter on SLO search) are KQL strings, because
the SLO API defines them that way. There is no ES|QL form of those fields. Write KQL there, and only there:
{
"type": "sli.kql.custom",
"params": {
"index": "traces-*.otel-*",
"filter": "service.name : \"cart\" and kind : \"Server\"",
"good": "http.response.status_code < 500",
"total": "*",
"timestampField": "@timestamp"
}
}
Do not translate these fields to ES|QL — the API will reject or silently mis-parse them. Do not translate exploratory
queries to KQL either.
How a missing field fails
The reason step 1 insists on confirming the field is that the two dialects fail in opposite ways, and the dangerous one
is the dialect that ends up in the SLO.
The exploratory ES|QL query fails loudly. Pointing the step 2 query at logs-*.otel-*, where
http.response.status_code does not exist, returns HTTP 400:
line 3:49: Unknown column [http.response.status_code]
The KQL in the indicator body does not fail at all. KQL compiles a numeric comparison on an unmapped field to a
range query that matches nothing, and the SLO API accepts the definition without complaint. Over the same one-hour
window on a cluster holding 101,920,257 documents in logs-*.otel-*, the good clause matched 0 documents while
total: "*" matched 93,847. The SLO computes cleanly, reports an SLI of 0% against its target, and burns the entire
error budget on the first transform run. Attach the burn-rate rule from the next section and the fast-burn window pages
continuously on an indicator that measures nothing.
Do not try to sanity-check the field with a Lucene query_string instead. The same predicate as Lucene returns 3,250
matches on that cluster — not because the field exists, but because Lucene tokenizes the expression and full-text
matches the fragments against the default fields. A non-zero count from Lucene is not evidence that a field is mapped.
Confirm with GET /_field_caps or GET /<index>/_mapping, which answer the question directly, and with a
COUNT(<field>) in ES|QL, which additionally tells you whether a mapped field is actually populated.
Process: alert on the SLO without adding noise
Creating an SLO does not create any alerting. Burn-rate rules are never auto-created by the SLO API and must be set
up separately with POST kbn:/api/alerting/rule/{id}, rule type slo.rules.burnRate.
Use multiple windows with different severities. A burn rate of 1 means the budget is being spent exactly fast
enough to exhaust at the end of the window. Fast-burn windows (roughly 14x over a one-hour long window with a
five-minute short window) catch outages and are worth paging on. Slow-burn windows (roughly 1x-3x over 24 to 72
hours) catch chronic degradation and belong in a ticket queue. Routing all windows to the same paging connector is
the single fastest way to make the SLO ignored. Window values and action groups are in
references/burn-rate-rules.md.
Keep the short window. Each window pairs a long window (does the burn rate justify alerting) with a short window
(is it still happening). The short window is what lets a resolved incident stop alerting on its own instead of
requiring someone to mute it.
Remove duplicate coverage. If a burn-rate rule and a raw threshold rule both watch the same signal, one incident
produces two pages. Find the overlap with GET kbn:/api/alerting/rules/_find before adding anything. Keep the
burn-rate rule as the pager and demote or delete the raw threshold — unless the threshold covers the no-data case the
SLO structurally cannot see, which is a real gap and should stay.
Snooze, do not disable. For a bounded pause on one rule use POST kbn:/api/alerting/rule/{id}/_snooze_schedule;
for planned change affecting many rules use a maintenance window via POST kbn:/api/maintenance_window. Both expire
on their own. Disabling a rule loses its alert history and depends on someone remembering to re-enable it, so reserve
POST kbn:/api/alerting/rule/{id}/_disable for rules that are wrong rather than rules that are temporarily
inconvenient.
Apply the actionability test. A rule that fires correctly but has no action the responder can take is a defect,
not a success. For each rule, name the first step the on-call would take. If the honest answer is "look at it later",
it is not a page — move it to a slow-burn window or a ticket, or delete it.
Process: monitor availability with synthetics
A synthetics monitor probes an endpoint on a schedule from one or more locations and writes results to synthetics-*.
It is the instrument that answers "is it up from the outside", which telemetry emitted by the service itself can never
answer. Manage monitors with GET kbn:/api/synthetics/monitors and POST kbn:/api/synthetics/monitors.
Budget the checks before choosing a target. Availability is measured in checks, not requests, so the check
frequency sets the resolution of the SLI. A ten-minute frequency over 30 days is 4,320 checks; at a 99.99% target the
entire error budget is under half a check, so one transient failure exhausts it. Require the budget to be worth at
least ten checks — raise the frequency or lower the target until it is.
Use at least three locations. A single-location monitor cannot distinguish a down service from a bad network path
out of one region. Three or more makes that call obvious and removes a whole class of false pages.
Feed it into an SLI. Create the SLO with sli.synthetics.availability, scoped by monitor ids, projects, or tags.
Leave groupBy unset; the indicator already produces one instance per monitor and location.
Process: operate and repair
- Update carefully.
PUT kbn:/api/observability/slos/{id} resets the underlying transform and recomputes
history from scratch. Changing the target or the indicator therefore discards the existing budget picture. Say so
before doing it, and prefer creating a second SLO when the old history still matters.
- Reset when stuck.
POST kbn:/api/observability/slos/{id}/_reset rebuilds the transform and rollup data. Use it
when an SLO stops updating, after index mapping changes, or after an upgrade leaves a definition outdated —
GET kbn:/api/observability/slos/_definitions reports which definitions are outdated.
- Pause versus retire.
POST kbn:/api/observability/slos/{id}/disable stops computation but keeps the definition;
POST kbn:/api/observability/slos/{id}/enable resumes it. DELETE kbn:/api/observability/slos/{id} is permanent —
confirm with the user first.
- Check the cluster shape. Every SLO is backed by a continuous transform, so the cluster needs nodes carrying both
the
transform and ingest roles. If SLOs never leave "no data", check this before debugging the indicator.
Examples
"Set up an SLO for the checkout service" — resolve traces-*.otel-* and confirm service.name and the outcome
field exist, measure the trailing 30-day success ratio with POST /_query, then recommend one design: if the measured
baseline is 99.6%, propose sli.apm.transactionErrorRate at 0.995 on a 30-day rolling window with occurrences
budgeting, and say that 99.9% is rejected because its 43-minute budget is smaller than the service's observed monthly
degradation. Create it, read it back, then create the burn-rate rule separately.
"We want a 99.99% SLO on the payments API" — measure first. If the trailing baseline is 99.7%, reject 99.99%
outright: it allows 4 minutes 19 seconds of budget over 30 days, less than a single rolling deploy. Recommend 99.5% now
with a ratchet plan, and state that the constraint is the deploy process rather than the target.
"Alert us when p95 latency goes above 500 ms" — this is a threshold on an aggregate, so it is sli.metric.timeslice
with a percentile metric, comparator: "LT", threshold: 500, and — required by that indicator —
budgetingMethod: "timeslices" with timesliceTarget and timesliceWindow set. Confirm the duration field and its
unit in the mapping first; a threshold written in milliseconds against a microsecond field is off by a thousand.
"Create an SLO per pod so we can see which pods are unreliable" — refuse the grouping. k8s.pod.name churns on
every deploy, so each groupBy value creates an SLO instance that is orphaned within days while the transform carries
the cardinality forever. Recommend grouping on k8s.namespace.name or service.name instead, and point out that
per-pod reliability is a symptom to investigate, not a target to budget.
"Disk on the log nodes keeps filling up — can we SLO that?" — no. Disk utilization is a capacity ceiling with a
known safe bound and an immediate operator action, so it is a threshold rule, not a reliability target. There is no
user-visible outcome to budget and no meaningful notion of spending 0.5% of disk-full.
"Our request rate looks weird but there is no threshold we can name" — no fixed bound means no SLO and no threshold
rule. Route it to an anomaly detection job, which learns the seasonal baseline, and note that it needs several weeks of
history before its scores are trustworthy.
"Why are we getting paged twice for every checkout incident?" — list rules with GET kbn:/api/alerting/rules/_find
and look for a burn-rate rule and a raw threshold rule on the same signal. Keep the burn-rate rule as the pager, demote
the threshold to a ticket, and verify no slow-burn window is routed to the paging connector.
Guidelines
- Choose the instrument before designing the SLO; a capacity ceiling is a threshold rule and an unbounded signal is an
anomaly job.
- Confirm every field against
GET /<index>/_mapping or GET /_field_caps before writing it into an indicator. Never
infer a field name from convention.
- Validate the SLI with ES|QL through
POST /_query first; set the target from the measured baseline, not from
ambition.
objective.target is a decimal between 0 and 1 — 0.995 for 99.5%.
- Timeslice metric indicators require
budgetingMethod: "timeslices".
- The
good, total, and filter fields of sli.kql.custom are KQL because the API defines them that way. Everything
else you query is ES|QL.
- Updating an SLO resets the underlying transform and recomputes history — warn the user before doing it.
- The cluster needs nodes with both the
transform and ingest roles.
- Use the reset operation when an SLO is stuck or after index mapping changes.
- Group-by SLOs create one instance per unique value — refuse high-cardinality fields rather than creating and warning.
- Synthetics SLOs are auto-grouped by monitor and location; do not set
groupBy manually.
- Burn-rate alert rules are not created by the SLO API — set them up separately, with fast-burn and slow-burn windows
routed to different severities.
- Prefer snoozing a rule or opening a maintenance window over disabling it.
- Confirm deletions before executing them.
Operations
| HTTP API (shorthand) |
elastic CLI command |
POST /_query |
elastic es esql query --format tsv --query '<esql>' |
GET /_resolve/index/<pattern> |
elastic es indices resolve-index --name '<pattern>' |
GET /<index>/_mapping |
elastic es indices get-mapping --index '<index>' |
GET /_field_caps |
elastic es field-caps --index '<index>' --fields '<fields>' |
GET /_ml/anomaly_detectors |
elastic es ml get-jobs |
GET kbn:/api/observability/slos |
elastic kb slo find-slos-op --space-id '<space>' --kql-query '<kql>' |
POST kbn:/api/observability/slos |
elastic kb slo create-slo-op --space-id '<space>' --input-file <json> |
GET kbn:/api/observability/slos/{id} |
elastic kb slo get-slo-op --space-id '<space>' --slo-id '<id>' |
PUT kbn:/api/observability/slos/{id} |
elastic kb slo update-slo-op --space-id '<space>' --slo-id '<id>' --input-file <json> |
DELETE kbn:/api/observability/slos/{id} |
elastic kb slo delete-slo-op --space-id '<space>' --slo-id '<id>' |
POST kbn:/api/observability/slos/{id}/_reset |
elastic kb slo reset-slo-op --space-id '<space>' --slo-id '<id>' |
POST kbn:/api/observability/slos/{id}/enable |
elastic kb slo enable-slo-op --space-id '<space>' --slo-id '<id>' |
POST kbn:/api/observability/slos/{id}/disable |
elastic kb slo disable-slo-op --space-id '<space>' --slo-id '<id>' |
GET kbn:/api/observability/slos/_definitions |
elastic kb slo get-definitions-op --space-id '<space>' |
GET kbn:/api/alerting/rules/_find |
elastic kb alerting get-alerting-rules-find --filter '<kql>' |
POST kbn:/api/alerting/rule/{id} |
elastic kb alerting post-alerting-rule-id --id '<id>' --input-file <json> |
POST kbn:/api/alerting/rule/{id}/_snooze_schedule |
elastic kb alerting post-alerting-rule-id-snooze-schedule --id '<id>' --input-file <json> |
POST kbn:/api/alerting/rule/{id}/_disable |
elastic kb alerting post-alerting-rule-id-disable --id '<id>' |
POST kbn:/api/maintenance_window |
elastic kb maintenance-window post-maintenance-window --title '<title>' --input-file <json> |
GET kbn:/api/synthetics/monitors |
no CLI binding — see CLI gaps below |
POST kbn:/api/synthetics/monitors |
no CLI binding — see CLI gaps below |
CLI gaps. As of elastic CLI 0.2.0 there is no binding for the Synthetics monitor management API — the stack kb
namespace exposes no synthetics commands. The HTTP shorthand above is still the correct contract and remains portable,
so keep using it when describing what must happen, but do not invent a CLI invocation for it. Create and edit monitors
through the Synthetics app in Kibana, through a Synthetics project, or by calling the Kibana API directly from tooling
that already holds credentials. Everything else in this skill, including the burn-rate rule and every SLO operation, has
a working CLI binding.
1---
2name: observability-service-reliability
3description: >
4 Design and operate service reliability targets in Elastic Observability: choose
5 an SLI type and a defensible target, pick a time window and budgeting method, create
6 and maintain SLOs through the Kibana API, attach burn-rate alert rules, and decide
7 when an SLO is the wrong instrument and a threshold rule, anomaly job, or synthetics
8 monitor is right. Use when defining or reviewing SLOs and error budgets, tuning
9 burn-rate alerting, reducing alert noise, or setting up availability monitoring
10 for a user-facing endpoint.
11compatibility: >
12 Requires Kibana 8.x or 9.x with a matching Elasticsearch cluster (self-managed,
13 Elastic Cloud Hosted, or Serverless) and the Observability solution enabled. The
14 cluster must have nodes carrying both the `transform` and `ingest` roles, since
15 every SLO is backed by a continuous transform. Needs the `elastic` CLI >= 0.2 with
16 `stack kb` and `stack es` support. Synthetics SLIs additionally require the Synthetics
17 app and at least one configured monitor location.
18metadata:
19 source: https://skillmd.com/skills/gabrielmoreira/observability-service-reliability
20 author: elastic
21 version: 0.4.1
22 universal: true
23---
24
25# Service Reliability
26
27Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before
28the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational,
29whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring
30SLOs through the Kibana API.
31
32Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a
33threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics
34monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong
35produces alerts that are technically correct and operationally useless. For diagnosing a service that is already
36degraded, and for the incident workflow itself, use the **observability-sre-triage** skill; for general rule lifecycle
37mechanics use the **kibana-alerting-rules** skill.
38
39<!-- begin-partial: preamble -->
40
41## Environment Configuration
42
43This skill executes Elasticsearch operations through the `elastic` CLI. If the
44[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
45not guess credentials, call the HTTP API directly, or attempt other workarounds.
46
47This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
48`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
49maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
50directly.
51
52<!-- end-partial: preamble -->
53
54### Analysis without cluster access
55
56The CLI check above gates _querying the cluster_ — it does not gate analysis. When the user has already supplied the
57evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason
58from that evidence and deliver the conclusion.
59
60When you genuinely do need data the user has not provided, still say what you would check and how — name the specific
61query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is
62useful without a cluster; one that only asks for setup is not.
63
64SLO and alerting operations run against Kibana and use the `kbn:` prefix (for example,
65`POST kbn:/api/observability/slos`); data validation runs against Elasticsearch with a bare path (for example,
66`POST /_query`). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path
67becomes `kbn:/s/<space_id>/api/observability/slos`.
68
69Full request-body schemas for every SLI type live in [references/slo-api-schemas.md](references/slo-api-schemas.md), the
70burn-rate rule schema in [references/burn-rate-rules.md](references/burn-rate-rules.md), and curated official
71documentation in [references/documentation.md](references/documentation.md).
72
73## Jobs to be done
74
75- Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or
76 synthetics monitor
77- Choose an SLI type for a given service and data shape, and validate the underlying fields before committing
78- Set a target, time window, and budgeting method that are defensible against measured history
79- Decide whether to group an SLO, and refuse high-cardinality grouping
80- Create, verify, update, reset, disable, and delete SLOs
81- Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities
82- Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable
83 rules
84- Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI
85
86## Output discipline
87
88Applies to every response produced under this skill.
89
90- **Commit to the best-supported conclusion.** Recommend one SLI design and defend it. Do not enumerate every SLI type
91 with equal weight and leave the choice to the user — that is not a recommendation, it is a menu.
92- **State the target and window you chose, and why, once.** Do not restate the justification per bullet.
93- **Do not invent field names.** Verify every field against the mapping before writing it into an indicator. If a field
94 cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap.
95- **Do not speculate past the evidence.** If the measured history does not support a target, say what it does support.
96- **Report absence as absence.** Zero events means the data is missing or the service is not emitting; it never means
97 the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%.
98- **Do not pad.** No restating the question, no narrating which queries were run unless the result mattered.
99- **End on the finding.** No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a
100 recommendations list, phrased as recommendations, not as questions.
101
102## Process: route the reliability concern to an instrument
103
104Do this before designing anything. Most bad SLOs are threshold rules wearing a costume.
105
1061. **Name the user-visible symptom.** Ask what a user or downstream consumer would notice and complain about. If the
107 answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a
108 capacity ceiling, not a reliability target. Route it to a threshold rule.
109
1102. **Check whether coverage already exists.** List SLOs with `GET kbn:/api/observability/slos`, alerting rules with
111 `GET kbn:/api/alerting/rules/_find`, and anomaly jobs with `GET /_ml/anomaly_detectors`. Adding a second instrument
112 on a signal that is already covered is the most common source of duplicate pages.
113
1143. **Pick the instrument.**
115
116 | Concern | Instrument | Why, and what it costs |
117 | ---------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
118 | A user-visible success or latency outcome you want to budget over weeks | SLO + burn-rate rule | Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window. |
119 | A hard bound with a known safe value and an immediate operator action | Threshold or custom rule | Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds. |
120 | A signal with no fixed threshold, strong seasonality, or many entities | Anomaly detection job | Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager. |
121 | Reachability of a user-facing endpoint from outside your own telemetry | Synthetics monitor | The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure. |
122 | An internal job with no consumer contract, or a service still changing daily | None | An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO. |
123
1244. **Do not skip the outage case.** An SLI built from a service's own logs or traces cannot see the service disappear:
125 with zero events there is no `total`, so the ratio is undefined and the SLO neither burns nor recovers. Pair every
126 request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule.
127 This is the one place where two instruments on one signal is correct rather than duplicative.
128
129## Process: design the SLO
130
1311. **Locate the data and confirm the fields exist.** Resolve the index pattern with `GET /_resolve/index/<pattern>`,
132 then confirm every field the indicator will reference with `GET /<index>/_mapping` or `GET /_field_caps`. OTel-native
133 data lives in `traces-*.otel-*`, `metrics-*.otel-*`, and `logs-*.otel-*`, with `service.name` populated on all three;
134 APM aggregate metrics live in `metrics-service_summary.1m.otel-*`, `metrics-service_transaction.1m.otel-*`,
135 `metrics-transaction.1m.otel-*`, and `metrics-service_destination.1m.otel-*`. Do not assume a status-code, duration,
136 or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes
137 cleanly and means nothing; [How a missing field fails](#how-a-missing-field-fails) shows exactly how.
138
139 Two field placements are worth knowing because they are commonly guessed wrong: `http.response.status_code` is on
140 `traces-*.otel-*` and not on `logs-*.otel-*`, and `transaction.duration.us` is on `traces-*.otel-*` and not on the
141 `metrics-*.1m.otel-*` rollups, which carry `transaction.duration.histogram` and `transaction.duration.summary`
142 instead. Confirm both against the deployment in front of you rather than trusting this list.
143
1442. **Validate the SLI with ES|QL before creating anything.** Run the good/total ratio over recent history with
145 `POST /_query` so you know the measured baseline. This query is exploratory and is written in ES|QL, not KQL:
146
147 ```esql
148 FROM traces-*.otel-*
149 | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 30 day
150 | STATS total = COUNT(*), good = COUNT(*) WHERE http.response.status_code < 500
151 | EVAL achieved = good::DOUBLE / total
152 ```
153
154 Against an OTel demo cluster this returns `total: 1393887`, `good: 1393887`, `achieved: 1.0`. That is the measured
155 input to step 4, not a target to copy — a clean 30 days argues for a target below 100%, not at it.
156
157 **HTTP status lives on traces, not on logs.** `http.response.status_code` is populated on `traces-*.otel-*`. OTel log
158 records do not carry an HTTP status, so the field does not exist in `logs-*.otel-*` at all, and its coverage on
159 traces is per-service: only spans emitted by an HTTP server carry it. Filtering `kind == "Server"` keeps the
160 service's own inbound requests and drops the client spans that report the status of calls it made to others. Confirm
161 coverage for the specific service before building an indicator on it:
162
163 ```esql
164 FROM traces-*.otel-*
165 | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 24 hour
166 | STATS spans = COUNT(*), with_status = COUNT(http.response.status_code)
167 ```
168
169 On the same cluster `cart` returns `spans: 46420, with_status: 46420`, while `checkout` — a gRPC service — returns
170 `spans: 4887, with_status: 0`. When `with_status` is 0 there is no HTTP status to budget. Use
171 `sli.apm.transactionErrorRate`, or build the ratio on `event.outcome`, which is populated on every OTel span
172 regardless of protocol:
173
174 ```esql
175 FROM traces-*.otel-*
176 | WHERE service.name == "checkout" AND kind == "Server" AND @timestamp >= NOW() - 30 day
177 | STATS total = COUNT(*), good = COUNT(*) WHERE event.outcome == "success"
178 | EVAL achieved = good::DOUBLE / total
179 ```
180
181 That returns `total: 147009`, `good: 146996`, `achieved: 0.9999`. Write `good` in the positive form (`== "success"`),
182 not as a negation of `"failure"` — a null outcome must not count as good.
183
184 This split is not specific to Serverless or to one demo application. The same measurement across every service on a
185 Stack 9.4.4 cluster, over `traces-*.otel-*` with `kind == "Server"`:
186
187 | Service | Server spans | `http.response.status_code` coverage | `event.outcome` coverage |
188 | --------------- | ------------ | ------------------------------------ | ------------------------ |
189 | catalog | 49,271 | 0% | 100% |
190 | gateway | 37,773 | 100% | 100% |
191 | orders | 37,754 | 100% | 100% |
192 | payments | 34,710 | 100% | 100% |
193 | recommendations | 34,692 | 0% | 100% |
194 | shipping | 34,531 | 100% | 100% |
195
196 Two of six services have no HTTP status at all while every one of the six has `event.outcome` on every span. An
197 availability SLO built on `http.response.status_code < 500` for `catalog` would compute `good: 0` against
198 `total: 49271` — an achieved SLI of 0%, a fully consumed error budget, and a burn-rate rule that pages continuously
199 against a healthy service. Nothing in the API response or the SLO UI flags this; the numerator is simply always zero.
200 Run the coverage check above for the specific service every time, and prefer `event.outcome` when you are writing one
201 indicator to cover several services.
202
203 Never substitute a plausible-looking status field for a missing one — see
204 [How a missing field fails](#how-a-missing-field-fails) for what that costs.
205
206 Also check that traffic is thick enough for a ratio to be meaningful. If the thinnest buckets carry only a handful of
207 events, a single failure swings the SLI by whole percentage points and the SLO will be noise:
208
209 ```esql
210 FROM traces-*.otel-*
211 | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 7 day
212 | STATS events = COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour)
213 | SORT events ASC
214 | LIMIT 10
215 ```
216
217 Bound this one explicitly. Without a `@timestamp` predicate it buckets the entire retention of the trace data
218 streams, which is cheap on a demo cluster and expensive on a customer's.
219
2203. **Choose the SLI type from the data shape, not from preference.**
221
222 | SLI type | Use when |
223 | ------------------------------ | ----------------------------------------------------------------------------------- |
224 | `sli.kql.custom` | Raw logs or documents where good and total are expressible as filters over events |
225 | `sli.metric.custom` | Pre-aggregated metric fields where good and total are equations over sums or counts |
226 | `sli.metric.timeslice` | A metric compared against a threshold per slice, such as a p95 latency ceiling |
227 | `sli.histogram.custom` | Histogram fields, using a range for good and a value count for total |
228 | `sli.apm.transactionDuration` | APM transaction latency against a millisecond threshold |
229 | `sli.apm.transactionErrorRate` | APM transaction success rate |
230 | `sli.synthetics.availability` | Synthetics monitor uptime for a user-facing endpoint |
231
232 Prefer an APM or synthetics type when it fits: they encode the service, environment, and transaction dimensions for
233 you and stay correct when the underlying index layout changes. Reach for `sli.kql.custom` when the outcome is only
234 visible in raw events, and for `sli.metric.custom` when the service already emits its own counters.
235
2364. **Set a target you can meet.** Take the measured baseline from step 2 and set the target at or just below it, then
237 ratchet upward once the service earns it. A target above the measured baseline burns the entire budget on day one,
238 the SLO stays red permanently, and the team stops looking at it. `objective.target` is a decimal between 0 and 1 —
239 `0.995`, not `99.5`. Sanity-check the target against the budget it implies over 30 days:
240
241 | Target | Error budget over 30 days |
242 | ------ | ------------------------- |
243 | 99% | 7h 12m |
244 | 99.5% | 3h 36m |
245 | 99.9% | 43m 12s |
246 | 99.95% | 21m 36s |
247 | 99.99% | 4m 19s |
248
249 If a single rolling deploy, a node restart, or one dependency blip costs more than the whole budget, the target is
250 unachievable and should be rejected, not accepted with a caveat.
251
2525. **Choose the time window.** `timeWindow.type` is `rolling` (`7d`, `30d`, `90d`) or `calendarAligned` (`1w`, `1M`).
253 Rolling windows move continuously, so budget recovers gradually and burn-rate alerting stays meaningful — this is the
254 default for anything operational. Calendar-aligned windows reset at the period boundary, which matches contractual or
255 monthly-reporting language but produces a budget cliff on the first of the month. Use rolling for paging, and add a
256 calendar-aligned SLO alongside it only when someone genuinely reports on calendar periods.
257
2586. **Choose the budgeting method.** `occurrences` divides good events by total events across the whole window, so a
259 high-traffic hour dominates and a quiet overnight outage barely registers. `timeslices` chops the window into slices,
260 marks each slice good or bad against `objective.timesliceTarget`, and divides good slices by total slices, so every
261 period counts equally. Choose `timeslices` when low-traffic periods matter or when the indicator is a threshold on an
262 aggregate rather than a countable good/total. **`sli.metric.timeslice` requires `budgetingMethod: "timeslices"`** —
263 the pairing is not optional, and `objective.timesliceTarget` and `objective.timesliceWindow` become required.
264
2657. **Decide grouping deliberately.** `groupBy` creates one independent SLO instance per unique value, each with its own
266 transform buckets and its own alerts. Measure the cardinality before setting it:
267
268 ```esql
269 FROM traces-*.otel-*
270 | WHERE @timestamp >= NOW() - 24 hour
271 | STATS instances = COUNT_DISTINCT(service.name)
272 ```
273
274 Keep the `@timestamp` bound. A cardinality aggregation with no time predicate runs over the whole retention of the
275 trace data streams; a recent window answers the same question at a fraction of the cost.
276
277 Cardinality is not the only check — the dimension also has to be **populated**. `groupBy` on a field that is mapped
278 but null produces one degenerate instance covering everything, which looks like a working grouped SLO and is not.
279 Confirm with `COUNT(<field>)` alongside `COUNT(*)` before setting it.
280
281 Group on stable, bounded dimensions such as `service.name`, `service.environment`, or `k8s.namespace.name`. Refuse
282 high-cardinality fields — `trace.id`, `url.full`, `user.id`, and churning identifiers like `k8s.pod.name` — and say
283 why rather than creating the SLO and warning afterward. If per-entity visibility is genuinely needed on a wide
284 dimension, filter to the entities that matter instead of grouping across all of them. **Synthetics SLOs are
285 auto-grouped by monitor and location; do not set `groupBy` manually.**
286
2878. **Create and verify.** Build the body per [references/slo-api-schemas.md](references/slo-api-schemas.md) and
288 `POST kbn:/api/observability/slos`. Then read it back with `GET kbn:/api/observability/slos/{id}` and confirm it is
289 computing before reporting success — a created SLO whose transform has not started yet returns no summary data.
290
291## KQL is an API contract here, not a style choice
292
293ES|QL is the query language for Observability, and every exploratory query you run to validate an SLI — baselines,
294cardinality checks, traffic distribution — must be ES|QL against `POST /_query`.
295
296The exception is inside the SLO request body. The `good`, `total`, and `filter` fields of `sli.kql.custom` (and the
297`filter` fields of the other indicator types, plus the `--kql-query` parameter on SLO search) are **KQL strings, because
298the SLO API defines them that way**. There is no ES|QL form of those fields. Write KQL there, and only there:
299
300```json
301{
302 "type": "sli.kql.custom",
303 "params": {
304 "index": "traces-*.otel-*",
305 "filter": "service.name : \"cart\" and kind : \"Server\"",
306 "good": "http.response.status_code < 500",
307 "total": "*",
308 "timestampField": "@timestamp"
309 }
310}
311```
312
313Do not translate these fields to ES|QL — the API will reject or silently mis-parse them. Do not translate exploratory
314queries to KQL either.
315
316## How a missing field fails
317
318The reason step 1 insists on confirming the field is that the two dialects fail in opposite ways, and the dangerous one
319is the dialect that ends up in the SLO.
320
321The exploratory ES|QL query fails loudly. Pointing the step 2 query at `logs-*.otel-*`, where
322`http.response.status_code` does not exist, returns HTTP 400:
323
324```text
325line 3:49: Unknown column [http.response.status_code]
326```
327
328**The KQL in the indicator body does not fail at all.** KQL compiles a numeric comparison on an unmapped field to a
329range query that matches nothing, and the SLO API accepts the definition without complaint. Over the same one-hour
330window on a cluster holding 101,920,257 documents in `logs-*.otel-*`, the `good` clause matched **0** documents while
331`total: "*"` matched **93,847**. The SLO computes cleanly, reports an SLI of 0% against its target, and burns the entire
332error budget on the first transform run. Attach the burn-rate rule from the next section and the fast-burn window pages
333continuously on an indicator that measures nothing.
334
335Do not try to sanity-check the field with a Lucene `query_string` instead. The same predicate as Lucene returns 3,250
336matches on that cluster — not because the field exists, but because Lucene tokenizes the expression and full-text
337matches the fragments against the default fields. A non-zero count from Lucene is not evidence that a field is mapped.
338
339Confirm with `GET /_field_caps` or `GET /<index>/_mapping`, which answer the question directly, and with a
340`COUNT(<field>)` in ES|QL, which additionally tells you whether a mapped field is actually populated.
341
342## Process: alert on the SLO without adding noise
343
344Creating an SLO does not create any alerting. **Burn-rate rules are never auto-created by the SLO API** and must be set
345up separately with `POST kbn:/api/alerting/rule/{id}`, rule type `slo.rules.burnRate`.
346
3471. **Use multiple windows with different severities.** A burn rate of 1 means the budget is being spent exactly fast
348 enough to exhaust at the end of the window. Fast-burn windows (roughly 14x over a one-hour long window with a
349 five-minute short window) catch outages and are worth paging on. Slow-burn windows (roughly 1x-3x over 24 to 72
350 hours) catch chronic degradation and belong in a ticket queue. Routing all windows to the same paging connector is
351 the single fastest way to make the SLO ignored. Window values and action groups are in
352 [references/burn-rate-rules.md](references/burn-rate-rules.md).
353
3542. **Keep the short window.** Each window pairs a long window (does the burn rate justify alerting) with a short window
355 (is it still happening). The short window is what lets a resolved incident stop alerting on its own instead of
356 requiring someone to mute it.
357
3583. **Remove duplicate coverage.** If a burn-rate rule and a raw threshold rule both watch the same signal, one incident
359 produces two pages. Find the overlap with `GET kbn:/api/alerting/rules/_find` before adding anything. Keep the
360 burn-rate rule as the pager and demote or delete the raw threshold — unless the threshold covers the no-data case the
361 SLO structurally cannot see, which is a real gap and should stay.
362
3634. **Snooze, do not disable.** For a bounded pause on one rule use `POST kbn:/api/alerting/rule/{id}/_snooze_schedule`;
364 for planned change affecting many rules use a maintenance window via `POST kbn:/api/maintenance_window`. Both expire
365 on their own. Disabling a rule loses its alert history and depends on someone remembering to re-enable it, so reserve
366 `POST kbn:/api/alerting/rule/{id}/_disable` for rules that are wrong rather than rules that are temporarily
367 inconvenient.
368
3695. **Apply the actionability test.** A rule that fires correctly but has no action the responder can take is a defect,
370 not a success. For each rule, name the first step the on-call would take. If the honest answer is "look at it later",
371 it is not a page — move it to a slow-burn window or a ticket, or delete it.
372
373## Process: monitor availability with synthetics
374
375A synthetics monitor probes an endpoint on a schedule from one or more locations and writes results to `synthetics-*`.
376It is the instrument that answers "is it up from the outside", which telemetry emitted by the service itself can never
377answer. Manage monitors with `GET kbn:/api/synthetics/monitors` and `POST kbn:/api/synthetics/monitors`.
378
3791. **Budget the checks before choosing a target.** Availability is measured in checks, not requests, so the check
380 frequency sets the resolution of the SLI. A ten-minute frequency over 30 days is 4,320 checks; at a 99.99% target the
381 entire error budget is under half a check, so one transient failure exhausts it. Require the budget to be worth at
382 least ten checks — raise the frequency or lower the target until it is.
383
3842. **Use at least three locations.** A single-location monitor cannot distinguish a down service from a bad network path
385 out of one region. Three or more makes that call obvious and removes a whole class of false pages.
386
3873. **Feed it into an SLI.** Create the SLO with `sli.synthetics.availability`, scoped by monitor ids, projects, or tags.
388 Leave `groupBy` unset; the indicator already produces one instance per monitor and location.
389
390## Process: operate and repair
391
392- **Update carefully.** `PUT kbn:/api/observability/slos/{id}` **resets the underlying transform and recomputes
393 history** from scratch. Changing the target or the indicator therefore discards the existing budget picture. Say so
394 before doing it, and prefer creating a second SLO when the old history still matters.
395- **Reset when stuck.** `POST kbn:/api/observability/slos/{id}/_reset` rebuilds the transform and rollup data. Use it
396 when an SLO stops updating, after index mapping changes, or after an upgrade leaves a definition outdated —
397 `GET kbn:/api/observability/slos/_definitions` reports which definitions are outdated.
398- **Pause versus retire.** `POST kbn:/api/observability/slos/{id}/disable` stops computation but keeps the definition;
399 `POST kbn:/api/observability/slos/{id}/enable` resumes it. `DELETE kbn:/api/observability/slos/{id}` is permanent —
400 confirm with the user first.
401- **Check the cluster shape.** Every SLO is backed by a continuous transform, so the cluster needs nodes carrying both
402 the `transform` and `ingest` roles. If SLOs never leave "no data", check this before debugging the indicator.
403
404## Examples
405
406**"Set up an SLO for the checkout service"** — resolve `traces-*.otel-*` and confirm `service.name` and the outcome
407field exist, measure the trailing 30-day success ratio with `POST /_query`, then recommend one design: if the measured
408baseline is 99.6%, propose `sli.apm.transactionErrorRate` at `0.995` on a 30-day rolling window with `occurrences`
409budgeting, and say that 99.9% is rejected because its 43-minute budget is smaller than the service's observed monthly
410degradation. Create it, read it back, then create the burn-rate rule separately.
411
412**"We want a 99.99% SLO on the payments API"** — measure first. If the trailing baseline is 99.7%, reject 99.99%
413outright: it allows 4 minutes 19 seconds of budget over 30 days, less than a single rolling deploy. Recommend 99.5% now
414with a ratchet plan, and state that the constraint is the deploy process rather than the target.
415
416**"Alert us when p95 latency goes above 500 ms"** — this is a threshold on an aggregate, so it is `sli.metric.timeslice`
417with a percentile metric, `comparator: "LT"`, `threshold: 500`, and — required by that indicator —
418`budgetingMethod: "timeslices"` with `timesliceTarget` and `timesliceWindow` set. Confirm the duration field and its
419unit in the mapping first; a threshold written in milliseconds against a microsecond field is off by a thousand.
420
421**"Create an SLO per pod so we can see which pods are unreliable"** — refuse the grouping. `k8s.pod.name` churns on
422every deploy, so each `groupBy` value creates an SLO instance that is orphaned within days while the transform carries
423the cardinality forever. Recommend grouping on `k8s.namespace.name` or `service.name` instead, and point out that
424per-pod reliability is a symptom to investigate, not a target to budget.
425
426**"Disk on the log nodes keeps filling up — can we SLO that?"** — no. Disk utilization is a capacity ceiling with a
427known safe bound and an immediate operator action, so it is a threshold rule, not a reliability target. There is no
428user-visible outcome to budget and no meaningful notion of spending 0.5% of disk-full.
429
430**"Our request rate looks weird but there is no threshold we can name"** — no fixed bound means no SLO and no threshold
431rule. Route it to an anomaly detection job, which learns the seasonal baseline, and note that it needs several weeks of
432history before its scores are trustworthy.
433
434**"Why are we getting paged twice for every checkout incident?"** — list rules with `GET kbn:/api/alerting/rules/_find`
435and look for a burn-rate rule and a raw threshold rule on the same signal. Keep the burn-rate rule as the pager, demote
436the threshold to a ticket, and verify no slow-burn window is routed to the paging connector.
437
438## Guidelines
439
440- Choose the instrument before designing the SLO; a capacity ceiling is a threshold rule and an unbounded signal is an
441 anomaly job.
442- Confirm every field against `GET /<index>/_mapping` or `GET /_field_caps` before writing it into an indicator. Never
443 infer a field name from convention.
444- Validate the SLI with ES|QL through `POST /_query` first; set the target from the measured baseline, not from
445 ambition.
446- `objective.target` is a decimal between 0 and 1 — `0.995` for 99.5%.
447- Timeslice metric indicators require `budgetingMethod: "timeslices"`.
448- The `good`, `total`, and `filter` fields of `sli.kql.custom` are KQL because the API defines them that way. Everything
449 else you query is ES|QL.
450- Updating an SLO resets the underlying transform and recomputes history — warn the user before doing it.
451- The cluster needs nodes with both the `transform` and `ingest` roles.
452- Use the reset operation when an SLO is stuck or after index mapping changes.
453- Group-by SLOs create one instance per unique value — refuse high-cardinality fields rather than creating and warning.
454- Synthetics SLOs are auto-grouped by monitor and location; do not set `groupBy` manually.
455- Burn-rate alert rules are not created by the SLO API — set them up separately, with fast-burn and slow-burn windows
456 routed to different severities.
457- Prefer snoozing a rule or opening a maintenance window over disabling it.
458- Confirm deletions before executing them.
459
460## Operations
461
462| HTTP API (shorthand) | `elastic` CLI command |
463| --------------------------------------------------- | --------------------------------------------------------------------------------------------- |
464| `POST /_query` | `elastic es esql query --format tsv --query '<esql>'` |
465| `GET /_resolve/index/<pattern>` | `elastic es indices resolve-index --name '<pattern>'` |
466| `GET /<index>/_mapping` | `elastic es indices get-mapping --index '<index>'` |
467| `GET /_field_caps` | `elastic es field-caps --index '<index>' --fields '<fields>'` |
468| `GET /_ml/anomaly_detectors` | `elastic es ml get-jobs` |
469| `GET kbn:/api/observability/slos` | `elastic kb slo find-slos-op --space-id '<space>' --kql-query '<kql>'` |
470| `POST kbn:/api/observability/slos` | `elastic kb slo create-slo-op --space-id '<space>' --input-file <json>` |
471| `GET kbn:/api/observability/slos/{id}` | `elastic kb slo get-slo-op --space-id '<space>' --slo-id '<id>'` |
472| `PUT kbn:/api/observability/slos/{id}` | `elastic kb slo update-slo-op --space-id '<space>' --slo-id '<id>' --input-file <json>` |
473| `DELETE kbn:/api/observability/slos/{id}` | `elastic kb slo delete-slo-op --space-id '<space>' --slo-id '<id>'` |
474| `POST kbn:/api/observability/slos/{id}/_reset` | `elastic kb slo reset-slo-op --space-id '<space>' --slo-id '<id>'` |
475| `POST kbn:/api/observability/slos/{id}/enable` | `elastic kb slo enable-slo-op --space-id '<space>' --slo-id '<id>'` |
476| `POST kbn:/api/observability/slos/{id}/disable` | `elastic kb slo disable-slo-op --space-id '<space>' --slo-id '<id>'` |
477| `GET kbn:/api/observability/slos/_definitions` | `elastic kb slo get-definitions-op --space-id '<space>'` |
478| `GET kbn:/api/alerting/rules/_find` | `elastic kb alerting get-alerting-rules-find --filter '<kql>'` |
479| `POST kbn:/api/alerting/rule/{id}` | `elastic kb alerting post-alerting-rule-id --id '<id>' --input-file <json>` |
480| `POST kbn:/api/alerting/rule/{id}/_snooze_schedule` | `elastic kb alerting post-alerting-rule-id-snooze-schedule --id '<id>' --input-file <json>` |
481| `POST kbn:/api/alerting/rule/{id}/_disable` | `elastic kb alerting post-alerting-rule-id-disable --id '<id>'` |
482| `POST kbn:/api/maintenance_window` | `elastic kb maintenance-window post-maintenance-window --title '<title>' --input-file <json>` |
483| `GET kbn:/api/synthetics/monitors` | _no CLI binding — see CLI gaps below_ |
484| `POST kbn:/api/synthetics/monitors` | _no CLI binding — see CLI gaps below_ |
485
486**CLI gaps.** As of `elastic` CLI 0.2.0 there is no binding for the Synthetics monitor management API — the `stack kb`
487namespace exposes no synthetics commands. The HTTP shorthand above is still the correct contract and remains portable,
488so keep using it when describing what must happen, but do not invent a CLI invocation for it. Create and edit monitors
489through the Synthetics app in Kibana, through a Synthetics project, or by calling the Kibana API directly from tooling
490that already holds credentials. Everything else in this skill, including the burn-rate rule and every SLO operation, has
491a working CLI binding.