Observability Service Reliability

Service Reliability

Design reliability targets that people will actually act on, then operate them. This skill covers the judgment before the API call — which service-level indicator fits the data you have, what target is achievable rather than aspirational, whether an SLO is even the right instrument — and then the mechanics of creating, alerting on, resetting, and retiring SLOs through the Kibana API.

Reliability instruments are not interchangeable. An SLO measures a user-visible outcome against a spendable budget; a threshold rule fires on a raw condition; an anomaly job finds deviations where no fixed threshold exists; a synthetics monitor is the only one of the four that can see a service that has stopped emitting telemetry entirely. Choosing wrong produces alerts that are technically correct and operationally useless. For diagnosing a service that is already degraded, and for the incident workflow itself, use the observability-sre-triage skill; for general rule lifecycle mechanics use the kibana-alerting-rules skill.

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

Analysis without cluster access

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

SLO and alerting operations run against Kibana and use the kbn: prefix (for example, POST kbn:/api/observability/slos); data validation runs against Elasticsearch with a bare path (for example, POST /_query). Kibana SLO commands are space-scoped — pass the space explicitly. For non-default spaces the HTTP path becomes kbn:/s/<space_id>/api/observability/slos.

Full request-body schemas for every SLI type live in references/slo-api-schemas.md, the burn-rate rule schema in references/burn-rate-rules.md, and curated official documentation in references/documentation.md.

Jobs to be done

• Translate a reliability concern into the right instrument: SLO, burn-rate rule, threshold rule, anomaly job, or synthetics monitor
• Choose an SLI type for a given service and data shape, and validate the underlying fields before committing
• Set a target, time window, and budgeting method that are defensible against measured history
• Decide whether to group an SLO, and refuse high-cardinality grouping
• Create, verify, update, reset, disable, and delete SLOs
• Attach burn-rate alert rules with fast-burn and slow-burn windows routed to different severities
• Keep the alerting surface signal-dense: remove duplicate coverage, snooze instead of disable, delete unactionable rules
• Monitor availability of user-facing endpoints with synthetics and feed that into an availability SLI

Output discipline

Applies to every response produced under this skill.

• Commit to the best-supported conclusion. Recommend one SLI design and defend it. Do not enumerate every SLI type with equal weight and leave the choice to the user — that is not a recommendation, it is a menu.
• State the target and window you chose, and why, once. Do not restate the justification per bullet.
• Do not invent field names. Verify every field against the mapping before writing it into an indicator. If a field cannot be confirmed, say so and stop; a plausible-looking field name is worse than an admitted gap.
• Do not speculate past the evidence. If the measured history does not support a target, say what it does support.
• Report absence as absence. Zero events means the data is missing or the service is not emitting; it never means the service is healthy. An SLI ratio computed over zero total events is undefined, not 100%.
• Do not pad. No restating the question, no narrating which queries were run unless the result mattered.
• End on the finding. No trailing offers such as "want me to set this up?". Actionable follow-ups belong in a recommendations list, phrased as recommendations, not as questions.

Process: route the reliability concern to an instrument

Do this before designing anything. Most bad SLOs are threshold rules wearing a costume.

1. Name the user-visible symptom. Ask what a user or downstream consumer would notice and complain about. If the answer is a resource number rather than an experience — disk at 90%, heap climbing, replica lag — the concern is a capacity ceiling, not a reliability target. Route it to a threshold rule.

2. Check whether coverage already exists. List SLOs with GET kbn:/api/observability/slos, alerting rules with GET kbn:/api/alerting/rules/_find, and anomaly jobs with GET /_ml/anomaly_detectors. Adding a second instrument on a signal that is already covered is the most common source of duplicate pages.

3. Pick the instrument.

   Concern	Instrument	Why, and what it costs
   A user-visible success or latency outcome you want to budget over weeks	SLO + burn-rate rule	Gives a spendable budget and history, and paces change velocity. Costs a transform, needs steady traffic, and is deliberately slow — even fast-burn uses a one-hour window.
   A hard bound with a known safe value and an immediate operator action	Threshold or custom rule	Fires within one schedule interval, no transform. But it has no budget and no memory, so it re-fires for as long as the condition holds.
   A signal with no fixed threshold, strong seasonality, or many entities	Anomaly detection job	Finds unknown-unknowns and adapts to seasonality. Needs weeks of history to be trustworthy and emits scores, not outcomes, so it is a poor pager.
   Reachability of a user-facing endpoint from outside your own telemetry	Synthetics monitor	The only instrument that detects a total outage. Costs a check budget (see the synthetics section) and cannot explain an internal partial failure.
   An internal job with no consumer contract, or a service still changing daily	None	An SLO with no owner and no stable baseline becomes permanently red and is then ignored, which is worse than no SLO.

4. Do not skip the outage case. An SLI built from a service's own logs or traces cannot see the service disappear: with zero events there is no total, so the ratio is undefined and the SLO neither burns nor recovers. Pair every request-based SLO on a user-facing service with either a synthetics availability monitor or a no-data threshold rule. This is the one place where two instruments on one signal is correct rather than duplicative.

Process: design the SLO

1. Locate the data and confirm the fields exist. Resolve the index pattern with GET /_resolve/index/<pattern>, then confirm every field the indicator will reference with GET /<index>/_mapping or GET /_field_caps. OTel-native data lives in traces-*.otel-*, metrics-*.otel-*, and logs-*.otel-*, with service.name populated on all three; APM aggregate metrics live in metrics-service_summary.1m.otel-*, metrics-service_transaction.1m.otel-*, metrics-transaction.1m.otel-*, and metrics-service_destination.1m.otel-*. Do not assume a status-code, duration, or outcome field exists because it is conventional — confirm it. A wrong field name produces an SLO that computes cleanly and means nothing; How a missing field fails shows exactly how.

   Two field placements are worth knowing because they are commonly guessed wrong: http.response.status_code is on traces-*.otel-* and not on logs-*.otel-*, and transaction.duration.us is on traces-*.otel-* and not on the metrics-*.1m.otel-* rollups, which carry transaction.duration.histogram and transaction.duration.summary instead. Confirm both against the deployment in front of you rather than trusting this list.

2. Validate the SLI with ES|QL before creating anything. Run the good/total ratio over recent history with POST /_query so you know the measured baseline. This query is exploratory and is written in ES|QL, not KQL:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 30 day
   | STATS total = COUNT(*), good = COUNT(*) WHERE http.response.status_code < 500
   | EVAL achieved = good::DOUBLE / total
   

   Against an OTel demo cluster this returns total: 1393887, good: 1393887, achieved: 1.0. That is the measured input to step 4, not a target to copy — a clean 30 days argues for a target below 100%, not at it.

   HTTP status lives on traces, not on logs. http.response.status_code is populated on traces-*.otel-*. OTel log records do not carry an HTTP status, so the field does not exist in logs-*.otel-* at all, and its coverage on traces is per-service: only spans emitted by an HTTP server carry it. Filtering kind == "Server" keeps the service's own inbound requests and drops the client spans that report the status of calls it made to others. Confirm coverage for the specific service before building an indicator on it:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 24 hour
   | STATS spans = COUNT(*), with_status = COUNT(http.response.status_code)
   

   On the same cluster cart returns spans: 46420, with_status: 46420, while checkout — a gRPC service — returns spans: 4887, with_status: 0. When with_status is 0 there is no HTTP status to budget. Use sli.apm.transactionErrorRate, or build the ratio on event.outcome, which is populated on every OTel span regardless of protocol:

   FROM traces-*.otel-*
   | WHERE service.name == "checkout" AND kind == "Server" AND @timestamp >= NOW() - 30 day
   | STATS total = COUNT(*), good = COUNT(*) WHERE event.outcome == "success"
   | EVAL achieved = good::DOUBLE / total
   

   That returns total: 147009, good: 146996, achieved: 0.9999. Write good in the positive form (== "success"), not as a negation of "failure" — a null outcome must not count as good.

   This split is not specific to Serverless or to one demo application. The same measurement across every service on a Stack 9.4.4 cluster, over traces-*.otel-* with kind == "Server":

   Service	Server spans	http.response.status_code coverage	event.outcome coverage
   catalog	49,271	0%	100%
   gateway	37,773	100%	100%
   orders	37,754	100%	100%
   payments	34,710	100%	100%
   recommendations	34,692	0%	100%
   shipping	34,531	100%	100%

   Two of six services have no HTTP status at all while every one of the six has event.outcome on every span. An availability SLO built on http.response.status_code < 500 for catalog would compute good: 0 against total: 49271 — an achieved SLI of 0%, a fully consumed error budget, and a burn-rate rule that pages continuously against a healthy service. Nothing in the API response or the SLO UI flags this; the numerator is simply always zero. Run the coverage check above for the specific service every time, and prefer event.outcome when you are writing one indicator to cover several services.

   Never substitute a plausible-looking status field for a missing one — see How a missing field fails for what that costs.

   Also check that traffic is thick enough for a ratio to be meaningful. If the thinnest buckets carry only a handful of events, a single failure swings the SLI by whole percentage points and the SLO will be noise:

   FROM traces-*.otel-*
   | WHERE service.name == "cart" AND kind == "Server" AND @timestamp >= NOW() - 7 day
   | STATS events = COUNT(*) BY bucket = BUCKET(@timestamp, 1 hour)
   | SORT events ASC
   | LIMIT 10
   

   Bound this one explicitly. Without a @timestamp predicate it buckets the entire retention of the trace data streams, which is cheap on a demo cluster and expensive on a customer's.

3. Choose the SLI type from the data shape, not from preference.

   SLI type	Use when
   sli.kql.custom	Raw logs or documents where good and total are expressible as filters over events
   sli.metric.custom	Pre-aggregated metric fields where good and total are equations over sums or counts
   sli.metric.timeslice	A metric compared against a threshold per slice, such as a p95 latency ceiling
   sli.histogram.custom	Histogram fields, using a range for good and a value count for total
   sli.apm.transactionDuration	APM transaction latency against a millisecond threshold
   sli.apm.transactionErrorRate	APM transaction success rate
   sli.synthetics.availability	Synthetics monitor uptime for a user-facing endpoint

   Prefer an APM or synthetics type when it fits: they encode the service, environment, and transaction dimensions for you and stay correct when the underlying index layout changes. Reach for sli.kql.custom when the outcome is only visible in raw events, and for sli.metric.custom when the service already emits its own counters.

4. Set a target you can meet. Take the measured baseline from step 2 and set the target at or just below it, then ratchet upward once the service earns it. A target above the measured baseline burns the entire budget on day one, the SLO stays red permanently, and the team stops looking at it. objective.target is a decimal between 0 and 1 — 0.995, not 99.5. Sanity-check the target against the budget it implies over 30 days:

   Target	Error budget over 30 days
   99%	7h 12m
   99.5%	3h 36m
   99.9%	43m 12s
   99.95%	21m 36s
   99.99%	4m 19s

   If a single rolling deploy, a node restart, or one dependency blip costs more than the whole budget, the target is unachievable and should be rejected, not accepted with a caveat.

5. Choose the time window. timeWindow.type is rolling (7d, 30d, 90d) or calendarAligned (1w, 1M). Rolling windows move continuously, so budget recovers gradually and burn-rate alerting stays meaningful — this is the default for anything operational. Calendar-aligned windows reset at the period boundary, which matches contractual or monthly-reporting language but produces a budget cliff on the first of the month. Use rolling for paging, and add a calendar-aligned SLO alongside it only when someone genuinely reports on calendar periods.

6. Choose the budgeting method. occurrences divides good events by total events across the whole window, so a high-traffic hour dominates and a quiet overnight outage barely registers. timeslices chops the window into slices, marks each slice good or bad against objective.timesliceTarget, and divides good slices by total slices, so every period counts equally. Choose timeslices when low-traffic periods matter or when the indicator is a threshold on an aggregate rather than a countable good/total. sli.metric.timeslice requires budgetingMethod: "timeslices" — the pairing is not optional, and objective.timesliceTarget and objective.timesliceWindow become required.

7. Decide grouping deliberately. groupBy creates one independent SLO instance per unique value, each with its own transform buckets and its own alerts. Measure the cardinality before setting it:

   FROM traces-*.otel-*
   | WHERE @timestamp >= NOW() - 24 hour
   | STATS instances = COUNT_DISTINCT(service.name)
   

   Keep the @timestamp bound. A cardinality aggregation with no time predicate runs over the whole retention of the trace data streams; a recent window answers the same question at a fraction of the cost.

   Cardinality is not the only check — the dimension also has to be populated. groupBy on a field that is mapped but null produces one degenerate instance covering everything, which looks like a working grouped SLO and is not. Confirm with COUNT(<field>) alongside COUNT(*) before setting it.

   Group on stable, bounded dimensions such as service.name, service.environment, or k8s.namespace.name. Refuse high-cardinality fields — trace.id, url.full, user.id, and churning identifiers like k8s.pod.name — and say why rather than creating the SLO and warning afterward. If per-entity visibility is genuinely needed on a wide dimension, filter to the entities that matter instead of grouping across all of them. Synthetics SLOs are auto-grouped by monitor and location; do not set groupBy manually.

8. Create and verify. Build the body per references/slo-api-schemas.md and POST kbn:/api/observability/slos. Then read it back with GET kbn:/api/observability/slos/{id} and confirm it is computing before reporting success — a created SLO whose transform has not started yet returns no summary data.

KQL is an API contract here, not a style choice

ES|QL is the query language for Observability, and every exploratory query you run to validate an SLI — baselines, cardinality checks, traffic distribution — must be ES|QL against POST /_query.

The exception is inside the SLO request body. The good, total, and filter fields of sli.kql.custom (and the filter fields of the other indicator types, plus the --kql-query parameter on SLO search) are KQL strings, because the SLO API defines them that way. There is no ES|QL form of those fields. Write KQL there, and only there:

{
  "type": "sli.kql.custom",
  "params": {
    "index": "traces-*.otel-*",
    "filter": "service.name : \"cart\" and kind : \"Server\"",
    "good": "http.response.status_code < 500",
    "total": "*",
    "timestampField": "@timestamp"
  }
}

Do not translate these fields to ES|QL — the API will reject or silently mis-parse them. Do not translate exploratory queries to KQL either.

How a missing field fails

The reason step 1 insists on confirming the field is that the two dialects fail in opposite ways, and the dangerous one is the dialect that ends up in the SLO.

The exploratory ES|QL query fails loudly. Pointing the step 2 query at logs-*.otel-*, where http.response.status_code does not exist, returns HTTP 400:

line 3:49: Unknown column [http.response.status_code]

The KQL in the indicator body does not fail at all. KQL compiles a numeric comparison on an unmapped field to a range query that matches nothing, and the SLO API accepts the definition without complaint. Over the same one-hour window on a cluster holding 101,920,257 documents in logs-*.otel-*, the good clause matched 0 documents while total: "*" matched 93,847. The SLO computes cleanly, reports an SLI of 0% against its target, and burns the entire error budget on the first transform run. Attach the burn-rate rule from the next section and the fast-burn window pages continuously on an indicator that measures nothing.

Do not try to sanity-check the field with a Lucene query_string instead. The same predicate as Lucene returns 3,250 matches on that cluster — not because the field exists, but because Lucene tokenizes the expression and full-text matches the fragments against the default fields. A non-zero count from Lucene is not evidence that a field is mapped.

Confirm with GET /_field_caps or GET /<index>/_mapping, which answer the question directly, and with a COUNT(<field>) in ES|QL, which additionally tells you whether a mapped field is actually populated.

Process: alert on the SLO without adding noise

Creating an SLO does not create any alerting. Burn-rate rules are never auto-created by the SLO API and must be set up separately with POST kbn:/api/alerting/rule/{id}, rule type slo.rules.burnRate.

1. Use multiple windows with different severities. A burn rate of 1 means the budget is being spent exactly fast enough to exhaust at the end of the window. Fast-burn windows (roughly 14x over a one-hour long window with a five-minute short window) catch outages and are worth paging on. Slow-burn windows (roughly 1x-3x over 24 to 72 hours) catch chronic degradation and belong in a ticket queue. Routing all windows to the same paging connector is the single fastest way to make the SLO ignored. Window values and action groups are in references/burn-rate-rules.md.

2. Keep the short window. Each window pairs a long window (does the burn rate justify alerting) with a short window (is it still happening). The short window is what lets a resolved incident stop alerting on its own instead of requiring someone to mute it.

3. Remove duplicate coverage. If a burn-rate rule and a raw threshold rule both watch the same signal, one incident produces two pages. Find the overlap with GET kbn:/api/alerting/rules/_find before adding anything. Keep the burn-rate rule as the pager and demote or delete the raw threshold — unless the threshold covers the no-data case the SLO structurally cannot see, which is a real gap and should stay.

4. Snooze, do not disable. For a bounded pause on one rule use POST kbn:/api/alerting/rule/{id}/_snooze_schedule; for planned change affecting many rules use a maintenance window via POST kbn:/api/maintenance_window. Both expire on their own. Disabling a rule loses its alert history and depends on someone remembering to re-enable it, so reserve POST kbn:/api/alerting/rule/{id}/_disable for rules that are wrong rather than rules that are temporarily inconvenient.

5. Apply the actionability test. A rule that fires correctly but has no action the responder can take is a defect, not a success. For each rule, name the first step the on-call would take. If the honest answer is "look at it later", it is not a page — move it to a slow-burn window or a ticket, or delete it.

Process: monitor availability with synthetics

A synthetics monitor probes an endpoint on a schedule from one or more locations and writes results to synthetics-*. It is the instrument that answers "is it up from the outside", which telemetry emitted by the service itself can never answer. Manage monitors with GET kbn:/api/synthetics/monitors and POST kbn:/api/synthetics/monitors.

1. Budget the checks before choosing a target. Availability is measured in checks, not requests, so the check frequency sets the resolution of the SLI. A ten-minute frequency over 30 days is 4,320 checks; at a 99.99% target the entire error budget is under half a check, so one transient failure exhausts it. Require the budget to be worth at least ten checks — raise the frequency or lower the target until it is.

2. Use at least three locations. A single-location monitor cannot distinguish a down service from a bad network path out of one region. Three or more makes that call obvious and removes a whole class of false pages.

3. Feed it into an SLI. Create the SLO with sli.synthetics.availability, scoped by monitor ids, projects, or tags. Leave groupBy unset; the indicator already produces one instance per monitor and location.

Process: operate and repair

• Update carefully. PUT kbn:/api/observability/slos/{id} resets the underlying transform and recomputes history from scratch. Changing the target or the indicator therefore discards the existing budget picture. Say so before doing it, and prefer creating a second SLO when the old history still matters.
• Reset when stuck. POST kbn:/api/observability/slos/{id}/_reset rebuilds the transform and rollup data. Use it when an SLO stops updating, after index mapping changes, or after an upgrade leaves a definition outdated — GET kbn:/api/observability/slos/_definitions reports which definitions are outdated.
• Pause versus retire. POST kbn:/api/observability/slos/{id}/disable stops computation but keeps the definition; POST kbn:/api/observability/slos/{id}/enable resumes it. DELETE kbn:/api/observability/slos/{id} is permanent — confirm with the user first.
• Check the cluster shape. Every SLO is backed by a continuous transform, so the cluster needs nodes carrying both the transform and ingest roles. If SLOs never leave "no data", check this before debugging the indicator.

Examples

"Set up an SLO for the checkout service" — resolve traces-*.otel-* and confirm service.name and the outcome field exist, measure the trailing 30-day success ratio with POST /_query, then recommend one design: if the measured baseline is 99.6%, propose sli.apm.transactionErrorRate at 0.995 on a 30-day rolling window with occurrences budgeting, and say that 99.9% is rejected because its 43-minute budget is smaller than the service's observed monthly degradation. Create it, read it back, then create the burn-rate rule separately.

"We want a 99.99% SLO on the payments API" — measure first. If the trailing baseline is 99.7%, reject 99.99% outright: it allows 4 minutes 19 seconds of budget over 30 days, less than a single rolling deploy. Recommend 99.5% now with a ratchet plan, and state that the constraint is the deploy process rather than the target.

"Alert us when p95 latency goes above 500 ms" — this is a threshold on an aggregate, so it is sli.metric.timeslice with a percentile metric, comparator: "LT", threshold: 500, and — required by that indicator — budgetingMethod: "timeslices" with timesliceTarget and timesliceWindow set. Confirm the duration field and its unit in the mapping first; a threshold written in milliseconds against a microsecond field is off by a thousand.

"Create an SLO per pod so we can see which pods are unreliable" — refuse the grouping. k8s.pod.name churns on every deploy, so each groupBy value creates an SLO instance that is orphaned within days while the transform carries the cardinality forever. Recommend grouping on k8s.namespace.name or service.name instead, and point out that per-pod reliability is a symptom to investigate, not a target to budget.

"Disk on the log nodes keeps filling up — can we SLO that?" — no. Disk utilization is a capacity ceiling with a known safe bound and an immediate operator action, so it is a threshold rule, not a reliability target. There is no user-visible outcome to budget and no meaningful notion of spending 0.5% of disk-full.

"Our request rate looks weird but there is no threshold we can name" — no fixed bound means no SLO and no threshold rule. Route it to an anomaly detection job, which learns the seasonal baseline, and note that it needs several weeks of history before its scores are trustworthy.

"Why are we getting paged twice for every checkout incident?" — list rules with GET kbn:/api/alerting/rules/_find and look for a burn-rate rule and a raw threshold rule on the same signal. Keep the burn-rate rule as the pager, demote the threshold to a ticket, and verify no slow-burn window is routed to the paging connector.

Guidelines

• Choose the instrument before designing the SLO; a capacity ceiling is a threshold rule and an unbounded signal is an anomaly job.
• Confirm every field against GET /<index>/_mapping or GET /_field_caps before writing it into an indicator. Never infer a field name from convention.
• Validate the SLI with ES|QL through POST /_query first; set the target from the measured baseline, not from ambition.
• objective.target is a decimal between 0 and 1 — 0.995 for 99.5%.
• Timeslice metric indicators require budgetingMethod: "timeslices".
• The good, total, and filter fields of sli.kql.custom are KQL because the API defines them that way. Everything else you query is ES|QL.
• Updating an SLO resets the underlying transform and recomputes history — warn the user before doing it.
• The cluster needs nodes with both the transform and ingest roles.
• Use the reset operation when an SLO is stuck or after index mapping changes.
• Group-by SLOs create one instance per unique value — refuse high-cardinality fields rather than creating and warning.
• Synthetics SLOs are auto-grouped by monitor and location; do not set groupBy manually.
• Burn-rate alert rules are not created by the SLO API — set them up separately, with fast-burn and slow-burn windows routed to different severities.
• Prefer snoozing a rule or opening a maintenance window over disabling it.
• Confirm deletions before executing them.

Operations

CLI gaps. As of elastic CLI 0.2.0 there is no binding for the Synthetics monitor management API — the stack kb namespace exposes no synthetics commands. The HTTP shorthand above is still the correct contract and remains portable, so keep using it when describing what must happen, but do not invent a CLI invocation for it. Create and edit monitors through the Synthetics app in Kibana, through a Synthetics project, or by calling the Kibana API directly from tooling that already holds credentials. Everything else in this skill, including the burn-rate rule and every SLO operation, has a working CLI binding.