Elasticsearch Anomaly Detection
Create, open, and start ML anomaly detection jobs on time-series data. Choose the right count-family detector direction,
configure bucket span and time field, wire the datafeed to the correct index, and confirm running state from stats — not
from assumptions.
Environment Configuration
This skill executes Elasticsearch operations through the elastic CLI. If the
elastic CLI is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,
GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document
maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API
directly.
Prerequisite: ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless
projects include ML. The caller needs manage_ml to create and manage jobs.
Related skill: For interpreting anomaly scores, influencers, and model behavior after a job is running, use
elasticsearch-anomaly-detection-explainer — not this skill.
Process
Discover the target index and time field. List candidate indices with GET /_cat/indices (pass a pattern when
the user names one). Fetch field types for the chosen index with GET /{index}/_mapping. The decision: confirm the
index exists, identify the time field (often @timestamp), and verify document volume is sufficient for baseline
learning. Never guess index or field names — they vary across deployments.
Choose detector function and direction. Match the user's intent to a count-family detector in
analysis_config.detectors:
- Spike, surge, unusual increase in event volume →
high_count (or count, which flags both directions but is
acceptable when the user cares about spikes). Do not use low_count — it will miss spikes.
- Drop, outage, absence of events, traffic stops →
low_count. Do not use high_count — it will miss drops
and silence.
- Metric deviation (CPU, latency, a numeric field) → mean-family functions (
mean, high_mean, low_mean) with
field_name set — only when the user asks about a numeric metric, not raw event volume.
The decision: pick one primary detector whose direction matches the anomaly type. For volume spike/drop questions on
document counts, stay in the count family — mean detectors are unsuited to "how many events" questions.
Set immutable job shape before creation. These fields cannot change after PUT /_ml/anomaly_detectors/{job_id}:
analysis_config.bucket_span — use the interval the user specifies (e.g. 15m for 15-minute buckets). Match the
granularity of anomalies they care about; too short is noisy, too long is slow to detect.
data_description.time_field — the time field from the mapping (commonly @timestamp).
analysis_config.detectors — the function and direction from step 2.
Example job body for a volume-spike detector:
{
"analysis_config": {
"bucket_span": "15m",
"detectors": [{ "function": "high_count" }]
},
"data_description": { "time_field": "@timestamp" }
}
Example for an outage / drop detector:
{
"analysis_config": {
"bucket_span": "15m",
"detectors": [{ "function": "low_count" }]
},
"data_description": { "time_field": "@timestamp" }
}
Create the job. Call PUT /_ml/anomaly_detectors/{job_id} with the job id the user requested (or a descriptive
id you propose). The job starts in closed state — creating it does not start analysis.
Create the datafeed. Call PUT /_ml/datafeeds/datafeed-{job_id} immediately after job creation. Set job_id to
the same id, indices to the target index (exact name or pattern from step 1), and a query that selects the relevant
documents (typically match_all). The datafeed id convention is datafeed-{job_id}.
{
"job_id": "{job_id}",
"indices": ["{index}"],
"query": { "match_all": {} }
}
Open the job, then start the datafeed — in that order. This sequence is mandatory; do not skip or reorder:
POST /_ml/anomaly_detectors/{job_id}/_open — transitions the job to opened.
POST /_ml/datafeeds/datafeed-{job_id}/_start — transitions the datafeed to started.
Opening before the datafeed exists fails. Starting the datafeed before opening the job fails. Do not report success
after only creating resources — the job is not running until both are active.
Confirm running state from stats. Verify the outcome with:
GET /_ml/anomaly_detectors/{job_id}/_stats — expect state: "opened".
GET /_ml/datafeeds/datafeed-{job_id}/_stats — expect state: "started".
Optionally call GET /_ml/anomaly_detectors/{job_id} to confirm configuration (detectors, bucket_span,
time_field, datafeed indices). Report both stats states explicitly — "created" is not the same as "opened" and
"started".
Teardown
When stopping or deleting a job, reverse the startup order:
POST /_ml/datafeeds/datafeed-{job_id}/_stop — stop the datafeed first.
POST /_ml/anomaly_detectors/{job_id}/_close — then close the job.
Stop the datafeed before closing the job. Close the job before resetting or deleting it.
Guidelines
- Required lifecycle order (create): job → datafeed → open job → start datafeed. Every new job follows this
sequence.
- Detector direction is the highest-impact decision for volume anomalies. Re-read the user's wording: "spike",
"surge", and "unusual increase" → high direction; "drop", "outage", "stops", "absence" → low direction.
- Immutable fields (
bucket_span, detectors, time_field) require delete-and-recreate if wrong — validate mapping
and intent before the first PUT.
- Datafeed index must match the user's target. Point
indices at the exact index or pattern they named — not a
nearby guess.
- Entity-level analysis (
by_field_name, over_field_name, partition_field_name) and advanced tuning live in
references/anomaly-detection-reference.md.
Full Reference
For API paths, request/response fields, score semantics, and field interactions, read
references/anomaly-detection-reference.md.
Operations
| HTTP API (shorthand) |
elastic CLI command |
GET /_cat/indices |
elastic es cat indices --index '<pattern>' |
GET /{index}/_mapping |
elastic es indices get-mapping --index '<index>' |
PUT /_ml/anomaly_detectors/{job_id} |
elastic es ml put-job --job-id '<job_id>' --analysis-config '<json>' --data-description '<json>' |
PUT /_ml/datafeeds/datafeed-{job_id} |
elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --job-id '<job_id>' --indices '<index>' --query '<json>' |
POST /_ml/anomaly_detectors/{job_id}/_open |
elastic es ml open-job --job-id '<job_id>' |
POST /_ml/datafeeds/datafeed-{job_id}/_start |
elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>' |
GET /_ml/anomaly_detectors/{job_id} |
elastic es ml get-jobs --job-id '<job_id>' |
GET /_ml/anomaly_detectors/{job_id}/_stats |
elastic es ml get-job-stats --job-id '<job_id>' |
GET /_ml/datafeeds/datafeed-{job_id}/_stats |
elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>' |
POST /_ml/datafeeds/datafeed-{job_id}/_stop |
elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>' |
POST /_ml/anomaly_detectors/{job_id}/_close |
elastic es ml close-job --job-id '<job_id>' |
1---
2name: elasticsearch-anomaly-detection
3description: >
4 Create and manage Elastic ML anomaly detection jobs via the API. Use when setting
5 up jobs on an index or data stream, configuring jobs and datafeeds, or opening,
6 starting, or stopping them.
7compatibility: >
8 Requires Elasticsearch 8.x+ or Elastic Cloud Serverless with ML anomaly detection.
9 Uses the ML REST API only. User needs manage_ml privilege to create and manage jobs.
10metadata:
11 source: https://skillmd.com/skills/gabrielmoreira/elasticsearch-anomaly-detection
12 author: elastic
13 version: 1.1.0
14 universal: true
15---
16
17# Elasticsearch Anomaly Detection
18
19Create, open, and start ML anomaly detection jobs on time-series data. Choose the right count-family detector direction,
20configure bucket span and time field, wire the datafeed to the correct index, and confirm running state from stats — not
21from assumptions.
22
23<!-- begin-partial: preamble -->
24
25## Environment Configuration
26
27This skill executes Elasticsearch operations through the `elastic` CLI. If the
28[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
29not guess credentials, call the HTTP API directly, or attempt other workarounds.
30
31This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
32`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
33maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
34directly.
35
36<!-- end-partial: preamble -->
37
38> **Prerequisite:** ML anomaly detection requires a Platinum-equivalent license on self-managed clusters. Serverless
39> projects include ML. The caller needs `manage_ml` to create and manage jobs.
40>
41> **Related skill:** For interpreting anomaly scores, influencers, and model behavior after a job is running, use
42> `elasticsearch-anomaly-detection-explainer` — not this skill.
43
44## Process
45
461. **Discover the target index and time field.** List candidate indices with `GET /_cat/indices` (pass a pattern when
47 the user names one). Fetch field types for the chosen index with `GET /{index}/_mapping`. The decision: confirm the
48 index exists, identify the time field (often `@timestamp`), and verify document volume is sufficient for baseline
49 learning. Never guess index or field names — they vary across deployments.
50
512. **Choose detector function and direction.** Match the user's intent to a count-family detector in
52 `analysis_config.detectors`:
53 - **Spike, surge, unusual increase in event volume** → `high_count` (or `count`, which flags both directions but is
54 acceptable when the user cares about spikes). Do **not** use `low_count` — it will miss spikes.
55 - **Drop, outage, absence of events, traffic stops** → `low_count`. Do **not** use `high_count` — it will miss drops
56 and silence.
57 - **Metric deviation** (CPU, latency, a numeric field) → mean-family functions (`mean`, `high_mean`, `low_mean`) with
58 `field_name` set — only when the user asks about a numeric metric, not raw event volume.
59
60 The decision: pick one primary detector whose direction matches the anomaly type. For volume spike/drop questions on
61 document counts, stay in the count family — mean detectors are unsuited to "how many events" questions.
62
633. **Set immutable job shape before creation.** These fields cannot change after `PUT /_ml/anomaly_detectors/{job_id}`:
64 - `analysis_config.bucket_span` — use the interval the user specifies (e.g. `15m` for 15-minute buckets). Match the
65 granularity of anomalies they care about; too short is noisy, too long is slow to detect.
66 - `data_description.time_field` — the time field from the mapping (commonly `@timestamp`).
67 - `analysis_config.detectors` — the function and direction from step 2.
68
69 Example job body for a volume-spike detector:
70
71 ```json
72 {
73 "analysis_config": {
74 "bucket_span": "15m",
75 "detectors": [{ "function": "high_count" }]
76 },
77 "data_description": { "time_field": "@timestamp" }
78 }
79 ```
80
81 Example for an outage / drop detector:
82
83 ```json
84 {
85 "analysis_config": {
86 "bucket_span": "15m",
87 "detectors": [{ "function": "low_count" }]
88 },
89 "data_description": { "time_field": "@timestamp" }
90 }
91 ```
92
934. **Create the job.** Call `PUT /_ml/anomaly_detectors/{job_id}` with the job id the user requested (or a descriptive
94 id you propose). The job starts in `closed` state — creating it does not start analysis.
95
965. **Create the datafeed.** Call `PUT /_ml/datafeeds/datafeed-{job_id}` immediately after job creation. Set `job_id` to
97 the same id, `indices` to the target index (exact name or pattern from step 1), and a query that selects the relevant
98 documents (typically `match_all`). The datafeed id convention is `datafeed-{job_id}`.
99
100 ```json
101 {
102 "job_id": "{job_id}",
103 "indices": ["{index}"],
104 "query": { "match_all": {} }
105 }
106 ```
107
1086. **Open the job, then start the datafeed — in that order.** This sequence is mandatory; do not skip or reorder:
109 1. `POST /_ml/anomaly_detectors/{job_id}/_open` — transitions the job to `opened`.
110 2. `POST /_ml/datafeeds/datafeed-{job_id}/_start` — transitions the datafeed to `started`.
111
112 Opening before the datafeed exists fails. Starting the datafeed before opening the job fails. Do not report success
113 after only creating resources — the job is not running until both are active.
114
1157. **Confirm running state from stats.** Verify the outcome with:
116 - `GET /_ml/anomaly_detectors/{job_id}/_stats` — expect `state: "opened"`.
117 - `GET /_ml/datafeeds/datafeed-{job_id}/_stats` — expect `state: "started"`.
118
119 Optionally call `GET /_ml/anomaly_detectors/{job_id}` to confirm configuration (detectors, `bucket_span`,
120 `time_field`, datafeed indices). Report both stats states explicitly — "created" is not the same as "opened" and
121 "started".
122
123## Teardown
124
125When stopping or deleting a job, reverse the startup order:
126
1271. `POST /_ml/datafeeds/datafeed-{job_id}/_stop` — stop the datafeed first.
1282. `POST /_ml/anomaly_detectors/{job_id}/_close` — then close the job.
129
130Stop the datafeed before closing the job. Close the job before resetting or deleting it.
131
132## Guidelines
133
134- **Required lifecycle order (create):** job → datafeed → open job → start datafeed. Every new job follows this
135 sequence.
136- **Detector direction is the highest-impact decision** for volume anomalies. Re-read the user's wording: "spike",
137 "surge", and "unusual increase" → high direction; "drop", "outage", "stops", "absence" → low direction.
138- **Immutable fields** (`bucket_span`, detectors, `time_field`) require delete-and-recreate if wrong — validate mapping
139 and intent before the first `PUT`.
140- **Datafeed index must match the user's target.** Point `indices` at the exact index or pattern they named — not a
141 nearby guess.
142- **Entity-level analysis** (`by_field_name`, `over_field_name`, `partition_field_name`) and advanced tuning live in
143 [references/anomaly-detection-reference.md](references/anomaly-detection-reference.md).
144
145## Full Reference
146
147For API paths, request/response fields, score semantics, and field interactions, read
148[references/anomaly-detection-reference.md](references/anomaly-detection-reference.md).
149
150## Operations
151
152| HTTP API (shorthand) | `elastic` CLI command |
153| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
154| `GET /_cat/indices` | `elastic es cat indices --index '<pattern>'` |
155| `GET /{index}/_mapping` | `elastic es indices get-mapping --index '<index>'` |
156| `PUT /_ml/anomaly_detectors/{job_id}` | `elastic es ml put-job --job-id '<job_id>' --analysis-config '<json>' --data-description '<json>'` |
157| `PUT /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --job-id '<job_id>' --indices '<index>' --query '<json>'` |
158| `POST /_ml/anomaly_detectors/{job_id}/_open` | `elastic es ml open-job --job-id '<job_id>'` |
159| `POST /_ml/datafeeds/datafeed-{job_id}/_start` | `elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>'` |
160| `GET /_ml/anomaly_detectors/{job_id}` | `elastic es ml get-jobs --job-id '<job_id>'` |
161| `GET /_ml/anomaly_detectors/{job_id}/_stats` | `elastic es ml get-job-stats --job-id '<job_id>'` |
162| `GET /_ml/datafeeds/datafeed-{job_id}/_stats` | `elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>'` |
163| `POST /_ml/datafeeds/datafeed-{job_id}/_stop` | `elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>'` |
164| `POST /_ml/anomaly_detectors/{job_id}/_close` | `elastic es ml close-job --job-id '<job_id>'` |