Elastic ML Anomaly Detection
Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose
job lifecycle failures, and manage jobs. Read anomaly results from POST /.ml-anomalies-*/_search (Serverless-safe)
and job/datafeed state from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the
prompt, apply the judgment below directly — do not re-fetch fields already supplied.
Environment Configuration
This skill executes Elasticsearch operations through the elastic CLI. If the
elastic CLI is not installed, tell the user what it is needed for. Do
not guess credentials, call the HTTP API directly, or attempt other workarounds.
This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping,
GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document
maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API
directly.
Mode selector
| User intent |
Mode |
| "What broke?" / RCA / cross-job / blast radius / influencers / log categories |
Investigate |
| "Why score high/low?" / renormalization / model bounds / forecasts |
Explain |
| Missing docs / memory limit / datafeed stopped / lifecycle / calendars |
Troubleshoot |
| Create a job / configure a datafeed / start analysis / retrieve results |
Manage |
| Security framing (attack chains, MITRE, exfil) |
Investigate + references/security-anomaly-expert.md |
| Observability/SRE framing (degradation, capacity, deployment regression) |
Investigate + references/observability-anomaly-expert.md |
When a question spans modes: Investigate → Explain → Troubleshoot. Finish one mode before blending logic.
Serverless note: Legacy /_ml/anomaly_detectors/{job_id}/results/* endpoints return HTTP 410 in Serverless.
Always query .ml-anomalies-* via POST /.ml-anomalies-*/_search with result_type filters.
Score quick reference
record_score bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational
multi_bucket_impact ≥ 3 → sustained shift (not a transient spike)
initial_record_score >> record_score → renormalization (model saw worse anomalies later)
actual << typical with count/low_count/low_mean → absence/outage, not just a low value
- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity
Full score definitions, renormalization mechanics, and anomaly_score_explanation components:
references/score-reference.md.
Core concepts
Treat .ml-anomalies-* as layered result types via result_type in search queries:
result_type |
Scope |
Key fields |
bucket |
Time window |
anomaly_score, initial_anomaly_score, timestamp |
record |
Detector row |
record_score, initial_record_score, actual, typical, anomaly_score_explanation |
influencer |
Entity × bucket |
influencer_field_name, influencer_field_value, influencer_score |
model_plot |
Bounds |
model_lower, model_upper, actual |
category_definition |
Log patterns |
category_id, terms, regex, examples |
Read scores this way:
anomaly_score / record_score = current normalized values (move as the model sees new extremes).
initial_anomaly_score / initial_record_score = immutable snapshots from detection time.
influencer_score ranks entity responsibility within a bucket — the highest score is the primary suspect, not the
bucket-level anomaly_score alone.
- Map entities via
partition_field_value / by_field_value / over_field_value.
- Read
multi_bucket_impact (-5 to +5) to separate single-bucket spikes from sustained trends.
Mode: Investigate — RCA
When: "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.
Process
Discover jobs. Call GET /_ml/anomaly_detectors when the job ID is unknown. Call
GET /_ml/anomaly_detectors/{job_id} and GET /_ml/datafeeds/datafeed-{job_id} to learn source indices, entity
fields (by_field_name, over_field_name, partition_field_name), and bucket_span. The decision: identify the
related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.
Scope the incident window. Call POST /.ml-anomalies-*/_search with result_type: bucket, a time range, and
optional minimum anomaly_score. The decision: fix the incident start/end and count how many jobs co-fire in that
window. Low scores across many jobs simultaneously often indicate a systemic root cause.
Attribute to entities (critical for RCA). For the anomalous bucket timestamp, call
POST /.ml-anomalies-*/_search with result_type: influencer, the job ID(s), and the bucket time range. Sort by
influencer_score descending. The decision: name the entity with the highest influencer_score as the
likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket anomaly_score
without attributing responsibility. Recommend drilling into that entity's records next.
Cross-job confirmation. Re-query influencers (or bucket records) across related job IDs for the same entity
values and time window. Entities anomalous in 2+ jobs are prime suspects (resource fault or systemic failure);
single-job entities are often downstream victims. See
references/protocols/investigation.md.
Drill into records. Call POST /.ml-anomalies-*/_search with result_type: record, exact job ID, entity filters
(partition_field_value, by_field_value), and low minimum record_score (25 or lower). Read
multi_bucket_impact ≥ 3 as sustained behavioral shift. Read actual vs typical for fault class (spike vs
absence/outage).
Confirm with source evidence. Call POST /{index}/_search on the datafeed source index for the suspect entity
and time window. Raw source documents are ground truth — never close an RCA without them.
Synthesize. Report: root cause entity · affected jobs · temporal progression · fault class · severity ·
recommended actions. Worked walkthrough: references/worked-example.md. Query
templates: references/investigation-queries.md.
Rules
- Rank by
influencer_score, not anomaly_score, for "which entity?" — bucket score is aggregate; influencer
score attributes cause.
- Multi-job entities are prime suspects; single-job entities are usually victims.
- Earliest anomaly timestamp wins — reconstruct chronology from record timestamps across jobs.
multi_bucket_impact ≥ 3 = sustained behavioral shift, weight higher than transient spikes.
- Use low score thresholds (25 or lower) for influencer/record queries — high thresholds miss correlated entities.
- Never close an RCA without source evidence from the datafeed index.
Mode: Explain — Score / model behavior
When: "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".
Process
Decide fetch vs interpret. If the user supplies a record with record_score, initial_record_score, actual,
and typical, interpret directly. Otherwise load config with GET /_ml/anomaly_detectors/{job_id} and records with
POST /.ml-anomalies-*/_search (result_type: record).
Always show both initial_record_score and record_score. The gap is the renormalization story. Large positive
drift (initial_record_score >> record_score) means a later, more extreme anomaly rescale this record downward —
expected healthy behavior, not a broken model.
Classify the pattern before speculating.
| Pattern |
Interpretation |
initial_record_score >> record_score |
Renormalization — explain before suggesting config changes |
actual << typical with low_count/count/low_mean |
Absence/outage anomaly — investigate the outage, not score tuning |
high_variance_penalty: true in anomaly_score_explanation |
Noisy metric — wide bounds absorbed the spike |
incomplete_bucket_penalty: true |
Ingest lag or sparse bucket — score legitimately reduced |
Only cite anomaly_score_explanation factors present in the record.
Quantify renormalization (optional). Re-query records sorted by timestamp; compute
score_drift = initial_record_score - record_score and flag large drift.
Add visual context when needed. If model_plot_config.enabled, query result_type: model_plot and compare
actual to model_lower/model_upper. For categorization jobs, query result_type: category_definition.
Check job health when scores look wrong persistently. Call GET /_ml/anomaly_detectors/{job_id}/_stats —
model_size_stats.memory_status of hard_limit corrupts learning and can invalidate scores. Escalate to
Troubleshoot mode.
anomaly_score_explanation components
| Component |
Effect |
What it means |
anomaly_length |
↑ score |
More consecutive anomalous buckets |
single_bucket_impact |
↑ score |
Lower probability → higher impact |
multi_bucket_impact |
↑ score |
Sustained pattern contribution |
anomaly_characteristics_impact |
↑ score |
Mean shift vs. variance change |
high_variance_penalty |
↓ score |
Noisy data → wide bounds → anomaly less surprising |
incomplete_bucket_penalty |
↓ score |
Bucket has less data than expected (ingest lag, sparse data) |
Rules
- Explain renormalization before diagnosing config — score drift is the most common "score dropped" cause.
actual << typical with count/low_count is an absence anomaly — distinguish outages from value spikes.
- Weekly seasonality needs ≥3 weeks of training data — flag young jobs as the cause.
- Detector function direction matters — see
references/anomaly-detection-functions.md.
Mode: Troubleshoot — Job lifecycle
When: "missing documents", "datafeed stopped", hard_limit, "results look wrong", lifecycle changes.
Process
Load job and datafeed state. Call GET /_ml/anomaly_detectors/{job_id}/_stats and
GET /_ml/datafeeds/datafeed-{job_id}/_stats. Read state, data_counts, model_size_stats, and datafeed
state. If the user embeds stats JSON, diagnose from memory_status and datafeed state directly.
Diagnose memory status first (critical). Inspect model_size_stats:
| Field |
Meaning |
memory_status |
ok / soft_limit (pruning) / hard_limit (critical) |
model_bytes |
Current memory used |
model_bytes_memory_limit |
Configured model_memory_limit |
When memory_status is hard_limit and model_bytes equals model_bytes_memory_limit, the model hit its
memory ceiling — it stops learning new entities and results degrade or stop. A stopped datafeed is often a
symptom, not the root cause. Do not recommend only restarting the datafeed — that alone does not clear a hard
limit.
Remediate hard_limit. The fix is to raise model_memory_limit (via job update) and/or reduce model size
by lowering cardinality (fewer partition/by/over field values, split into multiple jobs). Raising the limit requires
the lifecycle sequence below (stop datafeed → close job → update → open → start). Optionally call
POST /_ml/anomaly_detectors/_estimate_model_memory to size the new limit from source cardinality.
Diagnose missing documents / query timing. After memory is healthy, inspect datafeed query_delay and
delayed_data_check_config via GET /_ml/datafeeds/datafeed-{job_id}. Search .ml-annotations-* for delayed-data
events. Set query_delay to P95 ingest latency + buffer (default 60s–120s).
Read job messages. Search .ml-notifications-* for the job ID when errors are unclear.
Recover corrupted model state. Call POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert
to revert to a known-good snapshot when the model was corrupted during hard_limit.
Lifecycle for config changes (memory limit, query_delay)
Apply in order — skipping steps causes rejected updates:
POST /_ml/datafeeds/datafeed-{job_id}/_stop
POST /_ml/anomaly_detectors/{job_id}/_close
POST /_ml/anomaly_detectors/{job_id}/_update (memory limit) and/or POST /_ml/datafeeds/datafeed-{job_id}/_update
(query_delay)
POST /_ml/anomaly_detectors/{job_id}/_open
POST /_ml/datafeeds/datafeed-{job_id}/_start
Preview changes with POST /_ml/datafeeds/datafeed-{job_id}/_preview before restarting.
hard_limit corrupts model state and causes downstream missing-doc false alarms. Fix memory before fixing
query_delay. Full troubleshooting detail:
references/troubleshooting-reference.md.
Rules
- Ground lifecycle diagnosis in
memory_status — not generic "restart it" advice.
- Fix memory before
query_delay — hard_limit invalidates downstream diagnostics.
- Stop datafeed → close job → update → open → start for any memory or datafeed config change.
- Do not delete the job as first remediation for hard_limit — raise limit and/or reduce cardinality.
Mode: Manage — Create / configure jobs
When: "set up a job", "create an ML detector", "monitor X over time".
For the full create/open/start lifecycle, prefer the elasticsearch-anomaly-detection skill. This mode summarizes the
sequence and detector selection:
- Verify target index. Call
GET /{index}/_mapping — confirm time field and detector fields exist.
- Create job. Call
PUT /_ml/anomaly_detectors/{job_id} with analysis_config (detectors, bucket_span,
influencers) and data_description.time_field.
- Create datafeed. Call
PUT /_ml/datafeeds/datafeed-{job_id} with indices, query, and query_delay.
- Open and start. Call
POST /_ml/anomaly_detectors/{job_id}/_open, then
POST /_ml/datafeeds/datafeed-{job_id}/_start.
- Confirm. Call
GET /_ml/anomaly_detectors/{job_id}/_stats and GET /_ml/datafeeds/datafeed-{job_id}/_stats.
Choose detector functions from user intent — see
references/anomaly-detection-functions.md. Worked JSON bodies:
references/job-creation-recipes.md.
Rules
- Create job before datafeed. Open job before starting datafeed.
query_delay = P95 ingest latency + buffer (60s–120s safe default).
by_field_name vs over_field_name: by compares entity to its own history; over compares to peer group.
- Forecasts require non-population jobs — jobs with
over_field_name cannot be forecasted.
Examples
RCA: "Something caused a spike in checkout latency — which entity?" → Query influencers for the bucket → web-07
has highest influencer_score (91.5) vs 22.0 and 8.4 → name web-07 as likely cause → recommend drilling into its
records — do not answer with only bucket anomaly_score 88.
Score drop: "Score went from 90 to 55 — did the model change?" → Compare initial_record_score vs record_score →
explain renormalization if drift is large.
Memory limit: "Job shows hard_limit and datafeed stopped." → Diagnose
model_size_stats.memory_status = hard_limit → raise model_memory_limit via close/update/open lifecycle and/or reduce
cardinality — not "just restart the datafeed".
New job: "Detect unusual error rates per host." → high_count with by_field_name: host.keyword →
create/open/start sequence.
Guidelines
- Pick a mode first. Don't blend RCA logic with score-explanation logic in one response.
- For "which entity?" rank
influencer_score, not bucket anomaly_score.
- For lifecycle failures read
memory_status before recommending datafeed restarts.
- Show
initial_record_score alongside record_score — the gap tells the renormalization story.
- Fix memory before
query_delay. Hard_limit invalidates downstream diagnostics.
- Confirm RCAs with source evidence from the datafeed index.
Operations
| HTTP API (shorthand) |
elastic CLI command |
GET /{index}/_mapping |
elastic es indices get-mapping --index '<index>' |
POST /{index}/_search |
elastic es search --index '<index>' --input-file '<search-body.json>' |
GET /_ml/anomaly_detectors |
elastic es ml get-jobs |
GET /_ml/anomaly_detectors/{job_id} |
elastic es ml get-jobs --job-id '<job_id>' |
GET /_ml/anomaly_detectors/{job_id}/_stats |
elastic es ml get-job-stats --job-id '<job_id>' |
GET /_ml/datafeeds/datafeed-{job_id} |
elastic es ml get-datafeeds --datafeed-id 'datafeed-<job_id>' |
GET /_ml/datafeeds/datafeed-{job_id}/_stats |
elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>' |
POST /.ml-anomalies-*/_search |
elastic es search --index '.ml-anomalies-*' --input-file '<search-body.json>' |
POST /.ml-annotations-*/_search |
elastic es search --index '.ml-annotations-*' --input-file '<search-body.json>' |
POST /.ml-notifications-*/_search |
elastic es search --index '.ml-notifications-*' --input-file '<search-body.json>' |
POST /_ml/anomaly_detectors/_estimate_model_memory |
elastic es ml estimate-model-memory --analysis-config '<json>' |
PUT /_ml/anomaly_detectors/{job_id} |
elastic es ml put-job --job-id '<job_id>' --input-file '<job-body.json>' |
PUT /_ml/datafeeds/datafeed-{job_id} |
elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<datafeed-body.json>' |
POST /_ml/anomaly_detectors/{job_id}/_open |
elastic es ml open-job --job-id '<job_id>' |
POST /_ml/anomaly_detectors/{job_id}/_close |
elastic es ml close-job --job-id '<job_id>' |
POST /_ml/anomaly_detectors/{job_id}/_update |
elastic es ml update-job --job-id '<job_id>' --analysis-limits '<json>' |
POST /_ml/datafeeds/datafeed-{job_id}/_update |
elastic es ml update-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<update-body.json>' |
POST /_ml/datafeeds/datafeed-{job_id}/_start |
elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>' |
POST /_ml/datafeeds/datafeed-{job_id}/_stop |
elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>' |
POST /_ml/datafeeds/datafeed-{job_id}/_preview |
elastic es ml preview-datafeed --datafeed-id 'datafeed-<job_id>' |
POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert |
elastic es ml revert-model-snapshot --job-id '<job_id>' --snapshot-id '<snapshot_id>' |
Search body shapes for each result_type and troubleshooting queries are documented in
references/investigation-queries.md and
references/troubleshooting-reference.md.
1---
2name: kibana-anomaly-detection
3description: >
4 Elastic ML anomaly detection — investigation/RCA, score explanation, job lifecycle
5 troubleshooting, and job operations. Use when answering "what broke?"/"which entity?"/RCA,
6 "why is score high/low?"/renormalization, "datafeed stopped"/"memory limit"/hard_limit,
7 or configuring ML anomaly detection jobs. Reads results from `.ml-anomalies-*` and
8 job state from ML REST APIs.
9metadata:
10 source: https://skillmd.com/skills/gabrielmoreira/kibana-anomaly-detection
11 author: elastic
12 version: 0.3.0
13 universal: true
14compatibility: Elasticsearch 8.x–9.x or Elastic Cloud Serverless with ML anomaly detection;
15 Kibana 8.x–9.x for saved-object context only
16---
17
18# Elastic ML Anomaly Detection
19
20Expert process for ML anomaly detection: attribute incidents to entities, explain scores and model behavior, diagnose
21job lifecycle failures, and manage jobs. Read anomaly **results** from `POST /.ml-anomalies-*/_search` (Serverless-safe)
22and **job/datafeed state** from ML REST APIs. When the user embeds fixture evidence (influencer rows, job stats) in the
23prompt, apply the judgment below directly — do not re-fetch fields already supplied.
24
25<!-- begin-partial: preamble -->
26
27## Environment Configuration
28
29This skill executes Elasticsearch operations through the `elastic` CLI. If the
30[`elastic` CLI](https://github.com/elastic/cli#configuration) is not installed, tell the user what it is needed for. Do
31not guess credentials, call the HTTP API directly, or attempt other workarounds.
32
33This skill references operations in HTTP-shorthand form (e.g., `GET /`, `GET /_cat/indices`, `GET /{index}/_mapping`,
34`GET /{index}/_settings/index.mode`, `POST /_query`). The [Operations](#operations) table at the end of this document
35maps each shorthand to the equivalent `elastic` CLI command — always use the CLI rather than calling the HTTP API
36directly.
37
38<!-- end-partial: preamble -->
39
40## Mode selector
41
42| User intent | Mode |
43| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
44| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate** |
45| "Why score high/low?" / renormalization / model bounds / forecasts | **Explain** |
46| Missing docs / memory limit / datafeed stopped / lifecycle / calendars | **Troubleshoot** |
47| Create a job / configure a datafeed / start analysis / retrieve results | **Manage** |
48| Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) |
49| Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |
50
51When a question spans modes: **Investigate → Explain → Troubleshoot**. Finish one mode before blending logic.
52
53> **Serverless note:** Legacy `/_ml/anomaly_detectors/{job_id}/results/*` endpoints return HTTP 410 in Serverless.
54> Always query `.ml-anomalies-*` via `POST /.ml-anomalies-*/_search` with `result_type` filters.
55
56## Score quick reference
57
58- `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational
59- `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike)
60- `initial_record_score >> record_score` → renormalization (model saw worse anomalies later)
61- `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just a low value
62- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity
63
64> Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components:
65> [references/score-reference.md](references/score-reference.md).
66
67## Core concepts
68
69Treat `.ml-anomalies-*` as layered result types via `result_type` in search queries:
70
71| `result_type` | Scope | Key fields |
72| --------------------- | --------------- | ---------------------------------------------------------------------------------------- |
73| `bucket` | Time window | `anomaly_score`, `initial_anomaly_score`, `timestamp` |
74| `record` | Detector row | `record_score`, `initial_record_score`, `actual`, `typical`, `anomaly_score_explanation` |
75| `influencer` | Entity × bucket | `influencer_field_name`, `influencer_field_value`, **`influencer_score`** |
76| `model_plot` | Bounds | `model_lower`, `model_upper`, `actual` |
77| `category_definition` | Log patterns | `category_id`, `terms`, `regex`, `examples` |
78
79Read scores this way:
80
81- `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes).
82- `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time.
83- **`influencer_score` ranks entity responsibility within a bucket** — the highest score is the primary suspect, not the
84 bucket-level `anomaly_score` alone.
85- Map entities via `partition_field_value` / `by_field_value` / `over_field_value`.
86- Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends.
87
88---
89
90## Mode: Investigate — RCA
91
92**When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.
93
94### Process
95
961. **Discover jobs.** Call `GET /_ml/anomaly_detectors` when the job ID is unknown. Call
97 `GET /_ml/anomaly_detectors/{job_id}` and `GET /_ml/datafeeds/datafeed-{job_id}` to learn source indices, entity
98 fields (`by_field_name`, `over_field_name`, `partition_field_name`), and `bucket_span`. The decision: identify the
99 related job group — jobs sharing a datafeed index or entity field monitor the same system from different angles.
100
1012. **Scope the incident window.** Call `POST /.ml-anomalies-*/_search` with `result_type: bucket`, a time range, and
102 optional minimum `anomaly_score`. The decision: fix the incident start/end and count how many jobs co-fire in that
103 window. Low scores across many jobs simultaneously often indicate a systemic root cause.
104
1053. **Attribute to entities (critical for RCA).** For the anomalous bucket timestamp, call
106 `POST /.ml-anomalies-*/_search` with `result_type: influencer`, the job ID(s), and the bucket time range. Sort by
107 **`influencer_score` descending**. The decision: name the entity with the **highest `influencer_score`** as the
108 likely cause — it ranks how unusual each entity is in that bucket. Do not restate only the bucket `anomaly_score`
109 without attributing responsibility. Recommend drilling into that entity's records next.
110
1114. **Cross-job confirmation.** Re-query influencers (or bucket records) across related job IDs for the same entity
112 values and time window. Entities anomalous in **2+ jobs** are prime suspects (resource fault or systemic failure);
113 single-job entities are often downstream victims. See
114 [references/protocols/investigation.md](references/protocols/investigation.md).
115
1165. **Drill into records.** Call `POST /.ml-anomalies-*/_search` with `result_type: record`, exact job ID, entity filters
117 (`partition_field_value`, `by_field_value`), and low minimum `record_score` (25 or lower). Read
118 `multi_bucket_impact ≥ 3` as sustained behavioral shift. Read `actual` vs `typical` for fault class (spike vs
119 absence/outage).
120
1216. **Confirm with source evidence.** Call `POST /{index}/_search` on the datafeed source index for the suspect entity
122 and time window. Raw source documents are ground truth — never close an RCA without them.
123
1247. **Synthesize.** Report: **root cause entity · affected jobs · temporal progression · fault class · severity ·
125 recommended actions**. Worked walkthrough: [references/worked-example.md](references/worked-example.md). Query
126 templates: [references/investigation-queries.md](references/investigation-queries.md).
127
128### Rules
129
1301. **Rank by `influencer_score`, not `anomaly_score`, for "which entity?"** — bucket score is aggregate; influencer
131 score attributes cause.
1322. **Multi-job entities are prime suspects; single-job entities are usually victims.**
1333. **Earliest anomaly timestamp wins** — reconstruct chronology from record timestamps across jobs.
1344. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes.
1355. **Use low score thresholds (25 or lower) for influencer/record queries** — high thresholds miss correlated entities.
1366. **Never close an RCA without source evidence** from the datafeed index.
137
138---
139
140## Mode: Explain — Score / model behavior
141
142**When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".
143
144### Process
145
1461. **Decide fetch vs interpret.** If the user supplies a record with `record_score`, `initial_record_score`, `actual`,
147 and `typical`, interpret directly. Otherwise load config with `GET /_ml/anomaly_detectors/{job_id}` and records with
148 `POST /.ml-anomalies-*/_search` (`result_type: record`).
149
1502. **Always show both `initial_record_score` and `record_score`.** The gap is the renormalization story. Large positive
151 drift (`initial_record_score >> record_score`) means a later, more extreme anomaly rescale this record downward —
152 expected healthy behavior, not a broken model.
153
1543. **Classify the pattern before speculating.**
155
156 | Pattern | Interpretation |
157 | ------------------------------------------------------------ | ----------------------------------------------------------------- |
158 | `initial_record_score >> record_score` | Renormalization — explain before suggesting config changes |
159 | `actual << typical` with `low_count`/`count`/`low_mean` | Absence/outage anomaly — investigate the outage, not score tuning |
160 | `high_variance_penalty: true` in `anomaly_score_explanation` | Noisy metric — wide bounds absorbed the spike |
161 | `incomplete_bucket_penalty: true` | Ingest lag or sparse bucket — score legitimately reduced |
162
163 Only cite `anomaly_score_explanation` factors **present** in the record.
164
1654. **Quantify renormalization (optional).** Re-query records sorted by `timestamp`; compute
166 `score_drift = initial_record_score - record_score` and flag large drift.
167
1685. **Add visual context when needed.** If `model_plot_config.enabled`, query `result_type: model_plot` and compare
169 `actual` to `model_lower`/`model_upper`. For categorization jobs, query `result_type: category_definition`.
170
1716. **Check job health when scores look wrong persistently.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` —
172 `model_size_stats.memory_status` of `hard_limit` corrupts learning and can invalidate scores. Escalate to
173 Troubleshoot mode.
174
175### `anomaly_score_explanation` components
176
177| Component | Effect | What it means |
178| -------------------------------- | ------- | ------------------------------------------------------------ |
179| `anomaly_length` | ↑ score | More consecutive anomalous buckets |
180| `single_bucket_impact` | ↑ score | Lower probability → higher impact |
181| `multi_bucket_impact` | ↑ score | Sustained pattern contribution |
182| `anomaly_characteristics_impact` | ↑ score | Mean shift vs. variance change |
183| `high_variance_penalty` | ↓ score | Noisy data → wide bounds → anomaly less surprising |
184| `incomplete_bucket_penalty` | ↓ score | Bucket has less data than expected (ingest lag, sparse data) |
185
186### Rules
187
1881. **Explain renormalization before diagnosing config** — score drift is the most common "score dropped" cause.
1892. **`actual << typical` with count/low_count is an absence anomaly** — distinguish outages from value spikes.
1903. **Weekly seasonality needs ≥3 weeks of training data** — flag young jobs as the cause.
1914. **Detector function direction matters** — see
192 [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md).
193
194---
195
196## Mode: Troubleshoot — Job lifecycle
197
198**When:** "missing documents", "datafeed stopped", **`hard_limit`**, "results look wrong", lifecycle changes.
199
200### Process
201
2021. **Load job and datafeed state.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and
203 `GET /_ml/datafeeds/datafeed-{job_id}/_stats`. Read `state`, `data_counts`, **`model_size_stats`**, and datafeed
204 `state`. If the user embeds stats JSON, diagnose from `memory_status` and datafeed state directly.
205
2062. **Diagnose memory status first (critical).** Inspect `model_size_stats`:
207
208 | Field | Meaning |
209 | -------------------------- | ----------------------------------------------------------- |
210 | `memory_status` | `ok` / `soft_limit` (pruning) / **`hard_limit` (critical)** |
211 | `model_bytes` | Current memory used |
212 | `model_bytes_memory_limit` | Configured `model_memory_limit` |
213
214 When **`memory_status` is `hard_limit`** and `model_bytes` equals `model_bytes_memory_limit`, the model hit its
215 memory ceiling — it stops learning new entities and results degrade or stop. A stopped datafeed is often a
216 **symptom**, not the root cause. **Do not recommend only restarting the datafeed** — that alone does not clear a hard
217 limit.
218
2193. **Remediate hard_limit.** The fix is to **raise `model_memory_limit`** (via job update) **and/or reduce model size**
220 by lowering cardinality (fewer partition/by/over field values, split into multiple jobs). Raising the limit requires
221 the lifecycle sequence below (stop datafeed → close job → update → open → start). Optionally call
222 `POST /_ml/anomaly_detectors/_estimate_model_memory` to size the new limit from source cardinality.
223
2244. **Diagnose missing documents / query timing.** After memory is healthy, inspect datafeed `query_delay` and
225 `delayed_data_check_config` via `GET /_ml/datafeeds/datafeed-{job_id}`. Search `.ml-annotations-*` for delayed-data
226 events. Set `query_delay` to P95 ingest latency + buffer (default `60s`–`120s`).
227
2285. **Read job messages.** Search `.ml-notifications-*` for the job ID when errors are unclear.
229
2306. **Recover corrupted model state.** Call `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert`
231 to revert to a known-good snapshot when the model was corrupted during hard_limit.
232
233### Lifecycle for config changes (memory limit, query_delay)
234
235Apply in order — skipping steps causes rejected updates:
236
2371. `POST /_ml/datafeeds/datafeed-{job_id}/_stop`
2382. `POST /_ml/anomaly_detectors/{job_id}/_close`
2393. `POST /_ml/anomaly_detectors/{job_id}/_update` (memory limit) and/or `POST /_ml/datafeeds/datafeed-{job_id}/_update`
240 (query_delay)
2414. `POST /_ml/anomaly_detectors/{job_id}/_open`
2425. `POST /_ml/datafeeds/datafeed-{job_id}/_start`
243
244Preview changes with `POST /_ml/datafeeds/datafeed-{job_id}/_preview` before restarting.
245
246> **`hard_limit` corrupts model state** and causes downstream missing-doc false alarms. **Fix memory before fixing
247> `query_delay`.** Full troubleshooting detail:
248> [references/troubleshooting-reference.md](references/troubleshooting-reference.md).
249
250### Rules
251
2521. **Ground lifecycle diagnosis in `memory_status`** — not generic "restart it" advice.
2532. **Fix memory before `query_delay`** — hard_limit invalidates downstream diagnostics.
2543. **Stop datafeed → close job → update → open → start** for any memory or datafeed config change.
2554. **Do not delete the job** as first remediation for hard_limit — raise limit and/or reduce cardinality.
256
257---
258
259## Mode: Manage — Create / configure jobs
260
261**When:** "set up a job", "create an ML detector", "monitor X over time".
262
263For the full create/open/start lifecycle, prefer the `elasticsearch-anomaly-detection` skill. This mode summarizes the
264sequence and detector selection:
265
2661. **Verify target index.** Call `GET /{index}/_mapping` — confirm time field and detector fields exist.
2672. **Create job.** Call `PUT /_ml/anomaly_detectors/{job_id}` with `analysis_config` (detectors, `bucket_span`,
268 influencers) and `data_description.time_field`.
2693. **Create datafeed.** Call `PUT /_ml/datafeeds/datafeed-{job_id}` with `indices`, `query`, and `query_delay`.
2704. **Open and start.** Call `POST /_ml/anomaly_detectors/{job_id}/_open`, then
271 `POST /_ml/datafeeds/datafeed-{job_id}/_start`.
2725. **Confirm.** Call `GET /_ml/anomaly_detectors/{job_id}/_stats` and `GET /_ml/datafeeds/datafeed-{job_id}/_stats`.
273
274Choose detector functions from user intent — see
275[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md). Worked JSON bodies:
276[references/job-creation-recipes.md](references/job-creation-recipes.md).
277
278### Rules
279
2801. **Create job before datafeed.** Open job before starting datafeed.
2812. **`query_delay` = P95 ingest latency + buffer** (60s–120s safe default).
2823. **`by_field_name` vs `over_field_name`:** `by` compares entity to its own history; `over` compares to peer group.
2834. **Forecasts require non-population jobs** — jobs with `over_field_name` cannot be forecasted.
284
285---
286
287## Examples
288
289**RCA:** "Something caused a spike in checkout latency — which entity?" → Query influencers for the bucket → **web-07**
290has highest `influencer_score` (91.5) vs 22.0 and 8.4 → name web-07 as likely cause → recommend drilling into its
291records — do not answer with only bucket `anomaly_score` 88.
292
293**Score drop:** "Score went from 90 to 55 — did the model change?" → Compare `initial_record_score` vs `record_score` →
294explain renormalization if drift is large.
295
296**Memory limit:** "Job shows `hard_limit` and datafeed stopped." → Diagnose
297`model_size_stats.memory_status = hard_limit` → raise `model_memory_limit` via close/update/open lifecycle and/or reduce
298cardinality — **not** "just restart the datafeed".
299
300**New job:** "Detect unusual error rates per host." → `high_count` with `by_field_name: host.keyword` →
301create/open/start sequence.
302
303---
304
305## Guidelines
306
3071. **Pick a mode first.** Don't blend RCA logic with score-explanation logic in one response.
3082. **For "which entity?" rank `influencer_score`**, not bucket `anomaly_score`.
3093. **For lifecycle failures read `memory_status`** before recommending datafeed restarts.
3104. **Show `initial_record_score` alongside `record_score`** — the gap tells the renormalization story.
3115. **Fix memory before `query_delay`.** Hard_limit invalidates downstream diagnostics.
3126. **Confirm RCAs with source evidence** from the datafeed index.
313
314## Operations
315
316| HTTP API (shorthand) | `elastic` CLI command |
317| ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
318| `GET /{index}/_mapping` | `elastic es indices get-mapping --index '<index>'` |
319| `POST /{index}/_search` | `elastic es search --index '<index>' --input-file '<search-body.json>'` |
320| `GET /_ml/anomaly_detectors` | `elastic es ml get-jobs` |
321| `GET /_ml/anomaly_detectors/{job_id}` | `elastic es ml get-jobs --job-id '<job_id>'` |
322| `GET /_ml/anomaly_detectors/{job_id}/_stats` | `elastic es ml get-job-stats --job-id '<job_id>'` |
323| `GET /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml get-datafeeds --datafeed-id 'datafeed-<job_id>'` |
324| `GET /_ml/datafeeds/datafeed-{job_id}/_stats` | `elastic es ml get-datafeed-stats --datafeed-id 'datafeed-<job_id>'` |
325| `POST /.ml-anomalies-*/_search` | `elastic es search --index '.ml-anomalies-*' --input-file '<search-body.json>'` |
326| `POST /.ml-annotations-*/_search` | `elastic es search --index '.ml-annotations-*' --input-file '<search-body.json>'` |
327| `POST /.ml-notifications-*/_search` | `elastic es search --index '.ml-notifications-*' --input-file '<search-body.json>'` |
328| `POST /_ml/anomaly_detectors/_estimate_model_memory` | `elastic es ml estimate-model-memory --analysis-config '<json>'` |
329| `PUT /_ml/anomaly_detectors/{job_id}` | `elastic es ml put-job --job-id '<job_id>' --input-file '<job-body.json>'` |
330| `PUT /_ml/datafeeds/datafeed-{job_id}` | `elastic es ml put-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<datafeed-body.json>'` |
331| `POST /_ml/anomaly_detectors/{job_id}/_open` | `elastic es ml open-job --job-id '<job_id>'` |
332| `POST /_ml/anomaly_detectors/{job_id}/_close` | `elastic es ml close-job --job-id '<job_id>'` |
333| `POST /_ml/anomaly_detectors/{job_id}/_update` | `elastic es ml update-job --job-id '<job_id>' --analysis-limits '<json>'` |
334| `POST /_ml/datafeeds/datafeed-{job_id}/_update` | `elastic es ml update-datafeed --datafeed-id 'datafeed-<job_id>' --input-file '<update-body.json>'` |
335| `POST /_ml/datafeeds/datafeed-{job_id}/_start` | `elastic es ml start-datafeed --datafeed-id 'datafeed-<job_id>'` |
336| `POST /_ml/datafeeds/datafeed-{job_id}/_stop` | `elastic es ml stop-datafeed --datafeed-id 'datafeed-<job_id>'` |
337| `POST /_ml/datafeeds/datafeed-{job_id}/_preview` | `elastic es ml preview-datafeed --datafeed-id 'datafeed-<job_id>'` |
338| `POST /_ml/anomaly_detectors/{job_id}/model_snapshots/{snapshot_id}/_revert` | `elastic es ml revert-model-snapshot --job-id '<job_id>' --snapshot-id '<snapshot_id>'` |
339
340Search body shapes for each `result_type` and troubleshooting queries are documented in
341[references/investigation-queries.md](references/investigation-queries.md) and
342[references/troubleshooting-reference.md](references/troubleshooting-reference.md).