Elastic ML Anomaly Detection
Single skill covering all anomaly detection work against Kibana Agent Builder MCP at
{KIBANA_URL}/api/agent_builder/mcp. Use the Mode Selector below to pick the right approach for the user's question
— modes share the same tool surface and concepts.
Platform
- Read path: ES|QL against
.ml-anomalies-*, .ml-config, .ml-notifications-*, .ml-annotations-*
- Always-available:
platform.core.execute_esql (plus additional platform tools for search, index mapping, and
documentation — see scripts/agent_builder_constants.json)
- ML API spec (if available):
.kibana_ai_openapi_spec_elasticsearch — see
references/anomaly-detection-openapi-spec-discover.md for
discovery pattern.
- Run
ad_validate_ml_tool_permissions first when tools return empty/misleading results — missing privileges are
the most common cause of false negatives. Full permissions matrix:
references/permissions-matrix.md.
Mode Selector
| User intent |
Mode |
| "What broke?" / RCA / cross-job / blast radius / influencers / log categories |
Investigate |
| "Why score high/low?" / renormalization / model bounds / forecasts |
Explain |
| Missing docs / memory limit / datafeed stopped / CCS / lifecycle / calendars |
Troubleshoot |
| Create a job / configure a datafeed / start analysis / retrieve results |
Manage |
| Security framing (attack chains, MITRE, exfil) |
Investigate + references/security-anomaly-expert.md |
| Observability/SRE framing (degradation, capacity, deployment regression) |
Investigate + references/observability-anomaly-expert.md |
When a question spans modes: Investigate → Explain → Troubleshoot. Don't blend mode logic — finish one before moving
on.
Score Quick Reference
record_score bands: >75 critical · 50–75 warning · 25–50 minor · <25 informational
multi_bucket_impact ≥ 3 → sustained shift (not a transient spike)
initial_record_score >> record_score → renormalization (model saw worse anomalies later)
actual << typical with count/low_count/low_mean → absence/outage, not just low value
- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity
Full score definitions, renormalization mechanics, and anomaly_score_explanation components:
references/score-reference.md.
Core concepts
Treat .ml-anomalies-* as three layers, accessed via result_type:
bucket — bucket-level unusualness per bucket_span. anomaly_score is the aggregate across all detectors.
record — finest-grained rows with actual vs typical, probability, record_score,
anomaly_score_explanation.
influencer — entity contributions ranked within a bucket (influencer_score).
Read scores this way:
anomaly_score / record_score = current normalized values (move as the model sees new extremes).
initial_anomaly_score / initial_record_score = immutable snapshots from detection time.
- Compare
actual to typical; use probability for raw likelihood.
- Map entities via
partition_field_value / by_field_value / over_field_value.
- Read
multi_bucket_impact (-5 to +5) to separate single-bucket spikes from sustained trends.
Mode: Investigate — RCA
When: "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.
Tool chain
| Phase |
Tools |
| Discovery |
ad_get_available_metadata, ad_get_jobs, ad_discover_related_jobs, ad_discover_jobs_by_datafeed_index |
| Timeline / scope |
ad_query_anomaly_timeline |
| Cross-job / entities |
ad_rca_cross_job_entity_match, ad_rca_multi_job_entities, ad_rca_entity_profile |
| Records / influencers |
ad_query_anomaly_records, ad_query_influencers |
| RCA depth |
ad_rca_detector_fingerprint, ad_rca_correlation, ad_rca_blast_radius, ad_rca_score_reassessment |
| Evidence / categories |
ad_get_job_datafeed_config, ad_rca_source_evidence, ad_get_categories, ad_search_log_category_examples |
Protocol
Follow the 14-step sequence in references/protocols/investigation.md. High
level: ad_get_available_metadata → pair ad_discover_jobs_by_datafeed_index with ad_discover_related_jobs →
ad_query_anomaly_timeline → rank with ad_rca_multi_job_entities (min_job_count=2) → ad_rca_detector_fingerprint
→ drill with ad_query_anomaly_records + ad_query_influencers (low min_score=25) → profile with
ad_rca_entity_profile → order with ad_rca_correlation → confirm with ad_rca_source_evidence. When
by_field_name == "mlcategory", compare with ad_get_categories + paired ad_search_log_category_examples (baseline
vs. anomaly window).
Finish with a written RCA: root cause entity · affected jobs · temporal progression · fault class
(resource/network/application) · severity · recommended actions. Worked example:
references/worked-example.md. Full ES|QL templates and parameters:
references/investigate-anomaly-esql-tools.md.
Rules
- Multi-job entities are prime suspects; single-job entities are usually victims. Use
min_job_count=2.
- Earliest anomaly timestamp wins — sort
ad_rca_correlation by timestamp; first-appearing entity = origin.
multi_bucket_impact ≥ 3 = sustained behavioral shift, weight higher than transient spikes.
- Never close an RCA without
ad_rca_source_evidence — raw source documents are ground truth.
- Use low
min_score (25 or lower) for influencer queries — high thresholds miss correlated entities.
Mode: Explain — Score / model behavior
When: "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".
Score types
| Field |
Scope |
Meaning |
record_score |
Single record |
Normalized severity after renormalization. |
initial_record_score |
Single record |
Score at detection time. Gap vs record_score = renormalization drift. |
anomaly_score |
Bucket |
Aggregate severity across all detectors in a bucket. |
influencer_score |
Entity × bucket |
How anomalous a specific entity is in that bucket. |
anomaly_score_explanation components
| Component |
Effect |
What it means |
anomaly_length |
↑ score |
More consecutive anomalous buckets |
single_bucket_impact |
↑ score |
Lower probability → higher impact |
multi_bucket_impact |
↑ score |
Sustained pattern contribution |
anomaly_characteristics_impact |
↑ score |
Mean shift vs. variance change |
high_variance_penalty |
↓ score |
Noisy data → wide bounds → anomaly less surprising |
incomplete_bucket_penalty |
↓ score |
Bucket has less data than expected (ingest lag, sparse data) |
Why a score looks wrong
- Unexpectedly low:
high_variance_penalty, renormalization, <3 weeks training for weekly seasonality,
bucket_span too large, wrong detector function (mean vs high_mean), incomplete_bucket_penalty, suppression by
custom_rules.
- Unexpectedly high: insufficient history (early training over-flags), high-cardinality split (too few points per
entity),
use_null: true on a sparse field.
Tool chain
| Purpose |
Tools |
| Records + explanation |
ad_query_anomaly_records (exact job_id_pattern) |
| Renormalization drift |
ad_rca_score_reassessment (score_drift = initial_record_score - record_score) |
| Model bounds (visual) |
ad_get_model_plot — actual outside model_lower/model_upper = anomaly |
| Forecast overlap |
ad_get_forecast_results |
| Influencer attribution |
ad_query_influencers |
| Config & detector |
ad_get_job_datafeed_config — bucket_span, function, custom_rules, use_null |
| Categorization |
ad_get_categories |
| Model snapshots |
ad_get_model_snapshots |
| Structured diagnostic |
ad_wf_troubleshoot_anomaly_score (full decision tree) |
Decision tree (ad_wf_troubleshoot_anomaly_score)
ad_get_jobs — ≥3 weeks data for weekly seasonality?
ad_ts_model_memory_health — memory_status healthy?
ad_ts_delayed_data_annotations — no incomplete buckets?
ad_query_anomaly_records — compare record_score vs initial_record_score.
ad_get_job_datafeed_config — bucket_span, detector function, custom_rules, use_null.
ad_get_model_plot — wide bounds → high_variance_penalty.
ad_rca_score_reassessment — renormalization drift across history.
- Explain
anomaly_score_explanation factors.
Rules
- Always show both
initial_record_score and record_score — the gap is the renormalization story.
- Explain renormalization before diagnosing config — score drift is the most common "score dropped" cause and needs
no config change.
actual << typical with count/low_count is an absence anomaly — distinguish outages from value spikes.
high_variance_penalty and incomplete_bucket_penalty explain most "low score" surprises without remediation.
- Weekly seasonality needs ≥3 weeks of training data — flag young jobs as the cause.
For detector function selection details, see
references/anomaly-detection-functions.md.
Mode: Troubleshoot — Job ops
When: "missing documents", "datafeed stopped", "hard_limit", "results look wrong", lifecycle changes, calendars,
CCS.
Common issues → fast paths
| Issue |
Fast path |
Full decision tree |
Missing docs / query_delay warning |
ad_ts_delayed_data_annotations → ad_ts_bucket_event_gaps → ad_ts_ingest_latency_estimate → ad_update_datafeed_query_delay |
ad_wf_troubleshoot_query_delay |
Memory soft_limit / hard_limit |
ad_ts_model_memory_health → ad_wf_ts_field_cardinality → ad_estimate_memory_requirement → ad_update_model_memory_limit |
ad_wf_troubleshoot_memory_limit |
| Datafeed not running / job state |
ad_get_jobs (state) → ad_get_job_messages → ad_manage_datafeed |
— |
CCS / remote_cluster: indices |
ad_ts_ccs_diagnostics |
— |
| Score sanity check |
— |
ad_wf_troubleshoot_anomaly_score |
hard_limit corrupts model state and causes downstream missing-doc false alarms (categorizer silently skips events
for unknown categories). Fix memory before fixing query_delay.
Memory concepts
| Field |
Meaning |
model_bytes |
Current memory used |
peak_model_bytes |
High-water mark since job opened |
model_bytes_memory_limit |
Configured model_memory_limit |
memory_status |
ok / soft_limit (pruning) / hard_limit (critical) |
total_by_field_count > 100k |
by_field cardinality too high — dominant driver |
total_partition_field_count > 10k |
Partition explosion |
total_category_count > 10k |
Too many distinct log patterns |
Prefer ad_estimate_memory_requirement (samples cardinality from source, calls Estimate Model Memory API) over
heuristics like peak_model_bytes * 1.3 — the heuristic ignores pure influencer and categorization memory.
Datafeed & timing concepts
query_delay — how far behind real time the datafeed queries. Too small → missing docs; too large → slower
alerts. Set to P95 ingest latency + buffer (default 60s–120s).
delayed_data_check_config — how aggressively the datafeed checks for late data.
bucket_span — analysis interval. Align with data granularity and detection window.
frequency — defaults to min(query_delay, bucket_span / 2).
Lifecycle for config changes (memory limit, query_delay)
- Stop datafeed:
ad_manage_datafeed (action=_stop)
- Close job
- Update config:
ad_update_model_memory_limit, ad_update_datafeed_query_delay,
ad_update_delayed_data_check_config
- Open job:
ad_open_job
- Start datafeed:
ad_manage_datafeed (action=_start)
Recover a corrupted period without resetting the whole model: ad_revert_model_snapshot.
Tool surface
| Category |
Tools |
| Permissions / metadata |
ad_validate_ml_tool_permissions, ad_get_available_metadata, ad_get_jobs |
| Job + datafeed state |
ad_get_job_datafeed_config, ad_get_job_messages, ad_manage_datafeed, ad_preview_datafeed_with_latency |
| Timing / missing docs |
ad_ts_delayed_data_annotations, ad_ts_bucket_event_gaps, ad_ts_ingest_latency_estimate, ad_update_datafeed_query_delay, ad_update_delayed_data_check_config, ad_wf_troubleshoot_query_delay |
| Memory |
ad_ts_model_memory_health, ad_wf_ts_field_cardinality, ad_estimate_memory_requirement, ad_update_model_memory_limit, ad_wf_troubleshoot_memory_limit |
| Model / lifecycle |
ad_get_model_snapshots, ad_revert_model_snapshot, ad_open_job, ad_create_job |
| CCS |
ad_ts_ccs_diagnostics |
| Calendars |
ad_get_calendar_events, ad_create_calendar_event |
Full parameter tables, ES|QL templates, and REST step lists:
references/troubleshoot-anomaly-tool-reference.md.
Rules
ad_validate_ml_tool_permissions first — missing privileges produce misleading empty results.
- Fix memory before
query_delay — hard_limit corrupts state; query_delay fixes on a memory-limited job are
wasted.
- Stop the datafeed before updating it. Updating a running datafeed is rejected.
- Close the job before updating memory limit. Sequence above.
- Prefer workflow tools (
ad_wf_*) over manually chaining diagnostics for complex decisions.
ad_preview_datafeed_with_latency before starting — confirm the datafeed returns data after config changes.
Mode: Manage — Create / configure jobs
When: "set up a job", "create an ML detector", "monitor X over time", "detect rare/unusual/anomalous values".
4-step workflow
PUT _ml/anomaly_detectors/<job_id> # 1. Define job (ad_create_job)
PUT _ml/datafeeds/datafeed-<job_id> # 2. Define datafeed (ad_create_datafeed)
POST _ml/anomaly_detectors/<job_id>/_open # 3a. Open job (ad_open_job)
POST _ml/datafeeds/datafeed-<job_id>/_start # 3b. Start datafeed (ad_manage_datafeed action=_start)
GET _ml/anomaly_detectors/<job_id>/results/records # 4. Read results
Process
Build configs. Parse the user request into job + datafeed JSON with no null fields.
Apply smart defaults:
| Field |
Default |
Override when |
bucket_span |
"15m" |
User specifies a different span |
time_field |
"@timestamp" |
User names a different timestamp field |
index |
"logs-*" |
User specifies an index or pattern |
datafeed_query |
{"match_all": {}} |
User mentions filters, processes, or time windows |
influencers |
by/over/partition fields from detectors |
User adds extra influencer fields |
job_id |
Generated from user description |
User provides an explicit ID |
query_delay |
"60s" |
P95 ingest latency is higher |
Choose detector function from user intent — full table in
references/anomaly-detection-functions.md:
- "high CPU" / "unusually large" →
high_mean or high_sum
- "rare logins" / "unusual values" →
rare (variants below)
- "too many requests" / "spike in count" →
high_count
rare variants:
- Infrequent globally →
rare by_field_name: X
- Infrequent vs peers →
rare by_field_name: X over_field_name: Y
- Infrequent per segment →
rare by_field_name: X partition_field_name: Y
- Infrequent per segment vs peers →
rare by_field_name: X over_field_name: Y partition_field_name: Z
Validate. platform.core.get_index_mapping on the target index to verify field existence/types →
ad_validate_job_spec. If errors, fix and re-validate (max 3 attempts).
Present and confirm. Show the complete job + datafeed bodies formatted as the exact API calls. Ask for
approval once. If feedback, incorporate and re-present (up to 3 rounds).
Deploy. After confirmation: ad_create_job → ad_create_datafeed → ad_open_job → ad_manage_datafeed
(action=_start). Report final job_id and datafeed_id.
For batch analysis on historical data, pass start and end to the datafeed start call.
Worked examples (rare-username, DNS exfil, large-downloads) with full JSON bodies and datafeed filters:
references/job-creation-recipes.md.
Rules
- Create job before datafeed. Datafeed references job by ID.
- Open job before starting datafeed. Start on a closed job is rejected.
query_delay = P95 ingest latency + buffer (60s–120s safe default).
- Forecasts require non-population jobs —
over_field_name jobs cannot be forecasted; warn before attempting.
by_field_name vs over_field_name: by compares entity to its own history; over compares to peer group in
the same bucket. partition_field_name = fully independent sub-model with its own normalization.
bucket_span matches detection granularity — 15m for high-frequency, 1h for operational metrics, 1d for daily
patterns. Larger smooths short spikes; smaller increases noise.
Registration (Kibana Agent Builder)
Requires Node.js 18+. Defaults to elastic/changeme when no credentials are supplied.
cd skills/kibana/kibana-anomaly-detection
# tools → workflows → skills
node scripts/kibana-agent-builder.mjs all register --kibana-url http://localhost:5601
# HTTPS with self-signed cert
node scripts/kibana-agent-builder.mjs all register --kibana-url https://localhost:5601 --insecure
all register runs tools register, then workflows register, then skills register. Kibana allows at most five
tool_ids per skill; the script fills them by scanning SKILL.md for tool mentions (in document order), then appends
ids from references/kibana/tools/esql/*.json until the cap (workflow-only tools omitted by default). If you run
skills register alone, run tools register first so those ids exist.
Workflow tool exclusions and prefixes live in scripts/agent_builder_constants.json.
MCP API key permissions:
- Kibana:
read_onechat, space_read
- Index:
read, view_index_metadata on .ml-anomalies-*, .ml-annotations-*, .ml-notifications-*, .ml-config
- For source evidence:
read on source data indices
Tool inventory
ES|QL tool specs live under references/kibana/tools/esql/*.json; workflow definitions under
references/kibana/workflows/*.yaml. Each Mode section above lists the tools it uses. Full surface:
references/tools.md (ES|QL) and references/workflow-tools.md
(workflows).
Key system indices
| Index |
Relevant content |
.ml-anomalies-* |
record, bucket, influencer, model_plot, model_forecast, model_snapshot, category_definition, model_size_stats |
.ml-config |
job/datafeed documents (visible even for never-run jobs) |
.ml-annotations-* |
delayed data (event == "delayed_data") |
.ml-notifications-* |
job messages (level: info/warning/error) |
Examples
RCA: "Something caused a spike in our error rate at 2pm — what broke?" → Investigate → ad_get_available_metadata →
ad_query_anomaly_timeline → ad_rca_cross_job_entity_match → ad_rca_multi_job_entities → RCA report.
Score drop: "My anomaly score went from 90 to 55 — did the model change?" → Explain → ad_rca_score_reassessment
for drift → explain renormalization if score_drift is large.
Memory limit: "Job status shows hard_limit and results look wrong." → Troubleshoot → ad_ts_model_memory_health →
ad_wf_ts_field_cardinality → ad_estimate_memory_requirement → ad_update_model_memory_limit (lifecycle: stop
datafeed → close → update → open → start).
New job: "Detect unusual error rates per host on nginx access logs." → Manage → high_count detector with
by_field_name: "host.keyword" → validate → present → deploy.
Multi-mode: "We had an incident last night, scores were high but now low — is the job healthy?" → Investigate the
incident → Explain the score drift → Troubleshoot if hard_limit or delayed data is suspected.
Guidelines
- Pick a mode first. Don't blend RCA logic with score-explanation logic in one response.
ad_validate_ml_tool_permissions first on empty results — privileges are the most common false-negative cause.
- Score bands are absolute thresholds:
>75 critical, 50–75 warning, 25–50 minor, <25 informational.
- Multi-job entities are prime suspects. Use
min_job_count=2 in ad_rca_multi_job_entities.
- Show
initial_record_score alongside record_score — the gap tells the renormalization story.
- Fix memory before
query_delay. hard_limit invalidates downstream diagnostics.
- Stop datafeed → close job → update config → open job → start datafeed for any config change to memory or query
delay.
- Confirm RCAs with
ad_rca_source_evidence. Raw source documents are ground truth.
1---2name: kibana-anomaly-detection3description: Elastic ML anomaly detection skill — investigation/RCA, score explanation,4 job operations (create, datafeed, start/stop, results), and troubleshooting (missing5 docs, memory limits, datafeed health, lifecycle). Operates against Kibana Agent6 Builder MCP tools (`ad_*`) on `.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`,7 `.ml-annotations-*`. Use when answering "what broke?"/"which entity?"/RCA, "why8 is score high/low?"/renormalization, "datafeed stopped"/"memory limit", or any request9 to set up or configure an ML anomaly detection job.10metadata:11 source: https://skillmd.com/skills/zuoguyoupan2023/kibana-anomaly-detection12 author: elastic13 version: 0.2.014compatibility: Kibana 8.x–9.x with Agent Builder and Workflows; Elasticsearch 8.x–9.x15 with machine learning16---1718# Elastic ML Anomaly Detection1920Single skill covering all anomaly detection work against **Kibana Agent Builder** MCP at21`{KIBANA_URL}/api/agent_builder/mcp`. Use the **Mode Selector** below to pick the right approach for the user's question22— modes share the same tool surface and concepts.2324## Platform2526- Read path: ES|QL against `.ml-anomalies-*`, `.ml-config`, `.ml-notifications-*`, `.ml-annotations-*`27- Always-available: `platform.core.execute_esql` (plus additional platform tools for search, index mapping, and28 documentation — see `scripts/agent_builder_constants.json`)29- ML API spec (if available): `.kibana_ai_openapi_spec_elasticsearch` — see30 [references/anomaly-detection-openapi-spec-discover.md](references/anomaly-detection-openapi-spec-discover.md) for31 discovery pattern.32- **Run `ad_validate_ml_tool_permissions` first** when tools return empty/misleading results — missing privileges are33 the most common cause of false negatives. Full permissions matrix:34 [references/permissions-matrix.md](references/permissions-matrix.md).3536## Mode Selector3738| User intent | Mode |39| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |40| "What broke?" / RCA / cross-job / blast radius / influencers / log categories | **Investigate** |41| "Why score high/low?" / renormalization / model bounds / forecasts | **Explain** |42| Missing docs / memory limit / datafeed stopped / CCS / lifecycle / calendars | **Troubleshoot** |43| Create a job / configure a datafeed / start analysis / retrieve results | **Manage** |44| Security framing (attack chains, MITRE, exfil) | Investigate + [references/security-anomaly-expert.md](references/security-anomaly-expert.md) |45| Observability/SRE framing (degradation, capacity, deployment regression) | Investigate + [references/observability-anomaly-expert.md](references/observability-anomaly-expert.md) |4647When a question spans modes: **Investigate → Explain → Troubleshoot**. Don't blend mode logic — finish one before moving48on.4950---5152## Score Quick Reference5354- `record_score` bands: **>75** critical · **50–75** warning · **25–50** minor · **<25** informational55- `multi_bucket_impact ≥ 3` → sustained shift (not a transient spike)56- `initial_record_score >> record_score` → renormalization (model saw worse anomalies later)57- `actual << typical` with `count`/`low_count`/`low_mean` → absence/outage, not just low value58- Low scores across many jobs > one high score — composite cross-job signal often beats single-detector severity5960> Full score definitions, renormalization mechanics, and `anomaly_score_explanation` components:61> [references/score-reference.md](references/score-reference.md).6263## Core concepts6465Treat `.ml-anomalies-*` as three layers, accessed via `result_type`:6667- **`bucket`** — bucket-level unusualness per `bucket_span`. `anomaly_score` is the aggregate across all detectors.68- **`record`** — finest-grained rows with `actual` vs `typical`, `probability`, `record_score`,69 `anomaly_score_explanation`.70- **`influencer`** — entity contributions ranked within a bucket (`influencer_score`).7172Read scores this way:7374- `anomaly_score` / `record_score` = **current normalized** values (move as the model sees new extremes).75- `initial_anomaly_score` / `initial_record_score` = **immutable snapshots** from detection time.76- Compare `actual` to `typical`; use `probability` for raw likelihood.77- Map entities via `partition_field_value` / `by_field_value` / `over_field_value`.78- Read `multi_bucket_impact` (-5 to +5) to separate single-bucket spikes from sustained trends.7980---8182## Mode: Investigate — RCA8384**When:** "what broke?", "which entity caused this?", cross-job correlation, blast radius, attack/cascade chains.8586### Tool chain8788| Phase | Tools |89| --------------------- | -------------------------------------------------------------------------------------------------------------- |90| Discovery | `ad_get_available_metadata`, `ad_get_jobs`, `ad_discover_related_jobs`, `ad_discover_jobs_by_datafeed_index` |91| Timeline / scope | `ad_query_anomaly_timeline` |92| Cross-job / entities | `ad_rca_cross_job_entity_match`, `ad_rca_multi_job_entities`, `ad_rca_entity_profile` |93| Records / influencers | `ad_query_anomaly_records`, `ad_query_influencers` |94| RCA depth | `ad_rca_detector_fingerprint`, `ad_rca_correlation`, `ad_rca_blast_radius`, `ad_rca_score_reassessment` |95| Evidence / categories | `ad_get_job_datafeed_config`, `ad_rca_source_evidence`, `ad_get_categories`, `ad_search_log_category_examples` |9697### Protocol9899Follow the 14-step sequence in [references/protocols/investigation.md](references/protocols/investigation.md). High100level: `ad_get_available_metadata` → pair `ad_discover_jobs_by_datafeed_index` with `ad_discover_related_jobs` →101`ad_query_anomaly_timeline` → rank with `ad_rca_multi_job_entities` (`min_job_count=2`) → `ad_rca_detector_fingerprint`102→ drill with `ad_query_anomaly_records` + `ad_query_influencers` (low `min_score=25`) → profile with103`ad_rca_entity_profile` → order with `ad_rca_correlation` → confirm with `ad_rca_source_evidence`. When104`by_field_name == "mlcategory"`, compare with `ad_get_categories` + paired `ad_search_log_category_examples` (baseline105vs. anomaly window).106107Finish with a written RCA: **root cause entity · affected jobs · temporal progression · fault class108(resource/network/application) · severity · recommended actions**. Worked example:109[references/worked-example.md](references/worked-example.md). Full ES|QL templates and parameters:110[references/investigate-anomaly-esql-tools.md](references/investigate-anomaly-esql-tools.md).111112### Rules1131141. **Multi-job entities are prime suspects; single-job entities are usually victims.** Use `min_job_count=2`.1152. **Earliest anomaly timestamp wins** — sort `ad_rca_correlation` by timestamp; first-appearing entity = origin.1163. **`multi_bucket_impact ≥ 3` = sustained behavioral shift**, weight higher than transient spikes.1174. **Never close an RCA without `ad_rca_source_evidence`** — raw source documents are ground truth.1185. **Use low `min_score` (25 or lower) for influencer queries** — high thresholds miss correlated entities.119120---121122## Mode: Explain — Score / model behavior123124**When:** "why is my score 30/90?", "score dropped overnight", "what is renormalization?", "why wasn't this detected?".125126### Score types127128| Field | Scope | Meaning |129| ---------------------- | --------------- | ----------------------------------------------------------------------- |130| `record_score` | Single record | Normalized severity after renormalization. |131| `initial_record_score` | Single record | Score at detection time. Gap vs `record_score` = renormalization drift. |132| `anomaly_score` | Bucket | Aggregate severity across all detectors in a bucket. |133| `influencer_score` | Entity × bucket | How anomalous a specific entity is in that bucket. |134135### `anomaly_score_explanation` components136137| Component | Effect | What it means |138| -------------------------------- | ------- | ------------------------------------------------------------ |139| `anomaly_length` | ↑ score | More consecutive anomalous buckets |140| `single_bucket_impact` | ↑ score | Lower probability → higher impact |141| `multi_bucket_impact` | ↑ score | Sustained pattern contribution |142| `anomaly_characteristics_impact` | ↑ score | Mean shift vs. variance change |143| `high_variance_penalty` | ↓ score | Noisy data → wide bounds → anomaly less surprising |144| `incomplete_bucket_penalty` | ↓ score | Bucket has less data than expected (ingest lag, sparse data) |145146### Why a score looks wrong147148- **Unexpectedly low:** `high_variance_penalty`, renormalization, <3 weeks training for weekly seasonality,149 `bucket_span` too large, wrong detector function (`mean` vs `high_mean`), `incomplete_bucket_penalty`, suppression by150 `custom_rules`.151- **Unexpectedly high:** insufficient history (early training over-flags), high-cardinality split (too few points per152 entity), `use_null: true` on a sparse field.153154### Tool chain155156| Purpose | Tools |157| ---------------------- | ---------------------------------------------------------------------------------- |158| Records + explanation | `ad_query_anomaly_records` (exact `job_id_pattern`) |159| Renormalization drift | `ad_rca_score_reassessment` (`score_drift = initial_record_score - record_score`) |160| Model bounds (visual) | `ad_get_model_plot` — actual outside `model_lower`/`model_upper` = anomaly |161| Forecast overlap | `ad_get_forecast_results` |162| Influencer attribution | `ad_query_influencers` |163| Config & detector | `ad_get_job_datafeed_config` — `bucket_span`, function, `custom_rules`, `use_null` |164| Categorization | `ad_get_categories` |165| Model snapshots | `ad_get_model_snapshots` |166| Structured diagnostic | **`ad_wf_troubleshoot_anomaly_score`** (full decision tree) |167168### Decision tree (`ad_wf_troubleshoot_anomaly_score`)1691701. `ad_get_jobs` — ≥3 weeks data for weekly seasonality?1712. `ad_ts_model_memory_health` — `memory_status` healthy?1723. `ad_ts_delayed_data_annotations` — no incomplete buckets?1734. `ad_query_anomaly_records` — compare `record_score` vs `initial_record_score`.1745. `ad_get_job_datafeed_config` — `bucket_span`, detector function, `custom_rules`, `use_null`.1756. `ad_get_model_plot` — wide bounds → `high_variance_penalty`.1767. `ad_rca_score_reassessment` — renormalization drift across history.1778. Explain `anomaly_score_explanation` factors.178179### Rules1801811. **Always show both `initial_record_score` and `record_score`** — the gap is the renormalization story.1822. **Explain renormalization before diagnosing config** — score drift is the most common "score dropped" cause and needs183 no config change.1843. **`actual << typical` with `count`/`low_count` is an absence anomaly** — distinguish outages from value spikes.1854. **`high_variance_penalty` and `incomplete_bucket_penalty` explain most "low score" surprises** without remediation.1865. **Weekly seasonality needs ≥3 weeks of training data** — flag young jobs as the cause.187188For detector function selection details, see189[references/anomaly-detection-functions.md](references/anomaly-detection-functions.md).190191---192193## Mode: Troubleshoot — Job ops194195**When:** "missing documents", "datafeed stopped", "hard_limit", "results look wrong", lifecycle changes, calendars,196CCS.197198### Common issues → fast paths199200| Issue | Fast path | Full decision tree |201| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |202| Missing docs / `query_delay` warning | `ad_ts_delayed_data_annotations` → `ad_ts_bucket_event_gaps` → `ad_ts_ingest_latency_estimate` → `ad_update_datafeed_query_delay` | `ad_wf_troubleshoot_query_delay` |203| Memory `soft_limit` / `hard_limit` | `ad_ts_model_memory_health` → `ad_wf_ts_field_cardinality` → `ad_estimate_memory_requirement` → `ad_update_model_memory_limit` | `ad_wf_troubleshoot_memory_limit` |204| Datafeed not running / job state | `ad_get_jobs` (state) → `ad_get_job_messages` → `ad_manage_datafeed` | — |205| CCS / `remote_cluster:` indices | `ad_ts_ccs_diagnostics` | — |206| Score sanity check | — | `ad_wf_troubleshoot_anomaly_score` |207208> `hard_limit` corrupts model state and causes downstream missing-doc false alarms (categorizer silently skips events209> for unknown categories). **Fix memory before fixing `query_delay`.**210211### Memory concepts212213| Field | Meaning |214| ----------------------------------- | ------------------------------------------------------- |215| `model_bytes` | Current memory used |216| `peak_model_bytes` | High-water mark since job opened |217| `model_bytes_memory_limit` | Configured `model_memory_limit` |218| `memory_status` | `ok` / `soft_limit` (pruning) / `hard_limit` (critical) |219| `total_by_field_count > 100k` | `by_field` cardinality too high — dominant driver |220| `total_partition_field_count > 10k` | Partition explosion |221| `total_category_count > 10k` | Too many distinct log patterns |222223Prefer **`ad_estimate_memory_requirement`** (samples cardinality from source, calls Estimate Model Memory API) over224heuristics like `peak_model_bytes * 1.3` — the heuristic ignores pure influencer and categorization memory.225226### Datafeed & timing concepts227228- **`query_delay`** — how far behind real time the datafeed queries. Too small → missing docs; too large → slower229 alerts. Set to **P95 ingest latency + buffer** (default `60s`–`120s`).230- **`delayed_data_check_config`** — how aggressively the datafeed checks for late data.231- **`bucket_span`** — analysis interval. Align with data granularity and detection window.232- **`frequency`** — defaults to `min(query_delay, bucket_span / 2)`.233234### Lifecycle for config changes (memory limit, query_delay)2352361. Stop datafeed: `ad_manage_datafeed` (`action=_stop`)2372. Close job2383. Update config: `ad_update_model_memory_limit`, `ad_update_datafeed_query_delay`,239 `ad_update_delayed_data_check_config`2404. Open job: `ad_open_job`2415. Start datafeed: `ad_manage_datafeed` (`action=_start`)242243Recover a corrupted period without resetting the whole model: `ad_revert_model_snapshot`.244245### Tool surface246247| Category | Tools |248| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |249| Permissions / metadata | `ad_validate_ml_tool_permissions`, `ad_get_available_metadata`, `ad_get_jobs` |250| Job + datafeed state | `ad_get_job_datafeed_config`, `ad_get_job_messages`, `ad_manage_datafeed`, `ad_preview_datafeed_with_latency` |251| Timing / missing docs | `ad_ts_delayed_data_annotations`, `ad_ts_bucket_event_gaps`, `ad_ts_ingest_latency_estimate`, `ad_update_datafeed_query_delay`, `ad_update_delayed_data_check_config`, `ad_wf_troubleshoot_query_delay` |252| Memory | `ad_ts_model_memory_health`, `ad_wf_ts_field_cardinality`, `ad_estimate_memory_requirement`, `ad_update_model_memory_limit`, `ad_wf_troubleshoot_memory_limit` |253| Model / lifecycle | `ad_get_model_snapshots`, `ad_revert_model_snapshot`, `ad_open_job`, `ad_create_job` |254| CCS | `ad_ts_ccs_diagnostics` |255| Calendars | `ad_get_calendar_events`, `ad_create_calendar_event` |256257Full parameter tables, ES|QL templates, and REST step lists:258[references/troubleshoot-anomaly-tool-reference.md](references/troubleshoot-anomaly-tool-reference.md).259260### Rules2612621. **`ad_validate_ml_tool_permissions` first** — missing privileges produce misleading empty results.2632. **Fix memory before `query_delay`** — `hard_limit` corrupts state; `query_delay` fixes on a memory-limited job are264 wasted.2653. **Stop the datafeed before updating it.** Updating a running datafeed is rejected.2664. **Close the job before updating memory limit.** Sequence above.2675. **Prefer workflow tools (`ad_wf_*`) over manually chaining diagnostics** for complex decisions.2686. **`ad_preview_datafeed_with_latency` before starting** — confirm the datafeed returns data after config changes.269270---271272## Mode: Manage — Create / configure jobs273274**When:** "set up a job", "create an ML detector", "monitor X over time", "detect rare/unusual/anomalous values".275276### 4-step workflow277278```text279PUT _ml/anomaly_detectors/<job_id> # 1. Define job (ad_create_job)280PUT _ml/datafeeds/datafeed-<job_id> # 2. Define datafeed (ad_create_datafeed)281POST _ml/anomaly_detectors/<job_id>/_open # 3a. Open job (ad_open_job)282POST _ml/datafeeds/datafeed-<job_id>/_start # 3b. Start datafeed (ad_manage_datafeed action=_start)283GET _ml/anomaly_detectors/<job_id>/results/records # 4. Read results284```285286### Process2872881. **Build configs.** Parse the user request into job + datafeed JSON with no null fields.2892. **Apply smart defaults:**290291 | Field | Default | Override when |292 | ---------------- | --------------------------------------- | ------------------------------------------------- |293 | `bucket_span` | `"15m"` | User specifies a different span |294 | `time_field` | `"@timestamp"` | User names a different timestamp field |295 | `index` | `"logs-*"` | User specifies an index or pattern |296 | `datafeed_query` | `{"match_all": {}}` | User mentions filters, processes, or time windows |297 | `influencers` | by/over/partition fields from detectors | User adds extra influencer fields |298 | `job_id` | Generated from user description | User provides an explicit ID |299 | `query_delay` | `"60s"` | P95 ingest latency is higher |3003013. **Choose detector function** from user intent — full table in302 [references/anomaly-detection-functions.md](references/anomaly-detection-functions.md):303 - "high CPU" / "unusually large" → `high_mean` or `high_sum`304 - "rare logins" / "unusual values" → `rare` (variants below)305 - "too many requests" / "spike in count" → `high_count`306307 `rare` variants:308 - Infrequent globally → `rare by_field_name: X`309 - Infrequent vs peers → `rare by_field_name: X over_field_name: Y`310 - Infrequent per segment → `rare by_field_name: X partition_field_name: Y`311 - Infrequent per segment vs peers → `rare by_field_name: X over_field_name: Y partition_field_name: Z`3123134. **Validate.** `platform.core.get_index_mapping` on the target index to verify field existence/types →314 `ad_validate_job_spec`. If errors, fix and re-validate (max 3 attempts).3153165. **Present and confirm.** Show the **complete** job + datafeed bodies formatted as the exact API calls. Ask for317 approval **once**. If feedback, incorporate and re-present (up to 3 rounds).3183196. **Deploy.** After confirmation: `ad_create_job` → `ad_create_datafeed` → `ad_open_job` → `ad_manage_datafeed`320 (`action=_start`). Report final `job_id` and `datafeed_id`.321322For **batch analysis on historical data**, pass `start` and `end` to the datafeed start call.323324> Worked examples (rare-username, DNS exfil, large-downloads) with full JSON bodies and datafeed filters:325> [references/job-creation-recipes.md](references/job-creation-recipes.md).326327### Rules3283291. **Create job before datafeed.** Datafeed references job by ID.3302. **Open job before starting datafeed.** Start on a closed job is rejected.3313. **`query_delay` = P95 ingest latency + buffer** (60s–120s safe default).3324. **Forecasts require non-population jobs** — `over_field_name` jobs cannot be forecasted; warn before attempting.3335. **`by_field_name` vs `over_field_name`:** `by` compares entity to its own history; `over` compares to peer group in334 the same bucket. `partition_field_name` = fully independent sub-model with its own normalization.3356. **`bucket_span` matches detection granularity** — 15m for high-frequency, 1h for operational metrics, 1d for daily336 patterns. Larger smooths short spikes; smaller increases noise.337338---339340## Registration (Kibana Agent Builder)341342Requires Node.js 18+. Defaults to `elastic`/`changeme` when no credentials are supplied.343344```bash345cd skills/kibana/kibana-anomaly-detection346347# tools → workflows → skills348node scripts/kibana-agent-builder.mjs all register --kibana-url http://localhost:5601349350# HTTPS with self-signed cert351node scripts/kibana-agent-builder.mjs all register --kibana-url https://localhost:5601 --insecure352```353354`all register` runs `tools register`, then `workflows register`, then `skills register`. Kibana allows **at most five**355`tool_ids` per skill; the script fills them by scanning `SKILL.md` for tool mentions (in document order), then appends356ids from `references/kibana/tools/esql/*.json` until the cap (workflow-only tools omitted by default). If you run357`skills register` alone, run `tools register` first so those ids exist.358359Workflow tool exclusions and prefixes live in `scripts/agent_builder_constants.json`.360361**MCP API key permissions:**362363- Kibana: `read_onechat`, `space_read`364- Index: `read`, `view_index_metadata` on `.ml-anomalies-*`, `.ml-annotations-*`, `.ml-notifications-*`, `.ml-config`365- For source evidence: `read` on source data indices366367---368369## Tool inventory370371ES|QL tool specs live under `references/kibana/tools/esql/*.json`; workflow definitions under372`references/kibana/workflows/*.yaml`. Each Mode section above lists the tools it uses. Full surface:373[references/tools.md](references/tools.md) (ES|QL) and [references/workflow-tools.md](references/workflow-tools.md)374(workflows).375376### Key system indices377378| Index | Relevant content |379| --------------------- | ----------------------------------------------------------------------------------------------------------------------------- |380| `.ml-anomalies-*` | `record`, `bucket`, `influencer`, `model_plot`, `model_forecast`, `model_snapshot`, `category_definition`, `model_size_stats` |381| `.ml-config` | job/datafeed documents (visible even for never-run jobs) |382| `.ml-annotations-*` | delayed data (`event == "delayed_data"`) |383| `.ml-notifications-*` | job messages (`level`: info/warning/error) |384385---386387## Examples388389**RCA:** "Something caused a spike in our error rate at 2pm — what broke?" → Investigate → `ad_get_available_metadata` →390`ad_query_anomaly_timeline` → `ad_rca_cross_job_entity_match` → `ad_rca_multi_job_entities` → RCA report.391392**Score drop:** "My anomaly score went from 90 to 55 — did the model change?" → Explain → `ad_rca_score_reassessment`393for drift → explain renormalization if `score_drift` is large.394395**Memory limit:** "Job status shows `hard_limit` and results look wrong." → Troubleshoot → `ad_ts_model_memory_health` →396`ad_wf_ts_field_cardinality` → `ad_estimate_memory_requirement` → `ad_update_model_memory_limit` (lifecycle: stop397datafeed → close → update → open → start).398399**New job:** "Detect unusual error rates per host on nginx access logs." → Manage → `high_count` detector with400`by_field_name: "host.keyword"` → validate → present → deploy.401402**Multi-mode:** "We had an incident last night, scores were high but now low — is the job healthy?" → Investigate the403incident → Explain the score drift → Troubleshoot if `hard_limit` or delayed data is suspected.404405---406407## Guidelines4084091. **Pick a mode first.** Don't blend RCA logic with score-explanation logic in one response.4102. **`ad_validate_ml_tool_permissions` first** on empty results — privileges are the most common false-negative cause.4113. **Score bands are absolute thresholds**: `>75` critical, `50–75` warning, `25–50` minor, `<25` informational.4124. **Multi-job entities are prime suspects.** Use `min_job_count=2` in `ad_rca_multi_job_entities`.4135. **Show `initial_record_score` alongside `record_score`** — the gap tells the renormalization story.4146. **Fix memory before `query_delay`.** `hard_limit` invalidates downstream diagnostics.4157. **Stop datafeed → close job → update config → open job → start datafeed** for any config change to memory or query416 delay.4178. **Confirm RCAs with `ad_rca_source_evidence`.** Raw source documents are ground truth.