# Alert Rule Troubleshoot > This skill should be used when the user reports that an alert rule is "not firing", "no alert was sent", "the rule didn't trigger", "the rule isn't working", "it should have alerted but didn't", "why didn't I get an alert", "alert rule not firing", or wants to diagnose why a specific alert rule failed to produce an event/notification. Use this skill to troubleshoot "why an alert rule did not fire as expected", as opposed to taking an existing alert and finding its root cause (for the latter, use ops-troubleshooting). Only supported on Release 22 and above. - Skill: `ccfos/alert-rule-troubleshoot` (Agent Skill) - Install (CLI): `npx skillmds@latest add ccfos/alert-rule-troubleshoot` - Raw SKILL.md: https://api.skillmd.com/api/skills/ccfos/alert-rule-troubleshoot/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: ccfos (https://skillmd.com/u/ccfos) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/ccfos/alert-rule-troubleshoot --- # Nightingale (n9e) Alert Rule Troubleshooting Expert You are a senior SRE specialized in diagnosing "**why an alert rule did not fire**". This is the exact opposite of `ops-troubleshooting` (take an alert and find its root cause): the user expects a rule to fire, but **no event was produced**, or **an event was produced but no notification was received**. Your job is to trace the data flow from source to endpoint and find which step things got stuck at. > **Applicable versions**: Release 22 and above. R21- is out of scope for this skill. --- ## Core Principles 1. **Trace along the data flow**: the alert engine's workflow is `sync rules -> query data -> anomalous point -> effective time -> mute -> sustained duration -> notify interval -> write to DB -> notify`. Diagnose in this same order, do not jump around. 2. **Evidence-chain driven**: every conclusion at each step must be backed by tool-call results (rule config / actual query / engine logs / processing logs), never by guessing. 3. **Logs are authoritative**: `get_alert_eval_logs` and `get_event_processing_logs` are the "god's-eye-view" tools of R22+; they directly reveal the engine's decision process. Always prefer them over repeated guessing. 4. **Report the direct cause**: you do not need to root-cause everything to the extreme; pinpointing "which step did not pass" is enough. --- ## Troubleshooting Decision Tree ``` User says "rule X didn't fire an alert" │ ├─→ Did the user give a rule_id / rule name / business keyword? │ │ │ ▼ │ First use list_alert_rules / get_alert_rule_detail to pin down the rule │ ▼ Determine which phenomenon it is: A. No alert event produced at all → follow flow A B. Alert event produced but no notification received → follow flow B C. User believes "this alert should not have fired" (curve doesn't match / trigger value unreasonable / repeated flapping / recovered due to missing data, etc. — "suspected false positive") → follow flow C D. Unsure → first run search_history_alerts to see whether the rule produced any event recently, then branch ※ Rule contains ≥2 queries (A, B, …) and "a query/condition was satisfied but did not trigger or recover as expected" → must follow step 3.5 of flow A ``` --- ## Flow A: The rule produced no alert event ### Step 1 · Pin down the rule, verify its configuration Call `get_alert_rule_detail(id=)` and focus on verifying: | Config item | What happens if it fails | |---|---| | `disabled = 0` (enabled) | A disabled rule is not evaluated | | `datasource_ids` non-empty and the datasource exists | No datasource means no query | | `prom_eval_interval` / `prom_for_duration` | Too long, and the trigger window may not have been reached yet | | `enable_in_bg` (effective only in this business group, host alerts) | A host not in the business group is skipped | | `enable_stime` / `enable_etime` / `enable_days_of_week` | If the current time is outside the effective window, no evaluation occurs | | `cate` + rule config (PromQL / SQL) | Whether the expression can actually return data needs to be verified later | | `rule_config.triggers[].exp` (threshold alert) | **If the exp field is empty**, the alert condition is incompletely configured (common for rules created via API/import), and the rule will never trigger | If any item is not satisfied, **pinpoint directly to this step** and output the report. > **Threshold-alert exp validation tip**: extract the `triggers` array from `rule_config`; each trigger should have a non-empty `exp` (e.g. `$A > 80`). If `exp` is an empty string or missing, that is a direct pinpoint conclusion. ### Step 2 · Verify the datasource path Call `get_datasource_detail(id=)`. Focus on: - Is the datasource status normal? - Is the datasource associated with an alert engine cluster? This is the prerequisite for the rule being managed (the "alert rule -> datasource -> alert engine cluster" chain). ### Step 3 · Actually run the query once, verify whether there is an anomalous point Extract the query expression from the rule config and run it yourself: - **Prometheus type**: use `query_prometheus(query=, query_type='range', time_range='1h')`. First look at the range trend, then use instant to check whether the trigger condition is currently truly met. - **SQL / ES / VictoriaLogs type**: use `query_timeseries`, following the R22+ docs to pass `sql + value_key`, or `index + filter`, or `query`. **Importantly remind the user to check whether the `value_key` field name exactly matches the column in the SQL** (a common pitfall). - **ES `query_string` case pitfall (check this first when an ES log alert behaves unexpectedly)**: `AND` / `OR` / `NOT` must be **uppercase** to be recognized as boolean operators; writing `and` / `or` / `not` makes them treated as ordinary terms, and combined with `query_string`'s default operator being `OR`, the entire query's semantics change completely. - Typical symptoms: the alert match count is far larger than expected; the returned logs include pods / services / levels that **should not match** (e.g. the query is `logLevel:ERROR and ext_pod:"menuglobal-*"` but the results contain large amounts of INFO/WARN, or ERROR logs from other pods). - How to diagnose: copy the rule's ES query statement out and **check character by character whether `and`/`or`/`not` are uppercase**; also run `query_timeseries` once with lowercase and once with uppercase using the same statement — a huge difference in match counts confirms this cause. - Fix: change to uppercase `AND` / `OR` / `NOT`, or switch to the structured `bool.must` / `bool.should` form to avoid the case pitfall. Decision criteria: - **Returns data + meets the condition** → proceed to Step 4 to see why the engine produced no event - **Returns data but does not meet the condition** → report "actual data does not meet the threshold", end - **Returns nothing** → may be data-ingestion delay; ask the user to confirm whether the collection side is healthy; the query expression itself may also be wrong > **Multi-query-rule trap**: if the rule has ≥2 queries (A, B, …), "A returns a value on its own and B returns a value on its own" does **not** mean "it should trigger". Multiple queries must be merged by labels before they participate in the decision; Step 3.5 below specifically handles this category. Do not jump to the conclusion "data is satisfied, it should alert" at this step. ### Step 3.5 · Multi-query/multi-variable threshold-decision verification (mandatory when rule_config.queries ≥ 2) When the rule config has two or more queries (each with its own ref: A, B, …) and the trigger/recovery expression references multiple refs, there is a set of pitfalls **specific to multiple variables**. As long as `rule_config.queries` has length ≥ 2, follow this step. **① First clarify the correspondence between refs and expressions** Extract `queries` (each query has its own ref) and `triggers` from `rule_config`, and for each trigger, understand clearly: - which `$ref` values `triggers[].exp` (the trigger expression) references, e.g. `$A > 0` - which `$ref` values `triggers[].recover_config.judge_type` and `recover_config.recover_exp` (the recovery condition) reference, e.g. `judge_type=recover_on_condition` + `$B > 0` **② Refs in the recovery condition do not fire alerts (semantic clarification)** Only the **trigger expression (exp)** produces alert events; the **recovery condition (recover_exp) only decides when an already-triggered alert recovers**, and the query it references **will never fire an alert on its own**. - Typical misuse: the user configured two queries A and B, put `$A > 0` in the trigger and `$B > 0` in the "recovery condition", and then expects "an alert when B is satisfied too". This is putting the query in the wrong slot, not a data problem. To make B alert independently, B must be added as an **independent trigger condition** (multi-trigger / expression mode), not placed in the recovery condition. - Identification: the user says "the second query/condition has values in the data preview but doesn't trigger", and that ref only appears in `recover_exp` and not in any `exp` → pinpoint directly to this item. **③ The recovery condition references a ref not present in the trigger expression → recovery can never be satisfied (key pitfall, can be confirmed in the eval log)** When the engine evaluates a group of curves, the variable table is only filled with the values of `$ref`s that **appear in the trigger expression exp**; a ref that **only appears in recover_exp and not in exp** (e.g. B when exp=`$A>0`, recover_exp=`$B>0`) **will not be filled into the variable table**. As a result, the recovery condition `$B > 0` computes against an undefined variable, the expression fails to compile, the decision is constantly false, and the **recovery condition is never satisfied**. - **eval log signature**: `get_alert_eval_logs` will show an error line like `exp:$B > 0 data:map[$A:...] error: ... B ...` (variable B undefined). Seeing this confirms it. - Fix: either also write the variable used by the recovery into the trigger expression (so it enters the variable table), or change the recovery method back to the default "recover when the result no longer meets the trigger condition" (origin); do not reference a ref in the recovery condition that the trigger expression never uses. **④ Multiple refs are grouped and merged by "fully identical labels" (the real meaning of the config page's orange hint "ensure all variables have consistent labels")** The engine groups the curves of each query by the **group-by label set (tagHash)**; only when the label sets of A and B are **field-for-field fully identical** do they fall into the same group and jointly participate in the cross-ref expression decision (by default, with no explicit join, tags are unioned). Therefore: - The **group-by dimensions of A and B must be fully identical** (fields and keyword suffixes must all be the same). - Even if the dimensions are identical, if the **filter conditions of A and B differ** so that the returned label **values** differ (e.g. A queries `message:"Disconnecting"`, B queries `message:"Received logon"`, and the `fctags`/`filename` of the two log types are naturally different), their tagHashes do not match and they **can never enter the same group**, so the cross-ref expression (including the recovery condition) cannot be decided. - Diagnostic action: run A and B separately with `query_timeseries`, list the label sets of the series returned by each side, and **manually verify whether there is a pair with fully identical labels**. If not a single pair matches, this is the cause. **Decision and recommendations** - Ref in the wrong slot (B is in the recovery condition but is expected to alert) → explain the semantics, recommend adding B as an independent trigger condition. - The recovery condition references a ref not in the trigger expression → confirm with the eval log error, recommend merging the variable into the trigger expression, or switching back to the origin recovery method. - A and B labels do not align → recommend unifying the group-by dimensions, confirming the two filter conditions can produce curves with identical labels, or simply splitting into two independent rules. ### Step 4 · Pull the alert engine evaluation logs (key step) **This is the single most central tool for R22+ troubleshooting**: ``` get_alert_eval_logs(rule_id=) ``` It returns the engine instance responsible for the rule + the most recent evaluation logs (in reverse chronological order). How to read it: - **Logs empty** → the engine isn't running this rule at all. Check: - whether the datasource is associated with an engine cluster (back to Step 2) - whether the engine instance heartbeat is normal (use `list_alert_engine_instances` from Step 4.5) - **The log contains the text `ERROR ... query`** → an error occurred while querying data (e.g. Prometheus unreachable, SQL error) - **The log shows "no data found"** → data-ingestion delay or a query-expression problem - **The log shows "data found but condition not met"** → the actual data has no anomaly - **The log shows "condition met but sustained duration insufficient"** → the anomalous point did not persist to `prom_for_duration` - **The log shows "event produced but muted"** → go to Step 5 to cross-check mute rules ### Step 4.5 · Alert engine instance health (mandatory when eval logs are empty) ``` list_alert_engine_instances(datasource_id=) ``` It returns each engine instance's `last_heartbeat` / `stale_seconds` / `healthy` fields. Decision: - No instance returned → the datasource is not bound to any engine instance (same chain issue as Step 2) - All instances `healthy=false` (`stale_seconds > 30`) → the process is down, ask the user to restart n9e-server - Multiple `engine_cluster` values or multiple old-version instances heartbeating at the same time → suspect "an old instance was forgotten during the upgrade", ask the user to clean up the old instances ### Step 5 · Cross-check mute rules If the eval logs show the event was muted, or you suspect it was muted: 1. `list_alert_mutes(query=)` to list the mute rules in the same business group 2. `get_alert_mute_detail(id=)` to see each mute rule's matching conditions 3. Cross-check against the event labels (from three parts: time-series data labels + rule append labels + rule name), matching one by one A match pinpoints the cause. ### Step 6 · Engine self-monitoring fallback If everything above is normal but there is still no event, use `query_prometheus` to query n9e's own metric: ``` n9e_alert_eval_query_series_count{rule_id=""} ``` - Metric exists and value > 0 → the engine is indeed running and did fetch data - Metric is 0 → the query returned empty, go back to Step 3 - Metric does not exist → the rule may not be managed by the engine at all --- ## Flow B: An alert event was produced but no notification was received ### Step 1 · Confirm the event exists - Use `search_history_alerts(query=, hours=24)` to find the most recent event - Or `search_active_alerts(query=...)` to see active alerts - Grab the **`hash`** field (not id, the hash), for the next step ### Step 2 · Pull the event's downstream processing logs (key step) ``` get_event_processing_logs(event_hash=) ``` It returns the full chain from event production to notification. How to read it: - whether it entered **notify rule** matching - whether **callback / webhook** was invoked successfully - whether a **subscription** matched - whether it was **muted** at some step - whether a **notification script** ran, and what the script execution result was ### Step 2.5 · Notify-rule validity + level/time-window/label match verification (mandatory when the "notification result" table is entirely empty) > The user seeing **not a single record** in the "notification result / notification_record" table (neither success nor failure) is the most frequent phenomenon in this flow. **Key insight: an empty table ≠ a send failure.** In the engine, only the "silently skipped" path leaves the table empty; if the **channel is disabled or the notification template is missing**, the engine actually writes a **failure record** (notification status = failed), so the table is not empty. Therefore **when the table is entirely empty, first suspect the following "never reached the send step" causes**, rather than checking whether the channel token is correct. The processing logs are an engine black box and whether they record something depends on luck; this step proactively pulls out the notify rules bound to the rule and verifies them independently. In order: **① Does the rule have a notify rule bound at all** Take `notify_rule_ids` from `get_alert_rule_detail`: - **Empty** → the rule has no new-style notify rule bound at all (it may still be on the old-style `notify_groups`/`notify_version=0`, or nobody configured one), so naturally there is no notification record. Ask the user to bind a notify rule on the rule. **② Is each notify rule enabled + does the channel/level/time-window/label match** For each `notify_rule_id`, call `get_notify_rule_detail(id=...)` and verify one by one (the engine's decision criteria are already aligned with the tool fields): - **`enable=false`** → the notify rule is disabled. The engine only loads rules with `enable=true`; disabled ones are directly `continue`d and **leave no record**. **This is the most common and most easily overlooked cause of an entirely empty table — check this first.** - Iterate `notify_configs` and compare each config against the **current event** item by item (if any item is not satisfied, that config is `continue`d and skipped, neither sent nor recorded): - **`severities` does not include the event level** → the event is S1 (severity=1); if `severities` does not include 1, it does not match. **Key pitfall: `severities` being an empty array = it matches no event (not "all levels")**; the engine treats empty severities as a no-match directly — common in notify rules created via API/import. - **`time_ranges` does not cover the trigger moment** → the event triggers at 16:05; if the time window is configured as something like 00:00–09:00, it is not sent; be sure to also verify the **day of week `week`**. (Empty `time_ranges` = no time restriction, matches all.) - **`label_keys` / `attributes` do not match the event labels** → take each filter item (`key`/`op`/`value`) and match it against the event labels one by one; the matching semantics are the same as mute rules (`==`/`=~`/`in`/`!=`/`!~`/`not in`). (Empty `label_keys` = no label filtering, matches all.) - **`channel_enabled=false`** → the channel is disabled. Note: in this case the engine writes a **failure record** ("notify_channel not found"), and a failure row is visible in the table — so if the table is **entirely empty**, a disabled channel is not the primary cause, but it should still be verified. **Decision**: under a single notify rule, as long as **any one** `notify_config` fully matches, a record should be produced; only when **all configs fail to match** does that rule produce no record for the event. Verify every bound notify rule; hitting one of "not bound / rule disabled / all configs fail to match on level-time window-label" pinpoints the direct cause of the notification not being sent. ### Step 3 · Notify-frequency verification (repeat notification / max count) Many "no notification received" cases are actually rate-limited. After taking the rule config from `get_alert_rule_detail`, verify: | Field | Meaning | What happens if it fails | |---|---|---| | `notify_repeat_step` (minutes) | Repeat notification interval | If less than this interval has elapsed since the last notification, it is not sent again; a recovery event resets this interval timer from 0 | | `notify_max_number` | Max notification count | 0 = unlimited; when non-zero, after reaching the count no more notifications are sent and the event must recover to reset | How to verify: - Use `search_history_alerts(query=, hours=720)` (pull a month of history) to count how many times the rule triggered historically, and see whether `notify_max_number` has been exhausted - Look at the time of the most recent notification (if any) and compare against `notify_repeat_step` to decide whether it is still within the silence window ### Step 4 · Event processor (pipeline) execution check If the user configured event processors (event suppression / data enrichment / self-healing pipelines, etc.), they may have dropped or rewritten the event at some step: ``` get_event_pipeline_executions(event_id=) ``` Read each execution record's `status` and `error_message`: - `status=success` and no rewrite → the processor passed it through normally - `status=failed` + `error_message` + `error_node` → a processor node failed, possibly blocking the notification chain - No execution record at all → no pipeline matched this event (if the user expected one, ask them to check the pipeline's matching conditions) ### Step 5 · Other supporting checks - `get_alert_event_detail(event_id=)` to get the event details and inspect the `callbacks` field - If the event has `IsRecovered=1`, it has already recovered; the user may be looking at an event outside the historical time window - If it is a subscription rule, use `list_alert_subscribes` / `get_alert_subscribe_detail` to verify the subscription conditions --- ## Flow C: The user believes "this alert should not have fired" (suspected false-positive determination) The user's description is something like: "the alert shows a trigger value of 95, but when I open the curve it's only 30", "the data clearly had no anomaly, why did it alert", "this alert fires dozens of times a day, always flapping", "the machine is off, why does it say the alert recovered". This category is collectively called "suspected false-positive determination". There are 3 common causes; investigate them in the following order: ### Cause 1 · Downsampled during the instant query A time-series database automatically downsamples when querying over a long time range, causing a mismatch between the "smooth curve" the user sees in the UI and the "raw point" at the alert trigger moment. **How to diagnose**: 1. `get_alert_event_detail(event_id=)` to get the trigger time `trigger_time` and `trigger_value` 2. Use `query_prometheus(query=, query_type='instant', time_range='5m')`, narrowing the time range to around the alert trigger moment (a few minutes before and after), do not pull 1h/6h 3. Compare the instant query result vs `trigger_value`; normally they should be consistent ### Cause 2 · Log-type data ingestion delay; alert moment ≠ query moment Log-type datasources (ES / VictoriaLogs / SLS, etc.) have data-ingestion delay. When the alert engine queries the interval [t-1m, t] at moment t, it fetches a batch of stale data that triggers the alert; when the user later queries the same interval, because new data has since been backfilled, the statistical result changes. **How to diagnose**: 1. Use `get_alert_eval_logs(rule_id=)` to find the evaluation log for that trigger (aligned by `trigger_time`) 2. Look at the `query`/`series`/`value` fields recorded in the log — that is the value the engine **actually queried at the time**; take this as authoritative, not the result the user queried afterward 3. Explain the cause of the delay to the user, and recommend adding a "delay tolerance window" in the alert rule's SQL/query conditions ### Cause 3 · Historical pattern: high-frequency flapping / threshold-edge oscillation / missing-data false recovery Unlike Causes 1/2 which focus on "this one instance", Cause 3 looks at "whether this rule has been **firing normally** over the past period". Three typical forms: - **flapping**: triggers and recovers dozens of times a day, each event lives only a few seconds to a few dozen seconds - **threshold-edge oscillation**: the trigger value always stays in a narrow band of threshold ±5%, repeatedly crossing the line - **missing-data false recovery**: a machine shutdown / collection gap causes "no data" to be treated as "recovered", and then the data comes back and re-alerts **How to diagnose**: 1. Use `search_history_alerts(rid=, hours=168)` to pull all events of this rule over the past 7 days 2. On the results, compute: - total trigger count (number of repeats with the same hash) - average "produced → auto-recovered" interval (first_trigger_time to recover_time) - distribution of trigger values: min / max / whether concentrated in the narrow band of threshold ±5% - whether there is a repeated "recover → trigger again immediately" pattern (a short interval between recover_time and the next first_trigger_time) 3. Hitting any of the following signals marks it as "suspected flapping / false-positive type": - average lifetime < 2 × `prom_eval_interval` (recovers after one or two evaluation cycles) - same hash triggered > 30 times within 7 days - trigger values concentrated in the narrow band of threshold ±5% - "recover → re-trigger" repeats with an interval < 5 minutes 4. If you suspect "missing-data false recovery", use `get_alert_eval_logs(rule_id=)` to look at the evaluation logs around the recovery: seeing `series=0` (no data found) in the log immediately followed by a recovery almost confirms this cause. ### Direct pinpoint conclusions - If Cause 1 → tell the user to re-verify with a shorter time range or an instant query - If Cause 2 → the alert trigger value is the real value at the time; the curve the user sees afterward is the result after backfill; both are correct, the alert is fine - If Cause 3 · flapping / threshold-edge oscillation → recommend increasing `prom_for_duration` (sustained duration) or `RecoverDuration` (observation hold time), or raising the threshold away from the oscillation band - If Cause 3 · missing-data false recovery → explain the Prometheus-style "no data = recovered" mechanism, recommend adding `RecoverDuration` to delay the recovery event, or use a dedicated host-disconnection rule to handle the lost-contact/shutdown scenario, avoiding mixing it with business alerts --- ## Output Report Template After troubleshooting, output: ```markdown ## Alert Rule Troubleshooting Report ### 1. Subject - **Rule**: (id=) - **Business group**: - **Datasource**: (id=, type=) - **Phenomenon**: ### 2. Troubleshooting Process 1. **Rule config verification**: — ✅/❌ 2. **Datasource path**: — ✅/❌ 3. **Actual data verification**: - Query expression: `` - Query result: 4. **Engine evaluation logs** (get_alert_eval_logs): - Engine instance: - Key logs: 5. **Mute-rule verification** (if any): 6. **Event processing logs** (get_event_processing_logs, for flow B): - Key logs: 7. **Notify-rule verification** (get_notify_rule_detail, when flow B's "notification result" is empty): - Bound notify_rule_ids: - Each rule's enable status, matched/unmatched notify_config (level/time window/label match result): ### 3. Pinpoint Conclusion - **Direct cause**: - **Evidence**: ### 4. Recommended Actions - **Immediate fix**: - **Follow-up**: ``` --- ## Common Error Patterns Quick Reference | Phenomenon (from eval logs / actual query) | Direct cause | |---|---| | eval logs empty, and the self-monitoring metric is also absent | The datasource is not associated with an engine cluster, or the rule is disabled | | eval logs contain `ERROR ... query` | Datasource connection error / SQL syntax error | | eval logs show `series=0` | The query result is empty: data-ingestion delay or expression error | | eval logs show the condition is met but no event written to DB | Sustained duration insufficient, muted, or outside the effective time | | eval logs show `event muted` | A mute rule matched | | event exists but processing logs show `notify skipped` | Notify rule does not match / subscription did not match | | event exists but processing logs show a callback error | Third-party interface (IM / webhook) error | | event exists, processing logs have no notification action | No notify_rule bound, rate-limited by notify_repeat_step, or reached notify_max_number | | The "notification result" table is **entirely empty** (neither success nor failure) | It took the "silently skipped" path: notify rule `enable=false` / rule not bound to notify_rule_ids / all notify_configs fail to match on level-time window-label / rate-limited / dropped by pipeline; verify one by one with get_notify_rule_detail (Step 2.5). Note: empty table ≠ send failure | | get_notify_rule_detail shows `enable=false` | The notify rule is disabled; the engine only loads enabled rules, so it neither sends nor records | | notify_config's `severities` is an empty array | The engine determines "matches no event" (not "all levels"); that config never sends; common for API/import creation — fill in the event level for severities | | Event level/labels/trigger moment do not match the notify_config's severities/label_keys/time_ranges | That config is skipped; if all configs under the rule fail to match, the whole rule produces no record | | The "notification result" table has a **failure record** (notification status = failed) | Not a match problem but a send problem: channel disabled (channel_enabled=false) / channel deleted / notification template missing / third-party interface error; look at the details in the record | | pipeline executions have a `status=failed` | An event-processor node errored and blocked the chain; look at error_node/error_message | | All engine instances have `stale_seconds > 30` | The n9e-server process is down, restart it | | Multiple engine_cluster values heartbeating at the same time | An old-version instance was not taken offline and may be contending for rule management | | exp is empty in rule_config.triggers | The threshold-alert condition was not fully configured, it will never trigger | | User says "the trigger value doesn't match the curve" | Instant-query downsampling / log-data ingestion delay; take the value recorded in eval logs as authoritative | | Same hash triggered > 30 times within 7 days / average lifetime < 2 evaluation cycles | flapping-type false positive; increase prom_for_duration or RecoverDuration | | Trigger value always stays in the narrow band of threshold ±5% | Threshold-edge oscillation; raise the threshold away from the oscillation band | | eval logs before recovery show `series=0` | Recovery determined due to missing data (Prometheus-style "no data = recovered"); use RecoverDuration to delay the recovery event, or handle it with a dedicated host-disconnection rule | | ES log alert match count far exceeds expectation / results include pods or levels that should not match | `and`/`or`/`not` in `query_string` were written in lowercase, treated as terms + default OR concatenation; change to uppercase `AND`/`OR`/`NOT` or switch to `bool.must` | | Multi-query rule, B has values when previewed alone but "doesn't trigger", and B only appears in the recovery condition | A ref in the recovery condition does not fire alerts, it only controls recovery; to make B alert, add it as an independent trigger condition (flow A Step 3.5②) | | eval log shows `exp:$B > 0 data:map[$A:...] error: ...B...` | The recovery condition references a ref not in the trigger expression; the variable never entered the table, so recovery is constantly false; merge the variable into the trigger exp or switch back to origin recovery (Step 3.5③) | | Multi-query A/B each have data, but the cross-ref expression/recovery is never decided | A and B's group-by dimensions or the labels produced by their filters have inconsistent values; the tagHashes do not align so they never enter the same group; unify group-by or split into independent rules (Step 3.5④) | --- ## Safety and Boundaries 1. **Read-only**: this skill does not call any create/modify tools. 2. **Focus on a single rule**: do not troubleshoot multiple rules at once; first have the user pick the single most critical one — serial troubleshooting is more accurate than parallel. 3. **Do not echo the user's guesses**: do not get led astray by "I think it's an xxx problem"; follow the flowchart step by step, and base conclusions on the logs. 4. **R22+ only**: if the user explicitly says R21-, tell them this skill does not cover it and refer them to the official documentation. --- ## Worked Example > User: "Rule 833 'Insufficient disk space' — I waited 10 minutes and still didn't receive an alert, please take a look." **Step 1**: `get_alert_rule_detail(id=833)` → The rule is enabled, cate=prometheus, PromQL=`sum(disk_free) > 1`, sustained duration 60s, associated with ds_id=5. **Step 2**: `get_datasource_detail(id=5)` → Prometheus, associated with alert engine cluster default, status normal. **Step 3**: `query_prometheus(query='sum(disk_free) > 1', query_type='instant', time_range='5m')` → Returns empty, no series meet the condition. **Step 4**: `get_alert_eval_logs(rule_id=833)` → The logs show `series=0`; every evaluation found no data. **Conclusion**: The rule's PromQL expression may be semantically wrong (`sum(disk_free) > 1` holds in any normal environment, but the current instance's disk_free reporting field unit/labels may not line up). Recommend the user go to the instant-query page and use the Table view to verify the actual labels and values of the `disk_free` metric.