Investigate Alert
Quick investigation of a firing Prometheus alert. For deep investigation, use debug_prod directly.
Kubeconfig Rules
NEVER copy kubeconfig files. Use the correct file per environment:
| File |
Environment |
~/.kube/config.s |
Stage |
~/.kube/config.p |
Production |
Use --kubeconfig=~/.kube/config.s or config.p with kubectl/oc.
Inputs
| Input |
Type |
Default |
Purpose |
environment |
string |
required |
stage, production, or prod |
namespace |
string |
main |
main or billing |
alert_name |
string |
- |
Specific alert to investigate |
auto_escalate |
bool |
true |
Auto-run debug_prod if critical |
Persona
Load incident persona (prometheus, alertmanager, kibana, k8s tools).
Workflow
1. Bootstrap
persona_load("incident")check_known_issues("prometheus_query")check_known_issues("kubectl_get_pods")
2. Resolve Namespace
From config.json → namespaces:
- Stage →
tower-analytics-stage
- Prod main →
tower-analytics-prod
- Prod billing →
tower-analytics-prod-billing
3. Get Alerts
prometheus_alerts(environment, namespace, state="firing")- Parse severity (critical, high, low)
4. Quick Health Check
k8s_namespace_health(namespace, environment) or kubectl_get_pods(namespace, environment)kubectl_top_pods(namespace, environment) — CPU/memory usagekubectl_get_events(namespace, environment, field_selector="type=Warning")prometheus_query_range — error rate trend over last hour
5. Search Logs
kibana_search_logs(query="error OR exception OR critical", namespace, environment, limit=10)kibana_get_errors(namespace, environment)
6. Check Known Patterns
memory_read("learned/patterns") — match alerts/pod issues against error_patterns
7. Knowledge & Code Search
code_search(query=alert_name, project="automation-analytics-backend", limit=5)knowledge_query(project="automation-analytics-backend", persona="devops", section="gotchas")
8. Check Existing Silences
alertmanager_silences(environment)alertmanager_receivers(environment)
9. Assess Severity
- Critical/high + unhealthy pods → needs_escalation
- If
auto_escalate and production: skill_run("debug_prod", ...)
10. Notify & Log
- If critical:
skill_run("notify_team", template="alert", ...)
memory_session_log("Investigated alerts", ...)- Update
state/environments with status
11. Failure Recovery
- "connection refused" →
vpn_connect()
- "unauthorized" on k8s →
kube_login(cluster="stage"|"prod")
- Kibana 401 → open Kibana in browser first
learn_tool_fix(...) for recurring patterns
Output
Report with: alerts count, pod health, resource usage, events, Kibana errors, pattern matches, silence recommendation, escalation status.
Chains To
debug_prod — deep investigationsilence_alert — if known issuecreate_jira_issue — track alertrollout_restart, scale_deployment — remediation
1---2name: investigate-alert3description: Quick triage of firing Prometheus alerts. Gets alerts, pod health, events, Kibana logs, known patterns. Escalates to debug_prod for serious issues. Use when user says "what's firing?", "check alerts", "any alerts?", or "investigate alert".4---56# Investigate Alert78Quick investigation of a firing Prometheus alert. For deep investigation, use `debug_prod` directly.910## Kubeconfig Rules1112**NEVER copy kubeconfig files.** Use the correct file per environment:1314| File | Environment |15|------|-------------|16| `~/.kube/config.s` | Stage |17| `~/.kube/config.p` | Production |1819Use `--kubeconfig=~/.kube/config.s` or `config.p` with kubectl/oc.2021## Inputs2223| Input | Type | Default | Purpose |24|-------|------|---------|---------|25| `environment` | string | required | `stage`, `production`, or `prod` |26| `namespace` | string | main | `main` or `billing` |27| `alert_name` | string | - | Specific alert to investigate |28| `auto_escalate` | bool | true | Auto-run debug_prod if critical |2930## Persona3132Load **incident** persona (prometheus, alertmanager, kibana, k8s tools).3334## Workflow3536### 1. Bootstrap37- `persona_load("incident")`38- `check_known_issues("prometheus_query")`39- `check_known_issues("kubectl_get_pods")`4041### 2. Resolve Namespace42From `config.json` → `namespaces`:43- Stage → `tower-analytics-stage`44- Prod main → `tower-analytics-prod`45- Prod billing → `tower-analytics-prod-billing`4647### 3. Get Alerts48- `prometheus_alerts(environment, namespace, state="firing")`49- Parse severity (critical, high, low)5051### 4. Quick Health Check52- `k8s_namespace_health(namespace, environment)` or `kubectl_get_pods(namespace, environment)`53- `kubectl_top_pods(namespace, environment)` — CPU/memory usage54- `kubectl_get_events(namespace, environment, field_selector="type=Warning")`55- `prometheus_query_range` — error rate trend over last hour5657### 5. Search Logs58- `kibana_search_logs(query="error OR exception OR critical", namespace, environment, limit=10)`59- `kibana_get_errors(namespace, environment)`6061### 6. Check Known Patterns62- `memory_read("learned/patterns")` — match alerts/pod issues against error_patterns6364### 7. Knowledge & Code Search65- `code_search(query=alert_name, project="automation-analytics-backend", limit=5)`66- `knowledge_query(project="automation-analytics-backend", persona="devops", section="gotchas")`6768### 8. Check Existing Silences69- `alertmanager_silences(environment)`70- `alertmanager_receivers(environment)`7172### 9. Assess Severity73- Critical/high + unhealthy pods → needs_escalation74- If `auto_escalate` and production: `skill_run("debug_prod", ...)`7576### 10. Notify & Log77- If critical: `skill_run("notify_team", template="alert", ...)`78- `memory_session_log("Investigated alerts", ...)`79- Update `state/environments` with status8081### 11. Failure Recovery82- "connection refused" → `vpn_connect()`83- "unauthorized" on k8s → `kube_login(cluster="stage"|"prod")`84- Kibana 401 → open Kibana in browser first85- `learn_tool_fix(...)` for recurring patterns8687## Output8889Report with: alerts count, pod health, resource usage, events, Kibana errors, pattern matches, silence recommendation, escalation status.9091## Chains To9293- `debug_prod` — deep investigation94- `silence_alert` — if known issue95- `create_jira_issue` — track alert96- `rollout_restart`, `scale_deployment` — remediation
dmzoneill