Investigate Slack Alert
Investigate Prometheus alerts posted in Slack channels. Acknowledges immediately, checks pods/logs, creates or links Jira, replies with summary.
Kubeconfig Rules
| File |
Environment |
~/.kube/config.s |
Stage |
~/.kube/config.p |
Production |
Inputs
| Input |
Type |
Purpose |
channel_id |
string |
Slack channel ID |
message_ts |
string |
Message timestamp (threading) |
message_text |
string |
Alert message content |
alert_url |
string |
Optional AlertManager URL |
Persona Switches
- incident — K8s, kubectl, Jira
- slack —
slack_send_message for ack and reply
Workflow
1. Bootstrap
persona_load("incident")knowledge_query(project="automation-analytics-backend", persona="devops", section="gotchas")check_known_issues("alert", "")
2. Load Config
From config.json → slack.listener.alert_channels[channel_id]:
- environment, namespace, cluster
3. Parse Alert
Extract from message: alert_name, firing_count, description, links, is_billing, namespace.
Billing keywords: billing, subscription, vcpu, etc.
4. Acknowledge
persona_load("slack")slack_send_message(target=channel_id, thread_ts=message_ts, text="👀 Looking into this...")
5. Investigate
persona_load("incident")kubectl_get_pods(namespace, environment) — pod statuskubectl_logs(namespace, selector="app=automation-analytics-processor-ingress", tail=50) — if processor/error podscode_search(query=alert_name, project="automation-analytics-backend", limit=5)
6. Search Jira
jira_search(jql="project = AAP AND summary ~ '{alert_name}' AND status NOT IN (Done, Closed)")- If billing:
jira_search(jql="project = AAP AND summary ~ 'BillingEvent'") — get next BillingEvent number
7. Create or Link Jira
- If no match:
skill_run("create_jira_issue", summary, description, issue_type, labels)
- Billing format:
BillingEvent XXXXX - [Processor] Error: ...
8. Reply to Slack
persona_load("slack")- Build response: alert, env, namespace, pod status, errors, Jira link, quick links
slack_send_message(target=channel_id, thread_ts=message_ts, text=response)
9. Log & Update
memory_session_log("Investigated Slack alert", ...)- Update
state/environments
10. Failure Recovery
- k8s "forbidden"/"unauthorized" →
kube_login("stage"|"prod")
- "no route to host" →
vpn_connect()
- Kibana auth → open Kibana in browser
- Slack "not_in_channel" → invite bot to channel
Billing Alert Special Handling
- Higher priority
- Format:
BillingEvent XXXXX - [Processor] Error: description
- Numbered sequentially from existing billing events
Output
Summary with: alert, environment, unhealthy pods, error patterns, Jira key/url, Slack reply status.
1---2name: investigate-slack-alert3description: Investigate Prometheus alerts from Slack, create/link Jira issues, reply with findings. Triggered by app-sre-alerts in alert channels. Billing alerts get special format. Use when user says "investigate Slack alert" or "look into this alert".4---56# Investigate Slack Alert78Investigate Prometheus alerts posted in Slack channels. Acknowledges immediately, checks pods/logs, creates or links Jira, replies with summary.910## Kubeconfig Rules1112| File | Environment |13|------|-------------|14| `~/.kube/config.s` | Stage |15| `~/.kube/config.p` | Production |1617## Inputs1819| Input | Type | Purpose |20|-------|------|---------|21| `channel_id` | string | Slack channel ID |22| `message_ts` | string | Message timestamp (threading) |23| `message_text` | string | Alert message content |24| `alert_url` | string | Optional AlertManager URL |2526## Persona Switches2728- **incident** — K8s, kubectl, Jira29- **slack** — `slack_send_message` for ack and reply3031## Workflow3233### 1. Bootstrap34- `persona_load("incident")`35- `knowledge_query(project="automation-analytics-backend", persona="devops", section="gotchas")`36- `check_known_issues("alert", "")`3738### 2. Load Config39From `config.json` → `slack.listener.alert_channels[channel_id]`:40- environment, namespace, cluster4142### 3. Parse Alert43Extract from message: alert_name, firing_count, description, links, is_billing, namespace.44Billing keywords: billing, subscription, vcpu, etc.4546### 4. Acknowledge47- `persona_load("slack")`48- `slack_send_message(target=channel_id, thread_ts=message_ts, text="👀 Looking into this...")`4950### 5. Investigate51- `persona_load("incident")`52- `kubectl_get_pods(namespace, environment)` — pod status53- `kubectl_logs(namespace, selector="app=automation-analytics-processor-ingress", tail=50)` — if processor/error pods54- `code_search(query=alert_name, project="automation-analytics-backend", limit=5)`5556### 6. Search Jira57- `jira_search(jql="project = AAP AND summary ~ '{alert_name}' AND status NOT IN (Done, Closed)")`58- If billing: `jira_search(jql="project = AAP AND summary ~ 'BillingEvent'")` — get next BillingEvent number5960### 7. Create or Link Jira61- If no match: `skill_run("create_jira_issue", summary, description, issue_type, labels)`62- Billing format: `BillingEvent XXXXX - [Processor] Error: ...`6364### 8. Reply to Slack65- `persona_load("slack")`66- Build response: alert, env, namespace, pod status, errors, Jira link, quick links67- `slack_send_message(target=channel_id, thread_ts=message_ts, text=response)`6869### 9. Log & Update70- `memory_session_log("Investigated Slack alert", ...)`71- Update `state/environments`7273### 10. Failure Recovery74- k8s "forbidden"/"unauthorized" → `kube_login("stage"|"prod")`75- "no route to host" → `vpn_connect()`76- Kibana auth → open Kibana in browser77- Slack "not_in_channel" → invite bot to channel7879## Billing Alert Special Handling8081- Higher priority82- Format: `BillingEvent XXXXX - [Processor] Error: description`83- Numbered sequentially from existing billing events8485## Output8687Summary with: alert, environment, unhealthy pods, error patterns, Jira key/url, Slack reply status.
dmzoneill