Debug Production
Deep investigation of production issues in Automation Analytics. Gathers pod status, logs, metrics, alerts, recent deployments, and suggests likely causes.
Kubeconfig Rules
NEVER copy kubeconfig files.
| File |
Environment |
~/.kube/config.s |
Stage |
~/.kube/config.p |
Production |
Use --kubeconfig=~/.kube/config.p for prod.
Inputs
| Input |
Type |
Default |
Purpose |
namespace |
string |
- |
main or billing (ask if missing) |
alert_name |
string |
- |
Prometheus alert if triggered by alert |
pod_filter |
string |
- |
Filter pods (e.g., fastapi, processor) |
time_range |
string |
1h |
Log search window (15m, 1h, 6h, 24h) |
issue_key |
string |
- |
Jira key to attach session context |
Persona
Load incident persona (prometheus, alertmanager, kibana, k8s, grafana).
Workflow
1. Bootstrap
persona_load("incident")check_known_issues("kubectl_get_pods")check_known_issues("kibana_search_logs")- If no namespace: ask user (main vs billing)
2. Load Patterns
memory_read("learned/patterns") → error_patterns
3. Pod Status
kubectl_get_pods(namespace, environment="production")kubectl_top_pods(namespace, environment="production")- Parse: CrashLoopBackOff, OOMKilled, Error, Pending, ImagePullBackOff, high restarts
4. Events
kubectl_get_events(namespace, environment="production")- Filter: warning, error, failed, killed, backoff, unhealthy
5. Pod Logs
- For unhealthy pods (up to 3):
kubectl_describe_pod, kubectl_logs(pod, namespace, tail=500, since=time_range)
- Extract error/exception/traceback lines
6. Alerts
alertmanager_alerts(environment="production")- If alert_name: search app-interface alert YAML for definition
7. Kibana Logs
kibana_search_logs(query="error OR exception OR traceback", namespace, environment="production", limit=20)kibana_get_errors(namespace, environment="production")kibana_trace_request — if request_id in summary
8. Slack Context
persona_load("slack") → slack_find_channel, slack_list_messages (team-automation-analytics)persona_load("incident") — restore
9. Metrics
prometheus_query — 5xx error rateprometheus_query_range — error rate trend, memory trendprometheus_pod_health, prometheus_rulesgrafana_dashboard_list, grafana_dashboard_get, prometheus_grafana_link
10. Deployments
kubectl_get_deployments, kubectl_get(resource="replicasets")- Check app-interface CICD config for deployed SHA
- Check namespace config from app-interface
11. Knowledge & Code Search
code_search(query=error_from_logs, project="automation-analytics-backend")knowledge_query(project="automation-analytics-backend", persona="devops", section="patterns.deployment")knowledge_query(..., section="gotchas")
12. Match Patterns
Match pod issues against known_patterns. Add generic suggestions: OOMKilled, CrashLoopBackOff, ImagePullBackOff.
13. Attach to Jira
- If
issue_key: jira_attach_session(issue_key, include_transcript=true)
14. Log & Update
memory_session_log("Debugged production", ...)- Update
state/environments
15. Failure Recovery
- k8s "unauthorized" →
kube_login(cluster="prod")
- "no route to host" →
vpn_connect()
- Prometheus "connection refused" → VPN/auth
- Kibana 401 → check credentials
learn_tool_fix(...) for patterns
Output
Report: pod status, resource usage, events, error logs, Kibana errors, metrics, deployments, pattern matches, Grafana/Kibana links, suggested Prometheus queries.
Chains To
hotfix, rollout_restart, scale_deploymentsilence_alert, create_jira_issue, learn_pattern
1---2name: debug-prod3description: Systematic production investigation - pods, logs, metrics, alerts, deployments, Kibana, Slack context. Suggests likely causes from patterns. Use when user says "debug prod", "investigate production issue", or escalated from investigate_alert.4---56# Debug Production78Deep investigation of production issues in Automation Analytics. Gathers pod status, logs, metrics, alerts, recent deployments, and suggests likely causes.910## Kubeconfig Rules1112**NEVER copy kubeconfig files.**1314| File | Environment |15|------|-------------|16| `~/.kube/config.s` | Stage |17| `~/.kube/config.p` | Production |1819Use `--kubeconfig=~/.kube/config.p` for prod.2021## Inputs2223| Input | Type | Default | Purpose |24|-------|------|---------|---------|25| `namespace` | string | - | `main` or `billing` (ask if missing) |26| `alert_name` | string | - | Prometheus alert if triggered by alert |27| `pod_filter` | string | - | Filter pods (e.g., fastapi, processor) |28| `time_range` | string | 1h | Log search window (15m, 1h, 6h, 24h) |29| `issue_key` | string | - | Jira key to attach session context |3031## Persona3233Load **incident** persona (prometheus, alertmanager, kibana, k8s, grafana).3435## Workflow3637### 1. Bootstrap38- `persona_load("incident")`39- `check_known_issues("kubectl_get_pods")`40- `check_known_issues("kibana_search_logs")`41- If no namespace: ask user (main vs billing)4243### 2. Load Patterns44- `memory_read("learned/patterns")` → error_patterns4546### 3. Pod Status47- `kubectl_get_pods(namespace, environment="production")`48- `kubectl_top_pods(namespace, environment="production")`49- Parse: CrashLoopBackOff, OOMKilled, Error, Pending, ImagePullBackOff, high restarts5051### 4. Events52- `kubectl_get_events(namespace, environment="production")`53- Filter: warning, error, failed, killed, backoff, unhealthy5455### 5. Pod Logs56- For unhealthy pods (up to 3): `kubectl_describe_pod`, `kubectl_logs(pod, namespace, tail=500, since=time_range)`57- Extract error/exception/traceback lines5859### 6. Alerts60- `alertmanager_alerts(environment="production")`61- If alert_name: search app-interface alert YAML for definition6263### 7. Kibana Logs64- `kibana_search_logs(query="error OR exception OR traceback", namespace, environment="production", limit=20)`65- `kibana_get_errors(namespace, environment="production")`66- `kibana_trace_request` — if request_id in summary6768### 8. Slack Context69- `persona_load("slack")` → `slack_find_channel`, `slack_list_messages` (team-automation-analytics)70- `persona_load("incident")` — restore7172### 9. Metrics73- `prometheus_query` — 5xx error rate74- `prometheus_query_range` — error rate trend, memory trend75- `prometheus_pod_health`, `prometheus_rules`76- `grafana_dashboard_list`, `grafana_dashboard_get`, `prometheus_grafana_link`7778### 10. Deployments79- `kubectl_get_deployments`, `kubectl_get(resource="replicasets")`80- Check app-interface CICD config for deployed SHA81- Check namespace config from app-interface8283### 11. Knowledge & Code Search84- `code_search(query=error_from_logs, project="automation-analytics-backend")`85- `knowledge_query(project="automation-analytics-backend", persona="devops", section="patterns.deployment")`86- `knowledge_query(..., section="gotchas")`8788### 12. Match Patterns89Match pod issues against known_patterns. Add generic suggestions: OOMKilled, CrashLoopBackOff, ImagePullBackOff.9091### 13. Attach to Jira92- If `issue_key`: `jira_attach_session(issue_key, include_transcript=true)`9394### 14. Log & Update95- `memory_session_log("Debugged production", ...)`96- Update `state/environments`9798### 15. Failure Recovery99- k8s "unauthorized" → `kube_login(cluster="prod")`100- "no route to host" → `vpn_connect()`101- Prometheus "connection refused" → VPN/auth102- Kibana 401 → check credentials103- `learn_tool_fix(...)` for patterns104105## Output106107Report: pod status, resource usage, events, error logs, Kibana errors, metrics, deployments, pattern matches, Grafana/Kibana links, suggested Prometheus queries.108109## Chains To110111- `hotfix`, `rollout_restart`, `scale_deployment`112- `silence_alert`, `create_jira_issue`, `learn_pattern`
dmzoneill