K8S Incident Triage

Kubernetes Incident Triage

Performs read-only triage of Kubernetes workloads and namespaces using kubectl.

Supports:

• Natural language input (human mode)
• Structured JSON input (automation mode)

This skill NEVER mutates cluster state.

Tool Usage

This skill uses cli_execute with kubectl commands exclusively. NEVER use http_request or web_search to interact with Kubernetes. All cluster operations MUST go through kubectl via the cli_execute tool.

Tool: k8s_triage

Diagnose unhealthy Kubernetes workloads, pods, or namespaces.

Input Modes

1) Human Mode (Natural Language)

Input is a plain string.

Examples:

• triage payments-prod
• triage deployment payments-api in payments-prod
• why are pods pending in checkout-prod?
• investigate crashloop in payments-prod
• triage pod api-7c9f6d7f86-abcde in payments-prod
• check rollout of deployment payments-api in prod

Behavior:

• Parse namespace, workload, pod, or selector intent.
• If namespace omitted, use $DEFAULT_NAMESPACE if set.
• If ambiguity exists, default to namespace-level triage.
• Never require the user to remember JSON fields.

2) Automation Mode (Structured JSON)

Input JSON schema:

{ "namespace": "payments-prod", "workload_kind": "deployment", "workload_name": "payments-api", "pod_name": null, "label_selector": null, "include_logs": true, "logs_tail_lines": 200, "include_previous_logs": true, "events_limit": 50, "include_node_diagnostics": true, "include_metrics": false, "output_format": "markdown" }

Rules:

• namespace is required.
• If pod_name provided → pod-level triage.
• If workload fields provided → workload-level triage.
• Else → namespace scan.

Triage Process

Step 0 — Preconditions

Verify cluster access:

kubectl version --client kubectl cluster-info

If RBAC denies access:

• Continue with allowed operations.
• Explicitly report denied commands in output.

Step 1 — Fast Health Snapshot

Namespace scope:

kubectl get pods -n -o wide kubectl get deploy,sts,ds,job,cronjob -n

Workload scope:

kubectl get -n kubectl rollout status / -n --timeout=10s

Select pods in states:

• CrashLoopBackOff
• ImagePullBackOff
• ErrImagePull
• Pending / Unschedulable
• Error
• NotReady
• OOMKilled
• High restart count

Limit deep triage to $TRIAGE_MAX_PODS (default 5).

Step 2 — Events Timeline

kubectl get events -n --sort-by=.lastTimestamp | tail -n

Look for:

• FailedScheduling
• FailedMount
• Unhealthy (probe failures)
• Back-off pulling image
• Evicted
• Node pressure signals

Step 3 — Describe Pods & Workloads

For each selected pod:

kubectl describe pod -n

Capture:

• Container state and reason
• Restart count
• Last termination reason
• Probe failures
• Volume mount issues
• Node assignment
• Taints / tolerations
• Affinity constraints

If workload-level triage:

kubectl describe -n

Step 4 — Node Diagnostics (Optional)

kubectl get nodes -o wide kubectl describe node

Check for:

• NotReady
• MemoryPressure
• DiskPressure
• PIDPressure
• Evictions

Step 5 — Logs (Optional)

kubectl logs -n --all-containers --tail=

If restart loops and include_previous_logs=true:

kubectl logs -n --previous --all-containers --tail=

Rules:

• Prefer previous logs for CrashLoopBackOff.
• Redact obvious sensitive patterns.
• Never print Secret values.

Step 6 — Optional Metrics

If enabled:

kubectl top pods -n kubectl top node

Gracefully skip if metrics-server is unavailable.

Detection Heuristics

Classify detected issues into:

• CrashLoop / Application Crash
• OOMKilled / Resource Limits
• Image Pull Failure
• Scheduling Constraint
• Probe Failure
• PVC / Volume Failure
• Node Pressure / Eviction
• Rollout Stuck
• Unknown / Escalation Needed

For each issue provide:

• Hypothesis
• Supporting evidence
• Confidence score (0.0–1.0)
• Recommended next commands

Output Structure

1) Incident Summary

• Namespace
• Scope (pod/workload/namespace)
• Affected resources count
• Timestamp

2) Top Findings

Concise bullet summary.

3) Likely Root Causes (Top 3)

For each:

• Description
• Evidence excerpt
• Confidence
• Recommended actions

4) Next Commands

Copy-paste kubectl commands.

5) Evidence Appendix

• Events (recent)
• Describe excerpts
• Log excerpts

Safety Constraints

This skill MUST:

• Perform read-only kubectl operations only.
• Never execute:
  • apply
  • patch
  • delete
  • exec
  • port-forward
  • scale
  • rollout restart
• Never print Secret values.
• Avoid dumping full environment variables.

Autonomous Compatibility

This skill is designed to be invoked by:

• k8s_alert_handler (alert-triggered triage)
• k8s_patrol (scheduled patrol scans)
• Humans via natural language CLI

It must:

• Be idempotent
• Produce deterministic fingerprints
• Avoid excessive cluster load
• Limit deep triage scope