K8S Aiops

k8s AIops

Disclaimer: This is a community-maintained open-source project and is not affiliated with, endorsed by, or sponsored by the Cloud Native Computing Foundation, the Kubernetes project, or k3s/Rancher. "Kubernetes" and "k3s" are trademarks of their respective owners. Source code is publicly auditable at github.com/AIops-tools/K8s-AIops under the MIT license.

Governed Kubernetes operations — 55 MCP tools, every one wrapped with the bundled @governed_tool harness: a local unified audit log under ~/.k8s-aiops/, a token/runaway budget guard, undo-token recording, and a descriptive risk-tier label on every audit row. Works with any kubeconfig-reachable cluster (standard Kubernetes, k3s, EKS, GKE, AKS). Run k8s-aiops init for a friendly onboarding wizard that registers your kube contexts as named targets.

Standalone: the governance harness is bundled in the package (k8s_aiops.governance) — k8s-aiops has no external skill-family dependency. Coverage focuses on common operations and is not yet exhaustive.

What This Skill Does

Quick Install

uv tool install k8s-aiops
k8s-aiops init            # friendly wizard: register your kube contexts as targets
k8s-aiops doctor          # or skip init — works with your current kube-context too

When to Use This Skill

• List/inspect pods, deployments, services, nodes, namespaces and recent events
• Read a pod's recent log lines to diagnose a crash loop
• Run a read-only RCA sweep (diagnose pod-health / diagnose workload-readiness) to find the root cause worst-first
• Scale a deployment up/down, or trigger a rolling restart
• Delete a stuck pod (a controller recreates it) or a deployment
• Cordon a node before maintenance, then uncordon it after

Do NOT use when the target is not a Kubernetes cluster (hypervisor VM lifecycle, backup products, or cloud-provider consoles are out of scope for this skill).

Related Skills — Skill Routing

Common Workflows

Diagnose a crash-looping pod and restart its deployment

1. k8s-aiops pod list -n prod → find the pod with high restarts / non-Running phase
2. k8s-aiops pod logs <pod> -n prod --tail 200 → read the recent logs for the crash cause
3. k8s-aiops events -n prod → check for FailedScheduling / image-pull events
4. k8s-aiops deployment restart <deploy> -n prod → roll the deployment after fixing the cause
5. Failure branch: if logs/events show an RBAC 403, the kube context lacks the verb — run kubectl auth can-i get pods -n prod and switch to a context with adequate RBAC; the skill never retries a denied auth.

Triage an unhealthy namespace with RCA, then act on the worst finding

1. k8s-aiops diagnose pod-health -n prod → worst-first findings; a critical CrashLoopBackOff on prod/api cites restarts=9 and the exact kubectl logs … --previous action
2. k8s-aiops diagnose workload-readiness -n prod → confirm the blast radius: e.g. Deployment web ready 0/3 (critical, under-replicated)
3. k8s-aiops pod logs api-<hash> -n prod --tail 200 --previous-equivalent via k8s-aiops pod describe api-<hash> -n prod → read the crash cause the RCA pointed you at
4. k8s-aiops deployment restart web -n prod → roll the deployment once the root cause is fixed
5. Failure branch: if diagnose returns an RBAC 403, the kube context cannot list pods/deployments in that namespace — run kubectl auth can-i list pods -n prod and switch to a context with adequate RBAC; the RCA tools are read-only and never retry a denied auth.

Drain a node for maintenance, safely reversible

1. k8s-aiops node list → identify the node and confirm it is Ready/schedulable
2. k8s-aiops node cordon <node> --dry-run → preview, then k8s-aiops node cordon <node> (double confirm) — records an inverse uncordon_node undo descriptor
3. After maintenance: k8s-aiops node uncordon <node> → re-enable scheduling
4. Failure branch: if doctor shows the cluster unreachable, fix the kubeconfig context (kubectl config get-contexts) before retrying — cordon is never issued against an unauthenticated session.

Usage Mode

MCP environment caveat: MCP clients spawn the server with a CLEAN environment — shell exports may not reach it. Set K8S_AIOPS_HOME, K8S_AUDIT_APPROVED_BY, K8S_AUDIT_RATIONALE (and KUBECONFIG when the kubeconfig is not at ~/.kube/config) in the MCP server config's env block, not just in your terminal.

MCP Tools (55 — 39 read, 16 write)

Security — secrets: secret_list returns secret names, types, and key NAMES only. Secret VALUES are never read, returned, or logged, and there is deliberately no tool that returns secret values.

Dry-run previews: every write tool takes dry_run: bool = False. A dry run returns a {"dryRun": true, "wouldX": ...} preview without touching the cluster, and no undo descriptor is recorded for a preview.

Harness features that light up: write tools with a clean inverse pass an undo= lambda so the harness records an inverse descriptor (with _undo_id) to the undo store — scale_deployment/scale_statefulset record a scale-back to their returned previous_replicas, set_deployment_image records a restore to the captured previous_image, cordon_node ↔ uncordon_node and rollout_pause ↔ rollout_resume are mutual inverses, and create_namespace records a delete_namespace. drain_node records a partial uncordon_node inverse (cordon is reversible; evictions are not). delete_* and rollout_undo_deployment declare no undo. risk_level=high: delete_deployment, delete_job, delete_namespace, drain_node, rollout_undo_deployment. undo_list (read) lists recorded reversible writes whose undo tokens have not been applied yet, and undo_apply (write) executes a recorded inverse — itself governed, single-use, and supports dry_run. All 55 tools are audit-logged under ~/.k8s-aiops/ and pass through the budget/runaway guard, each recorded with a descriptive risk-tier label. pod_top/node_top return a clear "metrics-server not installed" message (not an error) when metrics-server is absent. Avoid tight poll loops (re-listing pods every second) — the runaway breaker backs this up.

CLI Quick Reference

k8s-aiops init                                            # interactive onboarding wizard
k8s-aiops pod list [-n <ns>] [-t <target>]
k8s-aiops pod get <name> [-n <ns>]
k8s-aiops pod describe <name> [-n <ns>]                   # status, container states, events
k8s-aiops pod logs <name> [-n <ns>] [--tail 200] [-c <container>]
k8s-aiops pod delete <name> [-n <ns>] [--dry-run]        # double confirm
k8s-aiops deployment list|get|scale|restart|delete ...    # scale/restart: single confirm + --dry-run; delete: double confirm
k8s-aiops rollout status|history|pause|resume <name> [-n <ns>]
k8s-aiops rollout set-image <name> <container> <image> [-n <ns>]
k8s-aiops rollout undo <name> [--to-revision N] [--dry-run]   # double confirm
k8s-aiops statefulset list|get|scale ...
k8s-aiops daemonset list|get ...
k8s-aiops job list|get|delete ...                         # delete: double confirm
k8s-aiops cronjob list|get ...
k8s-aiops service list [-n <ns>]
k8s-aiops ingress list|get [-n <ns>]
k8s-aiops configmap list|get [-n <ns>]
k8s-aiops secret list [-n <ns>]                           # names/keys only — never values
k8s-aiops storage pvc-list|pvc-get|pv-list|class-list
k8s-aiops top pod|node                                    # requires metrics-server
k8s-aiops node list|describe
k8s-aiops node cordon|drain <name> [--dry-run]           # double confirm
k8s-aiops node uncordon <name>
k8s-aiops namespace list|create
k8s-aiops namespace delete <name> [--dry-run]            # double confirm
k8s-aiops cluster-info
k8s-aiops api-resources
k8s-aiops events [-n <ns>]
k8s-aiops diagnose pod-health [-n <ns>] [-l <selector>]         # read-only RCA: crashloop/imagepull/OOM/unschedulable/restarts
k8s-aiops diagnose workload-readiness [-n <ns>]                 # read-only RCA: ready<desired / stuck rollouts
k8s-aiops doctor
k8s-aiops mcp                                             # start MCP server (stdio)

See references/cli-reference.md for the full command list.

Troubleshooting

"Could not load kubeconfig … context not found"

The named context does not exist in your kubeconfig. Run kubectl config get-contexts and set the target's context: to a listed name (or omit it to use current-context).

"Authentication/authorization failed (401/403)"

The kube context lacks the RBAC verb for the resource. Check with kubectl auth can-i <verb> <resource> -n <ns> and switch to a context/ServiceAccount with adequate roles. For EKS/GKE/AKS, confirm the exec-plugin (aws/gcloud/az CLI) is installed and logged in.

"Resource not found (404)"

The pod/deployment/node name or namespace is wrong, or the object was deleted. List the parent collection first (pod list, deployment list, node list) to get a current name. Remember most commands default to the default namespace unless -n is given.

"Conflict (409)"

The object changed concurrently (or already exists). Re-read it and retry the write.

Logs are empty or truncated

pod logs returns the trailing --tail lines (default 100); raise --tail. For a multi-container pod, pass -c <container> or the API returns an error naming the available containers.

Audit & Safety

All operations are automatically audited via the bundled @governed_tool decorator (k8s_aiops.governance):

• Every tool call logged to ~/.k8s-aiops/audit.db (local SQLite audit DB; relocate with K8S_AIOPS_HOME)
• Budget / runaway guard caps cumulative tool calls and wall-time, and trips on tight poll/retry loops — a safety backstop, not authorization
• Undo store records inverse descriptors for reversible writes (scale → previous replicas; cordon ↔ uncordon)
• Each write carries a descriptive risk-tier label into its audit row — a label, not a gate; K8S_AUDIT_APPROVED_BY / K8S_AUDIT_RATIONALE are optional annotations recorded when set, never required

Authorization is not this tool's job. There is no read-only switch, policy file, or approval gate. Whether a write is permitted is the agent's judgement or the RBAC of the kubeconfig context you connect with — give it a read-only ServiceAccount and writes fail at the apiserver, the place that owns the permission.

The harness is bundled in the package — no external dependency, no manual setup. See references/setup-guide.md for security details.

Driving these tools with a smaller / local model? See references/agent-guardrails.md — which guardrails the tool now enforces for you, plus a ready-to-paste system prompt.

Contributing & feature requests

Coverage is intentionally focused. Missing a device, action, or feature you need? Open an issue or pull request at github.com/AIops-tools/K8s-AIops — feature requests, contributions, and comments are all welcome.

License

MIT — github.com/AIops-tools/K8s-AIops