K8S Namespace Troubleshooting

Kubernetes Namespace Troubleshooting

Perform comprehensive, namespace-scoped investigation of Kubernetes workloads. Collect events, pod status, node health, resource consumption, configuration issues, and application logs, then summarise findings, diagnose root causes, and assess impact.

Keywords

kubernetes, namespace, troubleshoot, troubleshooting, debug, diagnose, investigate, problem, issue, error, unhealthy, failing, broken, pod, deployment, statefulset, daemonset, job, cronjob, service, ingress, events, logs, restart, crashloop, pending, oomkill, eviction, namespace health, application, workload

When to Use This Skill

• A user says "I have a problem in namespace X" or "something is wrong in namespace X"
• Pods are failing, restarting, pending, or evicted in a namespace
• Services or ingresses are not responding
• Applications are returning errors or degraded performance
• Namespace events show warnings or errors
• A general health check of everything running in a namespace is needed

Related Skills

• k8s-platform-operations - Cluster-wide health checks and incident response
• k8s-security-hardening - Security context and policy issues
• k8s-platform-tenancy - Resource quotas and limit ranges
• k8s-continual-improvement - SLOs and capacity planning
• k8s-security-redteam - Security-related namespace issues
• flux-troubleshooting - GitOps reconciliation failures
• Shared: Pod Security Context
• Shared: Network Policies

Quick Reference

Investigation Workflow

This skill follows a five-phase workflow. Each phase builds on the previous one.

Phase 1: COLLECT     → Gather all namespace data (resources, events, status)
    ↓
Phase 2: SUMMARISE   → Produce a namespace health summary
    ↓
Phase 3: DIAGNOSE    → Identify root causes from collected evidence
    ↓
Phase 4: IMPACT      → Assess severity and blast radius
    ↓
Phase 5: RECOMMEND   → Present recommended actions for the user

Phase 1: Collect

Gather comprehensive data from the namespace. Run all commands targeting the user-provided namespace. Collect the output of every section below before moving to Phase 2.

1.1 Namespace Status and Configuration

# Confirm namespace exists and check labels/annotations
kubectl get namespace ${NS} -o yaml

# Check resource quotas (may explain scheduling failures)
kubectl get resourcequota -n ${NS}
kubectl describe resourcequota -n ${NS}

# Check limit ranges (may explain OOMKills or throttling)
kubectl get limitrange -n ${NS}
kubectl describe limitrange -n ${NS}

1.2 Events (Critical — Always Check First)

# All events sorted by time (most recent last)
kubectl get events -n ${NS} --sort-by='.lastTimestamp'

# Warning events only
kubectl get events -n ${NS} --field-selector type=Warning --sort-by='.lastTimestamp'

# Event counts to spot recurring issues
kubectl get events -n ${NS} -o json | \
  jq -r '[.items[] | {reason: .reason, name: .involvedObject.name, count: .count}] | group_by(.reason) | map({reason: .[0].reason, total: (map(.count) | add)}) | sort_by(-.total) | .[] | "\(.total)\t\(.reason)"'

1.3 Pods — Status, Restarts, and Conditions

# Full pod listing with status
kubectl get pods -n ${NS} -o wide

# Unhealthy pods (not Running or Completed)
kubectl get pods -n ${NS} | grep -Ev 'Running|Completed'

# High restart counts
kubectl get pods -n ${NS} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.restartCount}{" "}{end}{"\n"}{end}' | awk '{sum=0; for(i=2;i<=NF;i++) sum+=$i; if(sum>0) print $1"\t"sum" restarts"}' | sort -t$'\t' -k2 -rn

# OOMKilled containers
kubectl get pods -n ${NS} -o json | \
  jq -r '.items[] | .metadata.name as $pod | .status.containerStatuses[]? | select(.lastState.terminated.reason == "OOMKilled") | "\($pod)\t\(.name)\tOOMKilled"'

# Pending pods — why are they not scheduled?
kubectl get pods -n ${NS} --field-selector=status.phase=Pending -o name | \
  xargs -I{} kubectl describe {} -n ${NS} | grep -A5 "Events:"

# Container image details (spot wrong tags, missing images)
kubectl get pods -n ${NS} -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}'

1.4 Workload Controllers

# Deployments — check desired vs ready replicas
kubectl get deployments -n ${NS} -o wide

# Deployments not at full availability
kubectl get deployments -n ${NS} -o json | \
  jq -r '.items[] | select(.status.availableReplicas != .status.replicas) | "\(.metadata.name)\tavailable=\(.status.availableReplicas // 0)/\(.status.replicas)"'

# StatefulSets
kubectl get statefulsets -n ${NS} -o wide

# DaemonSets — check desired vs ready
kubectl get daemonsets -n ${NS} -o wide

# Jobs — check for failures
kubectl get jobs -n ${NS}
kubectl get jobs -n ${NS} -o json | \
  jq -r '.items[] | select(.status.failed > 0) | "\(.metadata.name)\tfailed=\(.status.failed)"'

# CronJobs — check schedule and last run
kubectl get cronjobs -n ${NS}

1.5 Services, Endpoints, and Networking

# Services
kubectl get services -n ${NS} -o wide

# Endpoints — empty endpoints mean no backing pods
kubectl get endpoints -n ${NS}
kubectl get endpoints -n ${NS} -o json | \
  jq -r '.items[] | select((.subsets // []) | length == 0) | "\(.metadata.name)\tNO ENDPOINTS"'

# Ingresses
kubectl get ingress -n ${NS}

# Network policies (may be blocking traffic)
kubectl get networkpolicies -n ${NS}

1.6 Configuration — ConfigMaps, Secrets, PVCs

# ConfigMaps
kubectl get configmaps -n ${NS}

# Secrets (names only — never print secret data)
kubectl get secrets -n ${NS}

# PersistentVolumeClaims — check bound status and capacity
kubectl get pvc -n ${NS}
kubectl get pvc -n ${NS} -o json | \
  jq -r '.items[] | select(.status.phase != "Bound") | "\(.metadata.name)\t\(.status.phase)"'

1.7 Resource Consumption

# Pod CPU and memory (requires metrics-server)
kubectl top pods -n ${NS} --sort-by=memory
kubectl top pods -n ${NS} --sort-by=cpu

# Requests vs limits vs actual (identify over/under provisioning)
kubectl get pods -n ${NS} -o json | \
  jq -r '.items[] | .metadata.name as $pod | .spec.containers[] | "\($pod)\t\(.name)\treq_cpu=\(.resources.requests.cpu // "none")\tlim_cpu=\(.resources.limits.cpu // "none")\treq_mem=\(.resources.requests.memory // "none")\tlim_mem=\(.resources.limits.memory // "none")"'

1.8 Node Health (for Nodes Running Namespace Pods)

# Find which nodes host pods in this namespace
kubectl get pods -n ${NS} -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u

# Check conditions on those nodes
kubectl get pods -n ${NS} -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | \
  xargs -I{} sh -c 'echo "--- {} ---" && kubectl describe node {} | grep -A5 "Conditions:"'

# Node resource pressure
kubectl get pods -n ${NS} -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | \
  xargs -I{} kubectl top node {}

1.9 Application Logs

# Current logs, previous logs, and error grep — single pass over all pods
for pod in $(kubectl get pods -n ${NS} -o jsonpath='{.items[*].metadata.name}'); do
  echo "=== ${pod} ==="
  kubectl logs -n ${NS} ${pod} --all-containers --tail=100 2>&1
  echo "--- ${pod} (previous) ---"
  kubectl logs -n ${NS} ${pod} --all-containers --previous --tail=50 2>&1 || echo "No previous logs"
  echo "--- ${pod} (errors) ---"
  kubectl logs -n ${NS} ${pod} --all-containers --tail=200 2>/dev/null | \
    grep -iE 'error|exception|fatal|panic|timeout|refused|denied|failed|oom|kill' | \
    head -20
done

1.10 HPA and VPA (Autoscaling)

# Horizontal Pod Autoscalers
kubectl get hpa -n ${NS}
kubectl describe hpa -n ${NS}

# Vertical Pod Autoscalers (if installed)
kubectl get vpa -n ${NS} 2>/dev/null

1.11 Flux / GitOps Resources (If Present)

# Check if any Flux resources target this namespace
kubectl get kustomizations.kustomize.toolkit.fluxcd.io -A -o json | \
  jq -r '.items[] | select(.spec.targetNamespace == "'${NS}'" or .metadata.namespace == "'${NS}'") | "\(.metadata.namespace)/\(.metadata.name)\tReady=\(.status.conditions[-1].status)\tMessage=\(.status.conditions[-1].message)"' 2>/dev/null

kubectl get helmreleases.helm.toolkit.fluxcd.io -n ${NS} 2>/dev/null

Phase 2: Summarise

After collecting data, produce a structured health summary. This summary must be presented to the user before deeper diagnosis.

Namespace Health Summary Template

## Namespace Health Summary: ${NS}

### Overview
- **Total pods**: X (Y running, Z unhealthy)
- **Deployments**: X/Y at desired replicas
- **StatefulSets**: X/Y at desired replicas
- **Jobs**: X succeeded, Y failed
- **Services**: X total, Y with empty endpoints
- **PVCs**: X bound, Y unbound
- **Warning events**: X in last hour

### Unhealthy Resources
| Resource | Type | Status | Issue |
|----------|------|--------|-------|
| pod-name | Pod | CrashLoopBackOff | Exit code 1, 47 restarts |
| deploy-name | Deployment | 1/3 ready | 2 pods Pending |
| pvc-name | PVC | Pending | No matching PV |

### Resource Consumption
| Pod | CPU (actual/limit) | Memory (actual/limit) | Pressure |
|-----|--------------------|-----------------------|----------|
| pod-a | 450m/500m | 490Mi/512Mi | HIGH |
| pod-b | 10m/500m | 50Mi/512Mi | LOW |

### Recent Warning Events (Top 5)
| Count | Reason | Object | Message |
|-------|--------|--------|---------|
| 23 | BackOff | pod/app-x | Back-off restarting failed container |
| 12 | FailedScheduling | pod/app-y | Insufficient memory |

### Node Health (Hosting Namespace Pods)
| Node | CPU% | Mem% | Conditions |
|------|------|------|------------|
| node-1 | 82% | 91% | MemoryPressure=True |
| node-2 | 45% | 60% | Ready=True |

Phase 3: Diagnose

Analyse the collected evidence to identify root causes. Work through this decision tree for each unhealthy resource.

Pod Diagnostic Decision Tree

Pod not Running?
├─ Pending
│   ├─ "Insufficient cpu/memory" → Node capacity or resource quota exhausted
│   ├─ "no nodes match selector" → Node affinity/selector mismatch
│   ├─ "persistentvolumeclaim not found" → PVC missing or unbound
│   ├─ "0/N nodes are available" → Check taints, tolerations, affinity
│   └─ No events at all → Scheduler may be down
├─ CrashLoopBackOff
│   ├─ Exit code 1 → Application error (check logs)
│   ├─ Exit code 126/127 → Command not found (wrong image or entrypoint)
│   ├─ Exit code 137 → OOMKilled (raise memory limits)
│   ├─ Exit code 139 → Segfault (application bug or incompatible image)
│   └─ Exit code 143 → SIGTERM (graceful shutdown issue)
├─ ImagePullBackOff
│   ├─ "not found" → Wrong image name or tag
│   ├─ "unauthorized" → Missing or wrong imagePullSecret
│   └─ "timeout" → Registry unreachable (network/DNS)
├─ Init:Error / Init:CrashLoopBackOff
│   └─ Init container failing → Check init container logs separately
├─ CreateContainerConfigError
│   ├─ "secret not found" → Referenced Secret missing
│   └─ "configmap not found" → Referenced ConfigMap missing
├─ Evicted
│   └─ Node under pressure → Check node conditions, DiskPressure/MemoryPressure
└─ Terminating (stuck)
    ├─ Finalizer blocking deletion → Check finalizers
    └─ Process not responding to SIGTERM → Check graceful shutdown

Service Connectivity Diagnostic

Service not reachable?
├─ Endpoints empty?
│   ├─ Yes → Label selector mismatch between Service and Pods
│   └─ No → Endpoints exist, check:
│       ├─ Port mismatch (service port vs container port)
│       ├─ Network policy blocking traffic
│       ├─ Pod readiness probe failing
│       └─ DNS resolution issue
├─ Ingress not routing?
│   ├─ Ingress class missing or wrong
│   ├─ TLS certificate issue
│   ├─ Backend service name/port mismatch
│   └─ Ingress controller not running

Common Root Cause Patterns

Phase 4: Impact Assessment

For each identified problem, assess its severity and blast radius.

Severity Classification

Impact Assessment Format

For each problem, present: Problem | Severity | Affected Resources | User Impact | Blast Radius

Phase 5: Recommend

For each diagnosed problem, present recommended actions for the user to execute. Do not perform remediation directly.

Recommended Actions

Present each recommendation with the problem, the suggested command or change, and who should action it:

Recommendation Report Format

Structure the final report with two sections:

1. Recommended Actions — What the user should do (Priority, Action, Command/Change, Owner)
2. Preventive Recommendations — PDBs, resource limits, probes, HPA, right-sizing

Common Error Patterns — Quick Diagnosis

A quick-lookup table for when the user describes a specific symptom:

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query any Kubernetes resource
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve pod logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Get resource consumption metrics

When MCP tools are available, follow the same Phase 1 collection order using MCP equivalents (get_kubernetes_resources for each resource kind, get_kubernetes_metrics for usage, get_kubernetes_logs for targeted pod logs).

Common Mistakes

Behavioural Guidelines

When using this skill, follow these principles:

1. Always collect before concluding — Run Phase 1 fully. Do not guess root causes from partial data.
2. Present the summary first — Give the user the Phase 2 health summary before diving into diagnosis. Let them see the landscape.
3. Be explicit about severity — Every identified problem must have a severity rating from Phase 4.
4. Recommend, don't remediate — Present recommended actions with commands for the user to execute. Do not perform remediation directly.
5. Never print secret values — List secret names only. Never decode or display secret data.
6. Respect the namespace boundary — This is a namespace-scoped investigation. Only look at nodes or cluster resources when they directly affect this namespace's workloads.
7. Handle empty namespaces — If the namespace has no resources, say so. It may be newly created, or everything may have been deleted.
8. Handle large namespaces — If there are >50 pods, focus on unhealthy resources first. Summarise healthy resources in aggregate.
9. Correlate across resources — A pod failure may be caused by a node issue, a missing secret, or a quota limit. Cross-reference findings.
10. Recommend prevention — After diagnosing issues, suggest probes, PDBs, resource limits, and monitoring to prevent recurrence.