# K8S Resource Rightsizing

> Diagnose OOMKilled pods and fix resource configuration by analyzing actual usage patterns

- Skill: `castai/k8s-resource-rightsizing` (Agent Skill)
- Install (CLI): `npx skillmds@latest add castai/k8s-resource-rightsizing`
- Raw SKILL.md: https://api.skillmd.com/api/skills/castai/k8s-resource-rightsizing/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: castai (https://skillmd.com/u/castai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/castai/k8s-resource-rightsizing

---


## What I do

I diagnose workloads that are crashing due to incorrect resource limits. I analyze the actual usage pattern of the application, determine the correct resource values, and apply a fix that keeps the workload stable.

## My methodology

### Step 1: Identify the symptom

```
kubectl get pods -n <namespace>
```

Look for:
- `OOMKilled` status — the kernel killed the container for exceeding its memory limit
- High restart counts — the pod keeps crashing and restarting
- Pods cycling between `Running` and `OOMKilled`

### Step 2: Confirm OOMKill

```
kubectl describe pod -l app=<name> -n <namespace>
```

Look in the container status for:
```
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
```

Exit code 137 = SIGKILL from OOM killer. The container used more memory than its `limits.memory` allows.

### Step 3: Understand current resource configuration

```
kubectl get deployment <name> -n <namespace> -o yaml | grep -A 8 resources
```

Note:
- `requests.memory` — what the scheduler reserves (affects scheduling)
- `limits.memory` — the hard ceiling (exceeding this triggers OOMKill)
- The relationship: requests <= actual usage <= limits must hold for stability

### Step 4: Observe actual usage pattern

```
kubectl top pods -n <namespace>
```

Run this multiple times over 1-2 minutes. Many workloads have usage phases:
- **Startup phase**: low memory, application initializing
- **Ramp-up phase**: memory grows as workload increases
- **Steady state**: memory plateaus at operational level

The limit must accommodate the PEAK usage, not just the initial usage. If `kubectl top` shows 60Mi now but the pod OOMKills later, the workload ramps up over time.

Also check pod logs for clues about usage phases:
```
kubectl logs -l app=<name> -n <namespace>
```

### Step 5: Determine correct values

Calculate the right resource settings:

**Memory limit** = peak steady-state usage + 30% headroom
- If the workload peaks at 120Mi, set limit to ~160Mi
- This gives room for variance without waste

**Memory request** = typical steady-state usage
- Set this close to what the workload actually uses at steady state
- This affects scheduling — too low and the node gets overcommitted, too high and you waste capacity

**CPU request** = actual CPU usage (from `kubectl top`)
**CPU limit** = 2-4x the request to allow burst capacity

### Step 6: Apply the fix

```
kubectl patch deployment <name> -n <namespace> --type=json -p='[
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/memory", "value": "<new-request>"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "<new-limit>"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value": "<new-request>"},
  {"op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/cpu", "value": "<new-limit>"}
]'
```

Or use `kubectl edit deployment <name> -n <namespace>` to modify the values.

### Step 7: Verify stability

After applying the fix:

```
kubectl rollout status deployment/<name> -n <namespace>
kubectl get pods -n <namespace> -w
```

Watch for 2-3 minutes to confirm:
- No more OOMKills
- Restart count stays at 0 for the new pods
- All replicas reach and stay in Running+Ready state
- `kubectl top pods -n <namespace>` shows memory usage staying below the new limit

If pods still OOMKill, the limit is still too low — increase it further.

### What to report

After fixing, summarize:
- What the original resource config was (requests and limits)
- What the actual usage pattern looks like (phases, peak usage)
- Why the OOMKill happened (limit < peak usage)
- What the new values are and why
- Confirmation that pods are now stable

## When to use me

Use this skill when:
- Pods are being OOMKilled (exit code 137)
- Pods keep restarting with no obvious application errors
- You need to determine the correct memory/CPU values for a workload
- A workload runs fine initially but crashes after some time (phased usage pattern)

## Key principles

1. OOMKilled means the memory limit is too low for the actual workload — this is a configuration problem, not an application bug
2. Watch usage over time — many workloads have phases where memory grows after startup
3. Set limits based on peak usage + headroom, not on initial/average usage
4. Requests affect scheduling, limits affect stability — both matter
5. Always verify the fix by watching pods for several minutes to cover the full usage cycle
6. If metrics-server is not available, say so and fall back to log analysis and pod describe output