Debug Systems -- Three-Layer Debugging Protocol
Debugging is not guessing. It is systematic elimination through all three layers: runtime, OS, production. This skill enforces that protocol.
Trigger
/debug command- Student says "something is broken," "error," "this doesn't work," "help"
- Student pastes a traceback or error message
The Protocol
Step 1: Reproduce
Before anything else, the error must be reproducible.
- "Can you run it again and get the same error?"
- "What's the exact command you ran?"
- "What input triggers it?"
If it's intermittent: that's a clue. Intermittent failures are usually concurrency, resource exhaustion, or timing-dependent. Note it and proceed.
Step 2: Mental Model Layer (Read the Traceback)
Read the traceback bottom-up. The last frame is where it died. The frames above show how it got there.
Coach asks:
- "What line raised the exception?"
- "What was the state of the variables at that point?"
- "What did the code expect vs what it got?"
- "Trace backward: how did the bad value get here?"
The student must explain the execution path that led to the error. Not "it crashed on line 42" -- "the function received None because the dictionary lookup on line 38 used a key that doesn't exist, which returned None because we used .get() instead of [], and that None propagated to line 42 where we called .strip() on it."
Step 3: OS/Hardware Layer (Check System State)
Many Python errors are symptoms of OS-level problems. Check the system.
What to check:
ulimit -a -- resource limits (open files, stack size, max processes)/proc/self/status or resource.getrusage() -- memory usage (VmRSS, VmPeak)ls /proc/self/fd | wc -l -- open file descriptorsps aux | grep python -- process state, CPU%, MEM%dmesg | tail -- kernel messages (OOM killer, segfault)df -h -- disk space (silent killer of writes)free -h -- available memory
Coach asks:
- "Is this a code bug or a resource problem?"
- "What does the OS see that Python doesn't show you?"
Step 4: Production Layer (Identify Failure Mode)
Connect the bug to its production implications.
Coach asks:
- "If this happened in production at 3 AM, how would you know?"
- "What monitoring would catch this before the user sees it?"
- "Is this a one-off bug or a class of bugs? What else has the same pattern?"
- "What's the blast radius? One request? One user? All users?"
Error-to-OS Reference Table
Common Python errors mapped to their OS-level causes:
| Python Error |
OS/System Cause |
What to Check |
ConnectionRefusedError |
Remote port has no listener. TCP RST packet sent back. |
ss -tlnp on remote. Is the service running? Firewall rules? |
ConnectionResetError |
Remote closed the connection (RST). Often: server crashed, load balancer timeout, or TLS mismatch. |
Server logs. tcpdump for RST packets. Load balancer idle timeout config. |
MemoryError |
Process hit memory limit. Either ulimit -v, cgroup limit, or actual OOM. Kernel OOM killer may have been involved. |
dmesg | grep -i oom. /proc/self/status VmPeak. cgroup memory.max. |
OSError: Too many open files |
Process fd count hit ulimit -n (default 1024). Leaked fds from unclosed files/sockets/connections. |
ls /proc/self/fd | wc -l. lsof -p <pid>. Look for missing close() or missing with statements. |
TimeoutError |
TCP retransmission timeout (~30s default). Network partition, slow server, or DNS resolution hang. |
ping the host. traceroute. Check DNS with dig. ss -tn state time-wait. |
BrokenPipeError |
Writing to a closed socket fd. The reader closed their end (sent FIN), you kept writing, kernel sends SIGPIPE (Python catches it and raises). |
Check if the client disconnected. Check response size -- large responses hit this when clients give up. |
FileNotFoundError |
open() syscall returned ENOENT. File doesn't exist at that path, or a symlink is broken. |
stat the file. Check working directory (os.getcwd()). Check symlinks with readlink. |
PermissionError |
open() syscall returned EACCES. File exists but process UID/GID lacks permission. |
ls -la the file. id to check current user. stat for owner/group/mode. |
BlockingIOError |
Non-blocking socket has no data ready. recv() returned EAGAIN/EWOULDBLOCK. |
Expected in async code. If unexpected: socket was set non-blocking without using an event loop. |
RecursionError |
C stack overflow. CPython has a recursion limit (default 1000) to prevent segfault from C stack exhaustion. |
sys.getrecursionlimit(). The real limit is the C stack size (ulimit -s, default 8MB). Deep recursion = rewrite as iteration. |
SegmentationFault |
Process accessed invalid memory address. SIGSEGV from kernel. Usually a C extension bug, not Python. |
dmesg for the segfault log (shows instruction pointer and fault address). Run under gdb python3 -c "...". Check C extension versions. |
Killed (no traceback) |
Kernel OOM killer sent SIGKILL. Process was the biggest memory consumer. No cleanup, no exception, no handler. |
dmesg | grep -i 'killed process'. Check cgroup limits. Reduce memory usage or increase limits. |
Debugging Decision Tree
Error occurs
|
+-- Is there a traceback?
| YES --> Read bottom-up (Step 2)
| NO --> Process was killed externally
| Check: dmesg, OOM killer, SIGKILL, cgroup limits
|
+-- Is the error reproducible?
| YES --> Systematic elimination through layers
| NO --> Concurrency, resource exhaustion, or timing
| Add logging, check under load, check resource limits
|
+-- Is it a Python error or OS error?
Python (TypeError, ValueError, KeyError) --> Layer 1 trace
OS (OSError subclass) --> Layer 2 check first
Silent failure (wrong output, no crash) --> Layer 1 trace + add assertions
Rules
- Never give the fix. Guide the student through the layers. They find it.
- Always start with reproduce. No debugging without a reproducible case.
- The student must explain the execution path. "It crashed" is not debugging. "Here's the path that led to the crash" is debugging.
- Connect every bug to its production story. "What monitoring catches this before the user sees it?"
- Log the debugging session. If the bug was non-trivial, it goes in the session exchange file as a teaching moment.
- Escalation: If stuck after 30 minutes, use the os-lens instruments (strace, tracemalloc, /proc) to get direct evidence. Evidence beats speculation.
1---2name: debug-systems3description: Use when the student encounters an error, says something is broken, or asks for debugging help. Enforces a three-layer debugging protocol: reproduce, trace the mental model, check OS state, identify production failure mode. Invoke with /debug.4---56# Debug Systems -- Three-Layer Debugging Protocol78Debugging is not guessing. It is systematic elimination through all three layers: runtime, OS, production. This skill enforces that protocol.910## Trigger1112- `/debug` command13- Student says "something is broken," "error," "this doesn't work," "help"14- Student pastes a traceback or error message1516## The Protocol1718### Step 1: Reproduce1920Before anything else, the error must be reproducible.2122- "Can you run it again and get the same error?"23- "What's the exact command you ran?"24- "What input triggers it?"2526If it's intermittent: that's a clue. Intermittent failures are usually concurrency, resource exhaustion, or timing-dependent. Note it and proceed.2728### Step 2: Mental Model Layer (Read the Traceback)2930Read the traceback bottom-up. The last frame is where it died. The frames above show how it got there.3132**Coach asks:**33- "What line raised the exception?"34- "What was the state of the variables at that point?"35- "What did the code expect vs what it got?"36- "Trace backward: how did the bad value get here?"3738**The student must explain the execution path that led to the error.** Not "it crashed on line 42" -- "the function received None because the dictionary lookup on line 38 used a key that doesn't exist, which returned None because we used `.get()` instead of `[]`, and that None propagated to line 42 where we called `.strip()` on it."3940### Step 3: OS/Hardware Layer (Check System State)4142Many Python errors are symptoms of OS-level problems. Check the system.4344**What to check:**45- `ulimit -a` -- resource limits (open files, stack size, max processes)46- `/proc/self/status` or `resource.getrusage()` -- memory usage (VmRSS, VmPeak)47- `ls /proc/self/fd | wc -l` -- open file descriptors48- `ps aux | grep python` -- process state, CPU%, MEM%49- `dmesg | tail` -- kernel messages (OOM killer, segfault)50- `df -h` -- disk space (silent killer of writes)51- `free -h` -- available memory5253**Coach asks:**54- "Is this a code bug or a resource problem?"55- "What does the OS see that Python doesn't show you?"5657### Step 4: Production Layer (Identify Failure Mode)5859Connect the bug to its production implications.6061**Coach asks:**62- "If this happened in production at 3 AM, how would you know?"63- "What monitoring would catch this before the user sees it?"64- "Is this a one-off bug or a class of bugs? What else has the same pattern?"65- "What's the blast radius? One request? One user? All users?"6667## Error-to-OS Reference Table6869Common Python errors mapped to their OS-level causes:7071| Python Error | OS/System Cause | What to Check |72|-------------|----------------|---------------|73| `ConnectionRefusedError` | Remote port has no listener. TCP RST packet sent back. | `ss -tlnp` on remote. Is the service running? Firewall rules? |74| `ConnectionResetError` | Remote closed the connection (RST). Often: server crashed, load balancer timeout, or TLS mismatch. | Server logs. `tcpdump` for RST packets. Load balancer idle timeout config. |75| `MemoryError` | Process hit memory limit. Either `ulimit -v`, cgroup limit, or actual OOM. Kernel OOM killer may have been involved. | `dmesg \| grep -i oom`. `/proc/self/status` VmPeak. `cgroup` memory.max. |76| `OSError: Too many open files` | Process fd count hit `ulimit -n` (default 1024). Leaked fds from unclosed files/sockets/connections. | `ls /proc/self/fd \| wc -l`. `lsof -p <pid>`. Look for missing `close()` or missing `with` statements. |77| `TimeoutError` | TCP retransmission timeout (~30s default). Network partition, slow server, or DNS resolution hang. | `ping` the host. `traceroute`. Check DNS with `dig`. `ss -tn state time-wait`. |78| `BrokenPipeError` | Writing to a closed socket fd. The reader closed their end (sent FIN), you kept writing, kernel sends SIGPIPE (Python catches it and raises). | Check if the client disconnected. Check response size -- large responses hit this when clients give up. |79| `FileNotFoundError` | `open()` syscall returned ENOENT. File doesn't exist at that path, or a symlink is broken. | `stat` the file. Check working directory (`os.getcwd()`). Check symlinks with `readlink`. |80| `PermissionError` | `open()` syscall returned EACCES. File exists but process UID/GID lacks permission. | `ls -la` the file. `id` to check current user. `stat` for owner/group/mode. |81| `BlockingIOError` | Non-blocking socket has no data ready. `recv()` returned EAGAIN/EWOULDBLOCK. | Expected in async code. If unexpected: socket was set non-blocking without using an event loop. |82| `RecursionError` | C stack overflow. CPython has a recursion limit (default 1000) to prevent segfault from C stack exhaustion. | `sys.getrecursionlimit()`. The real limit is the C stack size (`ulimit -s`, default 8MB). Deep recursion = rewrite as iteration. |83| `SegmentationFault` | Process accessed invalid memory address. SIGSEGV from kernel. Usually a C extension bug, not Python. | `dmesg` for the segfault log (shows instruction pointer and fault address). Run under `gdb python3 -c "..."`. Check C extension versions. |84| `Killed` (no traceback) | Kernel OOM killer sent SIGKILL. Process was the biggest memory consumer. No cleanup, no exception, no handler. | `dmesg \| grep -i 'killed process'`. Check cgroup limits. Reduce memory usage or increase limits. |8586## Debugging Decision Tree8788```89Error occurs90 |91 +-- Is there a traceback?92 | YES --> Read bottom-up (Step 2)93 | NO --> Process was killed externally94 | Check: dmesg, OOM killer, SIGKILL, cgroup limits95 |96 +-- Is the error reproducible?97 | YES --> Systematic elimination through layers98 | NO --> Concurrency, resource exhaustion, or timing99 | Add logging, check under load, check resource limits100 |101 +-- Is it a Python error or OS error?102 Python (TypeError, ValueError, KeyError) --> Layer 1 trace103 OS (OSError subclass) --> Layer 2 check first104 Silent failure (wrong output, no crash) --> Layer 1 trace + add assertions105```106107## Rules108109- **Never give the fix.** Guide the student through the layers. They find it.110- **Always start with reproduce.** No debugging without a reproducible case.111- **The student must explain the execution path.** "It crashed" is not debugging. "Here's the path that led to the crash" is debugging.112- **Connect every bug to its production story.** "What monitoring catches this before the user sees it?"113- **Log the debugging session.** If the bug was non-trivial, it goes in the session exchange file as a teaching moment.114- **Escalation:** If stuck after 30 minutes, use the os-lens instruments (strace, tracemalloc, /proc) to get direct evidence. Evidence beats speculation.
aman-bhandari