# Systems Trace

> Use when introducing a new Python/ML/AI concept, explaining how code works, or when the student asks 'what happens when...'. Generates a three-layer explanation: Mental Model (runtime trace), OS/Hardware Model (kernel/memory), Production Model (what breaks at scale).

- Skill: `aman-bhandari/systems-trace` (Agent Skill)
- Install (CLI): `npx skillmds@latest add aman-bhandari/systems-trace`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aman-bhandari/systems-trace/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: aman-bhandari (https://skillmd.com/u/aman-bhandari)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aman-bhandari/systems-trace

---


# Systems Trace -- Three-Layer Concept Introduction

Every new concept passes through three layers before the student writes a single line of code. This is the "Live Inside the Machine" teaching method. See `.claude/rules/systems-thinking.md` for the full philosophy.

## When This Fires

- A new Python, ML, or AI concept is being introduced
- The student asks "what happens when..." or "how does this work"
- Phase 2 (Mental Model) of session design -- before the dynamic exercise
- Any time the student encounters a new stdlib module, framework feature, or runtime behavior

## The Three Layers

### Layer 1: Mental Model (Runtime Trace)

What the Python interpreter does, step by step. Not syntax -- execution. Trace state changes, show what's invisible.

**What to cover:**
- What objects are created/destroyed
- What names are bound/rebound
- What the call stack looks like
- What the return value path is
- Where exceptions would propagate

**Example -- `with open('data.json') as f:`**
> Python calls `open()` which returns a file object. The `with` statement calls `f.__enter__()`, which returns `f` itself. Your block runs. When the block exits (normally or via exception), Python calls `f.__exit__()`, which calls `f.close()`. The file object's finalizer is a safety net, but `__exit__` is the contract.

### Layer 2: OS/Hardware Model (Kernel and Memory)

What the operating system does. Syscalls, memory allocation, file descriptors, scheduling. The bridge between Python and physical reality.

**What to cover:**
- Which syscalls fire (`open`, `read`, `write`, `mmap`, `socket`, `connect`, `epoll_wait`)
- Memory: stack vs heap, pymalloc arenas/pools/blocks, RSS vs virtual
- File descriptors: what the kernel's fd table looks like
- Scheduling: GIL acquisition, thread state, context switches
- For ML: GPU memory, CUDA contexts, DMA transfers

**Example -- `with open('data.json') as f:`**
> The `open()` builtin triggers the `openat()` syscall. The kernel allocates a file descriptor (integer index into the process's fd table), checks permissions against the inode, and returns the fd. The Python file object wraps this fd. `f.read()` triggers `read()` syscall -- kernel copies bytes from the page cache (or faults them in from disk) into a userspace buffer. `f.close()` triggers `close()` syscall -- kernel releases the fd slot. If you forget to close: the fd stays open until process exit or GC finalizer runs. In a long-running server, leaked fds hit `ulimit -n` and you get `OSError: Too many open files`.

### Layer 3: Production Model (What Breaks at Scale)

Real incidents. Real failure modes. What happens when this code runs 10,000 times, with 10GB inputs, on 32 cores, behind a load balancer.

**What to cover:**
- Failure modes at scale (memory, concurrency, I/O)
- Real incident stories (cite source if available)
- What monitoring you'd add
- What the fix looks like
- Decision table: when to use this vs alternatives

**Example -- `with open('data.json') as f:`**
> A FastAPI endpoint that opens a file per request: at 1000 req/s, you have 1000 concurrent fds. Linux default `ulimit -n` is 1024. Request 1025 crashes with `OSError`. Fix: read once at startup, cache in memory. Or use `aiofiles` for async I/O. Google SRE documented a Shakespeare search outage where fd leaks under a new traffic pattern exhausted the limit -- 66-minute outage, ~1.21B queries affected.

## Teaching Sequence

For every new concept, follow this order:

1. **Production story first.** Start with a real incident or failure mode. This is the hook -- it answers "why should I care about the internals?"
2. **Mechanism trace (Layer 1).** Now trace the runtime behavior. The student understands the WHAT.
3. **OS/Hardware model (Layer 2).** Go one layer deeper. The student understands the HOW.
4. **Production consequence (Layer 3).** Connect the mechanism to the failure. The student understands the WHY.
5. **Decision table.** When to use this pattern vs alternatives. Concrete, not theoretical.
6. **Verify.** Ask the student to trace a variation: "What changes if this is async?" or "What if there are 50 threads doing this?"

## Concept Examples

### `async def` / `await`
- **L1:** Coroutine object created (not executed). `await` suspends, yields control to event loop. Event loop is a `while True` polling `epoll_wait`. Coroutine resumes when the awaited future resolves.
- **L2:** No threads. One OS thread. `epoll_wait()` syscall blocks until any registered fd is ready. The event loop is a userspace scheduler on top of kernel I/O multiplexing. Stack frames are heap-allocated (coroutine frame), not C stack.
- **L3:** 10K concurrent connections on one thread -- this is the C10K solution. But: one CPU-bound coroutine blocks the entire loop. CPU work needs `run_in_executor()` (thread pool) or `ProcessPoolExecutor`. Real failure: ML inference in an async endpoint without executor -- all other requests starve.

### `list.append(x)` in a loop
- **L1:** List object has an internal array (ob_item). When the array is full, Python over-allocates (growth pattern: 0, 4, 8, 16, 25, 35, ...). `append` is amortized O(1).
- **L2:** pymalloc handles small objects (<= 512 bytes). The list's ob_item array goes to the system allocator (`malloc`) when it exceeds pymalloc's range. Over-allocation means `realloc()` syscall, which may copy the entire array to a new location. For a 100M element list, that's a ~800MB memcpy during resize.
- **L3:** Building a 100M-row dataset with `list.append` in a loop: peak memory is 2x the final list size (during realloc copy). RSS may not decrease after the list is freed because pymalloc arenas are not returned to the OS if any block is still allocated (arena pinning). Fix: pre-allocate with `[None] * n`, or use generators, or use numpy arrays.

### `import torch`
- **L1:** Python loads `torch/__init__.py`, which triggers loading of C++ extensions (`torch._C`), which initializes the ATen tensor library, autograd engine, and (if available) CUDA runtime.
- **L2:** `dlopen()` loads `.so` shared libraries. CUDA initialization calls `cuInit()` which maps GPU device memory into the process's virtual address space. PyTorch's CUDA caching allocator pre-allocates a large segment (~20% of GPU memory) and manages it like pymalloc: split into blocks, cache freed blocks, avoid `cudaMalloc` per tensor.
- **L3:** In a forked worker pool (`multiprocessing.fork`): if `import torch` runs before `fork()`, children inherit the CUDA context. Any CUDA op in a child crashes: `RuntimeError: Cannot re-initialize CUDA in forked subprocess`. Fix: use `spawn` start method, or import torch after fork. This affects every ML serving framework.

## Rules

- Never skip layers. If Layer 2 is unknown for a concept, say so: "I don't know the exact syscall path here -- let's trace it together with strace."
- Keep each layer to 2-4 sentences. Dense, not verbose.
- Always end with a verify question. The student must trace, not just listen.
- Connect to `.claude/rules/systems-thinking.md` for the full reference table of per-topic systems focus.