Systems Trace

Systems Trace -- Three-Layer Concept Introduction

Every new concept passes through three layers before the student writes a single line of code. This is the "Live Inside the Machine" teaching method. See .claude/rules/systems-thinking.md for the full philosophy.

When This Fires

• A new Python, ML, or AI concept is being introduced
• The student asks "what happens when..." or "how does this work"
• Phase 2 (Mental Model) of session design -- before the dynamic exercise
• Any time the student encounters a new stdlib module, framework feature, or runtime behavior

The Three Layers

Layer 1: Mental Model (Runtime Trace)

What the Python interpreter does, step by step. Not syntax -- execution. Trace state changes, show what's invisible.

What to cover:

• What objects are created/destroyed
• What names are bound/rebound
• What the call stack looks like
• What the return value path is
• Where exceptions would propagate

Example -- with open('data.json') as f:

Python calls open() which returns a file object. The with statement calls f.__enter__(), which returns f itself. Your block runs. When the block exits (normally or via exception), Python calls f.__exit__(), which calls f.close(). The file object's finalizer is a safety net, but __exit__ is the contract.

Layer 2: OS/Hardware Model (Kernel and Memory)

What the operating system does. Syscalls, memory allocation, file descriptors, scheduling. The bridge between Python and physical reality.

What to cover:

• Which syscalls fire (open, read, write, mmap, socket, connect, epoll_wait)
• Memory: stack vs heap, pymalloc arenas/pools/blocks, RSS vs virtual
• File descriptors: what the kernel's fd table looks like
• Scheduling: GIL acquisition, thread state, context switches
• For ML: GPU memory, CUDA contexts, DMA transfers

Example -- with open('data.json') as f:

The open() builtin triggers the openat() syscall. The kernel allocates a file descriptor (integer index into the process's fd table), checks permissions against the inode, and returns the fd. The Python file object wraps this fd. f.read() triggers read() syscall -- kernel copies bytes from the page cache (or faults them in from disk) into a userspace buffer. f.close() triggers close() syscall -- kernel releases the fd slot. If you forget to close: the fd stays open until process exit or GC finalizer runs. In a long-running server, leaked fds hit ulimit -n and you get OSError: Too many open files.

Layer 3: Production Model (What Breaks at Scale)

Real incidents. Real failure modes. What happens when this code runs 10,000 times, with 10GB inputs, on 32 cores, behind a load balancer.

What to cover:

• Failure modes at scale (memory, concurrency, I/O)
• Real incident stories (cite source if available)
• What monitoring you'd add
• What the fix looks like
• Decision table: when to use this vs alternatives

Example -- with open('data.json') as f:

A FastAPI endpoint that opens a file per request: at 1000 req/s, you have 1000 concurrent fds. Linux default ulimit -n is 1024. Request 1025 crashes with OSError. Fix: read once at startup, cache in memory. Or use aiofiles for async I/O. Google SRE documented a Shakespeare search outage where fd leaks under a new traffic pattern exhausted the limit -- 66-minute outage, ~1.21B queries affected.

Teaching Sequence

For every new concept, follow this order:

1. Production story first. Start with a real incident or failure mode. This is the hook -- it answers "why should I care about the internals?"
2. Mechanism trace (Layer 1). Now trace the runtime behavior. The student understands the WHAT.
3. OS/Hardware model (Layer 2). Go one layer deeper. The student understands the HOW.
4. Production consequence (Layer 3). Connect the mechanism to the failure. The student understands the WHY.
5. Decision table. When to use this pattern vs alternatives. Concrete, not theoretical.
6. Verify. Ask the student to trace a variation: "What changes if this is async?" or "What if there are 50 threads doing this?"

Concept Examples

async def / await

• L1: Coroutine object created (not executed). await suspends, yields control to event loop. Event loop is a while True polling epoll_wait. Coroutine resumes when the awaited future resolves.
• L2: No threads. One OS thread. epoll_wait() syscall blocks until any registered fd is ready. The event loop is a userspace scheduler on top of kernel I/O multiplexing. Stack frames are heap-allocated (coroutine frame), not C stack.
• L3: 10K concurrent connections on one thread -- this is the C10K solution. But: one CPU-bound coroutine blocks the entire loop. CPU work needs run_in_executor() (thread pool) or ProcessPoolExecutor. Real failure: ML inference in an async endpoint without executor -- all other requests starve.

list.append(x) in a loop

• L1: List object has an internal array (ob_item). When the array is full, Python over-allocates (growth pattern: 0, 4, 8, 16, 25, 35, ...). append is amortized O(1).
• L2: pymalloc handles small objects (<= 512 bytes). The list's ob_item array goes to the system allocator (malloc) when it exceeds pymalloc's range. Over-allocation means realloc() syscall, which may copy the entire array to a new location. For a 100M element list, that's a ~800MB memcpy during resize.
• L3: Building a 100M-row dataset with list.append in a loop: peak memory is 2x the final list size (during realloc copy). RSS may not decrease after the list is freed because pymalloc arenas are not returned to the OS if any block is still allocated (arena pinning). Fix: pre-allocate with [None] * n, or use generators, or use numpy arrays.

import torch

• L1: Python loads torch/__init__.py, which triggers loading of C++ extensions (torch._C), which initializes the ATen tensor library, autograd engine, and (if available) CUDA runtime.
• L2: dlopen() loads .so shared libraries. CUDA initialization calls cuInit() which maps GPU device memory into the process's virtual address space. PyTorch's CUDA caching allocator pre-allocates a large segment (~20% of GPU memory) and manages it like pymalloc: split into blocks, cache freed blocks, avoid cudaMalloc per tensor.
• L3: In a forked worker pool (multiprocessing.fork): if import torch runs before fork(), children inherit the CUDA context. Any CUDA op in a child crashes: RuntimeError: Cannot re-initialize CUDA in forked subprocess. Fix: use spawn start method, or import torch after fork. This affects every ML serving framework.

Rules

• Never skip layers. If Layer 2 is unknown for a concept, say so: "I don't know the exact syscall path here -- let's trace it together with strace."
• Keep each layer to 2-4 sentences. Dense, not verbose.
• Always end with a verify question. The student must trace, not just listen.
• Connect to .claude/rules/systems-thinking.md for the full reference table of per-topic systems focus.