Explain Sample

explain-sample

Use this skill when the user asks about the macOS sample command, process profiling, diagnosing high CPU usage, understanding why a process is spinning, or wants to know what a running process is doing internally.

What is sample?

sample is a built-in macOS command-line profiler that answers the question: "What is this process doing right now?"

It works by taking stack trace snapshots of a running process at regular intervals (default: 1ms), then aggregating them into a call graph showing where CPU time is spent.

sample <PID> <duration_seconds> [-file output.txt]

How It Works (The Magic)

Every running program uses a call stack - it's fundamental to how CPUs execute code:

main() calls → processData() calls → parseJSON() calls → readBuffer()

Each function call pushes a return address onto the stack. sample uses macOS Mach kernel APIs (task_for_pid) to:

1. Pause the process briefly (~1ms)
2. Walk the stack - read the chain of return addresses
3. Resume the process
4. Repeat ~1000 times per second
5. Aggregate - tally which functions appeared most often

If a function appears in 900 out of 1000 samples, it's using ~90% of CPU time.

Basic Usage

# Sample process for 1 second, output to terminal
sample 12345 1

# Sample for 5 seconds, save to file
sample 12345 5 -file /tmp/profile.txt

# Sample by process name (if unique)
sample -p "node" 1

Reading the Output

The Call Graph

Call graph:
    1000 Thread_12345   ← 1000 samples from this thread
    + 1000 main
    +   950 doWork      ← 95% of time here
    +   ! 800 innerLoop ← 80% in this sub-function
    +   ! ! 800 spin    ← the actual culprit
    +   50 cleanup      ← only 5% here

Key insight: Follow the numbers. The deepest function with high counts is usually your bottleneck.

Symbols vs Addresses

Real-World Example: Diagnosing a Spinning Process

The Problem

A Claude Code process (PID 39779) was consuming 100% CPU despite being "idle":

ps -Ao pid,pcpu,comm | head -5
  PID  %CPU COMM
39779 102.1 claude   ← Why is this spinning?

The Investigation

sample 39779 1 -file /tmp/claude-sample.txt

The Output (abbreviated)

Call graph:
    805 Thread_223980213
    + 805 node::Start
    +   805 node::NodeMainInstance::Run()
    +     805 node::SpinEventLoopInternal
    +       733 uv_run                        ← libuv event loop
    +         733 uv__run_check               ← "check" phase
    +           730 node::CheckImmediate      ← setImmediate callbacks
    +             729 v8::Function::Call      ← executing JavaScript
    +               726 ???                   ← JIT code (no symbols)

The Diagnosis

• 733/805 samples (91%) in uv__run_check → CheckImmediate
• The process is busy-looping in setImmediate callbacks
• Combined with lsof -p 39779 showing stdin/stdout as (revoked)
• Root cause: The tmux session was killed, but the process didn't exit - it's spinning trying to read from a disconnected terminal

The Fix

kill 39779

What sample Can and Cannot Do

✅ Works On

❌ Limitations

🔑 Key Non-Limitation

You don't need source code, debug builds, or special compilation flags. sample works on any running process. Stripped binary? You still get the call stack - just addresses instead of names.

Pro Techniques

1. Quick CPU Hog Diagnosis

# Find top CPU process and sample it immediately
TOP_PID=$(ps -Ao pid,%cpu -r | awk 'NR==2 {print $1}')
sample $TOP_PID 3 -file /tmp/hot.txt && head -80 /tmp/hot.txt

2. Compare Before/After

# Baseline
sample $PID 5 -file /tmp/before.txt

# ... make changes ...

# After
sample $PID 5 -file /tmp/after.txt

# Diff the call graphs
diff <(grep -E '^\s+[0-9]+' /tmp/before.txt | head -50) \
     <(grep -E '^\s+[0-9]+' /tmp/after.txt | head -50)

3. Sample During Specific Operation

# Start sampling in background
sample $PID 30 -file /tmp/during-operation.txt &

# Trigger the slow operation
curl http://localhost:8080/slow-endpoint

# Wait for sample to finish
wait

4. Focus on Specific Thread

# Sample outputs all threads - grep for the hot one
sample $PID 5 | grep -A 100 "Thread_.*DispatchQueue.*main"

5. Combine with Other Tools

# Step 1: Find the suspect
ps -Ao pid,%cpu,command -r | head -10

# Step 2: Check what files it has open
lsof -p $PID | head -20

# Step 3: See its call stack
sample $PID 1

# Step 4: Watch syscalls (if SIP allows)
sudo dtruss -p $PID 2>&1 | head -50

6. Symbolicate Addresses Later

If you have addresses from a stripped binary:

# Get the load address
sample $PID 1 | grep "Load Address"
# Load Address:    0x100000000

# Symbolicate an address
atos -o /path/to/binary -l 0x100000000 0x100003f40

Related Tools

Quick Reference

# Basic: sample PID for N seconds
sample <PID> <seconds>

# Save to file
sample <PID> <seconds> -file /tmp/output.txt

# Sample with more frequent snapshots (microseconds between samples)
sample <PID> <seconds> -wait 500  # 500 microseconds = 2000 samples/sec

# Just see the summary
sample <PID> 1 2>&1 | head -50

TL;DR

sample answers "why is this process using CPU?" by showing you the call stack. It works on any process, any language, any binary - you just might get addresses instead of function names if symbols are stripped. When you see a process spinning at 100% CPU, sample is your first tool.