CPU/GPU Performance Discipline
When To Use
- At the beginning of every session (auto-load alongside
token-conservation).
- Whenever you plan to build, train, or test anything that could pin CPU cores
or GPUs for more than a minute.
- Before retrying a failing command that previously consumed significant resources.
When NOT To Use
- Simple operations with no resource impact
- Quick single-file operations
Required Progress Items
cpu-gpu-performance:baseline
cpu-gpu-performance:scope
cpu-gpu-performance:instrument
cpu-gpu-performance:throttle
cpu-gpu-performance:log
Step 1: Establish Current Baseline
Capture current utilization:
uptime
ps -eo pcpu,cmd | head
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv
Note which hosts/GPUs are already busy.
Record any CI/cluster budgets (time quotas, GPU hours) before launching work.
Set a per-task CPU minute / GPU minute budget that respects those limits.
Step 2: Narrow the Scope
- Avoid running "whole world" jobs after a small fix. Prefer diff-based
or tag-based selective testing:
pytest -k
- Bazel target patterns
cargo test <module>
- Batch low-level fixes so you can validate multiple changes with a single targeted command.
- For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before
scheduling the full training/eval sweep.
Step 3: Instrument Before You Optimize
- Pick the right profiler/monitor:
- CPU work:
perf
intel vtune
cargo flamegraph
- language-specific profilers
- GPU work:
nvidia-smi dmon
nsys
nvprof
- DLProf
- framework timeline tracers
- Capture kernel/ops timelines, memory footprints, and data pipeline latency
so you have evidence when throttling or parallelizing.
- Record hot paths and I/O bottlenecks in notes so future reruns can jump straight to the culprit.
Step 4: Throttle and Sequence Work
- Use
nice, ionice, or Kubernetes/Slurm quotas to prevent starvation of shared nodes.
- Chain heavy tasks with guardrails:
- Rerun only the failed test/module
- Then (optionally) escalate to the next-wider shard
- Reserve the full suite for the final gate
- Stagger GPU kernels (smaller batch sizes or gradient accumulation) when memory
pressure risks eviction. Prefer checkpoint/restore over restarts.
Step 5: Log Decisions and Next Steps
Conclude by documenting the commands that were run and their resource cost
(duration, CPU%, GPU%), confirming whether they remained within the per-task
budget. If a full suite or long training run was necessary, justify why selective
or staged approaches were not feasible. Capture any follow-up tasks, such as
adding a new test marker or profiling documentation, to simplify future sessions.
Output Expectations
- Brief summary covering:
- baseline metrics
- scope chosen
- instrumentation captured
- throttling tactics
- follow-up items
- Concrete example(s) of what ran (e.g.):
- "reran
pytest tests/test_orders.py -k test_refund instead of pytest -m slow"
- "profiled
nvidia-smi dmon output to prove GPU idle time before scaling"
Exit Criteria
1---2name: cpu-gpu-performance3description: Establishes CPU/GPU baselines before resource-intensive operations. Use before builds, training runs, or any task that pins cores or GPUs for over a minute.4alwaysApply: false5progressive_loading: true6dependencies:7 hub:8 - token-conservation9 modules: []10metadata:11 source: https://skillmd.com/skills/athola/cpu-gpu-performance12---1314# CPU/GPU Performance Discipline1516## When To Use17- At the beginning of every session (auto-load alongside `token-conservation`).18- Whenever you plan to build, train, or test anything that could pin CPU cores19 or GPUs for more than a minute.20- Before retrying a failing command that previously consumed significant resources.2122## When NOT To Use2324- Simple operations with no resource impact25- Quick single-file operations2627## Required Progress Items281. `cpu-gpu-performance:baseline`292. `cpu-gpu-performance:scope`303. `cpu-gpu-performance:instrument`314. `cpu-gpu-performance:throttle`325. `cpu-gpu-performance:log`3334## Step 1: Establish Current Baseline35- Capture current utilization:36 - `uptime`37 - `ps -eo pcpu,cmd | head`38 - `nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv`3940 Note which hosts/GPUs are already busy.41- Record any CI/cluster budgets (time quotas, GPU hours) before launching work.42- Set a per-task CPU minute / GPU minute budget that respects those limits.4344## Step 2: Narrow the Scope45- Avoid running "whole world" jobs after a small fix. Prefer diff-based46 or tag-based selective testing:47 - `pytest -k`48 - Bazel target patterns49 - `cargo test <module>`50- Batch low-level fixes so you can validate multiple changes with a single targeted command.51- For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before52 scheduling the full training/eval sweep.5354## Step 3: Instrument Before You Optimize55- Pick the right profiler/monitor:56 - CPU work:57 - `perf`58 - `intel vtune`59 - `cargo flamegraph`60 - language-specific profilers61 - GPU work:62 - `nvidia-smi dmon`63 - `nsys`64 - `nvprof`65 - DLProf66 - framework timeline tracers67- Capture kernel/ops timelines, memory footprints, and data pipeline latency68 so you have evidence when throttling or parallelizing.69- Record hot paths and I/O bottlenecks in notes so future reruns can jump straight to the culprit.7071## Step 4: Throttle and Sequence Work72- Use `nice`, `ionice`, or Kubernetes/Slurm quotas to prevent starvation of shared nodes.73- Chain heavy tasks with guardrails:74 - Rerun only the failed test/module75 - Then (optionally) escalate to the next-wider shard76 - Reserve the full suite for the final gate77- Stagger GPU kernels (smaller batch sizes or gradient accumulation) when memory78 pressure risks eviction. Prefer checkpoint/restore over restarts.7980## Step 5: Log Decisions and Next Steps8182Conclude by documenting the commands that were run and their resource cost83(duration, CPU%, GPU%), confirming whether they remained within the per-task84budget. If a full suite or long training run was necessary, justify why selective85or staged approaches were not feasible. Capture any follow-up tasks, such as86adding a new test marker or profiling documentation, to simplify future sessions.8788## Output Expectations89- Brief summary covering:90 - baseline metrics91 - scope chosen92 - instrumentation captured93 - throttling tactics94 - follow-up items95- Concrete example(s) of what ran (e.g.):96 - "reran `pytest tests/test_orders.py -k test_refund` instead of `pytest -m slow`"97 - "profiled `nvidia-smi dmon` output to prove GPU idle time before scaling"9899## Exit Criteria100101- [ ] `uptime` and `ps` baseline captured and recorded before any102 build, training run, or test suite starts103- [ ] Scope narrowed to diff-based or tag-based targets (e.g.,104 `pytest -k`, `cargo test <module>`); full-suite justification105 documented if selective approach was not feasible106- [ ] Output summary includes: duration, CPU% or GPU% consumed, and107 whether the run stayed within the per-task budget108- [ ] Any follow-up tasks (new test markers, profiling docs) written109 to a todo or issue so they survive the session