vamseeachanta Performance Tuning
When the user complains about slowness ("Hermes is slow", "work is lagging", "builds are crawling", "everything feels sluggish"), do not guess. Diagnose, then act.
Mental model
The most common failure mode is assuming "slow" means "CPU-bound". On these 32-thread Xeon boxes, CPU is rarely the bottleneck. The usual suspects in order:
- Filesystem driver (NTFS/ntfs-3g via FUSE on
/mnt/local-analysisis single-threaded and serializes IO across all 32 cores) - CPU frequency governor (
powersavedefault keeps cores at ~1.6 GHz) - Background indexers (
tracker-miner-fs-3,baloo,mlocate.updatedb) - Memory pressure / swap (rare on 31 GiB box, but check)
- Network mounts (sshfs, NFS — slow when remote host is unreachable)
- Genuinely CPU-bound (only after eliminating the above)
Diagnostic order (5 minutes total)
Run these in order. Stop as soon as you find a clear answer.
1. System pulse
uptime # load avg vs nproc — if load << threads, CPU is NOT the bottleneck
free -h # check available memory and swap usage
ps -eo pid,user,pcpu,pmem,etime,comm --sort=-pcpu | head -10
2. Filesystem of the working directory (the most-missed diagnostic)
mount | grep "$(git rev-parse --show-toplevel | cut -d/ -f1-3)"
If the line contains ntfs, ntfs-3g, or fuseblk → this is the bottleneck. No further CPU diagnosis needed; route to runbook section "Migrate /mnt/local-analysis from NTFS".
3. Governor state
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
lscpu | grep "scaling MHz"
If governor is not performance or scaling is <80%, install the cpupower service from the runbook.
4. Background indexer noise
ps -eo pcpu,comm --sort=-pcpu | grep -E "tracker|baloo|updatedb|spotlight" | head
5. Run the benchmark for hard numbers
scripts/benchmarks/perf-bench.sh diag-$(date +%s)
column -t -s $'\t' scripts/benchmarks/results.tsv | tail -10
Reference numbers on ace-linux-1 (governor=performance, NTFS-3G workspace):
cpu_single~500 ms — anything >800 ms means CPU contentioncpu_multi~1400 ms — should be close tocpu_singleon a dedicated box; >3× means IO/scheduling contentiongit_statuswas 155,707 ms with NTFS-3G — anything >2 s is a filesystem-driver smellgit_log~50 ms — should always be fast (cached)
If git_status is in the seconds, suspect IO contention. If in the minutes, suspect filesystem driver.
Fixes (apply via the runbook)
The runbook at docs/runbooks/perf-tuning.md is the canonical procedure. This skill's job is to reach the right runbook section quickly:
| Symptom | Runbook section |
|---|---|
git status >5 s on workspace-hub |
"Migrate /mnt/local-analysis from NTFS" + interim ntfs3 mitigation |
| Idle scaling MHz <80% | "CPU frequency governor → performance (persistent)" |
| Cross-socket memory thrashing on heavy job | "NUMA-pin heavy batch jobs" |
| Compute-bound solver (FEA/CFD) under-performing | "Disable Hyper-Threading" + benchmark |
tracker-miner-fs-3 >20% sustained |
"Turn off tracker3 for paths you don't search" |
Anti-patterns to avoid
- Don't reach for BIOS toggles before running the benchmark. HT-off blindly can hurt the overnight 5-terminal batch pattern. Numbers first.
- Don't recommend
tracker reset --hardwithout checkingtracker3 statusfirst — on ace-linux-1 it's idle and indexes only$HOME(not workspace-hub). Resetting it would just re-index later. - Don't remount the live workspace filesystem (
umount /mnt/local-analysis) while the user has a shell or editor in it. Plan filesystem changes for a maintenance window. - Don't skip the "filesystem of the working directory" check. This is the diagnostic most likely to surface a 100× win and the one most often missed.
Persisting findings
After diagnosis, append a row to scripts/benchmarks/results.tsv with a meaningful label so the next session can compare. Memorable labels: pre-{change}, post-{change}, diag-{date}.
If you discover a new bottleneck class not in the runbook, propose an addition rather than fixing in-place. The runbook is the durable artifact; this skill is the routing layer.