Gpu Undervolt

Autonomous GPU undervolting

You are going to tune real hardware that the user depends on. The value you add is measurement and judgment, not settings recommendations — anyone can read a forum post that says "try 875 mV @ 1850 MHz". What almost nobody does is establish, on this specific die, where the actual boundary is and how much margin the daily profile has from it.

Two things make this genuinely hard, and both are why the naive version of this task produces confidently wrong answers:

1. An unstable GPU returns wrong answers before it crashes. A benchmark that only reports FPS will score a data-corrupting undervolt as a clean pass. If your stability test cannot detect silent corruption, you are not testing stability.
2. Silicon varies enormously. Numbers from another card — including the ones in this repo — are a starting region to probe, never a setting to apply.

Before anything else

Read references/safety.md and hold to it. The short version: you may do reversible software tuning through the vendor tool. You may not touch VBIOS, firmware, BIOS/UEFI, PCI config, kernel parameters, module parameters, bootloader, CPU or RAM tuning, and you may not raise voltage above stock or disable any thermal or power protection.

Establish these gates before the first hardware write. If any fails, stop and tell the user rather than working around it:

• Identity — exact GPU, PCI BDF, driver, VBIOS. If more than one NVIDIA GPU is present, pin the target by BDF and use it everywhere. Never assume GPU 0.
• Reset works — apply a reset and confirm the curve reads back as stock. You are about to make the card unstable on purpose; prove you can undo it first.
• Boot persistence is off — an experimental curve must never survive a reboot. Verify, don't assume.
• Telemetry works — including hotspot and memory temperatures, which nvidia-smi does not expose. On NVIDIA this needs a helper such as gputemps; fan RPM needs NVML directly.
• Stock snapshot saved — the full VF curve, to a file.

Then show the user a compact preflight summary and proceed automatically. Do not ask permission for individual candidates; that turns a six-hour autonomous run into a six-hour conversation. Stop only for the conditions in references/safety.md.

Use the harness, don't improvise

This repo ships a deterministic harness in harness/. Drive it rather than inventing privileged commands per iteration — reproducibility is the entire point, and hand-typed nvidia-smi calls scattered through a session are not reproducible.

harness/preflight.py     identity, capability, reset-recovery proof
harness/baseline.py N    stock reference, N passes
harness/sweep.py         Stage A/B: voltage descent, then frequency frontier
harness/stagec.py        Stage C: fill frontier gaps
harness/extended.py      long soak of one candidate
harness/fandiscover.py   what fan control actually exists
harness/fantune.py       duty -> thermal equilibrium map
harness/rank.py          rank candidates by efficiency

If the hardware differs from what the harness assumes, extend the harness and say so in the report. Don't fall back to improvising.

Phase order, and why it is this order

0 — Calibrate the workloads before trusting them. Run each candidate workload once and record actual board power. A suite that peaks at 300 W against a 340 W limit is a screening tool, not a stability test. You need something that reaches the power limit and something that hammers clock/voltage transitions — see references/workloads.md. Getting this wrong invalidates everything downstream, which is why it comes first.

1 — Stock baseline, twice. Two passes, thermally settled. The second pass is your reference. Two passes also give you the run-to-run noise floor, without which you cannot tell a real 2% regression from measurement scatter. Record idle separately.

2 — Voltage descent at fixed frequency. Hold a frequency modestly below stock boost and step down one real voltage bin at a time until something breaks. Use bins the card actually reports; do not invent voltages.

3 — Frequency frontier. The lowest stable voltage is not the answer. At several voltages, climb frequency until failure. You are mapping a frontier, not finding a minimum — a point that keeps more performance for slightly more power is usually the better daily profile.

4 — Extended validation. Take the best 2–3 and soak each for 20+ minutes across every workload and every graphics API you have. Short runs under-detect the failure modes that matter.

5 — Fan work, only now. Changing fans before the undervolt is settled contaminates every thermal comparison you have made. Discover what control actually exists, map duty against steady-state temperature under load, then design the curve from that map.

6 — Combined validation. The undervolt and the fan curve must pass together, as a unit. They interact.

What to measure, and what it means

Log at 1 Hz throughout: clocks, power, utilisation, core temp, hotspot, memory temp, per-fan duty and RPM, P-state, throttle reasons.

Three interpretation traps that will otherwise cost you hours:

Averages, not peaks. A peak hotspot reading is a transient. Comparing peaks between configurations will tell you nothing changed when in fact the average dropped 10 °C. Report both; reason from averages.

Know which sensor is which. "Junction" from gputemps is the die hotspot, not memory. Memory is reported separately. Confusing them sends you after thermal pads when the problem is the main die's contact with the coldplate — or vice versa. Check the idle-vs-load delta to tell them apart.

Core-to-hotspot delta is a diagnostic. 10–15 °C is healthy on Ampere. If it sits above ~20 °C and does not improve when you add airflow, that is a thermal-interface problem. No fan curve can fix it, and saying so is more useful to the user than shaving 50 RPM.

Failure classification

A candidate fails on any of: workload crash or non-zero exit, checksum mismatch, new kernel Xid, curve read-back mismatch, performance collapse, rendering corruption, hang, or the GPU becoming unresponsive.

Match kernel errors against a journal cursor recorded per candidate, or old errors from earlier candidates will be attributed to innocent ones.

When something fails: stop the workload, record it, preserve telemetry, reset, verify the GPU responds, wait for it to normalise, and stop descending in that direction. Never re-run a known-failing configuration to "check".

Interpreting Xids and distinguishing hardware instability from application bugs: references/failure-modes.md.

Choosing the daily profile

Rank by efficiency, but never pick on one metric. Compute performance ratio, power ratio, efficiency gain, watts saved, and the temperature and RPM deltas.

Then apply margin. Do not recommend the discovered boundary. Back off at least one real voltage bin above the failure point, or 15–30 MHz below a borderline clock. When two candidates perform the same, take the one with more margin — margin is free performance you haven't spent.

State the margin honestly and from both directions. "45 MHz below its measured wall" and "one voltage bin above a fault at this same clock" can both be true of one profile, and the user deserves both framings.

If a measurement contradicts your earlier recommendation, change the recommendation and say that you did. That is the job.

Fan tuning

Read references/fan-tuning.md before designing a curve. The essentials:

• Measure the duty→RPM response first. Vendor curves often never use zero-RPM even when the card supports it, and there is usually a low-duty band where fans stall unevenly. Both are discoverable in ten minutes.
• NVML reports fan controllers, not fans. A three-fan card commonly exposes two. Do not claim fans are synchronised when you have only verified the controllers.
• Wait for equilibrium. Five minutes of sustained load per setting, and take the tail as steady state. Transient temperatures produce a curve that hunts.
• Under sustained load, temperature is often the controlled variable and clock is the dependent one. The card pins its hotspot and gives back MHz. When that is happening, fan duty trades against sustained clock speed, not against temperature — so a "quieter" curve silently costs performance. Detect it by watching whether hotspot is pinned at the same value across several duties while clock varies.
• Design the slope from the measured plant response so the loop converges instead of oscillating. Use hysteresis.

Reporting

Write a report that a sceptical reader could check. Include the hardware identity, methodology, every meaningful candidate, the failures, the discovered boundary, the final profile with its margin, before/after comparison with deltas, tests performed and their duration, Xids encountered with your interpretation, the fan curve, and what you did not do.

Two rules that matter more than they look:

Preserve failed results. They are the evidence that the boundary is real. Never silently discard an unstable candidate.

Document your own mistakes. If you misread data, wrote a bug that invalidated a run, or changed your recommendation, put it in the report. A write-up where the agent never errs is less trustworthy than one where the corrections are visible — and the reader needs to know which results were affected.

Never claim acoustic measurements without a microphone. RPM is a proxy; say so.

Finishing

Terminate every stress and telemetry process. Verify GPU health. Check the kernel log. Leave the validated profile applied for the session, save the profiles, and leave boot persistence disabled — then ask whether the user wants it made persistent. Their machine, their call.

Offer the user a real-world validation run (a game, a render) as the final check: it exercises code paths — ray tracing especially — that synthetic workloads miss, with a human watching for visual corruption.

References

• references/safety.md — the safety envelope and stop conditions
• references/workloads.md — building workloads that actually detect instability
• references/failure-modes.md — Xid interpretation, silent corruption, classification
• references/vf-curves.md — how VF curve editing actually behaves (offsets, drift, interpolation)
• references/fan-tuning.md — fan control discovery and curve design
• references/pitfalls.md — specific traps that cost time in the original run