Gpu Optimization Validation

GPU Optimization Validation

Skill navigation

• Parent/orchestrator: gpu-code-optimizer
• gpu-performance-evidence — load for baseline methodology and bottleneck evidence
• gpu-numerical-safety — load for C2+ error reports and semantic acceptance
• gpu-training-autodiff — load when the path is used for training
• gpu-resource-lifetime-allocation — load when accepting a lifetime, allocation, workspace, aliasing, or rematerialization policy
• gpu-virtual-memory-fragmentation — load when accepting a backing, VMM, fragmentation, or allocatability policy
• gpu-memory-tiering-migration — load when accepting a placement, residency, prefetch, offload, replication, or migration policy
• gpu-state-reuse-eviction — load when accepting state reuse, sharing, admission, retention, invalidation, or logical eviction
• gpu-persistent-state — load when accepting a cross-call state contract
• gpu-memory-scheduling — load when accepting a joint compute/memory schedule
• gpu-code-optimizer — return to orchestration when validation reveals a new bottleneck

Load linked skills only when their trigger applies. Do not duplicate their full workflow here.

Core principle

Treat every optimization as an experiment with an explicit hypothesis, guard domain, fallback, and acceptance criterion. The final artifact must let another engineer reproduce the claim and know when not to use the fast path.

Evidence-scoped experiment ledger

Maintain a compact ledger for every material probe or variant. The ledger is the memory of the search: it prevents repeated dead ends without turning a conditional rejection into a permanent blacklist.

Use one entry per one-factor variant and anchor all entries in the same baseline when they belong to one round:

Experiment ledger entry:
- ID: <stable identifier>
- Baseline ID: <stable identifier for the immutable snapshot>
- Baseline: <immutable snapshot: code, device, stack, workload, benchmark state, reference result>
- Reset/restore: <procedure that returns code, artifacts, allocator/cache state, inputs, and runtime mode>
- Evidence state / workload domain: <shapes, dtypes, layouts, modes, profiler/counter validity>
- Hypothesis / proposed mechanism: <causal explanation>
- Variant: <one changed relation, or observation-only probe>
- Prediction: <target metric and independent mechanism evidence>
- Falsifier: <observation that weakens or rejects the hypothesis>
- Observation: <result, uncertainty, and residual>
- Confounders checked: <compiler reachability, timing scope, cache/clock state, contention, etc.>
- Confidence: <high / medium / low, with update reason>
- Disposition: Keep as default / Keep behind guard/flag / Keep as local micro-optimization only / Reject / Need more evidence
- Rejection reason: <which prediction or acceptance gate failed>
- Reopen if: <specific changed fact that invalidates this rejection>
- Reopened from: <prior ledger entry ID, or none>
- Composition dependencies: <accepted or pending variants that may change the result>
- Constituent IDs: <variant IDs when this entry is a composition>
- Interaction result: <combined delta, uncertainty, and non-additive cost>
- New bottleneck: <post-composition or post-acceptance limiting resource>

Use the exact disposition vocabulary from the Acceptance matrix below. If a result is not yet attributable, reproducible, or sufficiently measured, use Need more evidence rather than silently treating it as rejected or accepted.

Rejection and reopening rules

Record a rejected hypothesis with its evidence state, falsifying observation, and confounders checked. Do not rerun it merely because the next optimization round has started. Reopen it only when a relevant fact changes, such as:

• compiler lowering, generated code, runtime schedule, or measurement validity;
• hardware, shape, dtype, layout, or representative workload domain;
• an accepted change moving the limiting resource or removing the cost that caused rejection;
• a semantic, numerical, guard, or fallback contract becoming different with explicit review.

When reopening, create a new ledger entry linked to the old one and compare it against the new baseline. Preserve the old result as historical evidence; do not overwrite it or present the new result as if it invalidated the original scope.

Composition revalidation

Independent wins do not prove that a combination wins. Keep variants attributable to the same baseline, then validate a composition as a new experiment:

1. Re-check that each constituent's guard, fallback, correctness contract, and intended lowering still hold together.
2. Measure the composition against the same baseline, record the constituent IDs and interaction result/uncertainty, and report the interaction rather than an assumed sum of isolated deltas.
3. Re-run resource, allocation, synchronization, compile/capture, and end-to-end checks for costs that only appear together.
4. Re-profile the combined path and record the new bottleneck or limiting resource.
5. Keep, guard, or reject the composition independently; retain useful constituents when the interaction is the problem.

If benchmark budget is tight, use a staged decision: finish the cheapest discriminating probes first, then test only compositions that remain plausible and relevant to the target metric. A composition that cannot be attributed, reproduced, or explained remains Need more evidence, not an automatic default.

Representative benchmark matrix

Benchmark at least:

• small, target, and maximum sizes;
• non-power-of-two and non-divisible dimensions;
• common and worst-case layouts/strides;
• supported dtypes and precision modes;
• relevant batch/sequence/channel/head dimensions;
• real input distributions plus extreme/pathological values;
• cold and warm states where compilation/autotuning/caching matters;
• training and inference modes if both are supported;
• multiple target devices if portability is claimed.

Do not optimize only a hand-picked benchmark shape and generalize the claim to all inputs.

Timing protocol

Use an appropriate device-side timer, framework benchmark utility, or synchronized timing method. Account for asynchronous execution. Warm up compilation, caches, allocators, and autotuners as appropriate before steady-state timing.

Report:

• sample count;
• median and a tail statistic such as p95 where latency matters;
• variance or confidence interval when noise is material;
• synchronization method;
• whether compilation/capture/autotuning is included;
• exact workload scope.

For very short kernels, measurement overhead can dominate. Prefer batched repetitions or profiler kernel timing while preserving realistic launch behavior.

Correctness protocol

Start from a known-good reference. For exact semantic paths, compare all relevant outputs and boundary conditions. For C2+ changes, use the error-reporting contract in gpu-numerical-safety.

Do not weaken existing tests merely to make the optimization pass. A tolerance change requires a documented numerical rationale and approval consistent with the semantic class.

Mode-specific recommendation

Many operators have multiple execution modes with different bottlenecks:

• Training forward / backward.
• Inference batch / single request.
• Online streaming / offline.
• Prefill / decode (autoregressive).
• Long-context / short-context.
• Low-latency / high-throughput.
• Memory-constrained.
• Multi-GPU.
• CPU fallback.
• Small batch / large batch.

The best path for one mode may be the worst for another. Report which mode(s) the optimization targets, and document which modes should use the fallback.

Decision record

Every optimization patch should include a short decision record:

Decision:
- Problem: <what was slow / memory-heavy>
- Bottleneck: <classified bottleneck>
- Change: <concrete description>
- Numerical class: C1 / C2 / C3 / C4 / N/A
- Semantic/runtime flags: <material flags only; see gpu-numerical-safety>
- Why safe: <numerical / semantic justification>
- Guard: <conditions enabling fast path>
- Fallback: <path taken when guard fails>
- Evidence: <profiler trace / IR / benchmark that confirms>
- Result: <before → after, with metric>
- Rejected alternatives: <what else was considered and why dismissed>
- Remaining risk: <open numerical / performance concerns>

For resource or runtime-state changes, append only the applicable specialist record. Do not force lifetime, backing, residency, reuse, persistent-state, and scheduling fields onto an ordinary kernel optimization.

Failure-case record

Optimization reports must include failure cases:

• Which shapes are not improved (or regress)?
• Which dtypes are not supported?
• Which layouts are not supported?
• Which input value ranges cause numerical issues?
• Which devices / architectures are not supported?
• Which precision modes break the fast path?
• Which batch sizes show no improvement?
• At what scale does the fallback become preferable?
• When is error too large?
• When does memory increase?
• When does compile time increase?
• When does end-to-end regress?

Failure cases define the guard conditions. Without them, the fast path will be misused.

Patch strategy

When editing code:

• Make one optimization at a time when possible.
• Preserve the original path as a fallback.
• Add shape, dtype, alignment, and layout guards.
• Keep boundary handling explicit.
• Keep synchronization explicit.
• Avoid hidden global state.
• Avoid undocumented assumptions.
• Prefer readable optimized code over fragile cleverness unless the hot path demands it.
• Include comments only for non-obvious performance or correctness constraints.

Review output contract

1. Bottleneck hypothesis: Classified bottleneck with evidence source.
2. Evidence: Code, profiler output, IR, byte/FLOP reasoning, or hardware counter.
3. Numerical class: C1 / C2 / C3 / C4 / N/A per proposed change, plus any material semantic/runtime flags (see gpu-numerical-safety).
4. Numerical behavior changed?: Yes / No, with justification.
5. Optimization plan: Ranked by impact, each with guard conditions.
6. Concrete patch or pseudocode.
7. Guard conditions: When is the fast path safe?
8. Fallback path: What runs when guard conditions are not met.
9. Bytes saved estimate: Intermediate tensor delta.
10. Kernel count before / after.
11. Peak memory before / after.
12. Allocation count before / after.
13. Expected speedup: Isolated + end-to-end estimate.
14. Correctness risks: Numerical behavior, boundary semantics, backward impact.
15. Error statistics: Per the error-reporting contract in gpu-numerical-safety (for C2+).
16. Unsupported cases: Shapes, dtypes, layouts, value ranges where fallback is required.
17. End-to-end impact: Full step / request / iteration time delta.
18. Decision: Keep / reject / behind flag, with rationale.
19. Next bottleneck: What to optimize after this change.
20. Benchmark plan: Isolated + end-to-end, shapes to test.

Answer format when generating optimized GPU code

1. Optimized code or pseudocode.
2. Numerical class (C1–C4, N/A) plus material semantic/runtime flags (see gpu-numerical-safety).
3. Assumptions (hardware, framework, shapes, dtypes, value range).
4. Guard conditions.
5. Fallback path.
6. Correctness test (forward + backward).
7. Error statistics (see gpu-numerical-safety).
8. Benchmark harness or measurement instructions.
9. Allocation + kernel-count delta.
10. Remaining bottlenecks.
11. Failure cases (when to use fallback).

Use proportionality: a quick review need not emit twenty empty sections. A production optimization claim, benchmark report, or merged fast path should include all material fields.

Red flags

Re-evaluate when:

• Fusion introduces register spills.
• Fusion reduces useful occupancy too much.
• Added branch divergence cancels saved memory traffic.
• Added random access cancels saved memory traffic.
• A local layout change slows the next major operation.
• A microbenchmark excludes conversion, allocation, synchronization, or auxiliary reduction cost.
• A kernel benchmark improves but end-to-end runtime regresses.
• A tuned library primitive is replaced without proving composition overhead.
• Boundary cases are slower or incorrect.
• Numerical tolerances change without approval.
• Determinism changes without approval.
• The profiler does not support the bottleneck hypothesis.
• The optimization only helps shapes that are not representative.
• The error grows with sequence length or batch size (systematic bias, not random noise).
• The fast path silently produces incorrect results on edge cases (all-zero, all-one, empty, singleton, padded).
• A matmul/conv is claimed to use Tensor Cores without profiler or IR evidence.
• Shared memory is used without evidence of improved reuse, coalescing, or scheduling.

Acceptance matrix

A change may be marked:

• Keep as default — correctness contract passes; representative end-to-end target improves; failure domain is acceptable.
• Keep behind guard/flag — useful fast path with narrower hardware/shape/numerical domain.
• Keep as local micro-optimization only — isolated kernel improves but end-to-end result is neutral; do not claim application speedup.
• Reject — correctness risk, end-to-end regression, excessive memory/compile cost, or unsupported bottleneck hypothesis.
• Need more evidence — measurement cannot yet distinguish the hypothesis from noise or confounding factors.

Conditional resource and state validation

Apply each row only when its specialist trigger is present.

Require the policy's no-change or fallback alternative in the same measurement scope. Reject default paging, offload, recency eviction, reuse, proactive migration, or overlap when workload evidence does not beat that alternative.

Separate projected, mechanism-feasible, and measured outcomes. A policy model does not become an optimization result until the runtime trace and end-to-end target confirm it.

Benchmark confounders

Common confounders include:

• JIT compilation or autotuning leaking into timed iterations;
• GPU clock/power state changes between runs;
• thermal throttling;
• background GPU work;
• different allocator/cache warmness;
• host scheduling noise for microsecond kernels;
• different synchronization scope;
• different data transfer inclusion;
• random inputs that trigger different sparsity or branch behavior.

Control what is practical and disclose what is not. For close results, repeat interleaved A/B runs rather than running all of A and then all of B.

A/B comparison design

Prefer identical environments and inputs. Randomize or interleave run order when drift is possible. Compare the same metric scope and report both absolute and relative changes.

For an end-to-end path, decompose enough to explain the result:

• target phase before/after;
• new overhead introduced elsewhere;
• kernel/launch count changes;
• memory/copy changes;
• compile/capture cost changes.

A surprising result is a reason to inspect the timeline, not to discard the benchmark.

Portability claims

Do not claim a GPU optimization is portable merely because it compiles on multiple backends. Distinguish:

• semantic portability;
• functional backend support;
• performance portability;
• architecture-specific fast paths.

For each claimed target, report the tested device family, software stack, supported shapes/dtypes/layouts, and fallback. A portable reference path plus architecture-specific guarded fast paths is often the cleanest design.

Reproducibility bundle

For production work, preserve enough information to reproduce the result:

• commit or code version;
• device model and relevant topology;
• driver/runtime/compiler/framework versions;
• benchmark command/configuration;
• shapes, dtypes, layouts, seeds;
• environment variables that affect compilation or math modes;
• profiler trace or summary when it supports a key claim.

Do not overload the user-facing summary with all raw data; keep the evidence available and present the decisive facts.

Regression checks

After the change, re-run:

• kernel/operator count;
• peak memory and allocations;
• copies and synchronization;
• compile/capture/autotune cost;
• key hardware counters for the new bottleneck;
• full application/request/iteration timing.

The bottleneck may move. Do not continue optimizing the old bottleneck without reclassification.

Final sign-off

Before presenting the result, verify:

• target metric is explicit;
• baseline and measurement scope are reproducible;
• optimization class is stated;
• guard conditions and fallback are documented;
• correctness and boundary tests pass;
• C2+ error statistics are complete;
• isolated and end-to-end results are both reported where relevant;
• unsupported devices/shapes/layouts/value ranges are listed;
• compiler/runtime evidence confirms the intended transformation;
• every applicable resource/state acceptance row passes without forcing unrelated rows;
• the decision is Keep as default / Keep behind guard/flag / Keep as local micro-optimization only / Reject / Need more evidence.