Jmh Advanced

JMH advanced

Purpose

Design experiments in which JVM compilation, state topology, fixture lifecycle, hardware, and instrumentation are explicit factors. Advanced annotations do not create realism automatically; they make it possible to represent and distinguish mechanisms.

Ownership boundary

• jmh-microbenchmarks owns semantic validity, observable work, basic modes/forks, uncertainty, and production extrapolation.
• This skill owns groups/topologies, parameters, auxiliary counters, profilers, compiler controls, cold-state protocols, and difficult variance.
• reading-jit-assembly owns instruction-level interpretation.
• performance-regression-ci owns automation and baseline governance.
• concurrency-testing owns correctness and schedule exploration; JMH is not a linearizability proof.

Advanced experiment contract

Inspect the project's compiler release, benchmark build/annotation processor, pinned JMH version and actual fork executable/arguments before changing the experiment. The Java snippets below were compiled with --release 17 and JMH 1.37, with a JDK 25 smoke run; this is not permission to upgrade the target toolchain. Preserve existing project compatibility. When measurement or profiler evidence is unavailable, return the experiment and missing checks, not a claimed winner.

hypothesis and competing mechanisms:
state ownership and sharing graph:
actor roles, ratios, threads, CPU/NUMA placement:
input/key/access distribution and mutation lifecycle:
success/failure/retry/drop counters and invariant oracle:
JVM/JDK/JMH factors and compiler context:
profiler/counter question, adequacy, overhead and control run:
cold/steady/transition cache and process state:
fork/block/randomization design and practical effect:

State topology before annotations

Thread count is a workload factor. Sweep across meaningful concurrency and topology points rather than publishing only the best saturation point. Record logical CPUs, SMT siblings, sockets/NUMA, cpuset/quota, and whether worker placement changes between forks.

Scope controls state-instance ownership, not the complete object graph. Scope.Thread states can still share static fields or referenced objects; trace those aliases before concluding that a benchmark has no contention.

Asymmetric groups

@Group and @GroupThreads model roles that share a Scope.Group instance. Define:

• actor ratio and whether it reflects arrivals or merely continuously looping workers;
• operation result and counters for success, miss, retry, full/empty, timeout, or failed CAS;
• initial occupancy and whether it drifts during an iteration;
• fairness/starvation per actor, not only total throughput;
• shutdown/progress behavior when one role stops or throws.

A 1 producer : 3 consumers closed loop is not automatically a production 1:3 arrival ratio. Backpressure and actor speed determine realized operations. Returned results prevent some dead-code elimination but are not automatically counted as successes/misses. Add explicit outcome counters to observe the realized mix. With a four-thread group, eight total workers normally form two independent groups; Scope.Group does not become one eight-worker object.

Parameters and experimental matrices

@Param expands combinations. Estimate run cost before launching:

cells = product(parameter cardinalities) * benchmark methods * modes * JVM variants
approximate time = cells * forks * (warm-up + measurement + lifecycle overhead)

Avoid a full Cartesian product when impossible combinations, redundant dimensions, or insufficient replication make it wasteful. Split experiments, generate a justified design, or use command-line parameter subsets. Keep one primary factor per causal comparison when interactions are not the question; use a factorial design when they are.

Input distributions need semantic names and reproducible generation. A fixed seed aids replay but does not provide population diversity; use multiple predeclared seeds/data cohorts when input variance matters and retain cohort identity.

Fixture arbitration and mutable state

Level.Trial, Iteration, and Invocation describe lifecycle, not whether costs are harmless. With shared state, helper invocation and teardown thread ownership can differ. Level.Invocation requires per-invocation timing and may require synchronization on the critical path; official JMH documentation also warns about overlap and coordinated omission.

For mutable structures, establish:

• reset point and whether reset is outside timing;
• cache/branch/type-profile/heap consequences of reset;
• data exhaustion, wraparound, ABA/version, and overflow behavior;
• invariant verification after an iteration/fork;
• whether reset creates an unrealistically pristine state.

Prebuilding a finite pool removes construction from the boundary but introduces reuse, cache locality, index coordination, memory footprint, and exhaustion. Measure or vary those factors.

Fork and JVM controls

Forks isolate JVM history and expose compilation/environment variation. Select counts from pilot variance and decision power, not a universal number. Warm-up forks, iteration warm-up, and measurement forks answer different lifecycle questions.

@Fork/CLI can select a JVM and append/prepend/replace arguments. Confirm the pinned JMH API and effective command; do not invent attributes such as jvmVersion. Run candidate JDKs as explicit blocks with otherwise controlled factors and retain vendor/build, flags, feature status, and hardware.

Controls such as equal -Xms/-Xmx, AlwaysPreTouch, fixed collector, affinity, performance governor, disabled turbo, or isolated host can reduce selected variability while changing the phenomenon. Use two layers when necessary:

1. mechanism experiment under controlled conditions;
2. representative experiment with production ergonomics and variance.

Profiler selection

Discover the pinned harness/environment:

java -jar benchmarks.jar -lprof
java -jar benchmarks.jar -prof <profiler>:help

Profiler names, options, prerequisites, and availability change. See references/profilers-and-hsdis.md. Separate:

• decision runs: minimal necessary instrumentation;
• diagnostic runs: profiler enabled to explain the mechanism;
• calibration runs: same workload with/without profiler to quantify interaction.

A profiler can change compilation, scheduling, allocation, cache pressure, timing, or the winner. Agreement is evidence; disagreement is a finding to reconcile.

Counters and denominators

Use secondary results only after defining their collection boundary and denominator:

• normalized allocation may reflect compiled escape context, TLAB accounting, rounding, and operations-per-invocation;
• hardware counters may multiplex, lack PMU support, include/exclude kernel/harness activity, and suffer skid;
• JMH 1.37 @AuxCounters require Scope.Thread; public numeric fields/methods become metrics. OPERATIONS normalizes to benchmark time, while EVENTS counts events without that time normalization. Neither automatically means a success fraction. Updates can perturb the path;
• total throughput can improve while successes per operation fall.

Report coverage/multiplex ratio, raw and normalized units, unsupported counters, and whether the counter population matches the timed operations.

Annotated assembly and compiler controls

Assembly profilers require a compatible OS collection path plus enough code/symbol/disassembly support. Annotation may be absent or partial while the numeric benchmark still succeeds. Validate:

• target compiled method/version and code-cache address mapping;
• sample coverage and unknown/unmapped share;
• compilation level, inlining, deoptimization, and multiple nmethods;
• hardware-counter event and sampling skid;
• architecture-specific disassembler/tool compatibility.

Use @CompilerControl only to answer a compiler-context hypothesis. Forcing or preventing inlining changes optimization scope, escape analysis, vectorization, register pressure, and call shape; it does not automatically make two variants fair. Preserve an unforced representative run.

Cold and transition experiments

SingleShotTime does not define “cold.” Declare which layers reset between observations:

fresh invocation / iteration / JVM / container / host
class initialization and compilation state
CDS and code/data/page cache
heap/allocator/GC state
connection/TLS/DNS/dependency state
CPU frequency and storage state

If reset cannot be proven, call the result first-use-under-specified-state rather than cold start. Startup questions often require process-level orchestration and JFR from launch, with JMH used only for the isolated mechanism.

Variance diagnostic tree

fork clusters
  -> compilation/deoptimization/type profile? inspect compilation/JFR/assembly
  -> host/core/NUMA/frequency/throttle? inspect OS placement/counters
  -> data/seed/state drift? compare cohort and invariant artifacts
  -> GC/heap lifecycle? compare allocation/occupancy/GC per fork

profiled and unprofiled winner differs
  -> profiler overhead/engine/compiler interaction? calibrate and separate claims
  -> inadequate samples/unknown symbols? validate coverage

thread sweep scales unexpectedly
  -> realized success/mix changed? auxiliary counters
  -> contention/coherence/false sharing? topology + counters/profile
  -> quota/SMT/NUMA placement changed? OS evidence
  -> correctness/progress failure? dedicated concurrency tests

Do not respond to unexplained variance by only lengthening iterations. More observations of a confounded state improve precision around the wrong mixture.

Anti-patterns

Definition of done

• State topology, actor mix, success counters, mutation, and invariants are explicit.
• Parameter matrix is feasible and interaction/seed strategy justified.
• Lifecycle/reset and environmental controls match the estimand.
• Fork/block/randomization and practical effect design are documented.
• Profiler/counter support, adequacy, overhead, and artifact integrity are validated.
• Diagnostic and decision runs are separated or combined with calibrated justification.
• Clusters, drift, and profiler sensitivity are explained, not averaged away.
• The claim is bounded and handed to the correct production/load/concurrency validation.

References

• Configuration recipes and variance diagnosis — read when implementing state, group, counter, seed or reset protocols.
• Profilers and annotated assembly — read when selecting a profiler or investigating coverage, counter or assembly failures.
• OpenJDK JMH project
• JMH profiler sample
• JMH asymmetric benchmark sample
• JMH Level API warnings