Java Performance

Java Performance

Purpose

Turn a vague symptom into a bounded investigation with competing hypotheses, a minimal evidence set, and the correct specialist owners. Do not route directly from one symptom to one cause: production regressions frequently combine load mix, queues, GC, compilation, I/O, dependencies, resource limits, observability, and deployment lifecycle.

This skill should leave context once the investigation protocol and specialist set are clear. performance-methodology owns the causal workflow from hypothesis through validation.

Triage contract

Before choosing a tool or owner, record:

user/business symptom and SLO impact:
metric definition, unit, aggregation, sample count, source, and clock:
affected percentiles/throughput/errors/cost and time interval:
offered load, completed work, operation/data/tenant mix, concurrency:
affected versions/instances/zones/hosts/clients and healthy controls:
startup/warm/steady/shutdown or incident lifecycle phase:
recent code/config/JDK/dependency/data/infrastructure/deploy changes:
resource demand/limits/queues and downstream health:
recovery deadline and evidence-preservation budget:

Start with available facts and mark unknowns; do not require every field before useful triage. For missing evidence, state which hypothesis remains unresolved and the smallest artifact or focused question that would distinguish it. A screenshot or unavailable process can support a collection plan, not a confirmed mechanism or an invented baseline.

Record the target JDK vendor/build, JVM implementation, collector, OS/container, agents and profiler versions separately from compiler --release. The linked tool documentation uses JDK 25/HotSpot; it is not a requirement to upgrade. Check the target's command help, event metadata and effective settings before transferring commands, flags or event assumptions. Preview/incubator APIs, attach access and platform-specific profilers need explicit target support.

If the process is degrading and remediation is imminent, invoke incident-evidence-capture before an ordinary investigation. Bounded capture must fit the recovery deadline: do not delay already authorized mitigation to complete this checklist or restart merely to enable a tool. If the measurement cannot be trusted, route first to latency-statistics, coordinated-omission, load-testing, or performance-methodology.

First classification

Use four independent axes rather than a single label:

“CPU low” is not “idle,” “throughput unchanged” is not equal work, and exit 137 is not by itself proof of OOM. Normalize resource by completed work where meaningful and preserve failures/ timeouts/cancellations.

Evidence selection

Choose the cheapest safe evidence that separates the leading hypotheses in this system. There is no universal bundle such as “GC log plus two-minute settings=profile JFR.” Existing continuous logs/JFR/profiles can be cheapest; a new profile/JFR can be inappropriate under disk, CPU, thread-count, privacy, or recovery pressure.

Useful parallel views, when already available or safely collectable:

• service-level latency/throughput/errors and workload mix;
• per-instance/container/cgroup CPU, memory, pressure, I/O, network, lifecycle;
• GC/JFR/runtime and thread/task evidence;
• dependency/database/queue/pool metrics and traces;
• code/config/JDK/dependency/deployment diff;
• one affected instance versus a genuinely comparable control.

Every collection has a time, overhead, privilege, survivability, and perturbation budget. Read-only diagnostics can still pause or trigger GC: inspect the target command's documented impact, not merely whether it changes application data. Keep files on bounded storage that survives the planned recovery action; use the evidence-capture owner for command selection.

Routing map

See Triage map for ambiguous symptom forks and Specialist map for the knowledge graph. The specialist map is not a mandatory shallow-to-deep ladder; load the minimum skills that own the actual questions.

Core rules

• Classify the measured outcome and work denominator before optimizing.
• Maintain multiple hypotheses until one is discriminated; correlated signals are not automatically causal.
• A deploy changes more than code: process age/JIT, caches, connections, traffic shifting, replicas, image/JDK/dependencies/config, data/schema, sidecars, and host placement.
• High latency with low average CPU can be downstream wait, a queue/pool, throttling, pauses, load-balancer skew, synchronization, network loss, client timing, or idle low demand. Measure.
• Worse scaling with more threads can be locks, shared resources, coherence/NUMA, GC, oversubscription, quotas, I/O, downstream saturation, or workload change—not automatically hardware cache effects.
• Pause frequency changes do not prove allocation changed; workload, heap occupancy, sizing, collector policy, humongous objects, promotion, concurrent-cycle progress, and explicit GC also matter.
• A restart that “fixes” performance resets much more than JIT/code cache: heap/live state, pools/connections, caches, leaks, queues, circuit state, load placement, native resources, and agents.
• A healthy CPU profile does not rule out elapsed wait; a healthy GC log does not rule out safepoint, OS pause, dependency, or measurement error.
• Never recommend a JVM flag, pool size, cache, virtual threads, SQL index, or architecture change from symptom shape alone.

Severity and decision path

live user impact and remediation imminent?
  -> preserve bounded evidence under incident authority

measurement invalid/ambiguous?
  -> repair definition/load/timing/omission before causal analysis

affected cohort and timeline known?
  -> compare affected versus compatible controls and recent-change epochs

leading mechanisms identified?
  -> collect one discriminating evidence set per surviving branch

mechanism materially contributes?
  -> hand off to owner; define intervention and validation

outcome improved under representative/repeated conditions?
  -> document evidence, trade-offs, guardrail/regression prevention

Troubleshooting triage failures

Handoff contract

Give the specialist:

precise symptom and affected cohort/window
metric and workload definitions with raw/normalized values
leading hypothesis and alternatives
evidence already collected, provenance, loss, and perturbation
why this evidence routes to that owner
recovery/experiment constraints
success, guardrail, and rollback criteria

Do not hand off only “high CPU” or a screenshot.

Definition of done

• Symptom, scope, time, work denominator, lifecycle, and recent changes are explicit.
• Measurement validity and live-incident evidence risk were checked first.
• At least plausible alternative hypotheses and discriminating signals are recorded.
• Affected/control cohorts and resource/dependency views are aligned.
• Specialist owners are selected by established questions, not presumed causes.
• No optimization or flag is proposed before material mechanism evidence.
• Handoff includes validation/rollback criteria and evidence limitations.

For this router, completion means a justified owner and bounded next discriminator, not a resolved performance incident. Unknowns may remain if their impact and next check are explicit. Do not keep loading the specialist catalog once one owner can advance the established question.

References

• Triage map — read for competing explanations of an ambiguous symptom.
• Specialist map — read when the owning question spans domains.
• Worked example: tail regression after deploy — read when separating logging, allocation, GC and rollout hypotheses after a deployment.
• JDK 25 diagnostic command impact — consult when budgeting collection, then check help on the target JVM.
• JDK 25 troubleshooting guide
• JDK Mission Control documentation