Async Profiler Advanced

Async-Profiler Advanced

Purpose

Choose a collection mechanism whose signal corresponds to the engineering question, prove what the runtime actually enabled, and state what the recording cannot establish. An async-profiler file is a sample of selected events under a particular stack walker, filter, rate limit, and access envelope—not a complete account of elapsed time.

Async-profiler evolves quickly. Pin its release, keep the matching binary and converter, run asprof -v, asprof list <pid>, and asprof --help, and consult that tag's documentation. Never make a runbook depend on master, a historical option name, or an assumed fallback.

The command reference is authored against async-profiler v4.5 for supported HotSpot targets on Linux/macOS; it is not a promise of support for native Windows, other JVM implementations, or every JDK update. Inspect the actual runtime build, container image, architecture and JVM flags; build toolchains alone may describe a different JVM. Keep profiler build requirements, target-JVM support and the converter's Java runtime separate. Do not upgrade the application, profiler or dependencies merely to make an example fit.

Ownership boundary

• Use jfr-and-async-profiler to choose the least-privileged first instrument.
• Use this skill to configure and validate async-profiler itself.
• Use flame-graph-analysis to reason from stack aggregates and differentials.
• Use allocation-profiling, concurrency-diagnostics, and off-heap-memory for the owning diagnosis once the event identifies the domain.
• Use ebpf-for-jvm when the question is system-wide or cannot be answered from the JVM process alone.

Start with a question contract

Write the target process, load window, event, interval/threshold, stack mode, filters, output, rate/memory limit, expected sample volume, and validation metric before collection.

If versions, command/logs, event weight, workload window, or session ownership are missing, request the smallest missing evidence and provide a conditional plan. Do not invent an engine, zero-loss result, safe interval, or diagnosis from an HTML filename. Separate observations from hypotheses; for each proposed adjustment name evidence that would confirm or refute it.

Engine selection

For CPU attribution on Linux, prefer the supported perf-events engine when kernel/user stack coverage and per-thread CPU accounting matter. Use ctimer when perf events are unavailable and current Linux/tool support is confirmed; use itimer where the platform lacks the better alternatives or for compatibility. These engines do not have identical selection probability, resolution, kernel visibility, resource cost, or failure behavior.

Use wall-clock collection for off-CPU residency and latency investigations. Wall mode samples eligible threads regardless of whether they are running, parked, sleeping, or blocked. --threads changes output grouping for non-JFR output; --filter changes eligible thread IDs where supported. They are different controls.

Do not assert that a requested event ran. Inspect start diagnostics, available status/metrics, output event types, weights, and kernel/native frame presence. Status and JFR event class alone may not identify the CPU engine. Mark unavailable loss counters or engine evidence as unknown. Current versions may choose a fallback for a generic CPU request; explicit event requests, platforms, and releases differ.

Read Sampling engines, events, and access when choosing an engine, interpreting lock/live/native events, diagnosing access or missing stacks, or checking PMU and virtual-thread coverage.

Access model and least privilege

Dynamic attach and event acquisition are independent:

• attach requires PID-namespace visibility, compatible filesystem /tmp view, attach enabled, target-compatible credentials, and readable/loadable profiler library/output paths;
• Linux perf events additionally depend on kernel policy, capabilities, seccomp/LSM, resource limits, PMU virtualization, and symbol policy;
• native/kernel symbolization requires matching binaries/build IDs and permissions beyond merely opening the event.

Diagnose the failing syscall/layer. SYS_PTRACE is not a universal fix for perf_event_open; CAP_PERFMON, CAP_SYS_ADMIN, --all-user, or an fd-transfer helper have different scope and security consequences. Avoid privileged containers when a CPU-timer or JFR recording answers the question. Record every exception and restore it after collection.

Sampling economics

Estimate event volume before profiling:

CPU-like samples ~= active CPU time / interval
wall candidates  ~= eligible threads * duration / interval
allocation events ~= allocated bytes / allocation interval
trace events      ~= selected invocations passing threshold

These are planning estimates, not guaranteed counts. Wall-clock cost can grow with eligible thread count; this matters acutely with large platform-thread populations. Batching, filters, rate limits, stack depth, memory limits, and longer intervals trade fidelity for overhead and file size. Instrumentation modes (--trace, native-allocation interception) incur work on intercepted calls, including calls rejected by duration/output filters. Bound a trial using total invocation rate, not just emitted events; method retransformation can also perturb compilation.

Calibrate overhead against the same workload using an unprofiled control and at least two collection intensities. Compare throughput, latency distribution, CPU, allocation/GC, and dropped/lost events. “Low overhead” is not an authorization to run every event in production.

Multi-event recordings and time

JFR output is appropriate when several async-profiler event classes or synchronized JDK JFR events need one timeline. Verify which combinations the pinned release supports; conflicts, rate budgets, and output semantics change. --jfrsync coordinates a JDK recording with the profiler, but configuration and event replacement semantics must be checked in that release.

Clock domains, timestamp preservation, chunking, conversion, and rate limits determine whether cross-event temporal claims remain valid. A collapsed stack file destroys most event metadata and timestamps. Preserve the original JFR plus exact command before conversion.

Differential evidence

A differential flame graph is descriptive evidence, not a performance test. Before/after must share workload, duration or valid normalization, warm-up state, event configuration, filters, stack semantics, build symbolization, and enough samples. Prefer paired repetitions and validate the performance outcome separately.

Normalization removes unequal total-sample tint; it cannot repair changed load mix or a different probability of selecting stacks. Positive/negative colors depend on converter and argument order—verify with a synthetic folded-stack pair rather than memorizing a palette.

Failure modes and troubleshooting

Anti-patterns

Anti-pattern: grant capabilities until it works. It conflates attach, event access, and symbols while expanding blast radius. Identify the failed layer, choose the least-privileged event, and make elevated access temporary and auditable.

Anti-pattern: fixed 60-second recording. Rare behavior may need longer or trigger-based capture; high-frequency instrumentation may need seconds. Choose duration from event rate, minimum useful sample count, incident window, overhead, and storage budget.

Anti-pattern: percentages without denominators. A 0.3% frame may be a handful of samples; wall and CPU percentages answer different questions. Report event, weight, total, interval, confidence limits where appropriate, and corroborating system metric.

Anti-pattern: latest converter over old recording without provenance. Converter semantics and event schemas evolve. Keep original file, producer version, converter version, command, and checksum; investigate differences before replacing prior output.

Minimum result

For a command request, return the bounded command, prerequisites, event/weight interpretation, ownership/stop rule, and success check. For a diagnostic review, add:

• observed evidence with artifact/window/versions and missing information;
• the leading hypothesis, competing explanation, and consequence;
• a proposed adjustment and a check that could confirm or refute it;
• what actually ran, artifact location, and remaining limits on the conclusion.

A proposed experiment is not a measured result. A successful conversion is not proof that collection captured the intended population.

Read Session, output, and conversion protocol before providing capture/lifecycle commands, multi-event settings, native/trace instrumentation, conversion, or a differential comparison. Read Validation cases when evaluating changes to this skill; these cases are not a required incident report.

References

• async-profiler repository — release, source, supported platforms, and matching documentation.
• Profiler options — use the document from the pinned release tag.
• Troubleshooting — official failure guidance; correlate with the installed release.
• Linux perf security — authoritative capability and perf_event_paranoid model.