# Linux For Jvm > The Linux side of a JVM incident: RSS versus virtual memory, page faults and swap, AlwaysPreTouch, transparent huge pages, cgroup CPU throttling, the two OOM killers, file-descriptor and process limits, signals and graceful shutdown, and PSI as a direct stall signal. Use when a process dies with exit code 137 or no log at all, when a GC pause in the log does not match the pause the client felt, when "too many open files" or "unable to create native thread" appears, when THP or swappiness is being changed by reflex, when kill -9 is the first response, or when container CPU limits may be throttling the JVM. Does not cover the JVM-side memory budget (jvm-memory-regions), collector behaviour (gc-fundamentals), or CPU cache and NUMA topology (cpu-cache-and-numa). What the JVM detects inside a cgroup is container-awareness, kernel-side tracing is ebpf-for-jvm, and the network stack is tcp-tuning. - Skill: `robsonkades/linux-for-jvm` (Agent Skill, multi-file: 4 files) - Install (CLI): `npx skillmds@latest add robsonkades/linux-for-jvm` - Raw SKILL.md: https://api.skillmd.com/api/skills/robsonkades/linux-for-jvm/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Finance & Business - Author: robsonkades (https://skillmd.com/u/robsonkades) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/robsonkades/linux-for-jvm --- # Linux for the JVM ## Purpose Separate what the JVM manages from what the kernel manages. The JVM administers virtual addresses and allocations; the kernel controls mappings, residency and scheduling. A client stall that overlaps GC is not by itself proof of a collector problem. ## Environment contract Inspect the deployed JDK/collector, kernel, cgroup version and mount/cgroup namespaces, service manager, container image and metric labels before applying commands. These are Linux shell fragments, not Java programs; cgroup v2 examples need the target process hierarchy, not an assumed host root. Missing permissions or files mean missing evidence, not zero. Do not upgrade the runtime or change host-wide policy merely to use this skill. ## Workflow 1. **On "it died with no log", check the container/runtime status and exit code first.** 137 conventionally means termination by `SIGKILL`; it does **not** distinguish cgroup OOM, global OOM, kubelet/runtime action, administrator action, or a wrapper that remapped a status. 2. **Classify with independent records**: cgroup v2 `memory.events`, pod termination reason, runtime/kubelet events, and kernel journal where permitted. `dmesg` can be inaccessible, rate-limited or already rotated. 3. **Compare memory accounting under real load**: NMT for tracked HotSpot allocations, RSS/PSS for residency, and cgroup usage for charged memory. NMT committed is not RSS, does not cover every native allocation, and cannot be subtracted from RSS to quantify unexplained native bytes. 4. **Correlate the logged GC pause with the observed pause** and attribute the difference to a layer only when evidence supports it — TTSP, throttling, swap, I/O or request queueing. Keep unresolved causes as hypotheses; do not subtract unrelated latency percentiles. 5. **Check throttling with deltas**: periods throttled says frequency; `throttled_usec` accumulates throttled run-queue time and can overlap across CPUs. Its delta divided by elapsed microseconds is not lost CPU capacity or request stall percentage and can exceed 1. Correlate with demand, ancestor quotas and latency. 6. **Check descriptor and thread counts against their limits** before believing a resource is exhausted. ## Rules - Page-fault counters are in `/proc//stat` (fields 10 and 12), **not** in `status`. And `VmPeak` is peak _virtual_ memory — peak RSS is `VmHWM`. A grep for the wrong field returns empty, and empty reads as zero. - `AlwaysPreTouch` pre-empts **minor** faults; it does not protect against swap. A pre-touched page is swapped out normally under memory pressure and the major fault happens just the same. The benefit is predictability — cost concentrated at startup instead of diffused through operation. - Swap can make JVM latency highly variable, but "incompatible" is too strong. Decide from the availability goal: no swap favors predictable latency but increases kill risk; bounded swap may preserve availability during transient pressure at a tail-latency cost. Monitor swap-in/out, `VmSwap`, faults, reclaim and PSI rather than prescribing one global `swappiness` value. - A major fault's cost depends on page source, device, queueing, reclaim and filesystem. Use fault deltas correlated with wall-clock stalls or block-I/O evidence; hardware-class latency ranges are not a substitute for measurement. - THP is a policy matrix, not a binary slogan. `always` can invoke direct reclaim/compaction; `madvise`, `defer`, `defer+madvise` and `never` make different latency/memory trade-offs, and modern kernels can expose multiple THP sizes. Verify kernel and JDK behaviour, measure `AnonHugePages`, TLB/CPU benefit and compaction stalls, then record the chosen policy. - Distinguish global OOM from memory-cgroup OOM by evidence. Victim selection can incorporate `oom_score_adj` within applicable constraints; it is not a protection against exceeding a container's or ancestor's `memory.max`. Kubernetes QoS influences scores; correlate `memory.events` with limit scope and kernel/runtime records to identify the failure. - Never `kill -9` first. `SIGKILL` cannot be intercepted: no shutdown hooks, no connection drain or final dump-on-exit. Previously completed JFR chunks/dumps may survive, while buffered events and the active chunk can be lost. Send `SIGTERM`, wait, then escalate — and make `terminationGracePeriodSeconds` match the real drain time. - Inspect the actual launcher limits. For systemd services, persist `LimitNOFILE` and use `TasksMax` for service task limits; `LimitNPROC` applies across the real UID and has privileged exemptions. Shell limits affect descendants, not an independently started systemd unit. Native-thread OOME needs PID/task, stack and memory evidence; heap retention can still explain unbounded thread creation. - Alert on OOM kills explicitly (`node_vmstat_oom_kill`, and the pod-level `OOMKilled` reason), with host and container attribution. Application logs can contain useful precursors even when they contain no final termination record. - NUMA behavior is collector/build-specific. Parallel GC and G1 are not the only users: Linux ZGC in JDK 25 consumes `UseNUMA`. Check effective flags and allocation policy rather than treating an accepted flag as proof of an optimization; detailed placement belongs to `cpu-cache-and-numa`. - Linux began transitioning the fair scheduler toward EEVDF in 6.6. Do not apply CFS tuning knobs from a runbook without checking the node kernel, scheduler documentation and whether the knob exists; `vruntime`, eligibility and virtual deadlines are related but not interchangeable models. - PSI (`/proc/pressure/*`, and cgroup-local `*.pressure` on cgroup v2) measures shares of time with some or all non-idle tasks stalled. It is a valuable saturation signal, not automatically the earliest one. `avg10/60/300` are percentages; `total` is cumulative microseconds. System-level CPU `full` is undefined and reported as zero for compatibility; do not interpret it as no CPU pressure. Compare the same scope and interval with symptoms. ## Output Return scoped, timestamped observations, supported hypotheses, the smallest justified action, and how to verify it. Report unavailable evidence and unresolved attribution explicitly. ## References - [Incident commands](references/incident-commands.md) — what to run, in order, for an OOM kill, a throttling suspicion, a descriptor exhaustion or an unexplained pause. Read during an incident. - [Host and container configuration](references/host-configuration.md) — the pre-deploy checklist, THP modes, swap, limits, signals and the alerting set. Read before deploying or when reviewing a host or pod spec.