Jvm Troubleshoot

JVM Production Troubleshooting Handbook

This skill is organized by "symptom -> diagnostic workflow -> root cause table -> fix", covering 7 categories of high-frequency JVM failures. All commands require first running jps -lvm to obtain the <pid>.

Boundary with performance-tuning:

• jvm-troubleshoot (this skill): reactive troubleshooting (OOM/CPU 100% has already occurred) -> follow "symptom -> diagnosis -> fix"
• performance-tuning: proactive optimization (selecting GC before deployment, tuning JVM parameters, writing high-performance code) -> follow "layer -> selection -> optimization -> verification"

Failure Category Quick Reference

OOM Troubleshooting

OOM types and identification:

Stop the bleeding: First enable -XX:+HeapDumpOnOutOfMemoryError to get a dump, then restart. Root fix: Use MAT to analyze the dump, find the GC Root chain, and locate the leak point.

CPU 100% Troubleshooting

Diagnostic workflow (4-step identification):

1. top to find the process PID with high CPU usage
2. top -Hp <pid> to find the thread TID with high CPU usage (decimal)
3. printf "%x\n" <tid> to convert to hexadecimal
4. jstack <pid> | grep <hexadecimal tid> -A 30 to see what the thread is doing

Common root causes:

Note: If CPU is high but business logic is not complex, first check whether GC threads are consuming all CPU (check GC logs).

High-Frequency GC Troubleshooting

Diagnostic workflow:

1. Enable GC logs: -Xlog:gc*:file=gc.log:time,uptime:filecount=10,filesize=10M (JDK9+)
2. Analyze logs with GCViewer or gceasy.io
3. Check key metrics:
   • Full GC frequency: Normal is once every few hours; once every few minutes is problematic
   • GC throughput: GC time ratio should be < 5%; > 10% requires investigation
   • Old generation occupancy: Continuously growing without dropping = memory leak
4. jstat -gcutil <pid> 1000 to observe occupancy changes across generations

Common root causes and countermeasures:

GC collector selection and production parameter configuration: Proactive configuration belongs to the performance-tuning skill, see ../performance-tuning/references/gc_tuning.md. This skill only covers reactive troubleshooting.

Performance Troubleshooting

When response is slow but CPU is not high and GC is normal:

1. jstack <pid> > thread.txt to see thread state distribution
2. Count BLOCKED / WAITING threads: grep "java.lang.Thread.State" thread.txt | sort | uniq -c
3. Many BLOCKED -> lock contention, see Thread Deadlock Troubleshooting
4. Many WAITING -> thread pool idle or waiting for external resources (DB/HTTP)
5. Check if threads are stuck in SocketRead -> RPC timeout or slow peer
6. jstat -gcutil <pid> 1000 to rule out GC issues

Thread Deadlock Troubleshooting

Diagnosis: jstack <pid> | grep -A 20 "Found Java deadlock" -- jstack automatically detects synchronized deadlocks and prints them.

synchronized deadlock:

• Manifestation: jstack reports "Found 1 deadlock", two threads waiting for each other's lock
• Root cause: Nested synchronized blocks with inconsistent lock ordering
• Fix: Unify lock ordering, or use tryLock(timeout)

Lock deadlock (ReentrantLock):

• Manifestation: Thread WAITING in AbstractQueuedSynchronizer, but jstack does not report deadlock (Lock deadlocks are not automatically detected)
• Diagnosis: Check jstack stack for threads stuck in lock() calls
• Fix: Use tryLock(timeout, unit) instead of lock(), release and retry on timeout

Thread leak (pseudo-deadlock):

• Manifestation: Thread count keeps growing, eventually unable to create thread
• Diagnosis: jstack <pid> | grep "java.lang.Thread.State" | wc -l to count total threads
• Fix: Check if thread pool is leaking (new Thread not in a pool, submit without shutdown)

Class Loading Troubleshooting

ClassNotFoundException / NoClassDefFoundError / LinkageError:

1. jmap -clstats <pid> to see ClassLoader count and loaded class count
2. Abnormally many ClassLoaders -> ClassLoader leak (common in hot deployment, dynamic proxies)
3. Class not found -> check classpath: jcmd <pid> VM.classloader_stats
4. Same class with multiple version conflicts -> jmap -clstats to see if multiple ClassLoaders load the same class

Metaspace leak:

• Symptom: Metaspace does not drop after Full GC, eventually OOM Metaspace
• Diagnosis: jmap -clstats <pid> to see if ClassLoader count keeps growing
• Root cause: Dynamic proxies (CGLIB/ByteBuddy) generating new classes each time without caching, or web container hot deployment not cleaning up old ClassLoaders
• Fix: Cache proxy classes, fix hot deployment leak, adjust -XX:MaxMetaspaceSize

Module System Troubleshooting

Failures specific to JDK 9+ module system (JPMS). Guaranteed to encounter on JDK 17.

InaccessibleObjectException

Symptom: Reflection call reports java.lang.reflect.InaccessibleObjectException: Unable to make field accessible: module java.base does not "opens java.lang" to unnamed module

Root cause: JDK 9+ module system protects non-exported packages; reflection access requires explicit opening.

Troubleshooting:

1. Check which line of the error mentions "module X does not opens Y"
2. Add the corresponding --add-opens for that module

Common --add-opens quick reference:

Configuration location:

• Command line: java --add-opens ... -jar app.jar
• MANIFEST.MF: Add-Opens: java.base/java.lang java.base/java.util
• K8s: JAVA_OPTS environment variable

--add-opens vs --add-exports

Choose: Use --add-opens for reflection scenarios; use --add-exports for direct imports.

Module Conflict

Symptom: Startup reports Module resolution failed or module not found.

Troubleshooting:

• Check if there are multiple versions of the same module
• Check module-path (--module-path) vs classpath
• Use java --list-modules to see loaded modules

Module System Diagnostic Commands

# View module system parameters
jcmd <pid> VM.system_properties | grep jdk.module

# View loaded modules
java --list-modules

# Startup debugging
java --show-module-resolution -jar app.jar

Container Troubleshooting

JVM failures specific to K8s / Docker containers. For detailed troubleshooting see references/container_troubleshooting.md.

Quick reference:

Startup Failure Troubleshooting

JVM exits immediately on startup; check the error keyword:

Command Quick Reference

JVM diagnostic commands organized by scenario:

# === Process Overview ===
jps -lvm                                      # List JVM processes
jcmd <pid> VM.flags                           # View JVM startup parameters
jcmd <pid> VM.system_properties               # View system properties

# === Heap Analysis ===
jmap -heap <pid>                              # Heap overview (generation occupancy)
jmap -histo:live <pid> | head -20             # Large object histogram (top 20)
jmap -dump:format=b,file=heap.hprof <pid>     # Export heap dump
jmap -finalizer_info <pid>                    # View finalizer queue

# === Thread Analysis ===
jstack <pid> > thread.txt                     # Thread stack
jstack -l <pid>                               # Thread stack with lock info
top -Hp <pid>                                 # Thread-level CPU usage

# === GC Analysis ===
jstat -gcutil <pid> 1000                      # Generation occupancy + GC count (refresh every second)
jstat -gccause <pid> 1000                     # Last GC cause
jcmd <pid> GC.heap_info                       # Heap info

# === Class Analysis ===
jmap -clstats <pid>                           # ClassLoader statistics
jcmd <pid> VM.classloader_stats               # ClassLoader status
jmap -permstat <pid>                          # Permanent generation statistics (JDK7)

# === System Level ===
top                                           # Process CPU/memory
iostat -x 1                                   # Disk IO
netstat -anp | grep <pid>                     # Network connections

JVM Parameters

Monitoring parameters commonly enabled during troubleshooting:

-XX:+HeapDumpOnOutOfMemoryError               # Auto dump on OOM (essential for OOM troubleshooting)
-XX:HeapDumpPath=/var/log/dumps/              # Dump path
-XX:ErrorFile=/var/log/hs_err_%p.log          # Fatal error log
-Xlog:gc*:file=gc.log:time,uptime             # GC log
-XX:+PrintFlagsFinal                          # Print all JVM parameter final values

Proactive tuning parameters (heap/GC collector/JIT configuration): Belongs to the performance-tuning skill, see ../performance-tuning/references/jvm_params.md.

Usage

1. Locate by symptom: First check the "Failure Category Quick Reference" table, jump to the corresponding section.
2. Execute diagnostic workflow: Each section has numbered steps; execute in order.
3. Check root cause table: After diagnosis, find countermeasures in the "Common Root Causes" table.
4. Command details: When command output is unclear, Read references/diagnostic_commands.md.
5. GC troubleshooting: When investigating GC issues, Read references/gc_tuning_guide.md (reactive diagnostic perspective). For proactive selection and configuration, see performance-tuning.
6. Container troubleshooting: For K8s/Docker container failures, Read references/container_troubleshooting.md.
7. JFR analysis: When using JFR for troubleshooting, Read references/jfr_analysis.md.
8. Troubleshooting cases: When an end-to-end reference is needed, Read references/troubleshooting_cases.md.
9. Do not fabricate when uncertain: JVM behavior is governed by source code and official documentation; this skill does not replace formal documentation.

Reference Entries

• Command details: references/diagnostic_commands.md (output field meanings and examples for each JVM command)
• GC troubleshooting guide: references/gc_tuning_guide.md (GC problem troubleshooting from a reactive diagnostic perspective)
• Container troubleshooting: references/container_troubleshooting.md (JVM failures inside K8s/Docker containers: OOM killed / PID 1 / JRE-only)
• JFR post-incident analysis: references/jfr_analysis.md (JFR event quick reference, online streaming, JMC flame graphs)
• Troubleshooting cases: references/troubleshooting_cases.md (4 end-to-end cases: heap leak / CPU 100% / deadlock / Metaspace leak)
• Proactive performance tuning: ../performance-tuning/SKILL.md (JVM parameter selection, GC collector selection, JIT tuning, JMH benchmarking)
• Official documentation: https://docs.oracle.com/en/java/javase/17/docs/specs/man/ (JDK 17 tools)
• GC log analysis: https://gceasy.io/
• Heap dump analysis: MAT (https://eclipse.org/mat/)
• Project coding standards: ../coding-standard/SKILL.md