Anantbhandarkar Make It Right Mir Backend Jvm

/mir-backend-jvm · Make It Right (JVM runtime)

The middle tier. mir-backend decides what is correct (any language). The framework module (e.g. mir-backend-jvm-spring) knows the library's mechanics. This tier owns what's true for all JVM backends because they run on the HotSpot JVM — the threading model, garbage collector, memory model, and process lifecycle that Spring Boot, Quarkus, Micronaut, and Vert.x all inherit.

Runtime assumed: Java 21+ (LTS) or Kotlin on the JVM. Notes reference OpenJDK/HotSpot defaults. Load order: mir-backend → mir-backend-jvm → <framework module>.

The JVM footguns AI walks into (framework-agnostic)

1. Thread-pool sizing and pool-exhaustion deadlock

Platform threads are expensive (≈1 MB stack each by default). The JVM's blocking I/O model means every request that waits on the DB, cache, or downstream HTTP holds a thread.

Little's Law sets the floor: threads_needed = throughput_rps × latency_s. At 500 req/s with 200 ms average latency you need 100 threads just to break even — the JVM default ForkJoinPool.commonPool() has only CPU_cores - 1.

Pool-exhaustion deadlock is the silent killer: a pooled request thread submits a subtask to the same pool and then get()s on the future. If all pool slots are occupied waiting on their own subtasks, no thread is available to run those subtasks — the whole pool gridlocks. Every .get() or join() inside a thread-pool-backed executor is a suspect.

// WRONG — if every request thread reaches this, pool deadlocks
ExecutorService pool = Executors.newFixedThreadPool(10);
Future<Result> f = pool.submit(() -> callDownstream()); // submitted to same pool
return f.get(); // blocks the submitting thread

// RIGHT — separate pools / bulkheads, or don't block at all
ExecutorService ioPool = Executors.newFixedThreadPool(50); // dedicated I/O pool
Future<Result> f = ioPool.submit(() -> callDownstream());
return f.get(); // request thread is distinct from the I/O pool

Bulkhead: give each downstream (DB, payment service, search) its own bounded pool. A single downstream going slow can only exhaust its own bulkhead, not the whole request path.

2. Blocking I/O ties up a platform thread — and virtual threads change the calculus, but introduce pinning

On platform threads, blocking I/O (JDBC, HttpClient with sync API, Thread.sleep) holds the OS thread for the full duration. That's why async frameworks (reactive, Vert.x, Quarkus reactive) exist: they free the thread during the wait.

Java 21 virtual threads (Project Loom) flip this: a virtual thread is parked (unmounted from its carrier thread) while blocked on I/O, so you can have millions of virtual threads with far fewer carrier threads. JDK HTTP client, Socket, file I/O all park correctly. This makes the "thread-per-request blocking model" viable again at high concurrency without reactive programming.

The PINNING trap — the one thing virtual threads do NOT fix:

A virtual thread that enters a synchronized block or calls a native method cannot be unmounted from its carrier thread — it pins the carrier for the duration of the lock. If many virtual threads pin simultaneously, you can exhaust all carrier threads despite having millions of virtual threads.

// WRONG — synchronized pins the carrier thread when blocking inside
synchronized (this) {
    result = jdbcStatement.executeQuery(); // I/O while pinned → carrier blocked
}

// RIGHT — ReentrantLock allows unmounting during I/O inside the lock
private final ReentrantLock lock = new ReentrantLock();
lock.lock();
try {
    result = jdbcStatement.executeQuery(); // virtual thread parks, carrier freed
} finally {
    lock.unlock();
}

Detect pinning: run with -Djdk.tracePinnedThreads=full to log pinning events. JDBC drivers that use synchronized internally (many older ones do) will pin — prefer drivers updated for Loom (PostgreSQL JDBC 42.7+, HikariCP 6+).

3. GC pauses: G1 vs ZGC/Shenandoah

The default collector (G1 in Java 9+) targets throughput with occasional stop-the-world pauses. A full GC can pause all application threads for hundreds of milliseconds to several seconds on large heaps.

Allocation pressure is the root cause: objects allocated at high rates fill regions faster than G1 can reclaim them, triggering full GC. Reduce pressure by: reusing objects (pooling byte[] buffers), avoiding unnecessary boxing (int → Integer in hot paths), and keeping short-lived objects truly short-lived so they die in Eden without being tenured.

Tune G1 targets before switching collectors:

-XX:MaxGCPauseMillis=100 -XX:G1HeapRegionSize=16m -XX:InitiatingHeapOccupancyPercent=35

4. Container memory: use -XX:MaxRAMPercentage, not a hard -Xmx

The JVM does not know it is in a container by default on older versions. On Java 8u191+ and Java 11+ container-awareness is on by default (-XX:+UseContainerSupport), but AI almost always hard-codes -Xmx — and that value is then wrong when the pod's memory limit is changed.

# WRONG — hardcoded; becomes wrong when limits change; ignores non-heap (metaspace, code cache, off-heap)
-Xmx512m

# RIGHT — fraction of the container limit; non-heap overhead is taken from the remainder
-XX:MaxRAMPercentage=75.0

Leave 20–25% for off-heap: metaspace, code cache (JIT), direct byte buffers, stack memory. If you set MaxRAMPercentage=90, metaspace growth will push you over the cgroup limit → the OOM killer terminates the container even though the heap looks fine in your metrics.

Also set -XX:MaxMetaspaceSize explicitly to prevent unbounded metaspace growth on hot-redeploy or heavy reflection use.

5. Cold start: JIT warmup cost and mitigations

The JVM starts fast enough for long-running servers but is hostile to serverless / scale-to-zero:

• Bytecode is interpreted until HotSpot's profiler decides a method is "hot" (C1 after ~2000 invocations, C2 after ~10000).
• A freshly started JVM can be 5–20× slower for the first few thousand requests as JIT compilation runs concurrently.

This is the runtime-map reason "SKIP JVM for serverless with zero cold-start tolerance."

Mitigations (in ascending order of complexity):

1. JVM warm-up in staging: replay production traffic on a warm instance before directing real traffic to it (cloud load-balancer health-check delay).
2. GraalVM native image: AOT-compiles to a native binary — startup in milliseconds, no JIT warmup. Trades: no dynamic class loading at runtime, reflection must be configured, peak throughput may be lower than a warmed JVM.
3. CRaC (Coordinated Restore at Checkpoint): checkpoint a warmed JVM process image, restore it in milliseconds. Requires Linux kernel ≥ 5.9 + libcriu. Spring Boot 3.2+, Micronaut 4.2+ support CRaC.
4. Class Data Sharing (AppCDS): dump and share class metadata across JVM restarts — reduces startup by ~30%.

6. Shared mutable state: happens-before, volatile, final, and data races

The Java Memory Model (JMM) does not guarantee that writes by one thread are visible to another without synchronization. Invisible writes cause stale reads silently — no exception, just wrong values.

Happens-before rules that matter:

• A write to a volatile field happens-before every subsequent read of that field.
• synchronized exit happens-before subsequent entry on the same monitor.
• Thread.start() happens-before any action in the started thread.
• Thread.join() happens-before code after the join.

// WRONG — not volatile; the reading thread may see stale `done`
private boolean done = false;
// Thread A: done = true;
// Thread B: while (!done) { ... } // may loop forever

// RIGHT
private volatile boolean done = false;

// RIGHT for compound check-then-act
private final AtomicBoolean done = new AtomicBoolean(false);

Immutability via final: fields written in a constructor and published safely (through a final reference or volatile) are guaranteed visible without further synchronization. Prefer immutable value types.

AtomicInteger, AtomicReference, LongAdder: use for lock-free counters/state. Use LongAdder over AtomicLong under high contention (stripes the counter).

7. ThreadLocal leaks: pooled threads and classloader leaks on redeploy

ThreadLocal state set in a pooled thread (Tomcat connector thread, HikariCP thread, thread-pool worker) persists across requests unless explicitly cleared. This causes two distinct failures:

Context bleed: request A stores tenant ID in ThreadLocal; the thread returns to the pool; request B gets the same thread and reads request A's tenant ID.

// WRONG — set but never cleared
MDC.put("tenantId", tenantId);
// ... handle request ...
// thread returns to pool with tenantId still set

// RIGHT — always remove in finally
MDC.put("tenantId", tenantId);
try {
    // ... handle request ...
} finally {
    MDC.remove("tenantId"); // or MDC.clear() if you own the full context
}

Classloader leak on hot-redeploy (Tomcat, JBoss): a ThreadLocal holding a reference to a class loaded by the web-app classloader prevents that classloader from being garbage-collected. The old classloader hangs around → metaspace fills → OutOfMemoryError: Metaspace after several redeployments. Always remove() ThreadLocal values in a finally block or Filter/Interceptor cleanup.

How this slots into the pipeline

• Gate 0/5 (model choice): state the threading model (platform threads, virtual threads, reactive/event-loop) and justify it against the workload. Check pool sizing against Little's Law. A hard-coded -Xmx in a container manifest is a Gate 5 defect — flag it.
• Gate 6 (implementation): apply ReentrantLock over synchronized in virtual-thread paths; verify ThreadLocal.remove() in finally; confirm MaxRAMPercentage rather than fixed heap.
• Gate 7 (review): additionally check items 1–7 here for any JVM service.

Edit boundary (what belongs here vs. above/below)

• Generic, all-language rules (idempotency, invariants, gates, observability principles) → up to mir-backend.
• A specific library's mechanics (Spring @Transactional, Quarkus @Blocking, Micronaut @ExecuteOn, Hibernate fetch strategies) → down to the framework module (mir-backend-jvm-<framework>).
• Here: only what every JVM backend shares because of HotSpot — threading model, GC, container memory, cold start, JMM visibility, ThreadLocal hygiene.
• A different runtime (Python, Go, Node…) → its own mir-backend-<runtime> tier. Never widen this one.

Source: anantbhandarkar/make-it-right — distributed by TomeVault.