JVM ML Inference
Purpose
Treat inference as a bounded native resource and queueing system. More request threads do not create
CPU, accelerator streams, native sessions or memory bandwidth; they can oversubscribe the engine and
worsen both throughput and tail latency.
Workload contract
Record model/version, engine/provider/version, target devices, input shapes and batch distribution,
pre/post-processing, in-process or remote boundary, session/predictor ownership, engine thread
settings, admitted concurrency/queue, warm-up state, latency SLO, useful throughput, heap/RSS/native
memory and failure/fallback semantics.
Inspect build toolchains, compiler release, runtime image, resolved Java/native artifacts and
provider/device support. This skill prescribes no universal JDK baseline; engine requirements
and virtual-thread guidance are version-specific (virtual threads are final in JDK 21).
Do not upgrade Java or the engine merely to apply an example from current documentation.
Workflow
- Decide in-process versus remote serving from latency, isolation, scaling, model cadence,
accelerator sharing, failure domain and operational ownership.
- Inventory every native resource and lifetime. Bound sessions, predictors, arenas, tensors and
direct buffers; close resources that expose ownership/close contracts and bound retained
storage where reclamation is GC-managed. Reuse only after actual native completion.
- Build a concurrency matrix across outer requests, session count, intra-op/inter-op threads and
device streams. Measure absolute goodput and tail latency, not speedup alone.
- If batching is used, bound size/storage and maximum wait, and dispatch before the earliest
member deadline minus the execution/remaining-work budget. Test sparse and burst traffic;
full-batch throughput is not a latency policy.
- Warm the JVM code path and the model/engine separately, then gate readiness on a representative
successful inference rather than model-file load.
- Overload deliberately. Use bounded admission, deadline-aware rejection/cancellation and an
explicit fallback whose quality is part of the contract.
Decision rules
- Pool only resources documented as non-thread-safe or expensive to create. Pool size must match a
measured useful concurrency limit, not request concurrency.
- Reuse direct, native-order buffers when the API permits, with exclusive ownership across
filling, native execution and result consumption. Moving allocation from heap to direct
memory inside the hot path does not remove allocation or guarantee zero device copies.
- Native CPU work can retain a virtual-thread carrier and does not gain throughput from virtual
threads. Isolate/admit it with a bounded executor when necessary.
- NMT excludes many third-party native allocations. Compare process/cgroup RSS with NMT categories
and application counters for live sessions/tensors; use native profilers where required.
jdk.VirtualThreadPinned absence cannot clear CPU-bound time inside native code; the event needs a
relevant park/block path to become visible.- A cancelled Java future may not stop native computation. Define abandonment, late completion and
resource reclamation explicitly.
Evidence output
Separate measured result from analytical ceiling. Pin environment and raw output; report confidence,
what the experiment cannot prove, and the same metrics after a change.
References
- Native resources, parallelism and batching — read
when sizing pools, engine threads, buffers or a dynamic batcher.
- Use
jni-and-ffm, off-heap-memory, concurrency-limiting-and-bulkheads and
load-testing-advanced for their owning mechanisms.
1---2name: jvm-ml-inference3description: Engineering CPU and accelerator-backed ML inference from JVM applications: choosing in-process versus remote serving, bounding native sessions and predictors, coordinating engine and request parallelism, batching under a latency deadline, reusing direct buffers, warming deployments and diagnosing native memory outside NMT. Use when DJL, ONNX Runtime or another native inference engine loses throughput as concurrency rises, leaks RSS, overloads a model pool or needs graceful degradation. Model quality and training pipelines are outside scope.4---56# JVM ML Inference78## Purpose910Treat inference as a bounded native resource and queueing system. More request threads do not create11CPU, accelerator streams, native sessions or memory bandwidth; they can oversubscribe the engine and12worsen both throughput and tail latency.1314## Workload contract1516Record model/version, engine/provider/version, target devices, input shapes and batch distribution,17pre/post-processing, in-process or remote boundary, session/predictor ownership, engine thread18settings, admitted concurrency/queue, warm-up state, latency SLO, useful throughput, heap/RSS/native19memory and failure/fallback semantics.2021Inspect build toolchains, compiler release, runtime image, resolved Java/native artifacts and22provider/device support. This skill prescribes no universal JDK baseline; engine requirements23and virtual-thread guidance are version-specific (virtual threads are final in JDK 21).24Do not upgrade Java or the engine merely to apply an example from current documentation.2526## Workflow27281. Decide in-process versus remote serving from latency, isolation, scaling, model cadence,29 accelerator sharing, failure domain and operational ownership.302. Inventory every native resource and lifetime. Bound sessions, predictors, arenas, tensors and31 direct buffers; close resources that expose ownership/close contracts and bound retained32 storage where reclamation is GC-managed. Reuse only after actual native completion.333. Build a concurrency matrix across outer requests, session count, intra-op/inter-op threads and34 device streams. Measure absolute goodput and tail latency, not speedup alone.354. If batching is used, bound size/storage and maximum wait, and dispatch before the earliest36 member deadline minus the execution/remaining-work budget. Test sparse and burst traffic;37 full-batch throughput is not a latency policy.385. Warm the JVM code path and the model/engine separately, then gate readiness on a representative39 successful inference rather than model-file load.406. Overload deliberately. Use bounded admission, deadline-aware rejection/cancellation and an41 explicit fallback whose quality is part of the contract.4243## Decision rules4445- Pool only resources documented as non-thread-safe or expensive to create. Pool size must match a46 measured useful concurrency limit, not request concurrency.47- Reuse direct, native-order buffers when the API permits, with exclusive ownership across48 filling, native execution and result consumption. Moving allocation from heap to direct49 memory inside the hot path does not remove allocation or guarantee zero device copies.50- Native CPU work can retain a virtual-thread carrier and does not gain throughput from virtual51 threads. Isolate/admit it with a bounded executor when necessary.52- NMT excludes many third-party native allocations. Compare process/cgroup RSS with NMT categories53 and application counters for live sessions/tensors; use native profilers where required.54- `jdk.VirtualThreadPinned` absence cannot clear CPU-bound time inside native code; the event needs a55 relevant park/block path to become visible.56- A cancelled Java future may not stop native computation. Define abandonment, late completion and57 resource reclamation explicitly.5859## Evidence output6061Separate measured result from analytical ceiling. Pin environment and raw output; report confidence,62what the experiment cannot prove, and the same metrics after a change.6364## References6566- [Native resources, parallelism and batching](references/native-resources-and-batching.md) — read67 when sizing pools, engine threads, buffers or a dynamic batcher.68- Use `jni-and-ffm`, `off-heap-memory`, `concurrency-limiting-and-bulkheads` and69 `load-testing-advanced` for their owning mechanisms.
robsonkades