Serialization Performance

Serialization performance

Purpose

Choose and operate a serialization boundary from total system cost and compatibility, not one library's small-payload throughput chart. The fastest encoder can lose after wire bytes, compression, copies, allocation, downstream parsing, schema migration, or recovery are included.

Ownership boundary

• This skill owns serialization performance models, experiments, buffer/copy strategy, and production attribution.
• schema-evolution-and-compatibility owns compatibility governance and rollout contracts.
• java-serialization-hardening owns deep ObjectInputStream security/migration.
• jmh-microbenchmarks owns harness validity; load-testing owns end-to-end arrivals/queueing.
• off-heap-memory owns native/direct memory lifecycle; serialization-performance owns how the codec uses those buffers.

Decision contract

Inspect the project's toolchain, resolved codec versions/modules, framework configuration and deployed readers before choosing an API. This skill has no universal Java/library baseline; documentation examples are not authorization to upgrade the target. Missing measurements leave performance benefits hypothetical; missing compatibility evidence can block a format migration.

boundary and trust zone: in-process/cache/process/network/storage/topic
producer/consumer languages, versions, ownership and deployment skew
payload schema, size/cardinality/nesting/optional-field and value distributions
read/write ratio and fields accessed
throughput/latency/tail/CPU/allocation/wire/storage objectives
streaming, framing, random access, compression and batching requirements
buffer ownership/lifetime/backpressure and maximum message policy
compatibility/registry/unknown-field/default/ordering/canonicalization rules
security/resource limits/privacy and malformed-input behavior
migration, dual-read/write, replay, rollback and retained-data horizon

Cost model

Measure stages separately and together:

end-to-end serialization cost =
  object/model construction
  + encode/decode CPU
  + allocation, retention and GC consequence
  + buffer growth/copy/reference-count/lifetime cost
  + compression/decompression
  + framing/checksum/encryption
  + wire/storage bytes and downstream I/O
  + schema lookup/validation/conversion
  + queueing/backpressure/retry/replay effects

Normalize per business message and per useful byte/field where appropriate. Batch-level results can hide per-message tail and oversized-item failure.

Eliminate by contract before speed

No format automatically supplies organizational compatibility. Registry policy, generated code, field IDs, defaults, unknown fields, enum evolution, maps/order, and implementation versions must be tested across deployed producers/consumers.

Format families and trade-offs

• Text/self-describing (for example JSON): broad interoperability and inspectability; repeated names and lexical conversion can increase bytes/CPU. Parsers may reuse field-name symbols and stream tokens, so “one String per key/value” is not a valid universal model.
• Schema-based formats: Protocol Buffers uses numbered field tags; Avro binary records encode values in writer-schema order without per-field tags. Their parsing, skipping and evolution rules differ. Generated objects may provide direct access after materialization. Use references/format-selection.md when comparing formats or changing codec configuration.
• Indexed/in-place access formats (for example FlatBuffers/Cap'n Proto designs): avoid full object materialization for some access patterns, while adding offset traversal, validation, alignment/layout, buffer-lifetime, implementation and mutation constraints.
• Object-graph/library-specific codecs (for example Kryo): flexible and often efficient within controlled ecosystems; class registration, graph/reference semantics and version compatibility become application protocol responsibilities.

“Zero-copy” is a claim about specific copies and stages, not end-to-end absence of copying. Kernel, TLS, framing, decompression, buffer conversion, alignment and application materialization may remain.

Buffer, ownership, and streaming

Prefer writing to the next stage's bounded buffer/stream when it eliminates a demonstrated copy. Before reuse/pooling, define:

• owner, thread-safety, handoff and release point;
• maximum retained capacity and oversized-message behavior;
• heap/direct/native accounting and container headroom;
• partial write/read, cancellation, timeout and exception cleanup;
• reference-count/use-after-release and data leakage between tenants;
• pool exhaustion/backpressure and shutdown/redeploy cleanup.

An asynchronous send must retain exclusive ownership or an appropriate lease until the transport has finished reading the bytes. Enqueue, timeout or cancellation alone does not prove that point. A read-only view can still observe mutations through another alias; copy or defer reuse when the handoff contract cannot establish safety. Discard/reset a failed codec only by its documented policy.

ThreadLocal avoids concurrent codec use but can retain large buffers per platform thread and behaves differently with virtual-thread workloads. Pools bound instances only if acquisition, capacity reset, eviction, failure and telemetry are designed. Reuse can reduce allocation while increasing retained memory or contention.

Library-specific caution

Do not infer protocol safety from a benchmark snippet:

• Kryo registrations can use compact/stable IDs for registered types even when registration is not globally required. setRegistrationRequired(true) rejects accidental unregistered types; it is not what makes existing registered types use their IDs.
• registration IDs and serializers must remain compatible with retained bytes and rolling versions; order-based implicit registration is fragile unless frozen and tested.
• Kryo instances are generally not thread-safe; choose confinement/pooling and test reset state.
• disabling graph reference tracking changes semantics for shared/cyclic graphs, not only speed.
• library defaults and version serializers are not substitutes for cross-version golden fixtures.

Never pin versions or capability claims from memory. Inspect the current official documentation, release artifact and supported JDK/platform matrix.

Measurement ladder

1. Corpus characterization: production-derived, privacy-safe cohorts for size, nesting, values, optional/unknown fields, compressibility, malformed and maximum inputs.
2. Semantic/conformance tests: an independent contract oracle plus round-trip and cross-language/version fixtures for unknown/default/null/presence and numeric fidelity; canonical bytes where required, corruption/resource limits. A same-codec round trip alone cannot prove interoperability or that both directions did not normalize away required data.
3. JMH mechanism benchmark: encode and decode separately plus round trip where relevant; CPU, allocation, output size, buffer mode, lifecycle, multiple forks and raw results.
4. Component benchmark: framing, registry, compression, buffer pool, network/storage and backpressure with realistic concurrency.
5. Production/canary evidence: profiles, allocation/GC, queue depth, payload sizes, errors, retries and SLOs normalized by useful work.
6. Migration/failure test: rolling versions, replayed old bytes, rollback, poison/max messages, dependency/registry outage and resource exhaustion.

Use references/benchmarking-serialisers.md for the experiment matrix.

Production attribution

CPU samples at ObjectMapper.readValue, a generated parser, or codec method show sampled CPU location, not automatically optimization value. Establish frequency per business operation, inclusive/self cost, payload cohort, compilation/native frames, allocation/GC consequence, queueing, and whether I/O or compression dominates end-to-end latency.

Allocation profiles find creation sites; they do not prove retained memory. Correlation between allocation and GC pauses is not additive causal attribution because thread durations overlap and collector work is phase-dependent. Use aligned work-normalized evidence and a controlled change.

Security and resource safety

Treat all deserialization across a trust boundary as parser attack surface:

• cap bytes, nesting/depth, collections/arrays, references and decompressed expansion;
• reject/route malformed, incompatible, unknown-type and oversized messages deterministically;
• bound time, memory, concurrency and retries; avoid poison-message loops;
• authenticate/integrity-check at the correct layer and protect sensitive payload/profile data;
• fuzz/property-test parsers and cross-version fixtures.

Avoid Java native serialization for new external boundaries. If legacy ObjectInputStream remains, use ObjectInputFilter with class and resource constraints, per-context policy where applicable, and a migration plan; follow java-serialization-hardening and official serialization-filter docs.

Decision framework

Prefer a candidate when it:

• satisfies trust, language, compatibility and retained-data constraints;
• meets CPU/allocation/wire/tail objectives over all important payload cohorts;
• has supported implementations, tooling and observable failure modes;
• integrates with bounded buffers/backpressure and operational recovery;
• survives mixed-version, rollback and malformed/max-input tests.

Reject or defer when the measured benefit is below migration risk/cost, only a toy corpus was tested, the producer/consumer rollout cannot be made compatible, or buffer/native headroom is unbounded. Return the contract, deployed baseline, supported observations, proposed change and validating experiment. Mark failed/missing measurements inconclusive; report measured gains separately from unexecuted rollout or failure tests. Scale the checks below to the changed boundary.

Anti-patterns

Definition of done

• Contract, trust boundary, compatibility horizon and migration are explicit.
• Representative corpus includes size/value/schema/malformed/max cohorts.
• Encode, decode, round trip, bytes, allocation, copies, compression and failure are measured as relevant.
• Buffer ownership, retention, backpressure, cancellation and shutdown are bounded/tested.
• JMH results preserve fork/corpus identity and component/load behavior validates impact.
• Cross-version/language, rollback/replay and registry/dependency failures pass.
• Security/resource limits and observability exist in production.

References

• Format-selection scorecard
• Benchmarking and profiling serializers
• Java serialization filtering
• Protocol Buffers encoding
• Apache Avro specification
• FlatBuffers internals
• Cap'n Proto encoding
• Kryo documentation