gRPC, HTTP/2 and Service-Mesh Performance
Purpose
Separate four costs that are often reported as one: payload encoding, HTTP/2 transport, RPC
semantics, and proxy/mesh policy. A change in one layer does not prove that another improved.
Investigation contract
Record the exact client/server/proxy versions, transport implementation, topology, request and
response sizes, unary/streaming shape, channel and connection counts, concurrent streams, flow-
control windows, TLS connection age, retries/hedges, offered and completed work, and per-hop
latency/CPU/memory. Preserve a direct-path control where possible.
Inspect the project's Java toolchain, resolved grpc-java/Netty artifacts (including shaded versus
unshaded transport), native TLS provider and deployed proxy version before naming knobs or defaults.
This skill has no executable Java baseline; protocol contracts do not authorize dependency/JDK
upgrades, and generic gRPC guidance does not establish a Java transport's exact behavior.
Workflow
- Draw
call -> channel -> transport connection -> HTTP/2 stream -> proxy hops -> backend.
Count each object; never use the terms interchangeably. A retry creates another attempt/stream;
a terminating proxy has separate downstream/upstream connections, settings and flow control.
- Locate the limit: application admission, executor/event loop, stream concurrency, connection
or stream window, socket/network, proxy, or backend.
- Compare aligned per-hop evidence. A smaller Protobuf payload can reduce encoding and bytes but
cannot establish that proxy policy or connection churn became cheaper.
- Check effective configuration from protocol negotiation, runtime metrics or proxy config dump.
A configuration key accepted by a framework or CRD is not evidence that it changed behavior.
- Change one layer and validate useful completion, tail latency, errors, retries, CPU and memory.
Decision rules
- Reuse long-lived channels by default. Create a pool only after one connection/event-loop path is
shown to bottleneck or routing requires more independent connections; size it from evidence.
- Increasing maximum concurrent streams does not create connection flow-control credit, event-loop
CPU or backend capacity. Identify which limit is binding first.
- Flow-control tuning follows bandwidth-delay product and observed stalls. Larger windows permit
more outstanding DATA and can worsen overload; credit is not a measurement of allocated
memory. Identify receiver, direction and hop before changing a window.
- A long-lived HTTP/2 connection through an L4 balancer can pin many calls to one backend. Route
connection distribution to
load-balancing-and-routing; adding streams to that connection does
not rebalance it.
- Treat TLS handshake cost separately from steady-state record protection. Connection churn,
certificate rotation and session resumption determine how often the expensive path occurs.
- Combine application and proxy retries into one attempt budget. Never retry or hedge a possibly
committed non-idempotent operation without a durable idempotency contract.
- A mesh is justified by security and policy as well as latency. Measure its marginal cost and
compare sidecar, node/ambient and direct paths without silently discarding required controls.
Evidence and output
For a material recommendation report evidence, direct observation, inference, alternative
hypotheses, the predicted metric change and a rollback trigger. Missing direct-path or effective-
configuration evidence makes the verdict inconclusive, not favourable.
Scope that uncertainty: missing a direct control prevents causal mesh-overhead attribution, but
does not invalidate a directly observed exhausted window or executor queue. Continue independent
diagnosis. Compare end-to-end distributions; adding/subtracting per-hop p99 values does not
produce a request's critical-path latency.
References
- HTTP/2 and gRPC mechanics — read when diagnosing stream stalls,
channel pools, flow control or Netty execution.
- Service-mesh cost and policy composition — read when a proxy,
mTLS, outlier detection or mesh retry participates in the path.
1---2name: grpc-http2-service-mesh-performance3description: Diagnosing and designing the performance of gRPC and HTTP/2 communication paths, including channel, connection and stream topology, flow control, serialization, Netty event loops, TLS connection churn and service-mesh proxy cost. Use when multiplexed traffic is skewed or stalls, a channel pool or HTTP/2 setting is proposed, mesh overhead consumes a material latency or CPU budget, or retries exist in both client and proxy. API semantics belong to rpc-and-api-contracts; TCP behavior to tcp-tuning; routing ownership to load-balancing-and-routing.4---56# gRPC, HTTP/2 and Service-Mesh Performance78## Purpose910Separate four costs that are often reported as one: payload encoding, HTTP/2 transport, RPC11semantics, and proxy/mesh policy. A change in one layer does not prove that another improved.1213## Investigation contract1415Record the exact client/server/proxy versions, transport implementation, topology, request and16response sizes, unary/streaming shape, channel and connection counts, concurrent streams, flow-17control windows, TLS connection age, retries/hedges, offered and completed work, and per-hop18latency/CPU/memory. Preserve a direct-path control where possible.1920Inspect the project's Java toolchain, resolved grpc-java/Netty artifacts (including shaded versus21unshaded transport), native TLS provider and deployed proxy version before naming knobs or defaults.22This skill has no executable Java baseline; protocol contracts do not authorize dependency/JDK23upgrades, and generic gRPC guidance does not establish a Java transport's exact behavior.2425## Workflow26271. Draw `call -> channel -> transport connection -> HTTP/2 stream -> proxy hops -> backend`.28 Count each object; never use the terms interchangeably. A retry creates another attempt/stream;29 a terminating proxy has separate downstream/upstream connections, settings and flow control.302. Locate the limit: application admission, executor/event loop, stream concurrency, connection31 or stream window, socket/network, proxy, or backend.323. Compare aligned per-hop evidence. A smaller Protobuf payload can reduce encoding and bytes but33 cannot establish that proxy policy or connection churn became cheaper.344. Check effective configuration from protocol negotiation, runtime metrics or proxy config dump.35 A configuration key accepted by a framework or CRD is not evidence that it changed behavior.365. Change one layer and validate useful completion, tail latency, errors, retries, CPU and memory.3738## Decision rules3940- Reuse long-lived channels by default. Create a pool only after one connection/event-loop path is41 shown to bottleneck or routing requires more independent connections; size it from evidence.42- Increasing maximum concurrent streams does not create connection flow-control credit, event-loop43 CPU or backend capacity. Identify which limit is binding first.44- Flow-control tuning follows bandwidth-delay product and observed stalls. Larger windows permit45 more outstanding DATA and can worsen overload; credit is not a measurement of allocated46 memory. Identify receiver, direction and hop before changing a window.47- A long-lived HTTP/2 connection through an L4 balancer can pin many calls to one backend. Route48 connection distribution to `load-balancing-and-routing`; adding streams to that connection does49 not rebalance it.50- Treat TLS handshake cost separately from steady-state record protection. Connection churn,51 certificate rotation and session resumption determine how often the expensive path occurs.52- Combine application and proxy retries into one attempt budget. Never retry or hedge a possibly53 committed non-idempotent operation without a durable idempotency contract.54- A mesh is justified by security and policy as well as latency. Measure its marginal cost and55 compare sidecar, node/ambient and direct paths without silently discarding required controls.5657## Evidence and output5859For a material recommendation report evidence, direct observation, inference, alternative60hypotheses, the predicted metric change and a rollback trigger. Missing direct-path or effective-61configuration evidence makes the verdict inconclusive, not favourable.62Scope that uncertainty: missing a direct control prevents causal mesh-overhead attribution, but63does not invalidate a directly observed exhausted window or executor queue. Continue independent64diagnosis. Compare end-to-end distributions; adding/subtracting per-hop p99 values does not65produce a request's critical-path latency.6667## References6869- [HTTP/2 and gRPC mechanics](references/http2-and-grpc.md) — read when diagnosing stream stalls,70 channel pools, flow control or Netty execution.71- [Service-mesh cost and policy composition](references/service-mesh.md) — read when a proxy,72 mTLS, outlier detection or mesh retry participates in the path.
robsonkades