Platform Service Connectivity

Platform Service Connectivity

• Code/test changes require self-checks; invoke code-review automatically before completion.

This skill owns how requests move between services: the transport layer (mesh), service discovery, multi-environment routing, retry/timeout/circuit-breaker policy, and the framework middleware that propagates app-level context across hops.

You do not own:

• Whether the right signals exist or how to alert on them — go to platform-observability.
• Whether a change is safe to promote across environments — platform-release-engineering consumes connectivity policy as input.
• HTTP handler shape, DB schema, or business contract design — language-specific service-architecture skills.
• Agent/MCP tool exposure, dynamic tool catalog/cache behavior, model-visible schemas, or execution-time tool authorization — route those to llm-inference-integration. This skill owns the connector's network path, egress policy, mTLS/TLS boundary, service identity, and transport observability, not whether the agent may call a tool.

Skill Routing

• Use this skill for: introducing or auditing service mesh, choosing service-discovery mechanism, setting retry/timeout/circuit-breaker defaults, designing lane-based or canary-based traffic shifts, debugging a "service A can't reach service B" failure, writing connectivity middleware, deciding where mTLS lives, propagating tenant/lane/log-id across hops.
• Use platform-observability for: anything about whether you can see a connectivity failure (envoy access log, kitex/grpc trace, Prom mesh metrics).
• Use platform-release-engineering for: shifting traffic during a deploy, promoting a canary, or rolling back via mesh VirtualService.
• Use llm-inference-integration for: agent tool catalogs, MCP tool/resource/prompt semantics, prompt-cache or manifest stability, tool-call approval policy, execution-time authorization, and model-visible connector schemas. Return here only for the connector's network reachability, service identity, egress allowlist, TLS/mTLS, timeout/retry transport policy, or mesh/discovery path.
• Use defect-diagnosis for: an active connectivity incident — come back here to land the recurring pattern.

Core Mental Model — Two Layers

A request crossing two services traverses two cooperating layers:

Service A app code
   │ writes ctx fields (log_id, lane, business ctx)
   ▼
Framework client middleware (kitex/hertz/equivalent)
   │ packs ctx fields into request headers / RPC base struct
   │ resolves callee via the chosen service-discovery mode
   ▼
A's mesh sidecar (Envoy)         ◄── owns: mTLS, retries, timeouts,
   │ TLS, traffic policy, lane     traffic policy, lane routing,
   │ routing, retry, timeout       circuit breaking, telemetry
   ▼
Network
   ▼
B's mesh sidecar (Envoy)
   │ TLS terminate, authz
   ▼
Framework server middleware
   │ extracts ctx fields from headers / base struct
   │ injects into ctx for handler
   ▼
Service B app code

Layer ownership is non-negotiable:

Confusing these is the source of most "why doesn't this work" connectivity bugs. When in doubt: mesh is transparent transport; framework is explicit context.

Non-Negotiable Rules

R1 — Mesh is opt-out, not opt-in

• Every service pod runs the mesh sidecar by default (sidecar.istio.io/inject: "true" or namespace label).
• Specific workloads MAY opt out via explicit annotation. Documented exceptions:
  • Telemetry collectors (would observe themselves).
  • File-log shippers (DaemonSet with hostPath).
  • Service discovery / registry pods (would bootstrap-cycle).
  • Mesh control plane itself.
• A new service that opts out of mesh without documented reason fails review.

R2 — Service discovery mode is explicit

• Pick and document the platform's service-discovery mode: registry-based, k8s-native, or mixed. Use references/service-discovery-choice.md before mandating a registry.
• App code MUST NOT bypass the chosen lane-routing mechanism with hard-coded cluster URLs or ad hoc hostnames.
• Registry mode carries lane and weight in instance metadata; k8s-native mode carries lane through pod labels, Services, EndpointSlices, and mesh subsets. Both require health to be honored before routing.

R3 — Lane is the canonical multi-env primitive

• "Lane" = a logical traffic isolation label: prod, test, pre, named per-developer lanes for staging, named per-test-run lanes for integration.
• Lane flows through:
  1. Request header (a platform-wide header name; treat the literal value as a constant) at the entry edge.
  2. context.Context field via framework server middleware.
  3. Outbound RPC base struct / outbound HTTP header via framework client middleware.
  4. Mesh VirtualService match rule that routes lane-tagged traffic to lane-specific endpoints or subsets.
  5. Discovery metadata (registry instance tag or k8s pod label) so callees are discoverable under the right lane.
• A service that does not propagate lane through ctx is a multi-env hazard. Lane must traverse every hop, including queue boundaries.

R4 — Default retry/timeout/circuit-breaker live at mesh, app overrides for business reasons

• On Istio HTTP/gRPC paths, VirtualService HTTP routes own request timeout and retries; DestinationRule owns connection-pool settings and outlierDetection. Other transports use their platform-owned equivalents.
• Before configuring retries or timers, read references/retry-timeout-circuit-breaker.md for field paths, single-owner retry selection, and idempotency checks. Both mesh and SDK follow them; a 5xx or missing response alone never proves replay safe.
• Framework client SDK may provide method-specific retry/hedging; its total call budget must fit the caller's remaining duration and any platform cap. A longer mesh timeout is a backstop, not an extended caller deadline.
• App code MAY override per-RPC. App code MUST NOT silently disable mesh-level outlier detection.
• Budget rule: downstream work, attempts, and backoff fit the remaining duration minus a safety margin (e.g. 100ms). Propagate deadline and cancellation; never reset the full budget at each hop.

R5 — mTLS is mesh-default, app cannot disable

• Mesh PeerAuthentication enforces STRICT mTLS namespace-wide.
• App code MUST NOT use plaintext HTTP/gRPC between services. Local dev can use plaintext (no mesh) but the build/staging path always tests with mesh on.

R6 — Framework client/server middleware chain is mandatory

Every service uses the platform framework client and server suites; ad-hoc construction is reviewed. The default chain (RPC side) MUST include:

If a developer can build a new client/server without these, the framework is broken.

R7 — Standard RPC base field contract

For RPC platforms that support a structured request "base" (Kitex base.Request, Thrift Base, protobuf-backed gRPC request fields, or metadata-only fallbacks), the platform defines a canonical shape:

Base {
  UnixTime  int64           // ms since epoch (caller's view)
  LogId     string          // platform log-id
  Caller    string          // PSM-style identity of caller
  Tags      map[string]string {
    "idc":     <region>,
    "cluster": <k8s cluster>,
    "lane":    <env/lane>,
    // optional: "stress_tag", "shadow"
  }
}

Whether a platform carries this metadata in an in-message base field at all, versus the transport metadata / context channel (gRPC HTTP/2 metadata + interceptors — the industry-standard default for ordinary unary and streaming gRPC), is a scenario decision, not a universal mandate; the propagation obligation (R6) holds on either carrier, but the in-message field-number contract applies only where the in-message base carrier is used. See the Carrier decision in references/rpc-framework-recipe.md.

For protobuf-backed internal RPC on a platform that has standardized the in-message base carrier, the protobuf base-field declarations are owned by references/rpc-framework-recipe.md. Field name, type, field number, request/response shape, exception handling, and generated-artifact provenance are fixed by that shared contract source; service repos and language-specific generated packages must consume them as generated, not rename, renumber, retype, or replace them locally.

The client middleware fills this from ctx automatically; the server middleware extracts it back. App code does not touch the base struct directly.

For non-protobuf or metadata-only transports, an equivalent header set is compliant only when it cites a resolvable platform owner record, such as a repo/path, document URL, registry id, gateway policy, or owner-suite id. The owner record must enumerate the concrete header names. In diff-only review without resolver tooling, the diff must cite a stable owner-record locator and list the concrete header names inline; with resolver tooling, the reviewer or owner-suite may resolve the locator to those names instead. The enumerated header names must then be checked against the actual propagated and exposed headers: caller-supplied identity headers are absent unless positive authenticated-caller evidence exists. An unresolvable, non-enumerating, uncited, service-local, or unchecked header set is an open gap, not a permitted alternative. For pure HTTP (no RPC base), the equivalent is a stable owner-recorded header set, also filled by middleware.

R8 — gRPC :authority compatibility follows the actual request path

HTTP/2 :authority carries URI authority, not a single DNS label. An underscore alone does not prove a gRPC protocol violation. DNS hostnames, certificate identity checks, SDK validation, and proxy routing can impose different constraints; preserve the constraints of the deployed path.

Before renaming a service or adding a rewrite, capture the failing request's authority, the rejecting layer and version, and the relevant error or trace. A platform may require DNS-compatible service names and enforce that policy at registration and CI, but a registry name need not be the wire authority.

A proxy rewrite is an option only when the request reaches that filter before the rejecting layer. It cannot fix a client rejection before transmission or an inbound parser rejection before the filter runs. Scope any verified rewrite to the affected traffic and explicit authority mappings; retain route, TLS identity, and authorization checks. Diagnosis, migration, and verification: references/grpc-authority-workaround.md.

R9 — Ingress and egress are explicit, not implicit

• Ingress: traffic from outside the cluster enters via a documented Gateway (Istio Gateway / k8s Ingress). One default gateway per environment is fine; ad-hoc NodePort exposure is not.
• Egress: traffic leaving the cluster goes through an egress gateway OR is explicitly allowed via mesh policy. "Service can reach the internet" is a configuration decision, not a default.
• A service that needs to call an external third-party API declares it; the platform sets up the egress path with mTLS termination, retry, observability.

R10 — Envoy access log is part of the observability contract

• Mesh access logs MUST include log-id and lane fields (read from request headers).
• Access logs ship to the same log pipeline as app logs so on-call can join "what mesh saw" with "what app saw" by log-id.
• enablePrometheusMerge (Istio) is enabled so Envoy stats appear in the same Prom scrape as the app — one less moving part.

R11 — Long-lived streaming control channels need transport contracts

• WebSocket, SSE, long-poll, or split read/write transports need explicit connection state: idle, connected, reconnecting, closing, and closed. Permanent auth/not-found/protocol rejections stop retry; retryable network, 429, and 5xx failures use bounded exponential backoff with jitter and a total reconnect horizon.
• Reconnect must refresh credentials, preserve or send a high-water mark that advances only after durable apply or write, replay only unconfirmed outbound messages whose original principal/workspace/session/policy/capability/auth-generation tuple still matches, deduplicate by stable event/request id, and fence callbacks from a previous transport incarnation.
  • If that tuple changed, cancel or fail closed and surface unknown finality instead of replaying under fresh credentials.
  • If multiple sessions run in one process, auth headers and tokens must be per transport instance, not process-global mutable state.
• Liveness is not optional. Define heartbeat or keepalive cadence, liveness timeout, proxy idle-timeout mitigation, sleep/wake behavior, listener/timer cleanup on reconnect and close, and graceful shutdown drain windows. A closed transport must release blocked writers and must not leave orphan listeners, timers, pings, or callbacks.
• Streaming writes need backpressure and queue bounds. Batches are serialized when the downstream store cannot safely handle concurrent writes; retry-after hints are clamped and jittered; permanent failures, max-failure drops, close-time drops, and best-effort flush semantics are surfaced as diagnostics rather than silently treated as delivered.

R12 — Local egress relays are credentialed transport, not environment sugar

• A local proxy or relay that injects credentials, rewrites trust roots, or tunnels subprocess traffic must bind only to loopback, fail open only before credentials are accepted or injected when optional setup fails, and fail closed once it has accepted a credentialed tunnel but cannot safely continue. Delete or revoke bootstrap secrets only after the listener, trust bundle, and cleanup path are proven ready, and before launching credentialed children or accepting traffic when possible; keep the secret out of model context, child-visible arguments, logs, dumps, crash reports, and fixture files.
• A local browser authorization callback listener is also credentialed transport.
  • Bind only to loopback, prefer an OS-assigned port unless a caller explicitly owns the port, validate callback path and host/port, accept only the pending flow's state, and reject duplicate or late callbacks after manual fallback, cancellation, timeout, success, or failure.
  • Pending HTTP responses must have exactly one terminal outcome, listeners must remove handlers and close on every exit path, and callback secrets or authorization codes must never be echoed into child process args, model-visible prompts, logs, telemetry, dumps, or fixture files.
• Trust-root and proxy environment changes are scoped runtime state. Publish them only to the intended child process tree, include explicit bypass rules for loopback, metadata, private networks, first-party control channels, package registries, and other direct paths, and never route plain traffic through a credential-injecting tunnel unless the protocol explicitly supports it. Per-runtime proxy support differences need tests; inherited proxy variables must be passed through only when the parent relay is still the intended authority.
• CONNECT-style or byte-tunnel relays need parser and stream contracts: cap header size, reject non-tunnel methods, preserve bytes that arrive before the upstream tunnel is open, chunk large payloads, validate tunnel frame encoding, implement keepalive inside proxy idle windows, handle partial writes and backpressure per runtime, avoid writing plaintext errors after encrypted payloads may have begun, and cleanup listeners, timers, sockets, and upstream sessions on every close path.

Workflow

Phase A — Identity and lane chain

1. Confirm lane and log-id propagate through ctx for every hop type (HTTP, RPC, queue). Spot-check by reading framework client/server middleware, not user code.
2. Confirm the RPC base struct (or HTTP header equivalent) carries log_id + tags {idc, cluster, lane}. A missing tag is a propagation bug.

Phase B — Mesh policy

1. PeerAuthentication = STRICT (mTLS namespace-wide).
2. DestinationRule per critical callee: connection-pool limits, connect timeout, outlier detection.
3. VirtualService HTTP routes: lane header match → lane subset, request timeout, and the R4 retry policy.
4. AuthorizationPolicy expresses which services may call which (zero-trust at network level).

Phase C — Service discovery

1. Confirm the chosen mode is documented: registry-based, k8s-native, or mixed.
2. Registry mode: registry is multi-replica, explicitly NOT in mesh; SDK resolver uses registry metadata.
3. K8s-native mode: Service / EndpointSlice / pod labels are the discovery source; app code does not bypass mesh/lane routing with raw cluster URLs.
4. Discovery metadata includes lane (registry tag or pod label) so VirtualService subsets can target correctly.
5. Healthcheck contract exists: registry heartbeat or k8s readiness removes unhealthy endpoints within the documented window.

Phase D — Framework middleware

1. Read the framework default client/server options module. Confirm middleware chain matches R6.
2. Confirm dev cannot build a client/server without inheriting these.
3. Test: induce the configured outlier threshold on one callee → verify ejection and recovery, the configured retry layer retries only eligible calls within its budget, and exhausted calls propagate a stable error-code.

Phase E — Failure modes

1. Mesh sidecar crash → app pod fails readiness. Configured behavior?
2. Discovery backend unreachable → resolver uses cached endpoints for a bounded TTL, or fails closed by documented policy?
3. Lane label drops mid-chain → service calls wrong env. Detected how?
4. Cascading timeout → caller deadline exceeded but downstream still running. Hedge or cancel?

Phase F — Evidence

Before marking work done:

• One request traced end-to-end across ≥ 2 services; mesh access log + app log + trace all share the same log-id.
• One synthetic outlier: pod 5xx → mesh ejects → recovers when pod healthy → metrics + access log show the pattern.
• One lane-routing verification: request with lane=test header → arrives only at lane=test instances. Verified via access log upstream_host.

Decision Points

• "Add a new retry policy for service X" → apply R4's replay-safety and single-owner checks. Use the matching VirtualService HTTP route for Istio retries; disable and verify route retries when the SDK owns them. DestinationRule limits concurrent retries, not per-request attempts.
• "Service A times out calling Service B" → check three layers in order: app deadline (ctx timeout) → framework client timeout → mesh request timeout. Whichever is smaller wins; align them.
• "Switch service discovery mode" → use references/service-discovery-choice.md; do not mandate registry unless the platform needs registry-specific capabilities such as per-instance drain, out-of-cluster lookup, or existing registry federation.
• "Need mTLS to a non-mesh external service" → egress gateway with terminating TLS, not app-managed certs.
• "Multiple registries (legacy + new)" → run them side-by-side, dual-write during migration, single read source per lane.

Sanitization and Provenance

This skill is product-agnostic. It must not contain:

• Repository names, service names, internal hostnames, internal IPs, namespaces, cluster names.
• Business-domain header values, tenant types, or vertical terms.
• Specific image registry URLs, ingress hostnames, or cloud-provider artifact paths.
• Concrete credentials or example certificates.

Reused industry patterns (PSM-style identity, <owner>.<class>.<env> shape, traceparent, :authority semantics) are fair game.

References

• references/mesh-architecture.md — Istio (or equivalent) topology, sidecar injection rules, gateway shape, EnvoyFilter use, what mesh owns vs framework owns.
• references/service-discovery-recipe.md — Registry-based SD (Nacos as recurring example), instance metadata, lane tag, healthcheck contract, dev-vs-prod differences.
• references/framework-middleware.md — Server and client middleware chain (HTTP + RPC); the canonical RPC base struct shape; verification commands.
• references/multi-env-routing.md — Lane label end-to-end recipe; VirtualService patterns; queue-boundary propagation; stress/shadow tags.
• references/retry-timeout-circuit-breaker.md — Mesh defaults vs SDK overrides; cascading timeout budgets; idempotency awareness; outlier detection tuning.
• references/grpc-authority-workaround.md — Locate authority rejection; distinguish naming policy from protocol syntax; verify scoped compatibility changes.
• references/dual-sidecar-and-traffic-config-center.md — Pod-level dual sidecar (mesh + platform), per-protocol mesh injection policy, per-caller-callee traffic config via config center (separate from mesh routing).
• references/rpc-framework-recipe.md — Concrete kitex/hertz default suite: shared RPC base field contract, RPC base.Request full schema (8 fields incl From/To), dual-channel ctx propagation (metainfo + grpc metadata), 9-code error enum + framework error mapping table, three resolver strategies (registry / FQDN fallback / proxy), platform latency histogram buckets, CORS defaults exposing log-id header, server boot sequence with graceful shutdown.
• references/service-discovery-choice.md — Decision framework: registry-based vs k8s-native SD; both support lane routing + canary + multi-env; pick by per-instance drain need, laptop access pattern, federation preference, operational burden; mixed mode (k8s east-west + thin registry for laptop) is workable; migration paths in both directions.
• references/service-discovery-migration-playbook.md — Step-by-step playbook: 6-phase registry-retirement (inventory / add k8s resolver / dual-resolve burn-in / switch registration off / migrate off-cluster tools / decommission); per-phase risk + rollback path; total ~3 months active work for medium platform; reverse direction (add thin sync-only registry) for off-cluster lookup; 6 anti-patterns to avoid during migration.
• references/protobuf-http-contract-signals.md — Cross-stack gate for classifying an HTTP surface as routine JSON/OpenAPI vs protobuf-backed: the in-scope/out-of-scope predicate, protobuf signal list, authoritative wire-format record, and owner-approval decision list. Shared by the Go/Python/web/mini-program/testing skills (each points here; do not redefine per stack).
• references/http-response-envelope-contract.md — Cross-stack JSON business-response envelope (code/message/data): canonical shape, new-service vs new-route vs existing-surface adoption/migration, non-JSON surfaces that record their own contract, and the invent/duplicate/hide anti-patterns. The response-side companion to the request-side base struct; shared by the same stack skills (each adds only its define/implement/consume/assert verb).

Verification before marking work done

1. Static: framework default options module includes all R6 middleware; mesh PeerAuthentication is STRICT; DestinationRule and VirtualService exist for every callee that participates in lane routing.
2. Live, identity: cross-service trace shows log-id + lane consistent across all hops; mesh access log lines include both.
3. Live, mesh policy: trigger the configured outlier threshold with an eligible pool size; verify ejection and recovery against the effective policy and metrics. Consecutive-error checks are inline, not delayed until the periodic analysis interval.
4. Live, lane routing: send request with non-default lane → confirm only matching-lane instances serve it.
5. Static, no leakage: grep this skill's content — zero internal hostnames, repo names, or business terms.

If any check fails, the work is interim, not done.