Fault Tolerant Distributed Integration

Fault-Tolerant Distributed Integration

Overview

Treat network boundaries as failure-prone by default. Add deadlines, telemetry, and bounded recovery paths before layering on resilience features.

Core Rules

1. Put explicit context.Context deadlines on outgoing Go calls.
2. Instrument cross-service paths with structured tracing and request correlation.
3. Send large files directly to storage instead of proxying them through the app server.
4. Retry only with idempotency or deduplication in place.
5. Use local mirrors such as httpbin to test failure paths before shipping.

Lifecycle And Partial Failure

• Make cancellation, timeout, connection close, and response-body ownership explicit for every network operation.
• Define what happens when a remote operation succeeds but the caller loses the response, or when a retry observes an unknown outcome.
• Preserve structured context at the boundary that handles the failure; do not hide partial failure behind a generic success or unbounded retry loop.

Guardrails

• Keep queues, retries, and worker pools bounded.
• Prefer direct-to-storage or queue-based designs over app-server relay for heavy transfers.
• Preserve clear error context across process boundaries.
• Do not add distributed machinery when the task is local or one-shot.

Flexible Guardrail

• For local-only helpers, skip the full telemetry stack.
• Do not wrap a tiny client in retry and circuit-breaker plumbing unless the boundary is real.
• Add the heavier patterns only when retries, file size, or latency make them necessary.

AGENTS.md Reference Entry

- **Skill Reference:** `$fault-tolerant-distributed-integration`
  - **When to invoke:** Use this when writing Go clients, direct-to-storage flows, or any network integration that must survive retries, timeouts, or partial failure.
  - **Prompt Hook:** "Act as a Distributed Infrastructure Developer. Add deadlines, telemetry, and bounded retry handling to the integration boundary."