# Debugging Network Issues

> Investigates network, streaming and protocol failures with falsification-first layered experiments, the general fallback for unknown root causes. Use for ECONNRESET / RST_STREAM / SSE stalls, CDN/CGNAT timeouts or certificate errors, throughput collapse where everything returns 200 but transfers crawl, or LAN identity/topology mysteries. Not for known Tailscale/TUN/DNS/proxy-path conflicts (use tunnel-doctor).

- Skill: `daymade/debugging-network-issues-2` (Agent Skill, multi-file: 18 files)
- Install (CLI): `npx get-skill add daymade/debugging-network-issues-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/daymade/debugging-network-issues-2/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: daymade (https://skillmd.com/u/daymade)
- Updated: 2026-11-16
- Page: https://skillmd.com/skills/daymade/debugging-network-issues-2

---


# Debugging Network Issues

Evidence-driven investigation methodology for incidents where the obvious cause is probably wrong. Built from a real 5-hour production case (see [references/case-sse-rst-130s.md](references/case-sse-rst-130s.md)) where assumption-stacking wasted hours that a 10-minute layered experiment would have resolved.

Apply this skill when the user reports a network/streaming/protocol symptom and the investigator feels tempted to diagnose from one log line or one circumstantial data point. The skill's job is to slow that reflex down.

## Triage first — is this a known domain?

Before applying the general methodology below, check whether the symptom points at a stack with a dedicated skill available to the consumer. Those carry the domain-specific symptom→cause→fix tables this skill deliberately stays general about — start there, and come back here for methodology if the root cause turns out to be elsewhere.

| If the symptom is…                                                                                                                                                                                       | Start with                                   |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| Codex session Bad Request / HTTP 400, native request or SSE failure, or a question about failure time versus a fix | **codex-request-diagnostics**, when available; otherwise continue this Skill's evidence-first method using available native/session evidence and explicit coverage gaps |
| macOS Tailscale ⨯ proxy/VPN conflict (Shadowrocket / Clash / Surge): `tailscale ping` works but SSH/curl/git fails, `Connection closed by 198.18.x.x`, TUN DNS hijack, ~60s `getaddrinfo` resolver stall | **tunnel-doctor**                            |
| Cloudflare config: `ERR_TOO_MANY_REDIRECTS`, SSL-mode mismatch, DNS / proxy-status issues behind the orange cloud                                                                                        | **cloudflare-troubleshooting**               |
| Windows App / AVD / W365 RDP connection quality: WebSocket instead of UDP Shortpath, high RTT, STUN/TURN interference                                                                                    | **windows-remote-desktop-connection-doctor** |
| Client-side proxy / VPN / TUN misrouting: one specific site fails with `ERR_CONNECTION_CLOSED` or `SSL_ERROR_SYSCALL`, other sites work, DNS returns fake/TUN IPs, and adding a PROXY rule did not help | **this skill** — read [references/case-proxy-tun-cname-override.md](references/case-proxy-tun-cname-override.md) first |
| TLS certificate-verification errors (`UNKNOWN_CERTIFICATE_VERIFICATION_ERROR`, a cert for the wrong site) or mid-handshake EOFs on **every** DIRECT-routed/domestic domain at once, while proxied domains work — and any proxy health watchdog still reports green                                        | **tunnel-doctor** (TUN DIRECT split-brain step) |
| The proxy is reachable and the OS/Tailscale route is healthy, but bulk transfers are slow and changing the active exit/node changes the measured rate | **tunnel-doctor** — proxy node / exit / chain capacity branch |
| A page or API completes slowly, including small responses that return 200; the slow layer is unknown | Read [latency investigation priorities](references/layered-isolation-experiment.md#latency-investigation-priorities) before selecting a lower-level probe |
| Bulk transfers crawl, jobs overrun their estimates, or files arrive truncated while latency and health checks look fine | **this skill** — go straight to [Step 0.7](#step-07-throughput-collapse--when-nothing-errors-and-everything-is-slow); preserve the throughput comparison |
| A whole group of smart-home / IoT devices is offline at once — often weeks after a router/AP swap or Wi-Fi settings change; the hub is "right there" but its children are all dead, plus some direct-Wi-Fi devices | **this skill** — read [references/case-iot-router-migration-wpa3.md](references/case-iot-router-migration-wpa3.md) first |

If none match — or you tried a domain skill and the evidence points elsewhere — continue below. The methodology generalizes to any multi-layer system.

> **Note for this skill specifically**: If the symptom is a Cloudflare 524/522 on a **large `POST` body** (e.g., `/<openrouter-path>` with `Content-Length` > 1 MB), the failure is often **upload time to origin exceeding Cloudflare's origin read timeout**, not backend slowness. Use the upload-vs-processing checklist below before assuming a backend stall.

## Core principles

### 1. Evidence over assumption

If you cannot point to a concrete artifact — log line, pcap frame, probe output, metric sample — you are guessing, not diagnosing. Before stating "X is the cause", require yourself to name the direct evidence. If it does not exist yet, add instrumentation (see [references/instrumentation-patterns.md](references/instrumentation-patterns.md)) or capture it (see [references/packet-capture-recipes.md](references/packet-capture-recipes.md)) before continuing.

### 2. Falsification over confirmation

N independent sources "confirming" a hypothesis does not make it true. One falsifying observation rules it out. Before acting on a hypothesis, answer:

> "What observation would make me abandon this hypothesis?"

If the answer is "nothing" or "I cannot think of one", the hypothesis is unfalsifiable and must not drive the investigation. If the answer is concrete, go look for that observation before committing to action.

### 3. Layered isolation

Multi-hop systems (client → CDN → LB → reverse proxy → app → upstream) concentrate bugs at the seams between layers. When a symptom could plausibly come from several layers, **do not reason about which layer; test**. Compare the same logical request across verified paths, name every changed hop, and attribute only the isolated difference. For intermittent failures, use a bounded return to the original condition before treating a successful variation as causal evidence. Read [references/layered-isolation-experiment.md](references/layered-isolation-experiment.md) for path design and A/B/A interpretation.

**The same technique isolates capacity, not just correctness.** When the symptom is a rate rather than an error, vary the *stack* instead of the *hop*: measure the same direction over two channels that share the network path but share no application code. Two slow channels make a shared-path limit the leading hypothesis; they do not exclude an application or workload factor common to both. A fast control supports a conclusion only for the transport, route and workload it actually matched. Two fast channels mean the probe did not reproduce the reported symptom, not that the incident is disproved. See Step 0.7 for how to read each of the four outcomes.

### 4. Counter-review before committing

Before committing to a root cause or shipping a fix, have independent reviewers challenge the conclusion — not confirm it. Agents are good at surfacing risks a single investigator did not think of; they are bad at weighing them. Apply the four-question filter (see [references/counter-review-pattern.md](references/counter-review-pattern.md)) to every finding before it shapes action.

### 5. A green health check certifies only the path it probes

Monitors watch the paths their authors thought to probe. Multi-plane systems — a TUN proxy with a DIRECT plane and a proxied plane, a service with a data plane and a control plane, a stack with a backend API and a web login page — fail one plane at a time, and a watchdog that probes only the other plane stays green through the entire outage. In the incident behind this principle, a proxy health daemon probed one overseas endpoint through the proxy every 5 minutes and logged "healthy" for 2+ hours while the direct plane was completely down.

Before accepting "the monitor says it's healthy" as evidence, ask: **which exact path does that check exercise?** Its green counts as evidence for that path alone. Enumerate the planes the system actually forwards or serves, and probe the failing one directly — the check that would have caught the outage is usually one curl away.

**A check also certifies only the *quantity* it is large enough to measure.** Path coverage is one axis; scale is the other, and it is the one that hides throughput collapse. A liveness probe returns a few hundred bytes, so its timing is dominated by handshake and round-trip — in the case study it answered in 40 ms all day, unchanged, across a 100× swing in the link's actual capacity, and it was not lying: it genuinely did what it measures. Same for `ping`/RTT, which times a small packet's round trip and says nothing about capacity.

The practical rule: **a probe measures the link only once transfer time dominates its total time.** So size it by the answer you need, not by convenience — if a probe returns in well under a second, essentially all of that was setup and you have measured setup. Sidestep the sizing question entirely by budgeting *time* instead of bytes (stream for N seconds, divide what arrived by N), which is what Step 0.7's commands do and why they stay cheap on a link that is already crawling. Extend the question to: **which path, and at what scale?**

### 6. Exhaust the instruments before asking the user — escalate only what only they can provide

A user's spoken observation ("the Wi-Fi drops for a few seconds", "some apps report connection errors") is a *lead*, not a question to hand back. Treat each open sub-question it contains as presumed observable, and search for the instrument before asking: monitor/probe ledgers (where a health daemon exists, its log holds the failure timetable of the paths it probes — in both directions: a green ledger never falsifies the user's observation (Principle 5), and a red one says nothing about paths it never probed), system logs (kernel link/disassociation events settle "did Wi-Fi physically drop"; per-process error bursts settle "which apps failed" only for apps that log there), and control channels on the same network (a second host over a different medium — a wired box with no proxy, a hotspot comparison, even the router's own status page; when no control channel exists, say so explicitly rather than treating the question as answered). When the instrument search comes back empty for a sub-question, the move is Step 4 — stand up instrumentation and wait for recurrence — not "ask the user to recall".

Only three categories legitimately reach the user mid-investigation:

1. **Decision rights** — production changes, restarts, spending, anything past your authorization boundary.
2. **Physical actions and first-person observations no instrument captured** — unplugging a box, reading a label password, whether the Wi-Fi icon itself disappeared on a device you have no telemetry for.
3. **Credentials or access you do not have** — a management password that exists only on a device label.

Observed 2026-09-10: a home-network investigation closed its report with four questions for the user. "Which apps reported errors" — its diagnostic content, a per-plane failure timetable — was already in the health daemon's ledger (the app-name list itself lives only on the user's screen, and adds no layer information the timetable lacks); "did Wi-Fi actually drop" was settled by twelve hours of kernel logs with zero disassociation events; both went to the user anyway. The fourth — "is the retired router still broadcasting" — is the honest counter-example: its ARP absence proved nothing (Trap 14: a probe certifies only the L2 domain it ran on), the platform redacted the Wi-Fi scan, so that one correctly reached the user as a five-second physical check.

For every question you hand the user, write one sentence in the report naming why no instrument can answer it and which of the three categories it falls in. If you cannot produce that sentence, the question is not exhausted — go back to the instruments; do not ask.

## Workflow

Copy this checklist into the investigation notes and check items off:

```
Investigation Progress:
- [ ] Step 0:   Scope the symptom (exact error, exact times, who, who-not, what changed)
- [ ] Step 0.5: Verify the premise — does direct evidence show the symptom is actually happening?
- [ ] Step 0.6: **For large POST bodies: distinguish upload-timeout from processing-timeout** (see recipe below)
- [ ] Step 0.7: **For bulk throughput collapse: measure a rate, then two-channel it** (see recipe below)
- [ ] Step 1:   Gather direct evidence at every hop before hypothesizing
- [ ] Step 2:   Frame ≥3 hypotheses; for each, name (a) what falsifies it, (b) which layer boundary the intervention would target
- [ ] Step 3:   Design a decisive experiment (for network: layered isolation)
- [ ] Step 4:   Add instrumentation if evidence gaps block direct observation
- [ ] Step 5:   Execute, record actual vs predicted
- [ ] Step 6:   Counter-review before acting
- [ ] Step 7:   Fix + re-run the same experiment to verify
- [ ] Step 8:   Document wrong turns as teaching material
```

### Step 0: Scope

A tight scope is the difference between a 20-minute investigation and a 5-hour one. Before looking at anything, extract:

- **Exact error string** (copy-paste, not paraphrase). `socket closed` is not the same as `ECONNRESET` is not the same as `HTTP/2 RST_STREAM INTERNAL_ERROR (err 2)`.
- **Exact timestamps** (ISO-8601 with timezone, not "yesterday evening")
- **Reproducibility** (every time / intermittent / only specific users)
- **Who is affected, who is not** (differential observations narrow the search)
- **What changed recently** (deploys, config, upstream dependencies, client versions)

Distinguish symptom from diagnosis. "Slow" is not a symptom. "Request took 130.898s then returned HTTP/2 INTERNAL_ERROR" is.

**Quantify the user's wait even when nothing errors.** For a page or API, record elapsed time to the named user-visible completion and use [latency investigation priorities](references/layered-isolation-experiment.md#latency-investigation-priorities). For a bulk transfer, record bytes per second over a stated payload size ("8.4 MB in 95 s = 0.09 MB/s, HTTP 200 throughout") and use Step 0.7. An unmeasured complaint starts at Step 0.5; HTTP 200 does not establish acceptable latency, throughput or complete content.

### Step 0.5: Verify the premise

Before investing in a full investigation, confirm the reported symptom is actually happening — not just inferred from downstream effects or user frustration. One cheap direct observation beats hours spent investigating a non-problem.

Ask: **"What direct evidence shows this symptom is real?"**

- If the user reports "timeout at 130s": is that from a timestamped log, a browser network panel, or a recollection?
- If the user reports "connection reset": did they see the packet or is it inferred from a retry spike?
- If the user reports "fails for some but not others": has it been reproduced in a controlled test, or is it anecdotal?

Acceptable premises:

- Log line with timestamp and error string
- Browser DevTools Network screenshot showing the failure
- Reproduction command that shows the symptom on demand
- Metrics chart showing the specific error count rising

Not sufficient as premise:

- "Users are saying it feels slow"
- "The alert fired but I did not check what actually failed"
- "Last week someone mentioned..."

If the premise fails verification, the fix is observation — not investigation. Add the missing telemetry, wait for the next occurrence with instrumentation in place, and return when you have real data. Resist the sunk-cost instinct to investigate anyway "since we are already here".

### Step 0.6: Upload-timeout vs processing-timeout for large POST bodies

For CDN-fronted `POST`/`PUT` endpoints with large bodies, the most common misdiagnosis is blaming backend slowness when the real problem is **time-to-upload-body exceeding the CDN/proxy origin timeout**.

Apply this sub-checklist when the symptom is a 524/522/504 on a request with `Content-Length > ~500 KB`:

1. **Locate the edge/reverse-proxy access log** (Caddy, nginx, Envoy, Cloudflare Logpush).
2. **Compare `bytes_read` (or equivalent) to `Content-Length`**:
   - `bytes_read == Content-Length` and `status` is an error → likely backend/processing problem.
   - `bytes_read < Content-Length` and the connection closed around the timeout window → **upload problem**.
3. **Check `duration` / `request_time` semantics**:
   - Caddy `duration` = wall time from first byte read to response end.
   - nginx `$request_time` = same.
   - <upstream-capture-service> / app `request_time` = time backend spent processing after body was fully received.
   - If proxy `duration` ≈ timeout but upstream `request_time` is short or never logged, the body upload is the bottleneck.
4. **Look for `status=0` (Caddy) or `-` (nginx)**:
   - `status=0` means the proxy never wrote an HTTP response, usually because the downstream/client side closed first.
5. **Correlate with upstream logs**:
   - If the request ID / ray ID / trace ID **does not appear** in upstream (<new-api-container>, <upstream-capture-service>, app) logs, the request never finished uploading.

**Example signature of an upload-timeout 524:**

```json
{
  "status": 0,
  "duration": 125.0,
  "bytes_read": 4111422,
  "request": {
    "headers": { "Content-Length": ["6042141"] }
  }
}
```

Interpretation: the proxy kept the connection for 125 s, read 4.1 MB of a 6 MB body, then Cloudflare closed it and returned 524.

**Example signature of a processing-timeout:**

```json
{
  "status": 504,
  "duration": 120.1,
  "bytes_read": 6042141,
  "request": { "headers": { "Content-Length": ["6042141"] } }
}
```

Interpretation: full body uploaded, but backend did not respond before proxy timeout → backend/processing problem.

### Step 0.7: Throughput collapse — when nothing errors and everything is slow

The recipe above (0.6) handles a *timeout* — an error code you can grep for. This one handles the family with **no error at all**: every request succeeds, every health check is green, and the system is unusable because bytes arrive at a fraction of the expected rate. It is the failure mode most likely to be misdiagnosed for hours, because the entire error-driven toolkit has nothing to bite on.

**Why the usual signals go green** — each of these was observed simultaneously with a 100×-degraded link:

| Signal | What it read | Why it is green anyway |
| --- | --- | --- |
| HTTP status | `200` on everything | Slow is not an error; the response completes, just late |
| Round-trip latency | 12 ms | RTT measures a small packet's round trip; it is **not** a capacity measurement |
| Liveness/health endpoint | `0.04 s` | Its response is handshake-sized, so its timing is all RTT and none of it transfer. **A probe that finishes before the link becomes the limiting factor cannot measure the link** — see Principle 5 |
| Transport status field | `active; direct <endpoint>` | Reports the **path type**, not the path's capacity — see trap 16 |

So the first move is to stop reading status and **measure a rate**.

For device or wired/Wi-Fi comparisons, load [device throughput fieldwork](references/device-throughput-fieldwork.md) before probing or changing association; bind the control/data paths, comparison and recovery window there.

**The decisive experiment: two independent channels, same direction, same time budget.**

This is Principle 3's layered isolation applied to capacity instead of correctness. Pick two channels that reach the same host over the same network path but share **no application code**. Agreement supports a shared-path limit, but it does not exclude an application/workload factor common to both channels or independent application bottlenecks. For either channel to serve as a fast control, record whether its transport, route and workload match the slow channel; an unmatched control supports a conclusion only about the conditions it actually exercised.

Both commands below are **time-budgeted, not size-budgeted**: they stream for a fixed maximum number of seconds and report whatever arrived. That keeps the probe's cost bounded no matter how bad the link is — a fixed 8 MB payload took under a second on the healthy path in the case study and about 90 seconds on the degraded one, and a degradation deeper than that scales the wait without bound, exactly when you can least afford it. Use each channel's observed elapsed time: a complete curl transfer may exit 0 before the cap, while a valid bounded sample uses the time actually received. Comparability comes from the same direction and receiving-end rate measurements, not from pretending every probe consumed the full budget.

```bash
BUDGET=20

# Channel A — the service under suspicion. Point it at the largest object you can name.
# Keep the body for payload validation. Capture curl's exit status before running
# another command; neither exit 0 nor exit 28 is sufficient by itself.
A_BODY=$(mktemp "${TMPDIR:-/tmp}/throughput-a.XXXXXX")
A_RC=0
curl -sS -o "$A_BODY" --max-time "$BUDGET" \
  -w 'A: %{size_download} B in %{time_total}s = %{speed_download} B/s; HTTP %{http_code}\n' \
  "http://<host>:<port>/<a-large-object>" || A_RC=$?
echo "A curl exit: $A_RC"

# Channel B — a completely different stack to the same host. `cat /dev/zero` streams
# until cut off, so the byte count is decided by the link, not by the payload size.
B=$(timeout "$BUDGET" ssh <host> 'cat /dev/zero' 2>/dev/null | wc -c)
echo "B: $B B in ${BUDGET}s = $((B / BUDGET)) B/s"
```

> **Do not substitute `dd`'s own summary line for channel B's number.** `dd if=/dev/zero … ` reports how fast it wrote into *its stdout*, which for a piped SSH transfer is absorbed by the pipe and SSH's channel window — it measures the local write, not the wire, and it never sees connection setup. Measured on the case study's healthy path, raw SSH moved 33.6 MB end-to-end at 44 MB/s while `dd` self-reported 54.9 MB/s on the same transfer. A sender-side rate is an upper bound on the wire rate, never a measurement of it. Time the whole invocation from the receiving end, as above.

**Expect the two channels to disagree in absolute terms — that is not a problem.** On the same healthy path, raw SSH measured ~44 MB/s while fetches through the media service measured 11–16 MB/s, because channel A pays for the service's own read and encode work on top of the wire. This is why the decision rule below is about **order of magnitude**, not equality: what you are testing is whether both channels sit in the same band, not whether they print the same number. Two channels that differ by 3× on a healthy link and both collapse to 0.1 MB/s on a degraded one suggest a shared constraint; check common application/workload factors and path evidence before treating that as causation.

**Before reading the table, confirm both probes actually ran.** This is trap 17 applied to your own recipe, and it is the easiest way to get a confidently wrong answer here: a probe that never transferred anything produces a *low or zero rate*, which can be misread as "slow" and push the investigation toward a shared-path hypothesis — a conclusion drawn from a measurement that never happened. Two checks, both cheap:

- **Channel A**: require the expected HTTP status and intended payload, not just a byte count. Exit 0 is a valid complete measurement only when the complete expected object — or the exact requested `Range` — arrived and its payload was verified by the object's own format, known content or digest. Exit 28 can yield a valid **bounded** sample only when `$A_BODY` contains intended-object bytes that actually arrived before the deadline; use the observed `time_total` and do not call the object complete. DNS/connection/TLS errors, zero bytes, redirects to a login page, and error bodies are invalid measurements, regardless of how fast or slow they were. Remove the temporary body after the payload check.
- **Channel B**: is `$B` plausibly non-zero? `2>/dev/null` in that command hides SSH's own errors, so a host you cannot authenticate to yields `B=0` silently. Re-run it once without `2>/dev/null` if the number looks wrong. Authentication/command errors and zero bytes are not slow-throughput measurements.

Read the result as a four-way test — all four outcomes are reachable, and three of them tell you to stop what you were about to do:

| Channel A | Channel B | Conclusion |
| --- | --- | --- |
| slow | slow | **A shared-path limit is likely, not proven.** Application/workload factors common to both channels and independent application bottlenecks remain possible. Confirm the shared transport/route and inspect those factors before changing the path; do not treat A-only tuning as futile on this evidence alone. If path evidence confirms the hypothesis, go to "what changed about the path" below. |
| slow | fast | **A-specific application/service or path is suspect only under a matched control.** If B matched A's transport, route and workload, its fast result supports an A-side bottleneck. Otherwise it establishes only that B's tested conditions were fast; align the control before acquitting A's path. |
| fast | slow | **Neither conclusion yet — your control is the anomaly.** A's fast result is scoped to its tested transport/route/workload. Check SSH-specific cost (cipher on a weak CPU, a hop that shapes SSH), or whether B crossed a different path. Do not conclude "the service is fine"; replace B with a third stack and re-run. |
| fast | fast | **The probe did not reproduce the symptom.** Both green results are scoped to the transport/route/workload actually tested. The real workload differs in some dimension you have not replicated — direction (upload vs download), concurrency, object size, or time of day. Change one of those and re-measure before believing the green result. |

**How close is "the same rate"?** Treat agreement within **~20%** as the same rate for this purpose — the two channels differ in framing, encryption and per-object overhead, so exact equality is not expected and not required. The judgement is order-of-magnitude: two stacks reading 0.09 and 0.11 MB/s agree; 0.1 and 12 MB/s do not. If the gap is between ~20% and ~2×, treat it as unresolved and widen the budget or the object size rather than picking a side.

**Pairing channel B when there is no SSH.** The requirement is only *same host, same direction, no shared application code* — SSH is convenient, not special. Any of these works: a second unrelated service on the host (a metrics endpoint, an object store, a static file server), a container-runtime transfer (`docker cp` from that host), a raw throughput tool if you can run one on both ends (`iperf3 -c`), or the host's own package/artifact mirror. What does **not** count is a second endpoint of the same service — that shares the code you are trying to exonerate, so agreement proves nothing.

**Then find what changed about the path.** Once the path is implicated, the variable is usually topology, not hardware: which route the traffic actually takes today versus yesterday. For mesh VPNs (Tailscale, Nebula, ZeroTier) and split-tunnel proxies, the same logical address can be served by a LAN-direct path, a WAN-direct path, or a relay — with order-of-magnitude different capacity and **no change in the status field or the address you connect to**. Ask the transport to report the path it is actually using (`tailscale ping <host>` prints the endpoint and whether the reply came via a relay).

If no prior known-good measurement exists to compare against — the common case in a first incident — you can still make progress without one, because the question "did this change?" has a cheaper substitute: **measure the same path from somewhere else.** A second client on a different network, or the host measuring *itself* over loopback, brackets the problem without any history. Then record today's number as the baseline you did not have; the second occurrence of this incident is much cheaper than the first, and only if someone writes the number down.

**Reuse a compatible successful transfer before inventing another.** For an upload,
check the existing successful command/configuration and its observed result first.
Match direction, route, object sizes, tool/SDK version, account permissions and
resume/completion semantics; reuse the compatible parts without copying credentials
or weakening overwrite protection. Treat a successful configuration as a starting
point to validate, not proof of today's capacity or a permanent optimum.

**Bound concurrency at both levels.** Record simultaneous files × in-flight parts
per file as the approximate part-request budget, plus retries and other traffic.
Vary one level at a time within the existing resource/permission limits. Compare
matched workload windows using newly acknowledged, non-duplicated payload bytes
divided by actual elapsed wall time; record units, window and errors. A sender's
queued bytes, cumulative totals including old work, or a short burst against a
degraded historical baseline cannot establish sustained benefit. Keep the simplest
configuration that improves the measured job without added failures; if the
benefit disappears or errors rise, restore the known working configuration.

Keep transfer evidence separate: part acknowledgements establish progress; a
successful completion response needs object-length/completeness verification;
matching remote size/type/metadata does not establish a whole-content digest.
Use the original job's required integrity check and user-visible result before
calling it delivered. Do not turn a throughput probe into an unnecessary full
redownload or duplicate payload backup.

**Finally, audit what the wrong assumption already broke.** A degraded link does not just make things slow — it silently invalidates every timeout you calibrated on the fast path, and those timeouts produce *corrupt artifacts that look like successes*. Before declaring the incident over, re-verify anything transferred during the degraded window; see trap 18 and Step 7.

**Replay the original workload before changing its semantics.** After a path or
node change restores a representative transfer rate, rerun the exact operation
that originally failed with its original completeness contract: a full Git clone
for a clone incident, the full body for an upload incident, or the original
streaming protocol for a stream incident. A small health request, `ls-remote`, or
partial/shallow clone proves only that its own smaller path works. If the original
workload now completes inside its existing deadline, the evidence refutes the need
for an application-level workaround such as partial clone or a relaxed timeout.
If it still fails, keep that result and only then investigate workload semantics.
This follows RFC 6349's separation of sustained TCP throughput from RTT/liveness:
the proof must exercise enough data for transfer capacity to dominate setup cost.

### Step 1: Gather direct evidence at every hop

Before calling an unfamiliar device or local-client HTTP/RPC endpoint for evidence, load [device throughput fieldwork](references/device-throughput-fieldwork.md#classify-an-unfamiliar-endpoint-before-calling-it); finish offline discovery and classify effects before a separate execution call.

Before framing hypotheses, collect:

- Server-side logs at every hop in the request path
- Client-side logs (browser devtools HAR, CLI debug log, SDK traces)
- Metrics over the incident window (RPS, latency, error rate, connection count, CPU/mem)
- Distributed trace if available
- Packet capture if the symptom is at the wire level (see [references/packet-capture-recipes.md](references/packet-capture-recipes.md))

If any of these is missing and relevant, **fill the gap before guessing**. Adding a `TRACE_*` env flag and restarting a container beats an hour of hypothesis-stacking. The instrumentation patterns in [references/instrumentation-patterns.md](references/instrumentation-patterns.md) are low-risk, env-gated, and safe to ship into production permanently.

#### Reading reverse-proxy access logs for upload/processing split

Caddy and nginx logs are the cheapest way to falsify "backend is slow". Focus on three fields:

| Field               | Caddy JSON key                   | nginx var                 | Meaning                                                            |
| ------------------- | -------------------------------- | ------------------------- | ------------------------------------------------------------------ |
| Total wall time     | `duration`                       | `$request_time`           | First byte from client → last byte to client (or connection close) |
| Body bytes received | `bytes_read`                     | `$request_length` (rough) | Bytes the proxy actually read from the client                      |
| Declared body size  | `request.headers.Content-Length` | `$content_length`         | What the client said it would send                                 |
| Response status     | `status`                         | `$status`                 | `0` / `-` means the proxy never wrote a response                   |

**Key patterns:**

- `bytes_read < Content-Length` and `duration ≈ timeout` → upload-timeout.
- `bytes_read == Content-Length` and `status` is 5xx → processing-timeout.
- `status == 0` and `bytes_read < Content-Length` → client/CDN closed before upload finished.

#### Tracing a single request across the stack

For the <project> stack (Cloudflare → Caddy → <provider-gateway-service> → <upstream-capture-service> → <new-api-container>), the canonical trace is:

1. **Cloudflare**: get `Cf-Ray` and timestamp from the client error or Cloudflare Logpush.
2. **Caddy**: `docker logs <gateway-container> | grep <Cf-Ray>` → extract `X-Request-Id` (Caddy `uuid`) and confirm `bytes_read`, `duration`, `status`.
3. **<provider-gateway-service>**: `docker logs <provider-gateway-service>` for `Client request error: aborted` or request/response logs.
4. **<upstream-capture-service>**: `grep <X-Request-Id or timestamp> /data/<upstream-capture-service>/log/access.log` → confirms whether the request reached <new-api-container> and how long upstream processing took.
5. **<new-api-container>**: `docker logs <new-api-container>` for billing/channel errors.

If the request ID never appears in steps 3–5, the failure happened at the edge or during body upload.

#### Aggregating by client IP to spot patterns

A single 524 can be a fluke; a pattern of 524s concentrated on one IP + one path is a smoking gun. Run an aggregation like:

```bash
# Caddy JSON example: count failures by IP and body size for an endpoint
python3 -c "
import sys, json
from collections import Counter, defaultdict
stats = defaultdict(lambda: {'total': 0, 'fail': 0, 'slow': 0, 'max_cl': 0})
for line in sys.stdin:
    d = json.loads(line)
    req = d.get('request', {})
    if req.get('uri', '').startswith('/<openrouter-path>'):
        ip = req.get('headers', {}).get('Cf-Connecting-Ip', [''])[0]
        cl = int(req.get('headers', {}).get('Content-Length', ['0'])[0] or 0)
        dur = d.get('duration', 0)
        status = d.get('status', 0)
        s = stats[ip]
        s['total'] += 1
        s['max_cl'] = max(s['max_cl'], cl)
        if status == 0:
            s['fail'] += 1
        elif status == 200 and dur > 60:
            s['slow'] += 1
for ip, s in sorted(stats.items(), key=lambda x: -x[1]['fail']):
    print(f\"{ip}: total={s['total']} fail={s['fail']} slow={s['slow']} max_cl={s['max_cl']}\")
" < caddy-access-log.jsonl
```

If one IP dominates failures and its `max_cl` is large, investigate upload bandwidth/path before backend.

### Step 2: Hypotheses with falsifiers and threat-model boundaries

List three or more plausible causes. For each, write three sentences:

- **What would confirm it?** (easy and often misleading)
- **What would refute it?** (the falsifier — this is what matters)
- **Which layer boundary would the intervention target?** (the threat-model question — forces you to be precise about where the fix would apply)

The third question prevents a common anti-pattern: proposing a fix that operates on the wrong hop. For example, a "keepalive" fix that writes bytes downstream to the client is useless for an _upstream_ idle timeout — the intervention targets a different boundary than the problem. Naming the boundary up-front surfaces this mismatch before coding starts.

If you cannot state a concrete refuter, the hypothesis is unfalsifiable. Flag it, but do not act on it. If you cannot state which boundary a proposed fix targets, you do not yet understand what the fix actually does.

### Step 3: Decisive experiment

For a request rejected only with a content encoding, follow the
[same-byte compression comparison](references/compressed-request-comparison.md).
It separates captured wire bytes from reconstructed JSON and synthetic fixtures,
and keeps transport selection and bounded A/B/A controls observable.

For network-layer problems, use **layered isolation** with verified path differences. Example for `Client → CDN → LB → Origin`:

| Path | Route | Difference from the full path |
| ---- | ----- | ----------------------------- |
| A | Full path via CDN and LB | Baseline |
| B | `--resolve` to verified origin IP | Remove CDN, LB and their connecting route |
| C | Server loopback | Also remove the external client-to-origin route |

If only A fails, localize the difference to the removed CDN/LB path; add a path retaining the LB to distinguish those layers. If A and B fail but C passes, investigate what C additionally removes. Read [references/layered-isolation-experiment.md](references/layered-isolation-experiment.md) for the full interpretation, bounded A/B/A checks for intermittent symptoms, and a mock idle upstream with a controlled idle interval.

For non-network domains:

- Performance: controlled benchmark with one variable changed
- Correctness bug: failing test case that reproduces
- Intermittent: sampled tracing plus a bounded return to the original condition; retain uncertainty when the contrast does not recur

### Step 4: Instrumentation when needed

For intermittent incidents that need unattended pre-failure evidence, first use
[the collection SOP](references/intermittent-evidence-collection.md). It covers
raw-before-parse preservation, independent planes, capture validation, bounded
shutdown, clock uncertainty and authentication/restart gaps.

If the decisive experiment requires an observation that cannot currently be made, add it — do not skip it. The canonical pattern is env-gated instrumentation that:

- Defaults off (zero runtime cost in steady state)
- Turns on via one environment variable, without code changes
- Writes greppable log tags (`[SSE-CHUNK] ts=... req=... bytes=...`)
- Ships into production permanently — future incidents reuse it

See [references/instrumentation-patterns.md](references/instrumentation-patterns.md) for the exact template used to diagnose the <upstream-provider> 125-second upstream silence in this incident.

### Step 5: Execute and record

Run the experiment once, fully documented: command, environment, inputs, observed outputs, wall-clock timestamps. Compare against the prediction made in Step 2. If actual matches predicted, the hypothesis is calibrated. If not, the hypothesis is wrong — **do not rescue it with ad-hoc auxiliary hypotheses** ("oh, but maybe X also interferes..."). Return to Step 2 and write new hypotheses from scratch.

### Step 6: Counter-review

Before committing to a root cause or shipping a fix, spawn independent reviewers to challenge the conclusion. Give them the s

…(truncated)
