Network diagnosis standards — the method, not the commands
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
1. Scope and triggers
Applies when something no longer works and you have to find out why: a method by layers and by
bisection, formulation of falsifiable hypotheses, canonical checking sequence
(resolution → outbound → return → acceptance by the application), choosing a tool by question and
which lie each one tells, a catalogue of the usual suspects with their characteristic symptom,
honest measurement of latency/loss/jitter, diagnosis in containers, Kubernetes, clouds and
overlays, and the discipline of recording and closing with a proven root cause.
Triggers: "it does not connect", "it is slow", "it is intermittent", "it drops every X minutes", "it works from here
but not from there", "it hangs while transferring", "it connects but does not load", ping, traceroute,
mtr --report, ss -tin/ss -s, ip route get, ip neigh, ip -s link, arping,
ethtool -S/ethtool -a, tshark, dumpcap, .pcap, curl -v --resolve, openssl s_client,
nc -zv, socat, iperf3 -R -P -u, /proc/net/nf_conntrack, /proc/net/sockstat,
net.ipv4.ip_local_port_range, somaxconn, TIME_WAIT, bpftrace, pwru, hubble observe,
kubectl debug, nicolaka/netshoot, tcp_retries2, "flow logs".
Guiding principle (inherited from networking-standards: the network is default-deny and documented
as code; what is not in the SoT does not exist): diagnosis ends in a proven root
cause, not in "it fixed itself". Every step answers a falsifiable hypothesis written before
typing, changes one variable, and leaves saved evidence. A problem that disappears with no
explanation is not solved: it is waiting.
Not applicable: see networking-standards (parent: the design —topology, addressing,
VLAN, routing/BGP, correct MTU/MSS values, proxies, overlays, OOB plane—; it says how the network
should be, you find out why today it is not; the structural fix goes back to it),
observability-standards (permanent telemetry and its alerts: metrics, logs, traces,
dashboards, SLI/SLO, sampling, retention and the design of what is instrumented before there is a
problem — all continuous monitoring is theirs; you are the reactive diagnosis once the alert
has fired. The exact line: if the question is "what is measured and at what threshold do we alert?", it is theirs;
if it is "this is broken now, why?", it is yours. And the evidence you need and that does not exist is a
finding for them: every blind diagnosis ends in an instrumentation requirement),
incident-management-standards (the process of the declared incident: severity, Incident
Commander, channel, communication, mitigate-before-diagnosing, postmortem — command and
communication are theirs; technical diagnosis inside the incident is yours. When the IC asks to
mitigate now, you mitigate and note down what you sacrifice in evidence),
firewall-policy-standards (the policy: when the diagnosis concludes "a rule is missing" or
"there is a rule too many", the fix, its approval and its negative test are theirs; proving
that the packet dies in the filter is yours),
dns-standards (the name service and its data: zone, TTL, DNSSEC, legitimate resolver,
delegation; you prove that resolution is the cause and who answers what, they decide
which is the right answer), vpn-standards (the tunnel: design, keys, correct MTU and keepalive,
concentrator redundancy; "the VPN connects but I cannot browse" is yours —it is
PMTU and it is proven with a capture at both ends—, likewise "it drops after 5 minutes" and "it resolves
badly inside the tunnel"), linux-administration-standards (resolvectl, systemd-resolved,
nsswitch.conf, systemd units and resolution from the host; dmesg and boot),
linux-storage-standards (when "the network is slow" turns out to be I/O: iostat, fio),
kubernetes-standards (CNI, NetworkPolicy, Service/Ingress, service mesh: their design),
container-runtime-security-standards (container isolation and the eBPF agent's privileges —
a privileged debugging pod is a security decision, not a shortcut),
detection-engineering-standards (analytics over network telemetry; a traffic capture
made to investigate a compromise, not a fault, belongs to them and to
incident-response-forensics-standards —chain of custody and preservation—),
sre-practice-standards (SLO, error budget, managing the toil of repeated diagnosis),
aws-standards/azure-standards/gcp-standards (flow logs, Reachability Analyzer/Network Watcher
and equivalents: the provider's tool and its configuration),
data-platform-standards (latency that turns out to be the database engine's, not the network's),
microservices-architecture-standards (timeouts, retries and circuit breakers as the application's
design; here you only prove that the failure comes from there),
appsec-standards (a failure that turns out to be the application's), onprem-standards (umbrella),
homelab-standards (your own lab), offensive-security-standards (scanning a network that is not
yours, or active sweeping without authorisation, is not diagnosis), and the three design and
proactive-operations skills the cause returns to once found: if the fault
is explained by the campus design or by a BGP policy, the structural fix belongs to
routing-switching-standards; if encapsulation MTU, asymmetric ECMP or EVPN appears, it belongs to
datacenter-fabric-standards; and if the answer to "what changed?" is a configuration
deployment, the rollback and the gate that should have prevented it belong to network-automation-standards.
Here the work ends when the cause is proven; the permanent fix lives there.
They also exist and are boundaries: ha-clustering-standards (split brain, fencing and the cluster
network: the expected behaviour of the cluster under partition is theirs; proving that there was a
network partition and why is yours) and podman-systemd-containers-standards (rootless networks with
netavark/pasta and their resolution: their configuration is theirs, the diagnosis of the packet
lost between namespaces belongs here).
2. Default decisions / Toolchain
Verify the latest version and maintenance status on the web before committing to anything (§8).
Dates obtained from api.github.com and the project's site, not from summarised HTML pages.
| Question you are answering |
Default tool |
Alternative |
Banned |
| What is configured on this machine? |
ip addr/ip link/ip route/ip neigh/ip -s link (iproute2) |
nmcli/networkctl depending on the manager |
ifconfig, route, arp, netstat (net-tools: no release since 2001, unmaintained; they hide namespaces, routing policies, multiple tables and modern socket state) |
| Which route would this packet take? |
ip route get <dst> from <src> |
ip rule show for policy routing |
Reading the main table and assuming |
| Which sockets and in which state? |
ss -tanp, ss -tin (RTT, cwnd, retransmissions), ss -s |
/proc/net/sockstat |
netstat -an |
| Does the packet arrive and come back? |
tshark/dumpcap capturing at BOTH ends at the same time |
A capture at each intermediate hop if the path has several |
Capturing at only one end and inferring |
| Analysing the capture? |
Wireshark 4.6.x (GUI) to analyse; tshark to filter and automate |
capinfos, editcap, mergecap |
Eyeballing 2 GB of pcap instead of filtering |
| Where is it lost and how much? |
mtr --report --report-cycles 100 -w (v0.96; repo with activity in Jun 2026) |
traceroute -T -p 443 when ICMP/UDP is filtered |
ping as the only quality measurement |
| Is layer 1-2 healthy? |
ethtool <if> (negotiation), ethtool -S (error/discard counters), ethtool -a (pause) |
ip -s link for aggregate errors |
Diagnosing layer 3 without having looked at interface errors |
| State of filtering and NAT? |
nft list ruleset with counter; /proc/net/nf_conntrack and nf_conntrack_count vs nf_conntrack_max (conntrack-tools 1.4.9) |
nft monitor trace to follow a packet through the chains |
Assuming "the firewall is open" because somebody said so |
| Does the application accept? |
curl -v with --resolve (isolates DNS from connectivity), openssl s_client -connect -servername |
nc -zv for a raw port; socat for relays and protocol tests |
telnet host port as a TLS test |
| How much bandwidth is there really? |
iperf3 3.21 (09 Apr 2026) with -P (parallel), -R (reverse direction) and long runs |
iperf3 -u -b for UDP with a fixed rate, measuring loss and jitter |
Measuring 10 s in one direction and calling it a baseline |
| Where does the packet die inside the kernel? |
pwru v1.0.12 (13 Jul 2026, kernel ≥5.3; --output-skb ≥5.9) |
bpftrace 0.26.1 (02 Jun 2026) and bcc 0.37.0 tools (02 Jul 2026) for bespoke cases |
Guessing between nftables, routing and the driver |
| And in Kubernetes? |
hubble observe --verdict DROPPED (Cilium 1.20.0, 29 Jul 2026); kubectl debug --image=nicolaka/netshoot with --target |
kubectl debug node/<n> for the node's netns; Retina where the CNI is not Cilium |
kubectl exec into a distroless container and giving up |
| And in the cloud? |
The provider's flow logs + its reachability analyser |
A capture on the instance if the provider does not offer a managed one |
Concluding "it is the cloud" without looking at the flow logs |
On eBPF (pwru, bpftrace, bcc, Hubble): recommended, but not as the first step.
They are the answer to "the packet enters the machine and does not come out, and no classic tool tells me
where" — a real and frequent problem on hosts with containers, and where classic capture does not
reach because the drop happens between capture points. Requirements and cost to accept
beforehand: a modern kernel with BTF and CONFIG_KPROBES/CONFIG_BPF, high privileges on the host
(a security decision, see container-runtime-security-standards), and non-zero overhead in
production. Criteria: layer 1-2, ss, capture at both ends and firewall counters
first; eBPF when those four do not close the case.
3. The method
3.1 Before typing anything: four questions
- What changed? — The cause is the last change until proven otherwise. The
diagnosis starts in change control: deployments, firewall or routing changes,
patching, certificate renewal, change of provider, firmware update, the expiry of
something. If nobody knows what changed, that is the first finding (and a governance problem, not a
network one).
- What exactly is "it does not work"? — Reproduce the symptom precisely: which source, which
destination, which port, which protocol, which client, at what time, with which literal error message.
"The network is bad" is not a symptom; "from host A,
curl to B:443 hangs after the
TLS handshake, in 30% of attempts, since 09:14" is.
- What is the scope? — One user or all of them? One destination or all of them? One protocol or all of them?
One VLAN, one node, one availability zone? Scope discards more hypotheses in 30
seconds than an hour of captures: a failure that affects a single client is not in the
server, and one that affects every destination is not in the destination.
- Did it ever work? — "It never worked" is a configuration or design problem;
"it worked and stopped working" is a change or resource exhaustion problem. They are
two different investigations and confusing them costs hours.
3.2 Bisection: divide the path, do not walk it
- You do not walk the path hop by hop. You split it in half: pick an intermediate point
with visibility (a router, a load balancer, a node) and determine whether the problem is before or
after. Repeat. With 8 hops, bisection is 3 tests; walking it linearly, 8 — and with more
opportunities to get it wrong.
- Bisection also applies to non-spatial dimensions: two clients (one that fails,
one that does not) → what differs?; two destinations; two protocols; two moments; IPv4 vs. IPv6. The cut
is by variable, not just by place.
- One variable at a time. Changing two things and having it work is not a diagnosis: it is a
coincidence you will have to deal with again. If pressure forces changing several at
once to mitigate, it is noted and reverted one by one afterwards to identify which one it was.
- A falsifiable hypothesis before typing: write down "if the cause is X, then doing Y I will see Z;
if I see W, X is ruled out". A test that cannot refute your hypothesis is not a test, it is
a ceremony. This is what separates diagnosis from "firing off commands".
- Confirmation bias is the main enemy: as soon as you have a suspect, every
test seems to confirm it. Antidote: define in advance what result would make you abandon that
hypothesis, and look for it explicitly.
3.3 The canonical sequence
Four questions, in order. Each one has a binary answer and eliminates half the universe.
1) Does the name resolve — and to the right thing?
- Isolate DNS from the rest from the first minute:
curl -v --resolve host:443:<ip> directly
compares "does not resolve" against "does not connect". If it works with --resolve, the problem is
resolution and not the network.
- Ask each resolver separately and compare them, and compare that with what the process
actually uses: the system may have a local stub, a cache, a
search that completes the name,
a forgotten /etc/hosts or a library that does not even go through the OS resolver.
Resolution "from the shell" and "from the application" are not the same.
- Negative caching and TTL explain the classic "it works for me and not for you" and "it took an hour to
fix itself". Zone, TTL and resolver design, in
dns-standards.
2) Does the packet reach the destination?
- Capture at the destination, filtering by source and port. If it does not appear, the packet dies on the
way: firewall, route, NAT, VLAN, layer 1-2.
- Check at the source that it leaves and through which interface:
ip route get <dst> from <src> answers
the real question (it includes policy routing and the effective table), not the main table.
- If it leaves and does not arrive, bisect the path and check the counters of the filtering rules at each
hop: a
counter incrementing on the drop rule is a proof, not a suspicion.
3) Does the response come back? — the forgotten half of the problem
- The response gets lost as much as the outbound, and almost nobody looks at it. If at the destination you see the
SYN
and the SYN/ACK leaves, but the source does not receive it, the problem is the return path and everything
you were looking at was irrelevant.
- Typical causes of asymmetric failure: the return takes a different path (multihoming, specific
routes, partial VPN, VRF); a stateful firewall that only sees one direction and drops the other
for lack of state —an unmistakable symptom: it works for a while and then cuts off, or it fails
intermittently and non-reproducibly—; NAT in one direction and not the other; uRPF dropping because of an
invalid reverse path.
- Operational rule: in every hard case, simultaneous capture at both ends with synchronised
clocks, and compare. This solves 90% of what looks impossible: it tells you in which half
of the path the packet disappears and with that the investigation is reduced to half the network. If
the path also has an intermediary (proxy, load balancer, NAT), capture on both of its sides.
- Correcting asymmetry is a design (
networking-standards) or policy
(firewall-policy-standards) matter: it is never fixed by adding a broad accept.
4) Does the application accept it?
- The packet arrives, the
SYN/ACK comes back, and the failure persists: check whether the process listens (ss -tanp), on which address (0.0.0.0 vs 127.0.0.1 vs :: — a service on loopback is
unreachable from outside and looks like a firewall), and whether the accept queue is full
(ss -lt shows Send-Q as the backlog and Recv-Q as pending: if it saturates, the kernel
drops SYNs and the client sees timeouts with the server "alive").
- Failures that look like network ones and are the application's or its edge's: an immediate
RST (nobody listens
or the proxy rejects), a TLS handshake failing because of SNI, certificate, version or chain
(openssl s_client -servername), redirects, authentication, or the application's own timeouts
shorter than its retry.
RST vs. timeout is the trade's most informative distinction: RST = something answered and
rejected (live host, closed port, a proxy or firewall that rejects); silence = something dropped
(a firewall that drops, a missing route, a downed host). They are not the same problem and they are not
diagnosed the same way.
3.4 Which lie each tool tells
No tool lies out of malice: they all answer a narrower question than the one you think you
are asking.
ping / ICMP — it measures that the destination answers ICMP, not that the service works. ICMP is usually
deprioritised in the control plane of routers and switches, or filtered by policy. A
ping with 200 ms and loss towards the network device may be perfectly normal while the
data traffic runs impeccably; and a perfect ping says nothing about port 443. Never use
ping as the only quality measurement or as a proof of service.
traceroute/mtr — the trade's most widespread lie: loss at intermediate hops
means nothing. Routers generate TTL exceeded responses in their CPU and rate-limit them;
seeing 40% loss at hop 5 and 0% at the destination means that router prioritises its
work, not that there is a problem. Only two things count: (a) loss at the last hop (the
destination), and (b) loss that persists from one hop to the end. Besides, ECMP makes
each probe take a different path (use fixed-flow mode if your tool supports it),
the return path is invisible, and MPLS can hide hops entirely. When ICMP/UDP is
filtered, traceroute -T over the application's real port is what reflects the path that
matters.
ss — it tells you the socket's local state. ESTABLISHED at one end does not imply the other
is still there: a connection whose peer disappeared without a FIN keeps showing as established until a
keepalive or a write discovers it. ss -tin does give gold: RTT, cwnd, retransmissions — which
distinguish "the network is losing" from "the application is slow".
tcpdump/tshark — they capture where you are and after the kernel has decided
some things and before others: a packet dropped by the filter may or may not appear in the
capture depending on the hook point, and offload (GRO/GSO/TSO/LRO) shows giant "packets" that
do not exist on the wire (disable it if you are going to analyse sizes or MTU). With sampling or a badly
written filter, your "it does not show up" may be yours, not the network's. And an unfiltered capture on a loaded
link loses itself: use a capture filter (BPF) for what you want, and a display
filter to analyse.
ip route — it shows tables; ip route get shows the decision. With policy rules,
VRF or several tables, reading the main one and assuming is a classic mistake.
ip neigh/ARP — a STALE or FAILED entry tells you more than a REACHABLE one; and a correct entry
with the wrong MAC (duplicate IP, unexpected proxy ARP) is a silent failure
that no layer 3 test gives away. arping from the same segment reveals duplicate IPs
(two answers, two MACs) in one second, which is what no other tool does.
ethtool — the only one that sees layer 1-2: negotiation (duplex/speed; a badly negotiated
half duplex shows up as "slow and intermittent" under load), CRC errors, drops due to
lack of buffer, and per-queue counters. Cumulative counters: what matters is the
delta during the failure, not the total since boot.
iperf3 — it measures what you ask it to, and by default that is not what you think. Classic mistakes: measuring
10 s (all slow start), a single flow (limited by RTT and window, not by the link), only in
one direction (-R measures the other, which may be the broken one), in UDP without setting the rate (-b) or setting it
above capacity and calling your own saturation "network loss", with iperf3 itself
as the CPU bottleneck, or measuring against a shared public server.
iperf3 measures one path between two points at one instant; it does not measure "the network".
curl -v — without --resolve it mixes DNS, connection, TLS and HTTP into a single result; with
--resolve it separates the first. Its per-phase timings (-w) are a diagnosis in themselves: if the
time goes on the connect, it is the network; on the appconnect, it is TLS; on the starttransfer, it is
the application.
- The application logs — they say "connection timeout" for half a dozen mutually incompatible
causes. Useful for the exact time and the scope; useless as a diagnosis.
3.5 The usual suspects and their characteristic symptom
A recognition table. The symptom is what makes you suspect; the proof is what demonstrates it.
| Suspect |
Characteristic symptom |
Proof that demonstrates it |
| MTU / PMTU black hole |
The connection opens and hangs while transferring: SSH connects but scp stalls; the web page loads the HTML and not the images; "the VPN connects but I cannot browse". Typically after a tunnel or a change of encapsulation |
ping with a large packet and the DF bit, increasing until you find the cut-off; a capture showing retransmissions of the same large segment with no ACK; absence of ICMP fragmentation needed coming back |
| Filtered ICMP that breaks PMTUD |
Identical to the previous one, and it never fixes itself |
The intermediary blocking ICMP type 3 code 4 (or ICMPv6 packet-too-big): it is seen by its absence in the capture on the side that should receive it |
| DNS |
Intermittency that looks like the network's; "sometimes it takes exactly 5 seconds" (resolver timeout); it works by IP and not by name |
curl --resolve versus a normal curl; querying each resolver separately |
| Ephemeral port / NAT exhaustion |
Failures that increase with load, on the side that initiates many connections (proxy, NAT, API client) |
Socket count against net.ipv4.ip_local_port_range; on the NAT, active sessions against its capacity |
| Full conntrack table |
Silent drops under load, with a line in dmesg nobody looks at |
nf_conntrack_count against nf_conntrack_max; dmesg with table full |
| Different return path |
"It works for a while and then cuts off"; intermittent and irreproducible; it works in one direction |
Simultaneous capture at both ends: you see the outbound and the response that does not arrive |
TIME_WAIT / exhausted backlog |
Connection timeouts with the server alive and CPU low; it worsens at peaks |
ss -s (count by state), ss -lt with Recv-Q growing, the dropped-SYN counter |
| Keepalive vs. an intermediary's idle timeout |
"The session drops exactly after N minutes" of inactivity. A firewall, NAT, load balancer or cloud with an idle timeout shorter than the client's keepalive |
Reproduce with an idle session and a stopwatch; compare the intermediary's timeout with the TCP/application keepalive. It is fixed by lowering the keepalive, not by raising everybody's timeout |
| Duplicate IP |
Unexplained intermittency that changes with the ARP cache lifetime; "sometimes it lands on a different server" |
arping from the segment: two answers with different MACs |
| Duplicate MAC / MAC flapping |
Massive loss on a segment; the switch logs learning the same MAC on different ports |
Switch log; address table |
| Layer 2 loop / broadcast storm |
The whole VLAN goes down or crawls; switch CPU at 100%; it starts right after plugging something in |
Broadcast/multicast counters per port shooting up; STP with constant topology changes. It is an emergency: the port is isolated first |
| Aged or incomplete ARP/ND |
A host unreachable from its own segment while everything else is fine |
ip neigh in FAILED/INCOMPLETE state |
| IPv6 enabled and failing while IPv4 works |
"It is slow" with delays of exact seconds at the start; it works with -4; it fails only on some clients |
Compare curl -4 and curl -6; route and ND on IPv6. Usual cause: AAAA published without real IPv6 connectivity, or an IPv6 firewall without parity with IPv4 |
| Happy Eyeballs masking the failure |
The IPv6 failure is almost unnoticeable (the client retries over IPv4 after a short delay), so nobody fixes it and the initial latency is worse for everyone |
A capture showing the abandoned IPv6 attempt. RFC 8305 is the current specification; v3 is still a draft (§8) |
| A proxy or TLS inspection in the middle |
An unexpected certificate, a forced TLS version, rewritten ALPN, HTTP/3 that does not work, mTLS that fails, a Server different from the expected one |
openssl s_client -servername and compare the certificate issuer with the expected one |
| Load balancer with one bad backend |
A constant fraction of requests fails (1 in N) |
Repeat the test N+ times recording which backend each one goes to |
| Expired certificate or incomplete chain |
It fails at an exact time, for everybody at once, with nobody having touched anything |
openssl s_client showing the chain and the dates |
| Layer 1 |
Growing CRC errors, slow only under load, badly negotiated duplex, degraded optics |
ethtool -S (delta during the failure), ethtool (negotiation), optical power on the device |
| Saturation / bufferbloat |
Latency that shoots up only when there is traffic; the ping goes from 10 ms to 300 ms when a download starts |
Latency under load versus at rest; link utilisation by percentiles |
3.6 Latency, loss and jitter: measuring them properly
- Percentiles, never averages. The average hides exactly what breaks the experience. Look at
p50, p95, p99 and the maximum; and compare with the at-rest state, not with an absolute number. A p50 of 20 ms
with a p99 of 2 s is a broken system that averages well.
- Distinguish the three causes of "it is slow", because they have opposite fixes:
- Latency (RTT): it limits a TCP flow's throughput through the window. If the RTT is high, more
bandwidth fixes nothing; more parallel flows or a larger window do.
- Loss: it sinks TCP throughput disproportionately (1% loss can cost
most of the performance on a high-RTT link). It is seen in retransmissions (
ss -tin),
not in the ping.
- Jitter: irrelevant for a download, lethal for voice and video. It is measured with UDP at a fixed
rate, not with TCP.
- The measurement must reproduce the real case: same source-destination pair, same protocol, same
transfer size, same time of day. A test at rest does not reproduce a saturation problem,
and a 10-second test does not reproduce a 20-minute problem.
- If "it is slow" turns out to be I/O, CPU or the database, say so and close the case there: the network is
blamed by default, and proving that it is not the network is as valid a result as any
other (with evidence, not with a denial).
3.7 Modern environments: containers, Kubernetes, clouds and overlays
- The packet crosses several network namespaces. Before capturing, decide which one you are
capturing in: inside the container, on the host side of the
veth, on the bridge, in the node's netns,
on the physical interface, on the overlay interface. "It does not show up in the capture" almost always
means "you captured in the wrong namespace".
- Kubernetes: a checking order by the cluster's own layers — does the
Service name
resolve? → does the Service have Endpoints/EndpointSlice (a Service with no endpoints from a badly
written selector is the most common failure and does not look like a network one)? → is there a NetworkPolicy that
denies it (the drop happens in the datapath, before reaching the pod)? → does the readinessProbe
take the pod out of rotation? → do the CNI or kube-proxy have the rule? → does the node route?
hubble observe --verdict DROPPED gives the verdict and the reason for the drop in one step; beware of
monitoring aggregation, which can hide individual events from you.
- Distroless containers with no tools:
kubectl debug with an ephemeral container
(GA since Kubernetes 1.25) and a diagnostic image (nicolaka/netshoot), sharing the
pod's namespace, and kubectl debug node/<node> for the node's netns. Two operational warnings:
an ephemeral container cannot be removed until the pod is deleted and has no resource
limits; and to extract a capture, stream it over standard output instead of writing the
file inside (copying from an ephemeral container does not work).
- Clouds: the VPC/VNet flow logs are the first stop —they say whether the packet was
accepted or rejected and by which rule— followed by the provider's reachability analyser. Their
limits, which must be known: aggregation into windows (you do not see the packet, you see the flow), possible
sampling, a delay of minutes, and no payload. The specific configuration, in
aws-standards/azure-standards/gcp-standards.
- Overlays and tunnels (VXLAN, GENEVE, IPsec, WireGuard): capture inside the tunnel and
outside —they are two different questions: "does the traffic enter the tunnel?" and "does the tunnel reach the other
side?". And as soon as there is encapsulation, MTU is the first suspect, always.
- Service mesh / sidecar: the proxy may terminate TLS, rewrite headers, apply its
own timeouts and retries and return errors that look like the application's. Its access logs
are the source, not the capture.
4. Quality of the diagnosis (gates)
A diagnosis is accepted when it meets all of this. It is not bureaucracy: it is what prevents
the same incident from coming back in three weeks.
- Reproducibility: there is an exact command or procedure that produces the symptom at will
(or, if it is intermittent, a documented condition that triggers it and a measured rate). Without
reproduction, the fix cannot be validated.
- Hypotheses recorded with their result: what was tried, what was expected, what was seen and which
hypothesis was ruled out. A record of exclusions is worth as much as the finding, and it prevents the
next shift from repeating the same tests.
- Evidence saved, not described: captures (
.pcap) from both ends with a timestamp,
complete command outputs, counters before/after, screenshots of graphs with their
time range. "We saw packets being lost" is not evidence. Save the evidence before
mitigating: mitigation destroys the state that proves it.
- Root cause proven, not inferred — the central gate. "Proven" means you can
explain the complete mechanism from the change or the condition to the symptom, and that you can
reproduce the failure by activating the cause and make it disappear by deactivating it. Temporal correlation
is not causation: "we rebooted and it was fixed" is a data point, not a conclusion.
- Fix validated by the test that was failing, and additionally with the corresponding
negative test (what was supposed to stay blocked is still blocked). And validated from the real
affected source, not from the engineer's bastion.
- Absence of side effects verified: what was touched did not break anything else. Temporary
diagnostic changes (open rules, offload disabled,
tcpdump running, raised timeouts,
logs at debug) are explicitly reverted and the reversion is verified.
- Derived findings with an owner: the configuration drift, the missing rule, the metric
that did not exist, the runbook that was useless, the alert that did not fire. Each one is opened as work
with an owner and a date — towards
observability-standards, firewall-policy-standards,
networking-standards or whoever is relevant.
- If the case is closed without a root cause (it happens, and it is legitimate), it is closed saying so: what was
ruled out, what instrumentation is missing to diagnose it next time, and what trigger is left
armed to capture evidence when it comes back. That is a result; "it fixed itself" is not.
5. Security of the diagnosis
- A traffic capture contains personal data, credentials and business content. It is not
an innocuous technical file. Treat it as classified data: controlled storage, restricted
access, minimum retention and deletion when the case closes. If you are going to share it, anonymise it or
trim to headers (
-s to limit the capture) and remove the payload. Legal basis and
minimisation, in privacy-engineering-standards.
- Capturing in production is an action with impact: it consumes CPU and disk and can fill a
filesystem. Use a capture filter, a size limit and rotation, and give it a time limit
from the start. A forgotten
tcpdump on a server is a future incident.
- Distinguish a fault from a compromise from the first minute. If there is any sign of intrusion,
the objective stops being "restore the service" and becomes "preserve the evidence": the procedure changes,
do not reboot, do not delete, and escalate to
incident-response-forensics-standards (order of volatility and chain of custody).
- Port sweeping, scanning or injecting traffic into networks that are not yours —or without internal
authorisation— is not diagnosis: it is offensive activity and it is governed in
offensive-security-standards. Inside your network, warn whoever is monitoring so that your test does not
show up as an attack.
- Privileges: capturing requires
CAP_NET_RAW/CAP_NET_ADMIN and eBPF requires more. Grant them
temporarily and by name, not as a permanent configuration nor with a privileged container
that stays there (container-runtime-security-standards). Withdraw them when the case closes.
- Do not weaken controls to diagnose and then leave it like that: opening a rule "to test" is the
fastest route to a permanent
any/any (firewall-policy-standards). If you open, you open with
an automatic expiry.
6. Operability: preparing before it fails
- Diagnosis is prepared, not improvised. What must exist before the incident:
an up-to-date inventory and SoT (
networking-standards), a diagram of the traffic's real path,
OOB access, telemetry with sufficient retention (observability-standards), flows, and baselines:
normal RTT, normal throughput, normal error rate. Without a baseline, "it is high" is an
opinion.
- A synchronised clock on everything that logs or captures (healthy NTP/chrony). Without it, correlating
two captures or two logs is impossible, and that is precisely the method that solves the hard cases.
- An available observation point: SPAN/mirror, TAP, or at least a host with access to the segment and
tools installed. If the first step of your diagnosis is "install
tcpdump on the production
server", you are already too late.
- A minimum kit preinstalled or a diagnostic image ready: iproute2, capture,
mtr, curl,
nc/socat, ethtool, iperf3, and a diagnostic container image for environments with no
tools.
- Runbooks by symptom, not by tool: "it does not resolve", "it connects and hangs while transferring",
"intermittent", "slow only under load", "it drops every N minutes", "1 in N fails", "it works
from one host and not from another". Each one with the canonical sequence adapted and its escalation
criterion.
- A rescue window open when touching remote access, routes or the firewall — an inherited and non-
negotiable rule (
firewall-policy-standards, vpn-standards): OOB console, a second path, or
a timed rollback. Locking yourself out during a diagnosis is the most predictable and most
avoidable incident there is.
- When the diagnosis happens inside a declared incident: command and communication belong
to
incident-management-standards. You give hypotheses with a confidence level and an estimated time,
not premature certainties; and if the IC decides to mitigate before understanding, you mitigate — but you capture
the evidence first and leave the pending diagnosis written down. Mitigating does not close the root cause.
- Toil: the same diagnosis repeated three times is an instrumentation or design failure, not
bad luck. It becomes an alert, an automatic check or a structural fix
(
sre-practice-standards).
7. Sustainability and prohibitions
- The postmortem feeds the method: every hard case leaves either a new runbook, or a new
metric, or a design change. If it leaves nothing, it will happen again.
- Tool migration:
net-tools (ifconfig, netstat, route, arp) has been
unmaintained for years and is absent by default in modern distributions; scripts and
runbooks that still use it are migrated to iproute2 with a date. It is not purism: it hides namespaces,
policy routing and state that today determines the diagnosis.
- Cadence: review the version and CVEs of the capture and analysis tools (Wireshark/
tshark,
libpcap/tcpdump) along with the rest of the stack — they are parsers that process hostile input by
definition, and 2026 brings a flood of LLM-assisted findings in that family. Analysing an
untrusted capture with an unpatched Wireshark is exposing yourself.
- Training with real cases: keeping captures and timelines of solved cases as
team training material is worth more than any course.
FORBIDDEN
- ❌ Rebooting as the first step. It destroys the state you need (sockets, conntrack,
counters, ARP cache, in-memory logs) and turns the problem into something irreproducible. It is the last resort,
and with the evidence already collected.
- ❌ Changing several things at once and declaring victory when it works.
- ❌ Touching production blind: diagnostic changes with no hypothesis, no record and no rollback
plan.
- ❌ Blaming the network without evidence — and also absolving it without evidence. Both are the same
failing.
- ❌ Capturing at only one end in a hard case, or capturing without clock synchronisation.
- ❌ Concluding from loss at intermediate hops in
traceroute/mtr.
- ❌ Using
ping as proof that a service works, or as the only quality measurement.
- ❌
ifconfig, netstat, route, arp (net-tools) in diagnosis or in new runbooks.
- ❌ Reading
ip route and assuming, instead of asking with ip route get.
- ❌ Measuring with
iperf3 for 10 s, a single flow, a single direction, and calling it a baseline.
- ❌ Reporting latency averages instead of percentiles.
- ❌ Analysing packet sizes or MTU without disabling the interface's offload.
- ❌ Diagnosing layer 3 without having looked at layer 1-2 errors and negotiation.
- ❌ Closing the case with "it fixed itself", "it was a network thing" or "we rebooted and it works now".
- ❌ Leaving the diagnosis's temporary changes in place (open rules, captures running,
offload disabled, logs at debug, elevated privileges, a privileged debugging pod).
- ❌
…(truncated)
1---2name: network-troubleshooting-standards3description: Reactive network fault diagnosis method — bisecting the path, forming a falsifiable hypothesis and proving root cause. Use when something "doesn't connect", is intermittent, slow or hangs mid-transfer and you are reaching for ping, traceroute, mtr --report, ss -tin, ip route get, ip neigh, arping, ethtool -S, tshark or dumpcap, capturing pcap simultaneously at both endpoints, curl -v --resolve, openssl s_client, nc -zv, socat, iperf3 -R with -P and -u, /proc/net/nf_conntrack and ephemeral port exhaustion, PMTU blackhole with ICMP fragmentation-needed filtered, TIME_WAIT and listen-backlog overflow, keepalive versus middlebox idle timeout, IP address conflict or MAC flapping, broadcast storms and layer-2 loops, ARP/ND caches, IPv6 preferred over working IPv4 and Happy Eyeballs, TLS-inspecting proxies, tail-latency percentiles versus averages, bpftrace, bcc tools, pwru, hubble observe, kubectl debug --image=nicolaka/netshoot across pod and node netns, or VPC flow logs — and when you must record what was tried, 4---56# Network diagnosis standards — the method, not the commands78Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).910## 1. Scope and triggers1112Applies when **something no longer works and you have to find out why**: a method by layers and by13bisection, formulation of falsifiable hypotheses, canonical checking sequence14(resolution → outbound → return → acceptance by the application), choosing a tool by question and15**which lie each one tells**, a catalogue of the usual suspects with their characteristic symptom,16honest measurement of latency/loss/jitter, diagnosis in containers, Kubernetes, clouds and17overlays, and the discipline of recording and closing with a **proven root cause**.1819Triggers: "it does not connect", "it is slow", "it is intermittent", "it drops every X minutes", "it works from here20but not from there", "it hangs while transferring", "it connects but does not load", `ping`, `traceroute`,21`mtr --report`, `ss -tin`/`ss -s`, `ip route get`, `ip neigh`, `ip -s link`, `arping`,22`ethtool -S`/`ethtool -a`, `tshark`, `dumpcap`, `.pcap`, `curl -v --resolve`, `openssl s_client`,23`nc -zv`, `socat`, `iperf3 -R -P -u`, `/proc/net/nf_conntrack`, `/proc/net/sockstat`,24`net.ipv4.ip_local_port_range`, `somaxconn`, `TIME_WAIT`, `bpftrace`, `pwru`, `hubble observe`,25`kubectl debug`, `nicolaka/netshoot`, `tcp_retries2`, "flow logs".2627**Guiding principle** (inherited from `networking-standards`: *the network is default-deny and documented28as code; what is not in the SoT does not exist*): **diagnosis ends in a proven root29cause, not in "it fixed itself"**. Every step answers a falsifiable hypothesis written before30typing, changes **one** variable, and leaves saved evidence. A problem that disappears with no31explanation is not solved: it is waiting.3233**Not applicable**: see `networking-standards` (**parent**: the **design** —topology, addressing,34VLAN, routing/BGP, correct MTU/MSS values, proxies, overlays, OOB plane—; **it says how the network35should be, you find out why today it is not**; the structural fix goes back to it),36`observability-standards` (**permanent telemetry and its alerts**: metrics, logs, traces,37dashboards, SLI/SLO, sampling, retention and the design of what is instrumented **before** there is a38problem — **all continuous monitoring is theirs; you are the reactive diagnosis once the alert39has fired**. The exact line: if the question is "what is measured and at what threshold do we alert?", it is theirs;40if it is "this is broken now, why?", it is yours. And the evidence you need and that does not exist is a41finding **for** them: every blind diagnosis ends in an instrumentation requirement),42`incident-management-standards` (**the process of the declared incident**: severity, Incident43Commander, channel, communication, mitigate-before-diagnosing, postmortem — **command and44communication are theirs; technical diagnosis inside the incident is yours**. When the IC asks to45mitigate now, you mitigate and **note down** what you sacrifice in evidence),46`firewall-policy-standards` (**the policy**: when the diagnosis concludes "a rule is missing" or47"there is a rule too many", the fix, its approval and its negative test are theirs; **proving48that the packet dies in the filter is yours**),49`dns-standards` (**the name service and its data**: zone, TTL, DNSSEC, legitimate resolver,50delegation; **you prove that resolution is the cause and who answers what**, they decide51which is the right answer), `vpn-standards` (**the tunnel**: design, keys, correct MTU and keepalive,52concentrator redundancy; **"the VPN connects but I cannot browse" is yours** —it is53PMTU and it is proven with a capture at both ends—, likewise "it drops after 5 minutes" and "it resolves54badly inside the tunnel"), `linux-administration-standards` (`resolvectl`, `systemd-resolved`,55`nsswitch.conf`, systemd units and resolution **from the host**; `dmesg` and boot),56`linux-storage-standards` (when "the network is slow" turns out to be I/O: `iostat`, `fio`),57`kubernetes-standards` (CNI, `NetworkPolicy`, `Service`/`Ingress`, service mesh: their **design**),58`container-runtime-security-standards` (container isolation and the eBPF agent's privileges —59a privileged debugging pod is a security decision, not a shortcut),60`detection-engineering-standards` (analytics over network telemetry; **a traffic capture61made to investigate a compromise, not a fault, belongs to them and to**62`incident-response-forensics-standards` —chain of custody and preservation—),63`sre-practice-standards` (SLO, error budget, managing the *toil* of repeated diagnosis),64`aws-standards`/`azure-standards`/`gcp-standards` (flow logs, Reachability Analyzer/Network Watcher65and equivalents: the provider's tool and its configuration),66`data-platform-standards` (latency that turns out to be the database engine's, not the network's),67`microservices-architecture-standards` (timeouts, retries and circuit breakers as the application's68**design**; here you only prove that the failure comes from there),69`appsec-standards` (a failure that turns out to be the application's), `onprem-standards` (umbrella),70`homelab-standards` (your own lab), `offensive-security-standards` (**scanning a network that is not71yours, or active sweeping without authorisation, is not diagnosis**), and the three design and72proactive-operations skills the cause **returns to once found**: if the fault73is explained by the campus design or by a BGP policy, the structural fix belongs to74`routing-switching-standards`; if encapsulation MTU, asymmetric ECMP or EVPN appears, it belongs to75`datacenter-fabric-standards`; and **if the answer to "what changed?" is a configuration76deployment, the rollback and the gate that should have prevented it belong to `network-automation-standards`**.77Here the work ends when the cause is proven; the permanent fix lives there.7879They also exist and are boundaries: `ha-clustering-standards` (*split brain*, fencing and the cluster80network: **the expected behaviour of the cluster under partition is theirs; proving that there was a81network partition and why is yours**) and `podman-systemd-containers-standards` (rootless networks with82`netavark`/`pasta` and their resolution: their **configuration** is theirs, the diagnosis of the packet83lost between namespaces belongs here).8485## 2. Default decisions / Toolchain8687> Verify the latest version and maintenance status on the web before committing to anything (§8).88> Dates obtained from `api.github.com` and the project's site, not from summarised HTML pages.8990| Question you are answering | Default tool | Alternative | Banned |91|---|---|---|---|92| What is configured on this machine? | `ip addr`/`ip link`/`ip route`/`ip neigh`/`ip -s link` (**iproute2**) | `nmcli`/`networkctl` depending on the manager | `ifconfig`, `route`, `arp`, `netstat` (**net-tools**: no release since 2001, unmaintained; they **hide** namespaces, routing policies, multiple tables and modern socket state) |93| Which route would **this** packet take? | `ip route get <dst> from <src>` | `ip rule show` for policy routing | Reading the main table and assuming |94| Which sockets and in which state? | `ss -tanp`, `ss -tin` (RTT, cwnd, retransmissions), `ss -s` | `/proc/net/sockstat` | `netstat -an` |95| Does the packet arrive and come back? | **`tshark`/`dumpcap` capturing at BOTH ends at the same time** | A capture at each intermediate hop if the path has several | Capturing at only one end and inferring |96| Analysing the capture? | **Wireshark 4.6.x** (GUI) to analyse; `tshark` to filter and automate | `capinfos`, `editcap`, `mergecap` | Eyeballing 2 GB of pcap instead of filtering |97| Where is it lost and how much? | `mtr --report --report-cycles 100 -w` (**v0.96**; repo with activity in Jun 2026) | `traceroute -T -p 443` when ICMP/UDP is filtered | `ping` as the only quality measurement |98| Is layer 1-2 healthy? | `ethtool <if>` (negotiation), `ethtool -S` (error/discard counters), `ethtool -a` (pause) | `ip -s link` for aggregate errors | Diagnosing layer 3 without having looked at interface errors |99| State of filtering and NAT? | `nft list ruleset` with `counter`; `/proc/net/nf_conntrack` and `nf_conntrack_count` vs `nf_conntrack_max` (**conntrack-tools 1.4.9**) | `nft monitor trace` to follow a packet through the chains | Assuming "the firewall is open" because somebody said so |100| Does the application accept? | `curl -v` with **`--resolve`** (isolates DNS from connectivity), `openssl s_client -connect -servername` | `nc -zv` for a raw port; `socat` for relays and protocol tests | `telnet host port` as a TLS test |101| How much bandwidth is there really? | **`iperf3 3.21`** (09 Apr 2026) with `-P` (parallel), `-R` (reverse direction) and long runs | `iperf3 -u -b` for UDP with a fixed rate, measuring loss and jitter | Measuring 10 s in one direction and calling it a *baseline* |102| Where does the packet die **inside** the kernel? | **`pwru` v1.0.12** (13 Jul 2026, kernel ≥5.3; `--output-skb` ≥5.9) | `bpftrace 0.26.1` (02 Jun 2026) and `bcc 0.37.0` tools (02 Jul 2026) for bespoke cases | Guessing between `nftables`, routing and the driver |103| And in Kubernetes? | `hubble observe --verdict DROPPED` (Cilium **1.20.0**, 29 Jul 2026); `kubectl debug --image=nicolaka/netshoot` with `--target` | `kubectl debug node/<n>` for the node's netns; Retina where the CNI is not Cilium | `kubectl exec` into a *distroless* container and giving up |104| And in the cloud? | The provider's **flow logs** + its reachability analyser | A capture on the instance if the provider does not offer a managed one | Concluding "it is the cloud" without looking at the flow logs |105106**On eBPF (`pwru`, `bpftrace`, `bcc`, Hubble): recommended, but not as the first step.**107They are the answer to "the packet enters the machine and does not come out, and no classic tool tells me108where" — a real and frequent problem on hosts with containers, and where classic capture does not109reach because the drop happens between capture points. Requirements and cost to accept110beforehand: a modern kernel with **BTF** and `CONFIG_KPROBES`/`CONFIG_BPF`, high privileges on the host111(a security decision, see `container-runtime-security-standards`), and non-zero overhead in112production. **Criteria**: layer 1-2, `ss`, capture at both ends and firewall counters113first; eBPF when those four do not close the case.114115## 3. The method116117### 3.1 Before typing anything: four questions1181191. **What changed?** — The cause is the last change until proven otherwise. The120 diagnosis **starts in change control**: deployments, firewall or routing changes,121 patching, certificate renewal, change of provider, firmware update, the expiry of122 something. If nobody knows what changed, that is the first finding (and a governance problem, not a123 network one).1242. **What exactly is "it does not work"?** — Reproduce the symptom precisely: which source, which125 destination, which port, which protocol, which client, at what time, with which literal error message.126 "The network is bad" is not a symptom; "from host A, `curl` to B:443 hangs after the127 TLS handshake, in 30% of attempts, since 09:14" is.1283. **What is the scope?** — One user or all of them? One destination or all of them? One protocol or all of them?129 One VLAN, one node, one availability zone? Scope **discards more hypotheses in 30130 seconds than an hour of captures**: a failure that affects a single client is not in the131 server, and one that affects every destination is not in the destination.1324. **Did it ever work?** — "It never worked" is a **configuration or design** problem;133 "it worked and stopped working" is a **change or resource exhaustion** problem. They are134 two different investigations and confusing them costs hours.135136### 3.2 Bisection: divide the path, do not walk it137138- **You do not walk the path hop by hop.** You **split it in half**: pick an intermediate point139 with visibility (a router, a load balancer, a node) and determine whether the problem is before or140 after. Repeat. With 8 hops, bisection is 3 tests; walking it linearly, 8 — and with more141 opportunities to get it wrong.142- **Bisection also applies to non-spatial dimensions**: two clients (one that fails,143 one that does not) → what differs?; two destinations; two protocols; two moments; IPv4 vs. IPv6. The cut144 is by *variable*, not just by *place*.145- **One variable at a time.** Changing two things and having it work is not a diagnosis: it is a146 coincidence you will have to deal with again. If pressure forces changing several at147 once to mitigate, it is **noted** and reverted one by one afterwards to identify which one it was.148- **A falsifiable hypothesis before typing**: write down "if the cause is X, then doing Y I will see Z;149 if I see W, X is ruled out". A test that cannot refute your hypothesis is not a test, it is150 a ceremony. This is what separates diagnosis from "firing off commands".151- **Confirmation bias is the main enemy**: as soon as you have a suspect, every152 test seems to confirm it. Antidote: define in advance **what result would make you abandon** that153 hypothesis, and look for it explicitly.154155### 3.3 The canonical sequence156157Four questions, in order. Each one has a binary answer and eliminates half the universe.158159**1) Does the name resolve — and to the right thing?**160- Isolate DNS from the rest **from the first minute**: `curl -v --resolve host:443:<ip>` directly161 compares "does not resolve" against "does not connect". If it works with `--resolve`, the problem is162 resolution and not the network.163- Ask **each** resolver separately and compare them, and **compare that with what the process164 actually uses**: the system may have a local stub, a cache, a `search` that completes the name,165 a forgotten `/etc/hosts` or a library that does not even go through the OS resolver.166 Resolution "from the shell" and "from the application" **are not the same**.167- Negative caching and TTL explain the classic "it works for me and not for you" and "it took an hour to168 fix itself". Zone, TTL and resolver design, in `dns-standards`.169170**2) Does the packet reach the destination?**171- Capture **at the destination**, filtering by source and port. If it does not appear, the packet dies on the172 way: firewall, route, NAT, VLAN, layer 1-2.173- Check at the source that it **leaves** and through which interface: `ip route get <dst> from <src>` answers174 the real question (it includes policy routing and the effective table), not the main table.175- If it leaves and does not arrive, bisect the path and check the counters of the filtering rules at each176 hop: a `counter` incrementing on the drop rule is a proof, not a suspicion.177178**3) Does the response come back? — the forgotten half of the problem**179- **The response gets lost as much as the outbound, and almost nobody looks at it.** If at the destination you see the `SYN`180 and the `SYN/ACK` leaves, but the source does not receive it, the problem is the **return path** and everything181 you were looking at was irrelevant.182- Typical causes of asymmetric failure: the return takes a different path (multihoming, specific183 routes, partial VPN, VRF); a stateful firewall that only sees one direction and drops the other184 for lack of state —**an unmistakable symptom: it works for a while and then cuts off, or it fails185 intermittently and non-reproducibly**—; NAT in one direction and not the other; uRPF dropping because of an186 invalid reverse path.187- **Operational rule**: in every hard case, simultaneous capture at both ends **with synchronised188 clocks**, and compare. This solves 90% of what looks impossible: it tells you in which half189 of the path the packet disappears and with that the investigation is reduced to half the network. If190 the path also has an intermediary (proxy, load balancer, NAT), capture on both of its sides.191- Correcting asymmetry is a design (`networking-standards`) or policy192 (`firewall-policy-standards`) matter: it is **never** fixed by adding a broad `accept`.193194**4) Does the application accept it?**195- The packet arrives, the `SYN/ACK` comes back, and the failure persists: check whether the process **listens** (`ss196 -tanp`), on which address (`0.0.0.0` vs `127.0.0.1` vs `::` — a service on loopback is197 unreachable from outside and looks like a firewall), and whether the accept queue is full198 (`ss -lt` shows `Send-Q` as the backlog and `Recv-Q` as pending: if it saturates, the kernel199 drops `SYN`s and the client sees *timeouts* with the server "alive").200- Failures that **look** like network ones and are the application's or its edge's: an immediate `RST` (nobody listens201 or the proxy rejects), a TLS handshake failing because of SNI, certificate, version or chain202 (`openssl s_client -servername`), redirects, authentication, or the application's own *timeouts*203 shorter than its retry.204- **`RST` vs. *timeout* is the trade's most informative distinction**: `RST` = something answered and205 rejected (live host, closed port, a proxy or firewall that **rejects**); silence = something dropped206 (a firewall that **drops**, a missing route, a downed host). They are not the same problem and they are not207 diagnosed the same way.208209### 3.4 Which lie each tool tells210211No tool lies out of malice: they all answer a narrower question than the one you think you212are asking.213214- **`ping` / ICMP** — it measures *that the destination answers ICMP*, not that the service works. ICMP is usually215 **deprioritised** in the control plane of routers and switches, or filtered by policy. A216 `ping` with 200 ms and loss towards the network device may be perfectly normal while the217 data traffic runs impeccably; and a perfect `ping` says nothing about port 443. **Never use218 `ping` as the only quality measurement or as a proof of service.**219- **`traceroute`/`mtr`** — the trade's most widespread lie: **loss at intermediate hops220 means nothing**. Routers generate `TTL exceeded` responses in their CPU and rate-limit them;221 seeing 40% loss at hop 5 and 0% at the destination means **that router prioritises its222 work, not that there is a problem**. Only two things count: (a) loss at the **last hop** (the223 destination), and (b) loss that **persists** from one hop to the end. Besides, ECMP makes224 each probe take a different path (use fixed-flow mode if your tool supports it),225 the return path is invisible, and MPLS can hide hops entirely. When ICMP/UDP is226 filtered, `traceroute -T` over the application's real port is what reflects the path that227 matters.228- **`ss`** — it tells you the socket's **local** state. `ESTABLISHED` at one end does not imply the other229 is still there: a connection whose peer disappeared without a `FIN` keeps showing as established until a230 keepalive or a write discovers it. `ss -tin` does give gold: RTT, `cwnd`, retransmissions — which231 distinguish "the network is losing" from "the application is slow".232- **`tcpdump`/`tshark`** — they capture where you are and **after** the kernel has decided233 some things and **before** others: a packet dropped by the filter may or may not appear in the234 capture depending on the hook point, and offload (GRO/GSO/TSO/LRO) shows giant "packets" that235 do not exist on the wire (disable it if you are going to analyse sizes or MTU). With sampling or a badly236 written filter, your "it does not show up" may be yours, not the network's. And an unfiltered capture on a loaded237 link loses itself: use a capture filter (BPF) for what you want, and a display238 filter to analyse.239- **`ip route`** — it shows tables; **`ip route get`** shows the decision. With policy rules,240 VRF or several tables, reading the main one and assuming is a classic mistake.241- **`ip neigh`/ARP** — a `STALE` or `FAILED` entry tells you more than a `REACHABLE` one; and a correct entry242 with the **wrong MAC** (duplicate IP, unexpected proxy ARP) is a silent failure243 that no layer 3 test gives away. `arping` from the same segment reveals **duplicate IPs**244 (two answers, two MACs) in one second, which is what no other tool does.245- **`ethtool`** — the only one that sees layer 1-2: negotiation (duplex/speed; a badly negotiated246 *half duplex* shows up as "slow and intermittent" under load), CRC errors, drops due to247 lack of buffer, and per-queue counters. **Cumulative counters**: what matters is the248 **delta** during the failure, not the total since boot.249- **`iperf3`** — it measures what you ask it to, and by default that is not what you think. Classic mistakes: measuring250 10 s (all *slow start*), a single flow (limited by RTT and window, not by the link), only in251 one direction (`-R` measures the other, which may be the broken one), in UDP without setting the rate (`-b`) or setting it252 above capacity and calling your own saturation "network loss", with `iperf3` itself253 as the CPU bottleneck, or measuring against a shared public server.254 **`iperf3` measures one path between two points at one instant; it does not measure "the network".**255- **`curl -v`** — without `--resolve` it mixes DNS, connection, TLS and HTTP into a single result; with256 `--resolve` it separates the first. Its per-phase timings (`-w`) are a diagnosis in themselves: if the257 time goes on the `connect`, it is the network; on the `appconnect`, it is TLS; on the `starttransfer`, it is258 the application.259- **The application logs** — they say "connection timeout" for half a dozen mutually incompatible260 causes. Useful for the exact time and the scope; useless as a diagnosis.261262### 3.5 The usual suspects and their characteristic symptom263264A recognition table. The symptom is what makes you suspect; the proof is what demonstrates it.265266| Suspect | Characteristic symptom | Proof that demonstrates it |267|---|---|---|268| **MTU / PMTU black hole** | **The connection opens and hangs while transferring**: SSH connects but `scp` stalls; the web page loads the HTML and not the images; "the VPN connects but I cannot browse". Typically after a tunnel or a change of encapsulation | `ping` with a large packet and the DF bit, increasing until you find the cut-off; a capture showing retransmissions of the same large segment with no ACK; absence of ICMP *fragmentation needed* coming back |269| **Filtered ICMP that breaks PMTUD** | Identical to the previous one, and **it never fixes itself** | The intermediary blocking ICMP type 3 code 4 (or ICMPv6 *packet-too-big*): it is seen by its absence in the capture on the side that should receive it |270| **DNS** | Intermittency that looks like the network's; "sometimes it takes exactly 5 seconds" (resolver timeout); it works by IP and not by name | `curl --resolve` versus a normal `curl`; querying each resolver separately |271| **Ephemeral port / NAT exhaustion** | Failures that increase with load, on the side that **initiates** many connections (proxy, NAT, API client) | Socket count against `net.ipv4.ip_local_port_range`; on the NAT, active sessions against its capacity |272| **Full conntrack table** | **Silent** drops under load, with a line in `dmesg` nobody looks at | `nf_conntrack_count` against `nf_conntrack_max`; `dmesg` with `table full` |273| **Different return path** | "It works for a while and then cuts off"; intermittent and irreproducible; it works in one direction | Simultaneous capture at both ends: you see the outbound and the response that does not arrive |274| **`TIME_WAIT` / exhausted backlog** | Connection *timeouts* with the server alive and CPU low; it worsens at peaks | `ss -s` (count by state), `ss -lt` with `Recv-Q` growing, the dropped-`SYN` counter |275| **Keepalive vs. an intermediary's *idle timeout*** | **"The session drops exactly after N minutes"** of inactivity. A firewall, NAT, load balancer or cloud with an idle timeout shorter than the client's keepalive | Reproduce with an idle session and a stopwatch; compare the intermediary's timeout with the TCP/application keepalive. It is fixed by lowering the keepalive, not by raising everybody's timeout |276| **Duplicate IP** | Unexplained intermittency that changes with the ARP cache lifetime; "sometimes it lands on a different server" | `arping` from the segment: two answers with different MACs |277| **Duplicate MAC / *MAC flapping*** | Massive loss on a segment; the switch logs learning the same MAC on different ports | Switch log; address table |278| **Layer 2 loop / broadcast storm** | The whole VLAN goes down or crawls; switch CPU at 100%; it starts right after plugging something in | Broadcast/multicast counters per port shooting up; STP with constant topology changes. It is an emergency: the port is isolated first |279| **Aged or incomplete ARP/ND** | A host unreachable from its own segment while everything else is fine | `ip neigh` in `FAILED`/`INCOMPLETE` state |280| **IPv6 enabled and failing while IPv4 works** | "It is slow" with delays of exact seconds at the start; it works with `-4`; it fails only on some clients | Compare `curl -4` and `curl -6`; route and ND on IPv6. Usual cause: `AAAA` published without real IPv6 connectivity, or an IPv6 firewall without parity with IPv4 |281| **Happy Eyeballs masking the failure** | The IPv6 failure is **almost** unnoticeable (the client retries over IPv4 after a short delay), so nobody fixes it and the initial latency is worse for everyone | A capture showing the abandoned IPv6 attempt. **RFC 8305** is the current specification; v3 is still a **draft** (§8) |282| **A proxy or TLS inspection in the middle** | An unexpected certificate, a forced TLS version, rewritten ALPN, HTTP/3 that does not work, mTLS that fails, a `Server` different from the expected one | `openssl s_client -servername` and compare the certificate issuer with the expected one |283| **Load balancer with one bad backend** | A **constant fraction** of requests fails (1 in N) | Repeat the test N+ times recording which backend each one goes to |284| **Expired certificate or incomplete chain** | It fails at an exact time, for everybody at once, with nobody having touched anything | `openssl s_client` showing the chain and the dates |285| **Layer 1** | Growing CRC errors, slow only under load, badly negotiated duplex, degraded optics | `ethtool -S` (delta during the failure), `ethtool` (negotiation), optical power on the device |286| **Saturation / bufferbloat** | Latency that shoots up **only when there is traffic**; the `ping` goes from 10 ms to 300 ms when a download starts | Latency under load versus at rest; link utilisation by percentiles |287288### 3.6 Latency, loss and jitter: measuring them properly289290- **Percentiles, never averages.** The average hides exactly what breaks the experience. Look at291 p50, p95, p99 and **the maximum**; and compare with the at-rest state, not with an absolute number. A p50 of 20 ms292 with a p99 of 2 s is a broken system that averages well.293- **Distinguish the three causes of "it is slow"**, because they have opposite fixes:294 - **Latency (RTT)**: it limits a TCP flow's throughput through the window. If the RTT is high, more295 bandwidth fixes nothing; more parallel flows or a larger window do.296 - **Loss**: it sinks TCP throughput disproportionately (1% loss can cost297 most of the performance on a high-RTT link). It is seen in retransmissions (`ss -tin`),298 not in the `ping`.299 - **Jitter**: irrelevant for a download, lethal for voice and video. It is measured with UDP at a fixed300 rate, not with TCP.301- **The measurement must reproduce the real case**: same source-destination pair, same protocol, same302 transfer size, same time of day. A test at rest does not reproduce a saturation problem,303 and a 10-second test does not reproduce a 20-minute problem.304- **If "it is slow" turns out to be I/O, CPU or the database, say so and close the case there**: the network is305 blamed by default, and proving that it is **not** the network is as valid a result as any306 other (with evidence, not with a denial).307308### 3.7 Modern environments: containers, Kubernetes, clouds and overlays309310- **The packet crosses several *network namespaces*.** Before capturing, decide **which one** you are311 capturing in: inside the container, on the host side of the `veth`, on the bridge, in the node's netns,312 on the physical interface, on the overlay interface. "It does not show up in the capture" almost always313 means "you captured in the wrong namespace".314- **Kubernetes: a checking order by the cluster's own layers** — does the `Service` name315 resolve? → does the `Service` have `Endpoints`/`EndpointSlice` (a `Service` with no endpoints from a badly316 written selector is the most common failure and does not look like a network one)? → is there a `NetworkPolicy` that317 denies it (the drop happens in the datapath, **before** reaching the pod)? → does the `readinessProbe`318 take the pod out of rotation? → do the CNI or `kube-proxy` have the rule? → does the node route?319 `hubble observe --verdict DROPPED` gives the verdict and the reason for the drop in one step; beware of320 **monitoring aggregation**, which can hide individual events from you.321- ***Distroless* containers with no tools**: `kubectl debug` with an ephemeral container322 (**GA since Kubernetes 1.25**) and a diagnostic image (`nicolaka/netshoot`), sharing the323 pod's namespace, and `kubectl debug node/<node>` for the node's netns. Two operational warnings:324 an ephemeral container **cannot be removed** until the pod is deleted and **has no resource325 limits**; and to extract a capture, **stream it over standard output** instead of writing the326 file inside (copying from an ephemeral container does not work).327- **Clouds**: the VPC/VNet **flow logs** are the first stop —they say whether the packet was328 accepted or rejected and by which rule— followed by the provider's reachability analyser. Their329 limits, which must be known: aggregation into windows (you do not see the packet, you see the flow), possible330 sampling, a delay of minutes, and no payload. The specific configuration, in331 `aws-standards`/`azure-standards`/`gcp-standards`.332- **Overlays and tunnels** (VXLAN, GENEVE, IPsec, WireGuard): capture **inside** the tunnel and333 **outside** —they are two different questions: "does the traffic enter the tunnel?" and "does the tunnel reach the other334 side?". And as soon as there is encapsulation, **MTU is the first suspect, always**.335- **Service mesh / sidecar**: the proxy may terminate TLS, rewrite headers, apply its336 own timeouts and retries and return errors that look like the application's. Its access logs337 are the source, not the capture.338339## 4. Quality of the diagnosis (gates)340341A diagnosis is accepted when it meets **all** of this. It is not bureaucracy: it is what prevents342the same incident from coming back in three weeks.3433441. **Reproducibility**: there is an exact command or procedure that produces the symptom at will345 (or, if it is intermittent, a documented condition that triggers it and a measured rate). Without346 reproduction, the fix cannot be validated.3472. **Hypotheses recorded with their result**: what was tried, what was expected, what was seen and which348 hypothesis was ruled out. A record of exclusions is worth as much as the finding, and it prevents the349 next shift from repeating the same tests.3503. **Evidence saved, not described**: captures (`.pcap`) from **both** ends with a timestamp,351 complete command outputs, counters before/after, screenshots of graphs with their352 time range. "We saw packets being lost" is not evidence. Save the evidence **before**353 mitigating: mitigation destroys the state that proves it.3544. **Root cause proven, not inferred** — the central gate. "Proven" means you can355 explain the complete mechanism from the change or the condition to the symptom, **and** that you can356 reproduce the failure by activating the cause and make it disappear by deactivating it. Temporal correlation357 is not causation: "we rebooted and it was fixed" is a data point, not a conclusion.3585. **Fix validated by the test that was failing**, and additionally with the corresponding359 negative test (what was supposed to stay blocked is still blocked). And validated from the **real360 affected source**, not from the engineer's bastion.3616. **Absence of side effects verified**: what was touched did not break anything else. Temporary362 diagnostic changes (open rules, offload disabled, `tcpdump` running, raised timeouts,363 logs at debug) **are explicitly reverted** and the reversion is verified.3647. **Derived findings with an owner**: the configuration drift, the missing rule, the metric365 that did not exist, the runbook that was useless, the alert that did not fire. Each one is opened as work366 with an owner and a date — towards `observability-standards`, `firewall-policy-standards`,367 `networking-standards` or whoever is relevant.3688. **If the case is closed without a root cause** (it happens, and it is legitimate), it is closed **saying so**: what was369 ruled out, what instrumentation is missing to diagnose it next time, and what trigger is left370 armed to capture evidence when it comes back. That is a result; "it fixed itself" is not.371372## 5. Security of the diagnosis373374- **A traffic capture contains personal data, credentials and business content.** It is not375 an innocuous technical file. Treat it as classified data: controlled storage, restricted376 access, minimum retention and deletion when the case closes. If you are going to share it, anonymise it or377 trim to headers (`-s` to limit the capture) and remove the payload. Legal basis and378 minimisation, in `privacy-engineering-standards`.379- **Capturing in production is an action with impact**: it consumes CPU and disk and can fill a380 filesystem. Use a capture filter, a size limit and rotation, and give it a time limit381 from the start. A forgotten `tcpdump` on a server is a future incident.382- **Distinguish a fault from a compromise from the first minute.** If there is any sign of intrusion,383 the objective stops being "restore the service" and becomes "preserve the evidence": the procedure changes,384 do not reboot, do not delete, and escalate to385 `incident-response-forensics-standards` (order of volatility and chain of custody).386- **Port sweeping, scanning or injecting traffic into networks that are not yours —or without internal387 authorisation— is not diagnosis**: it is offensive activity and it is governed in388 `offensive-security-standards`. Inside your network, warn whoever is monitoring so that your test does not389 show up as an attack.390- **Privileges**: capturing requires `CAP_NET_RAW`/`CAP_NET_ADMIN` and eBPF requires more. Grant them391 **temporarily and by name**, not as a permanent configuration nor with a privileged container392 that stays there (`container-runtime-security-standards`). Withdraw them when the case closes.393- **Do not weaken controls to diagnose and then leave it like that**: opening a rule "to test" is the394 fastest route to a permanent `any/any` (`firewall-policy-standards`). If you open, you open with395 an automatic expiry.396397## 6. Operability: preparing before it fails398399- **Diagnosis is prepared, not improvised.** What must exist **before** the incident:400 an up-to-date inventory and SoT (`networking-standards`), a diagram of the traffic's real path,401 OOB access, telemetry with sufficient retention (`observability-standards`), flows, and **baselines**:402 normal RTT, normal throughput, normal error rate. **Without a baseline, "it is high" is an403 opinion.**404- **A synchronised clock on everything that logs or captures** (healthy NTP/chrony). Without it, correlating405 two captures or two logs is impossible, and that is precisely the method that solves the hard cases.406- **An available observation point**: SPAN/mirror, TAP, or at least a host with access to the segment and407 tools installed. If the first step of your diagnosis is "install `tcpdump` on the production408 server", you are already too late.409- **A minimum kit preinstalled or a diagnostic image ready**: iproute2, capture, `mtr`, `curl`,410 `nc`/`socat`, `ethtool`, `iperf3`, and a diagnostic container image for environments with no411 tools.412- **Runbooks by symptom, not by tool**: "it does not resolve", "it connects and hangs while transferring",413 "intermittent", "slow only under load", "it drops every N minutes", "1 in N fails", "it works414 from one host and not from another". Each one with the canonical sequence adapted and its escalation415 criterion.416- **A rescue window open when touching remote access, routes or the firewall** — an inherited and non-417 negotiable rule (`firewall-policy-standards`, `vpn-standards`): OOB console, a second path, or418 a timed rollback. Locking yourself out during a diagnosis is the most predictable and most419 avoidable incident there is.420- **When the diagnosis happens inside a declared incident**: command and communication belong421 to `incident-management-standards`. You give **hypotheses with a confidence level and an estimated time**,422 not premature certainties; and if the IC decides to mitigate before understanding, you mitigate — but **you capture423 the evidence first** and leave the pending diagnosis written down. Mitigating does not close the root cause.424- **Toil**: the same diagnosis repeated three times is an instrumentation or design failure, not425 bad luck. It becomes an alert, an automatic check or a structural fix426 (`sre-practice-standards`).427428## 7. Sustainability and prohibitions429430- **The postmortem feeds the method**: every hard case leaves either a new runbook, or a new431 metric, or a design change. If it leaves nothing, it will happen again.432- **Tool migration**: `net-tools` (`ifconfig`, `netstat`, `route`, `arp`) has been433 unmaintained for years and is absent by default in modern distributions; scripts and434 runbooks that still use it are migrated to **iproute2** with a date. It is not purism: it **hides** namespaces,435 policy routing and state that today determines the diagnosis.436- **Cadence**: review the version and CVEs of the capture and analysis tools (Wireshark/`tshark`,437 `libpcap`/`tcpdump`) along with the rest of the stack — **they are parsers that process hostile input by438 definition**, and 2026 brings a flood of LLM-assisted findings in that family. Analysing an439 untrusted capture with an unpatched Wireshark is exposing yourself.440- **Training with real cases**: keeping captures and timelines of solved cases as441 team training material is worth more than any course.442443**FORBIDDEN**444- ❌ **Rebooting as the first step.** It destroys the state you need (sockets, conntrack,445 counters, ARP cache, in-memory logs) and turns the problem into something irreproducible. It is the last resort,446 and with the evidence already collected.447- ❌ Changing **several things at once** and declaring victory when it works.448- ❌ Touching production blind: diagnostic changes with no hypothesis, no record and no rollback449 plan.450- ❌ **Blaming the network without evidence** — and also absolving it without evidence. Both are the same451 failing.452- ❌ **Capturing at only one end** in a hard case, or capturing without clock synchronisation.453- ❌ Concluding from loss at **intermediate hops** in `traceroute`/`mtr`.454- ❌ Using `ping` as proof that a service works, or as the only quality measurement.455- ❌ `ifconfig`, `netstat`, `route`, `arp` (`net-tools`) in diagnosis or in new runbooks.456- ❌ Reading `ip route` and assuming, instead of asking with `ip route get`.457- ❌ Measuring with `iperf3` for 10 s, a single flow, a single direction, and calling it a *baseline*.458- ❌ Reporting latency averages instead of percentiles.459- ❌ Analysing packet sizes or MTU without disabling the interface's offload.460- ❌ Diagnosing layer 3 without having looked at layer 1-2 errors and negotiation.461- ❌ Closing the case with "it fixed itself", "it was a network thing" or "we rebooted and it works now".462- ❌ Leaving the diagnosis's temporary changes in place (open rules, captures running,463 offload disabled, logs at debug, elevated privileges, a privileged debugging pod).464- ❌ 465466…(truncated)