Network diagnosis standards — the method, not the commands
Criteria verified as of August 2026. Re-verify on the web before committing to anything (§8).
1. Scope and triggers
Applies when something no longer works and you have to find out why: a method by layers and by
bisection, formulation of falsifiable hypotheses, canonical checking sequence
(resolution → outbound → return → acceptance by the application), choosing a tool by question and
which lie each one tells, a catalogue of the usual suspects with their characteristic symptom,
honest measurement of latency/loss/jitter, diagnosis in containers, Kubernetes, clouds and
overlays, and the discipline of recording and closing with a proven root cause.
Triggers: "it does not connect", "it is slow", "it is intermittent", "it drops every X minutes", "it works from here
but not from there", "it hangs while transferring", "it connects but does not load", ping, traceroute,
mtr --report, ss -tin/ss -s, ip route get, ip neigh, ip -s link, arping,
ethtool -S/ethtool -a, tshark, dumpcap, .pcap, curl -v --resolve, openssl s_client,
nc -zv, socat, iperf3 -R -P -u, /proc/net/nf_conntrack, /proc/net/sockstat,
net.ipv4.ip_local_port_range, somaxconn, TIME_WAIT, bpftrace, pwru, hubble observe,
kubectl debug, nicolaka/netshoot, tcp_retries2, "flow logs".
Guiding principle (inherited from networking-standards: the network is default-deny and documented
as code; what is not in the SoT does not exist): diagnosis ends in a proven root
cause, not in "it fixed itself". Every step answers a falsifiable hypothesis written before
typing, changes one variable, and leaves saved evidence. A problem that disappears with no
explanation is not solved: it is waiting.
Not applicable: see networking-standards (parent: the design —topology, addressing,
VLAN, routing/BGP, correct MTU/MSS values, proxies, overlays, OOB plane—; it says how the network
should be, you find out why today it is not; the structural fix goes back to it),
observability-standards (permanent telemetry and its alerts: metrics, logs, traces,
dashboards, SLI/SLO, sampling, retention and the design of what is instrumented before there is a
problem — all continuous monitoring is theirs; you are the reactive diagnosis once the alert
has fired. The exact line: if the question is "what is measured and at what threshold do we alert?", it is theirs;
if it is "this is broken now, why?", it is yours. And the evidence you need and that does not exist is a
finding for them: every blind diagnosis ends in an instrumentation requirement),
incident-management-standards (the process of the declared incident: severity, Incident
Commander, channel, communication, mitigate-before-diagnosing, postmortem — command and
communication are theirs; technical diagnosis inside the incident is yours. When the IC asks to
mitigate now, you mitigate and note down what you sacrifice in evidence),
firewall-policy-standards (the policy: when the diagnosis concludes "a rule is missing" or
"there is a rule too many", the fix, its approval and its negative test are theirs; proving
that the packet dies in the filter is yours),
dns-standards (the name service and its data: zone, TTL, DNSSEC, legitimate resolver,
delegation; you prove that resolution is the cause and who answers what, they decide
which is the right answer), vpn-standards (the tunnel: design, keys, correct MTU and keepalive,
concentrator redundancy; "the VPN connects but I cannot browse" is yours —it is
PMTU and it is proven with a capture at both ends—, likewise "it drops after 5 minutes" and "it resolves
badly inside the tunnel"), linux-administration-standards (resolvectl, systemd-resolved,
nsswitch.conf, systemd units and resolution from the host; dmesg and boot),
linux-storage-standards (when "the network is slow" turns out to be I/O: iostat, fio),
kubernetes-standards (CNI, NetworkPolicy, Service/Ingress, service mesh: their design),
container-runtime-security-standards (container isolation and the eBPF agent's privileges —
a privileged debugging pod is a security decision, not a shortcut),
detection-engineering-standards (analytics over network telemetry; a traffic capture
made to investigate a compromise, not a fault, belongs to them and to
incident-response-forensics-standards —chain of custody and preservation—),
sre-practice-standards (SLO, error budget, managing the toil of repeated diagnosis),
aws-standards/azure-standards/gcp-standards (flow logs, Reachability Analyzer/Network Watcher
and equivalents: the provider's tool and its configuration),
data-platform-standards (latency that turns out to be the database engine's, not the network's),
microservices-architecture-standards (timeouts, retries and circuit breakers as the application's
design; here you only prove that the failure comes from there),
appsec-standards (a failure that turns out to be the application's), onprem-standards (umbrella),
homelab-standards (your own lab), offensive-security-standards (scanning a network that is not
yours, or active sweeping without authorisation, is not diagnosis), and the three design and
proactive-operations skills the cause returns to once found: if the fault
is explained by the campus design or by a BGP policy, the structural fix belongs to
routing-switching-standards; if encapsulation MTU, asymmetric ECMP or EVPN appears, it belongs to
datacenter-fabric-standards; and if the answer to "what changed?" is a configuration
deployment, the rollback and the gate that should have prevented it belong to network-automation-standards.
Here the work ends when the cause is proven; the permanent fix lives there.
They also exist and are boundaries: ha-clustering-standards (split brain, fencing and the cluster
network: the expected behaviour of the cluster under partition is theirs; proving that there was a
network partition and why is yours) and podman-systemd-containers-standards (rootless networks with
netavark/pasta and their resolution: their configuration is theirs, the diagnosis of the packet
lost between namespaces belongs here).
2. Default decisions / Toolchain
Verify the latest version and maintenance status on the web before committing to anything (§8).
Dates obtained from api.github.com and the project's site, not from summarised HTML pages.
| Question you are answering |
Default tool |
Alternative |
Banned |
| What is configured on this machine? |
ip addr/ip link/ip route/ip neigh/ip -s link (iproute2) |
nmcli/networkctl depending on the manager |
ifconfig, route, arp, netstat (net-tools: no release since 2001, unmaintained; they hide namespaces, routing policies, multiple tables and modern socket state) |
| Which route would this packet take? |
ip route get <dst> from <src> |
ip rule show for policy routing |
Reading the main table and assuming |
| Which sockets and in which state? |
ss -tanp, ss -tin (RTT, cwnd, retransmissions), ss -s |
/proc/net/sockstat |
netstat -an |
| Does the packet arrive and come back? |
tshark/dumpcap capturing at BOTH ends at the same time |
A capture at each intermediate hop if the path has several |
Capturing at only one end and inferring |
| Analysing the capture? |
Wireshark 4.6.x (GUI) to analyse; tshark to filter and automate |
capinfos, editcap, mergecap |
Eyeballing 2 GB of pcap instead of filtering |
| Where is it lost and how much? |
mtr --report --report-cycles 100 -w (v0.96; repo with activity in Jun 2026) |
traceroute -T -p 443 when ICMP/UDP is filtered |
ping as the only quality measurement |
| Is layer 1-2 healthy? |
ethtool <if> (negotiation), ethtool -S (error/discard counters), ethtool -a (pause) |
ip -s link for aggregate errors |
Diagnosing layer 3 without having looked at interface errors |
| State of filtering and NAT? |
nft list ruleset with counter; /proc/net/nf_conntrack and nf_conntrack_count vs nf_conntrack_max (conntrack-tools 1.4.9) |
nft monitor trace to follow a packet through the chains |
Assuming "the firewall is open" because somebody said so |
| Does the application accept? |
curl -v with --resolve (isolates DNS from connectivity), openssl s_client -connect -servername |
nc -zv for a raw port; socat for relays and protocol tests |
telnet host port as a TLS test |
| How much bandwidth is there really? |
iperf3 3.21 (09 Apr 2026) with -P (parallel), -R (reverse direction) and long runs |
iperf3 -u -b for UDP with a fixed rate, measuring loss and jitter |
Measuring 10 s in one direction and calling it a baseline |
| Where does the packet die inside the kernel? |
pwru v1.0.12 (13 Jul 2026, kernel ≥5.3; --output-skb ≥5.9) |
bpftrace 0.26.1 (02 Jun 2026) and bcc 0.37.0 tools (02 Jul 2026) for bespoke cases |
Guessing between nftables, routing and the driver |
| And in Kubernetes? |
hubble observe --verdict DROPPED (Cilium 1.20.0, 29 Jul 2026); kubectl debug --image=nicolaka/netshoot with --target |
kubectl debug node/<n> for the node's netns; Retina where the CNI is not Cilium |
kubectl exec into a distroless container and giving up |
| And in the cloud? |
The provider's flow logs + its reachability analyser |
A capture on the instance if the provider does not offer a managed one |
Concluding "it is the cloud" without looking at the flow logs |
On eBPF (pwru, bpftrace, bcc, Hubble): recommended, but not as the first step.
They are the answer to "the packet enters the machine and does not come out, and no classic tool tells me
where" — a real and frequent problem on hosts with containers, and where classic capture does not
reach because the drop happens between capture points. Requirements and cost to accept
beforehand: a modern kernel with BTF and CONFIG_KPROBES/CONFIG_BPF, high privileges on the host
(a security decision, see container-runtime-security-standards), and non-zero overhead in
production. Criteria: layer 1-2, ss, capture at both ends and firewall counters
first; eBPF when those four do not close the case.
3. The method
3.1 Before typing anything: four questions
- What changed? — The cause is the last change until proven otherwise. The
diagnosis starts in change control: deployments, firewall or routing changes,
patching, certificate renewal, change of provider, firmware update, the expiry of
something. If nobody knows what changed, that is the first finding (and a governance problem, not a
network one).
- What exactly is "it does not work"? — Reproduce the symptom precisely: which source, which
destination, which port, which protocol, which client, at what time, with which literal error message.
"The network is bad" is not a symptom; "from host A,
curl to B:443 hangs after the
TLS handshake, in 30% of attempts, since 09:14" is.
- What is the scope? — One user or all of them? One destination or all of them? One protocol or all of them?
One VLAN, one node, one availability zone? Scope discards more hypotheses in 30
seconds than an hour of captures: a failure that affects a single client is not in the
server, and one that affects every destination is not in the destination.
- Did it ever work? — "It never worked" is a configuration or design problem;
"it worked and stopped working" is a change or resource exhaustion problem. They are
two different investigations and confusing them costs hours.
3.2 Bisection: divide the path, do not walk it
- You do not walk the path hop by hop. You split it in half: pick an intermediate point
with visibility (a router, a load balancer, a node) and determine whether the problem is before or
after. Repeat. With 8 hops, bisection is 3 tests; walking it linearly, 8 — and with more
opportunities to get it wrong.
- Bisection also applies to non-spatial dimensions: two clients (one that fails,
one that does not) → what differs?; two destinations; two protocols; two moments; IPv4 vs. IPv6. The cut
is by variable, not just by place.
- One variable at a time. Changing two things and having it work is not a diagnosis: it is a
coincidence you will have to deal with again. If pressure forces changing several at
once to mitigate, it is noted and reverted one by one afterwards to identify which one it was.
- A falsifiable hypothesis before typing: write down "if the cause is X, then doing Y I will see Z;
if I see W, X is ruled out". A test that cannot refute your hypothesis is not a test, it is
a ceremony. This is what separates diagnosis from "firing off commands".
- Confirmation bias is the main enemy: as soon as you have a suspect, every
test seems to confirm it. Antidote: define in advance what result would make you abandon that
hypothesis, and look for it explicitly.
3.3 The canonical sequence
Four questions, in order. Each one has a binary answer and eliminates half the universe.
1) Does the name resolve — and to the right thing?
- Isolate DNS from the rest from the first minute:
curl -v --resolve host:443:<ip> directly
compares "does not resolve" against "does not connect". If it works with --resolve, the problem is
resolution and not the network.
- Ask each resolver separately and compare them, and compare that with what the process
actually uses: the system may have a local stub, a cache, a
search that completes the name,
a forgotten /etc/hosts or a library that does not even go through the OS resolver.
Resolution "from the shell" and "from the application" are not the same.
- Negative caching and TTL explain the classic "it works for me and not for you" and "it took an hour to
fix itself". Zone, TTL and resolver design, in
dns-standards.
2) Does the packet reach the destination?
- Capture at the destination, filtering by source and port. If it does not appear, the packet dies on the
way: firewall, route, NAT, VLAN, layer 1-2.
- Check at the source that it leaves and through which interface:
ip route get <dst> from <src> answers
the real question (it includes policy routing and the effective table), not the main table.
- If it leaves and does not arrive, bisect the path and check the counters of the filtering rules at each
hop: a
counter incrementing on the drop rule is a proof, not a suspicion.
3) Does the response come back? — the forgotten half of the problem
- The response gets lost as much as the outbound, and almost nobody looks at it. If at the destination you see the
SYN
and the SYN/ACK leaves, but the source does not receive it, the problem is the return path and everything
you were looking at was irrelevant.
- Typical causes of asymmetric failure: the return takes a different path (multihoming, specific
routes, partial VPN, VRF); a stateful firewall that only sees one direction and drops the other
for lack of state —an unmistakable symptom: it works for a while and then cuts off, or it fails
intermittently and non-reproducibly—; NAT in one direction and not the other; uRPF dropping because of an
invalid reverse path.
- Operational rule: in every hard case, simultaneous capture at both ends with synchronised
clocks, and compare. This solves 90% of what looks impossible: it tells you in which half
of the path the packet disappears and with that the investigation is reduced to half the network. If
the path also has an intermediary (proxy, load balancer, NAT), capture on both of its sides.
- Correcting asymmetry is a design (
networking-standards) or policy
(firewall-policy-standards) matter: it is never fixed by adding a broad accept.
4) Does the application accept it?
- The packet arrives, the
SYN/ACK comes back, and the failure persists: check whether the process listens (ss -tanp), on which address (0.0.0.0 vs 127.0.0.1 vs :: — a service on loopback is
unreachable from outside and looks like a firewall), and whether the accept queue is full
(ss -lt shows Send-Q as the backlog and Recv-Q as pending: if it saturates, the kernel
drops SYNs and the client sees timeouts with the server "alive").
- Failures that look like network ones and are the application's or its edge's: an immediate
RST (nobody listens
or the proxy rejects), a TLS handshake failing because of SNI, certificate, version or chain
(openssl s_client -servername), redirects, authentication, or the application's own timeouts
shorter than its retry.
RST vs. timeout is the trade's most informative distinction: RST = something answered and
rejected (live host, closed port, a proxy or firewall that rejects); silence = something dropped
(a firewall that drops, a missing route, a downed host). They are not the same problem and they are not
diagnosed the same way.
3.4 Which lie each tool tells
No tool lies out of malice: they all answer a narrower question than the one you think you
are asking.
ping / ICMP — it measures that the destination answers ICMP, not that the service works. ICMP is usually
deprioritised in the control plane of routers and switches, or filtered by policy. A
ping with 200 ms and loss towards the network device may be perfectly normal while the
data traffic runs impeccably; and a perfect ping says nothing about port 443. Never use
ping as the only quality measurement or as a proof of service.
traceroute/mtr — the trade's most widespread lie: loss at intermediate hops
means nothing. Routers generate TTL exceeded responses in their CPU and rate-limit them;
seeing 40% loss at hop 5 and 0% at the destination means that router prioritises its
work, not that there is a problem. Only two things count: (a) loss at the last hop (the
destination), and (b) loss that persists from one hop to the end. Besides, ECMP makes
each probe take a different path (use fixed-flow mode if your tool supports it),
the return path is invisible, and MPLS can hide hops entirely. When ICMP/UDP is
filtered, traceroute -T over the application's real port is what reflects the path that
matters.
ss — it tells you the socket's local state. ESTABLISHED at one end does not imply the other
is still there: a connection whose peer disappeared without a FIN keeps showing as established until a
keepalive or a write discovers it. ss -tin does give gold: RTT, cwnd, retransmissions — which
distinguish "the network is losing" from "the application is slow".
tcpdump/tshark — they capture where you are and after the kernel has decided
some things and before others: a packet dropped by the filter may or may not appear in the
capture depending on the hook point, and offload (GRO/GSO/TSO/LRO) shows giant "packets" that
do not exist on the wire (disable it if you are going to analyse sizes or MTU). With sampling or a badly
written filter, your "it does not show up" may be yours, not the network's. And an unfiltered capture on a loaded
link loses itself: use a capture filter (BPF) for what you want, and a display
filter to analyse.
ip route — it shows tables; ip route get shows the decision. With policy rules,
VRF or several tables, reading the main one and assuming is a classic mistake.
ip neigh/ARP — a STALE or FAILED entry tells you more than a REACHABLE one; and a correct entry
with the wrong MAC (duplicate IP, unexpected proxy ARP) is a silent failure
that no layer 3 test gives away. arping from the same segment reveals duplicate IPs
(two answers, two MACs) in one second, which is what no other tool does.
ethtool — the only one that sees layer 1-2: negotiation (duplex/speed; a badly negotiated
half duplex shows up as "slow and intermittent" under load), CRC errors, drops due to
lack of buffer, and per-queue counters. Cumulative counters: what matters is the
delta during the failure, not the total since boot.
iperf3 — it measures what you ask it to, and by default that is not what you think. Classic mistakes: measuring
10 s (all slow start), a single flow (limited by RTT and window, not by the link), only in
one direction (-R measures the other, which may be the broken one), in UDP without setting the rate (-b) or setting it
above capacity and calling your own saturation "network loss", with iperf3 itself
as the CPU bottleneck, or measuring against a shared public server.
iperf3 measures one path between two points at one instant; it does not measure "the network".
curl -v — without --resolve it mixes DNS, connection, TLS and HTTP into a single result; with
--resolve it separates the first. Its per-phase timings (-w) are a diagnosis in themselves: if the
time goes on the connect, it is the network; on the appconnect, it is TLS; on the starttransfer, it is
the application.
- The application logs — they say "connection timeout" for half a dozen mutually incompatible
causes. Useful for the exact time and the scope; useless as a diagnosis.
3.5 The usual suspects and their characteristic symptom
A recognition table. The symptom is what makes you suspect; the proof is what demonstrates it.
| Suspect |
Characteristic symptom |
Proof that demonstrates it |
| MTU / PMTU black hole |
The connection opens and hangs while transferring: SSH connects but scp stalls; the web page loads the HTML and not the images; "the VPN connects but I cannot browse". Typically after a tunnel or a change of encapsulation |
ping with a large packet and the DF bit, increasing until you find the cut-off; a capture showing retransmissions of the same large segment with no ACK; absence of ICMP fragmentation needed coming back |
| Filtered ICMP that breaks PMTUD |
Identical to the previous one, and it never fixes itself |
The intermediary blocking ICMP type 3 code 4 (or ICMPv6 packet-too-big): it is seen by its absence in the capture on the side that should receive it |
| DNS |
Intermittency that looks like the network's; "sometimes it takes exactly 5 seconds" (resolver timeout); it works by IP and not by name |
curl --resolve versus a normal curl; querying each resolver separately |
| Ephemeral port / NAT exhaustion |
Failures that increase with load, on the side that initiates many connections (proxy, NAT, API client) |
Socket count against net.ipv4.ip_local_port_range; on the NAT, active sessions against its capacity |
| Full conntrack table |
Silent drops under load, with a line in dmesg nobody looks at |
nf_conntrack_count against nf_conntrack_max; dmesg with table full |
| Different return path |
"It works for a while and then cuts off"; intermittent and irreproducible; it works in one direction |
Simultaneous capture at both ends: you see the outbound and the response that does not arrive |
TIME_WAIT / exhausted backlog |
Connection timeouts with the server alive and CPU low; it worsens at peaks |
ss -s (count by state), ss -lt with Recv-Q growing, the dropped-SYN counter |
| Keepalive vs. an intermediary's idle timeout |
"The session drops exactly after N minutes" of inactivity. A firewall, NAT, load balancer or cloud with an idle timeout shorter than the client's keepalive |
Reproduce with an idle session and a stopwatch; compare the intermediary's timeout with the TCP/application keepalive. It is fixed by lowering the keepalive, not by raising everybody's timeout |
| Duplicate IP |
Unexplained intermittency that changes with the ARP cache lifetime; "sometimes it lands on a different server" |
arping from the segment: two answers with different MACs |
| Duplicate MAC / MAC flapping |
Massive loss on a segment; the switch logs learning the same MAC on different ports |
Switch log; address table |
| Layer 2 loop / broadcast storm |
The whole VLAN goes down or crawls; switch CPU at 100%; it starts right after plugging something in |
Broadcast/multicast counters per port shooting up; STP with constant topology changes. It is an emergency: the port is isolated first |
| Aged or incomplete ARP/ND |
A host unreachable from its own segment while everything else is fine |
ip neigh in FAILED/INCOMPLETE state |
| IPv6 enabled and failing while IPv4 works |
"It is slow" with delays of exact seconds at the start; it works with -4; it fails only on some clients |
Compare curl -4 and curl -6; route and ND on IPv6. Usual cause: AAAA published without real IPv6 connectivity, or an IPv6 firewall without parity with IPv4 |
| Happy Eyeballs masking the failure |
The IPv6 failure is almost unnoticeable (the client retries over IPv4 after a short delay), so nobody fixes it and the initial latency is worse for everyone |
A capture showing the abandoned IPv6 attempt. RFC 8305 is the current specification; v3 is still a draft (§8) |
| A proxy or TLS inspection in the middle |
An unexpected certificate, a forced TLS version, rewritten ALPN, HTTP/3 that does not work, mTLS that fails, a Server different from the expected one |
openssl s_client -servername and compare the certificate issuer with the expected one |
| Load balancer with one bad backend |
A constant fraction of requests fails (1 in N) |
Repeat the test N+ times recording which backend each one goes to |
| Expired certificate or incomplete chain |
It fails at an exact time, for everybody at once, with nobody having touched anything |
openssl s_client showing the chain and the dates |
| Layer 1 |
Growing CRC errors, slow only under load, badly negotiated duplex, degraded optics |
ethtool -S (delta during the failure), ethtool (negotiation), optical power on the device |
| Saturation / bufferbloat |
Latency that shoots up only when there is traffic; the ping goes from 10 ms to 300 ms when a download starts |
Latency under load versus at rest; link utilisation by percentiles |
3.6 Latency, loss and jitter: measuring them properly
- Percentiles, never averages. The average hides exactly what breaks the experience. Look at
p50, p95, p99 and the maximum; and compare with the at-rest state, not with an absolute number. A p50 of 20 ms
with a p99 of 2 s is a broken system that averages well.
- Distinguish the three causes of "it is slow", because they have opposite fixes:
- Latency (RTT): it limits a TCP flow's throughput through the window. If the RTT is high, more
bandwidth fixes nothing; more parallel flows or a larger window do.
- Loss: it sinks TCP throughput disproportionately (1% loss can cost
most of the performance on a high-RTT link). It is seen in retransmissions (
ss -tin),
not in the ping.
- Jitter: irrelevant for a download, lethal for voice and video. It is measured with UDP at a fixed
rate, not with TCP.
- The measurement must reproduce the real case: same source-destination pair, same protocol, same
transfer size, same time of day. A test at rest does not reproduce a saturation problem,
and a 10-second test does not reproduce a 20-minute problem.
- If "it is slow" turns out to be I/O, CPU or the database, say so and close the case there: the network is
blamed by default, and proving that it is not the network is as valid a result as any
other (with evidence, not with a denial).
3.7 Modern environments: containers, Kubernetes, clouds and overlays
- The packet crosses several network namespaces. Before capturing, decide which one you are
capturing in: inside the container, on the host side of the
veth, on the bridge, in the node's netns,
on the physical interface, on the overlay interface. "It does not show up in the capture" almost always
means "you captured in the wrong namespace".
- Kubernetes: a checking order by the cluster's own layers — does the
Service name
resolve? → does the Service have Endpoints/EndpointSlice (a Service with no endpoints from a badly
written selector is the most common failure and does not look like a network one)? → is there a NetworkPolicy that
denies it (the drop happens in the datapath, before reaching the pod)? → does the readinessProbe
take the pod out of rotation? → do the CNI or kube-proxy have the rule? → does the node route?
hubble observe --verdict DROPPED gives the verdict and the reason for the drop in one step; beware of
monitoring aggregation, which can hide individual events from you.
- Distroless containers with no tools:
kubectl debug with an ephemeral container
(GA since Kubernetes 1.25) and a diagnostic image (nicolaka/netshoot), sharing the
pod's namespace, and kubectl debug node/<node> for the node's netns. Two operational warnings:
an ephemeral container cannot be removed until the pod is deleted and has no resource
limits; and to extract a capture, stream it over standard output instead of writing the
file inside (copying from an ephemeral container does not work).
- Clouds: the VPC/VNet flow logs are the first stop —they say whether the packet was
accepted or rejected and by which rule— followed by the provider's reachability analyser. Their
limits, which must be known: aggregation into windows (you do not see the packet, you see the flow), possible
sampling, a delay of minutes, and no payload. The specific configuration, in
aws-standards/azure-standards/gcp-standards.
- Overlays and tunnels (VXLAN, GENEVE, IPsec, WireGuard): capture inside the tunnel and
outside —they are two different questions: "does the traffic enter the tunnel?" and "does the tunnel reach the other
side?". And as soon as there is encapsulation, MTU is the first suspect, always.
- Service mesh / sidecar: the proxy may terminate TLS, rewrite headers, apply its
own timeouts and retries and return errors that look like the application's. Its access logs
are the source, not the capture.
4. Quality of the diagnosis (gates)
A diagnosis is accepted when it meets all of this. It is not bureaucracy: it is what prevents
the same incident from coming back in three weeks.
- Reproducibility: there is an exact command or procedure that produces the symptom at will
(or, if it is intermittent, a documented condition that triggers it and a measured rate). Without
reproduction, the fix cannot be validated.
- Hypotheses recorded with their result: what was tried, what was expected, what was seen and which
hypothesis was ruled out. A record of exclusions is worth as much as the finding, and it prevents the
next shift from repeating the same tests.
- Evidence saved, not described: captures (
.pcap) from both ends with a timestamp,
complete command outputs, counters before/after, screenshots of graphs with their
time range. "We saw packets being lost" is not evidence. Save the evidence before
mitigating: mitigation destroys the state that proves it.
- Root cause proven, not inferred — the central gate. "Proven" means you can
explain the complete mechanism from the change or the condition to the symptom, and that you can
reproduce the failure by activating the cause and make it disappear by deactivating it. Temporal correlation
is not causation: "we rebooted and it was fixed" is a data point, not a conclusion.
- Fix validated by the test that was failing, and additionally with the corresponding
negative test (what was supposed to stay blocked is still blocked). And validated from the real
affected source, not from the engineer's bastion.
- Absence of side effects verified: what was touched did not break anything else. Temporary
diagnostic changes (open rules, offload disabled,
tcpdump running, raised timeouts,
logs at debug) are explicitly reverted and the reversion is verified.
- Derived findings with an owner: the configuration drift, the missing rule, the metric
that did not exist, the runbook that was useless, the alert that did not fire. Each one is opened as work
with an owner and a date — towards
observability-standards, firewall-policy-standards,
networking-standards or whoever is relevant.
- If the case is closed without a root cause (it happens, and it is legitimate), it is closed saying so: what was
ruled out, what instrumentation is missing to diagnose it next time, and what trigger is left
armed to capture evidence when it comes back. That is a result; "it fixed itself" is not.
5. Security of the diagnosis
- A traffic capture contains personal data, credentials and business content. It is not
an innocuous technical file. Treat it as classified data: controlled storage, restricted
access, minimum retention and deletion when the case closes. If you are going to share it, anonymise it or
trim to headers (
-s to limit the capture) and remove the payload. Legal basis and
minimisation, in privacy-engineering-standards.
- Capturing in production is an action with impact: it consumes CPU and disk and can fill a
filesystem. Use a capture filter, a size limit and rotation, and give it a time limit
from the start. A forgotten
tcpdump on a server is a future incident.
- Distinguish a fault from a compromise from the first minute. If there is any sign of intrusion,
the objective stops being "restore the service" and becomes "preserve the evidence": the procedure changes,
do not reboot, do not delete, and escalate to
incident-response-forensics-standards (order of volatility and chain of custody).
- Port sweeping, scanning or injecting traffic into networks that are not yours —or without internal
authorisation— is not diagnosis: it is offensive activity and it is governed in
offensive-security-standards. Inside your network, warn whoever is monitoring so that your test does not
show up as an attack.
- Privileges: capturing requires
CAP_NET_RAW/CAP_NET_ADMIN and eBPF requires more. Grant them
temporarily and by name, not as a permanent configuration nor with a privileged container
that stays there (container-runtime-security-standards). Withdraw them when the case closes.
- Do not weaken controls to diagnose and then leave it like that: opening a rule "to test" is the
fastest route to a permanent
any/any (firewall-policy-standards). If you open, you open with
an automatic expiry.
6. Operability: preparing before it fails
- Diagnosis is prepared, not improvised. What must exist before the incident:
an up-to-date inventory and SoT (
networking-standards), a diagram of the traffic's real path,
OOB access, telemetry with sufficient retention (observability-standards), flows, and baselines:
normal RTT, normal throughput, normal error rate. Without a baseline, "it is high" is an
opinion.
- A synchronised clock on everything that logs or captures (healthy NTP/chrony). Without it, correlating
two captures or two logs is impossible, and that is precisely the method that solves the hard cases.
- An available observation point: SPAN/mirror, TAP, or at least a host with access to the segment and
tools installed. If the first step of your diagnosis is "install
tcpdump on the production
server", you are already too late.
- A minimum kit preinstalled or a diagnostic image ready: iproute2, capture,
mtr, curl,
nc/socat, ethtool, iperf3, and a diagnostic container image for environments with no
tools.
- Runbooks by symptom, not by tool: "it does not resolve", "it connects and hangs while transferring",
"intermittent", "slow only under load", "it drops every N minutes", "1 in N fails", "it works
from one host and not from another". Each one with the canonical sequence adapted and its escalation
criterion.
- A rescue window open when touching remote access, routes or the firewall — an inherited and non-
negotiable rule (
firewall-policy-standards, vpn-standards): OOB console, a second path, or
a timed rollback. Locking yourself out during a diagnosis is the most predictable and most
avoidable incident there is.
- When the diagnosis happens inside a declared incident: command and communication belong
to
incident-management-standards. You give hypotheses with a confidence level and an estimated time,
not premature certainties; and if the IC decides to mitigate before understanding, you mitigate — but you capture
the evidence first and leave the pending diagnosis written down. Mitigating does not close the root cause.
- Toil: the same diagnosis repeated three times is an instrumentation or design failure, not
bad luck. It becomes an alert, an automatic check or a structural fix
(
sre-practice-standards).
7. Sustainability and prohibitions
- The postmortem feeds the method: every hard case leaves either a new runbook, or a new
metric, or a design change. If it leaves nothing, it will happen again.
- Tool migration:
net-tools (ifconfig, netstat, route, arp) has been
unmaintained for years and is absent by default in modern distributions; scripts and
runbooks that still use it are migrated to iproute2 with a date. It is not purism: it hides namespaces,
policy routing and state that today determines the diagnosis.
- Cadence: review the version and CVEs of the capture and analysis tools (Wireshark/
tshark,
libpcap/tcpdump) along with the rest of the stack — they are parsers that process hostile input by
definition, and 2026 brings a flood of LLM-assisted findings in that family. Analysing an
untrusted capture with an unpatched Wireshark is exposing yourself.
- Training with real cases: keeping captures and timelines of solved cases as
team training material is worth more than any course.
FORBIDDEN
- ❌ Rebooting as the first step. It destroys the state you need (sockets, conntrack,
counters, ARP cache, in-memory logs) and turns the problem into something irreproducible. It is the last resort,
and with the evidence already collected.
- ❌ Changing several things at once and declaring victory when it works.
- ❌ Touching production blind: diagnostic changes with no hypothesis, no record and no rollback
plan.
- ❌ Blaming the network without evidence — and also absolving it without evidence. Both are the same
failing.
- ❌ Capturing at only one end in a hard case, or capturing without clock synchronisation.
- ❌ Concluding from loss at intermediate hops in
traceroute/mtr.
- ❌ Using
ping as proof that a service works, or as the only quality measurement.
- ❌
ifconfig, netstat, route, arp (net-tools) in diagnosis or in new runbooks.
- ❌ Reading
ip route and assuming, instead of asking with ip route get.
- ❌ Measuring with
iperf3 for 10 s, a single flow, a single direction, and calling it a baseline.
- ❌ Reporting latency averages instead of percentiles.
- ❌ Analysing packet sizes or MTU without disabling the interface's offload.
- ❌ Diagnosing layer 3 without having looked at layer 1-2 errors and negotiation.
- ❌ Closing the case with "it fixed itself", "it was a network thing" or "we rebooted and it works now".
- ❌ Leaving the diagnosis's temporary changes in place (open rules, captures running,
offload disabled, logs at debug, elevated privileges, a privileged debugging pod).
- ❌
…(truncated)
1---2name: network-troubleshooting-standards3description: Reactive network fault diagnosis method — bisecting the path, forming a falsifiable hypothesis and proving root cause. Use when something "doesn't connect", is intermittent, slow or hangs mid-transfer and you are reaching for ping, traceroute, mtr --report, ss -tin, ip route get, ip neigh, arping, ethtool -S, tshark or dumpcap, capturing pcap simultaneously at both endpoints, curl -v --resolve, openssl s_client, nc -zv, socat, iperf3 -R with -P and -u, /proc/net/nf_conntrack and ephemeral port exhaustion, PMTU blackhole with ICMP fragmentation-needed filtered, TIME_WAIT and listen-backlog overflow, keepalive versus middlebox idle timeout, IP address conflict or MAC flapping, broadcast storms and layer-2 loops, ARP/ND caches, IPv6 preferred over working IPv4 and Happy Eyeballs, TLS-inspecting proxies, tail-latency percentiles versus averages, bpftrace, bcc tools, pwru, hubble observe, kubectl debug --image=nicolaka/netshoot across pod and node netns, or VPC flow logs — and when you must record what was tried, what it proved and why the fix was the fix.4metadata:5 source: https://skillmd.com/skills/serialexperimentslainnnn/network-troubleshooting-standards6---78# Network diagnosis standards — the method, not the commands910Criteria verified as of **August 2026**. Re-verify on the web before committing to anything (§8).1112## 1. Scope and triggers1314Applies when **something no longer works and you have to find out why**: a method by layers and by15bisection, formulation of falsifiable hypotheses, canonical checking sequence16(resolution → outbound → return → acceptance by the application), choosing a tool by question and17**which lie each one tells**, a catalogue of the usual suspects with their characteristic symptom,18honest measurement of latency/loss/jitter, diagnosis in containers, Kubernetes, clouds and19overlays, and the discipline of recording and closing with a **proven root cause**.2021Triggers: "it does not connect", "it is slow", "it is intermittent", "it drops every X minutes", "it works from here22but not from there", "it hangs while transferring", "it connects but does not load", `ping`, `traceroute`,23`mtr --report`, `ss -tin`/`ss -s`, `ip route get`, `ip neigh`, `ip -s link`, `arping`,24`ethtool -S`/`ethtool -a`, `tshark`, `dumpcap`, `.pcap`, `curl -v --resolve`, `openssl s_client`,25`nc -zv`, `socat`, `iperf3 -R -P -u`, `/proc/net/nf_conntrack`, `/proc/net/sockstat`,26`net.ipv4.ip_local_port_range`, `somaxconn`, `TIME_WAIT`, `bpftrace`, `pwru`, `hubble observe`,27`kubectl debug`, `nicolaka/netshoot`, `tcp_retries2`, "flow logs".2829**Guiding principle** (inherited from `networking-standards`: *the network is default-deny and documented30as code; what is not in the SoT does not exist*): **diagnosis ends in a proven root31cause, not in "it fixed itself"**. Every step answers a falsifiable hypothesis written before32typing, changes **one** variable, and leaves saved evidence. A problem that disappears with no33explanation is not solved: it is waiting.3435**Not applicable**: see `networking-standards` (**parent**: the **design** —topology, addressing,36VLAN, routing/BGP, correct MTU/MSS values, proxies, overlays, OOB plane—; **it says how the network37should be, you find out why today it is not**; the structural fix goes back to it),38`observability-standards` (**permanent telemetry and its alerts**: metrics, logs, traces,39dashboards, SLI/SLO, sampling, retention and the design of what is instrumented **before** there is a40problem — **all continuous monitoring is theirs; you are the reactive diagnosis once the alert41has fired**. The exact line: if the question is "what is measured and at what threshold do we alert?", it is theirs;42if it is "this is broken now, why?", it is yours. And the evidence you need and that does not exist is a43finding **for** them: every blind diagnosis ends in an instrumentation requirement),44`incident-management-standards` (**the process of the declared incident**: severity, Incident45Commander, channel, communication, mitigate-before-diagnosing, postmortem — **command and46communication are theirs; technical diagnosis inside the incident is yours**. When the IC asks to47mitigate now, you mitigate and **note down** what you sacrifice in evidence),48`firewall-policy-standards` (**the policy**: when the diagnosis concludes "a rule is missing" or49"there is a rule too many", the fix, its approval and its negative test are theirs; **proving50that the packet dies in the filter is yours**),51`dns-standards` (**the name service and its data**: zone, TTL, DNSSEC, legitimate resolver,52delegation; **you prove that resolution is the cause and who answers what**, they decide53which is the right answer), `vpn-standards` (**the tunnel**: design, keys, correct MTU and keepalive,54concentrator redundancy; **"the VPN connects but I cannot browse" is yours** —it is55PMTU and it is proven with a capture at both ends—, likewise "it drops after 5 minutes" and "it resolves56badly inside the tunnel"), `linux-administration-standards` (`resolvectl`, `systemd-resolved`,57`nsswitch.conf`, systemd units and resolution **from the host**; `dmesg` and boot),58`linux-storage-standards` (when "the network is slow" turns out to be I/O: `iostat`, `fio`),59`kubernetes-standards` (CNI, `NetworkPolicy`, `Service`/`Ingress`, service mesh: their **design**),60`container-runtime-security-standards` (container isolation and the eBPF agent's privileges —61a privileged debugging pod is a security decision, not a shortcut),62`detection-engineering-standards` (analytics over network telemetry; **a traffic capture63made to investigate a compromise, not a fault, belongs to them and to**64`incident-response-forensics-standards` —chain of custody and preservation—),65`sre-practice-standards` (SLO, error budget, managing the *toil* of repeated diagnosis),66`aws-standards`/`azure-standards`/`gcp-standards` (flow logs, Reachability Analyzer/Network Watcher67and equivalents: the provider's tool and its configuration),68`data-platform-standards` (latency that turns out to be the database engine's, not the network's),69`microservices-architecture-standards` (timeouts, retries and circuit breakers as the application's70**design**; here you only prove that the failure comes from there),71`appsec-standards` (a failure that turns out to be the application's), `onprem-standards` (umbrella),72`homelab-standards` (your own lab), `offensive-security-standards` (**scanning a network that is not73yours, or active sweeping without authorisation, is not diagnosis**), and the three design and74proactive-operations skills the cause **returns to once found**: if the fault75is explained by the campus design or by a BGP policy, the structural fix belongs to76`routing-switching-standards`; if encapsulation MTU, asymmetric ECMP or EVPN appears, it belongs to77`datacenter-fabric-standards`; and **if the answer to "what changed?" is a configuration78deployment, the rollback and the gate that should have prevented it belong to `network-automation-standards`**.79Here the work ends when the cause is proven; the permanent fix lives there.8081They also exist and are boundaries: `ha-clustering-standards` (*split brain*, fencing and the cluster82network: **the expected behaviour of the cluster under partition is theirs; proving that there was a83network partition and why is yours**) and `podman-systemd-containers-standards` (rootless networks with84`netavark`/`pasta` and their resolution: their **configuration** is theirs, the diagnosis of the packet85lost between namespaces belongs here).8687## 2. Default decisions / Toolchain8889> Verify the latest version and maintenance status on the web before committing to anything (§8).90> Dates obtained from `api.github.com` and the project's site, not from summarised HTML pages.9192| Question you are answering | Default tool | Alternative | Banned |93|---|---|---|---|94| What is configured on this machine? | `ip addr`/`ip link`/`ip route`/`ip neigh`/`ip -s link` (**iproute2**) | `nmcli`/`networkctl` depending on the manager | `ifconfig`, `route`, `arp`, `netstat` (**net-tools**: no release since 2001, unmaintained; they **hide** namespaces, routing policies, multiple tables and modern socket state) |95| Which route would **this** packet take? | `ip route get <dst> from <src>` | `ip rule show` for policy routing | Reading the main table and assuming |96| Which sockets and in which state? | `ss -tanp`, `ss -tin` (RTT, cwnd, retransmissions), `ss -s` | `/proc/net/sockstat` | `netstat -an` |97| Does the packet arrive and come back? | **`tshark`/`dumpcap` capturing at BOTH ends at the same time** | A capture at each intermediate hop if the path has several | Capturing at only one end and inferring |98| Analysing the capture? | **Wireshark 4.6.x** (GUI) to analyse; `tshark` to filter and automate | `capinfos`, `editcap`, `mergecap` | Eyeballing 2 GB of pcap instead of filtering |99| Where is it lost and how much? | `mtr --report --report-cycles 100 -w` (**v0.96**; repo with activity in Jun 2026) | `traceroute -T -p 443` when ICMP/UDP is filtered | `ping` as the only quality measurement |100| Is layer 1-2 healthy? | `ethtool <if>` (negotiation), `ethtool -S` (error/discard counters), `ethtool -a` (pause) | `ip -s link` for aggregate errors | Diagnosing layer 3 without having looked at interface errors |101| State of filtering and NAT? | `nft list ruleset` with `counter`; `/proc/net/nf_conntrack` and `nf_conntrack_count` vs `nf_conntrack_max` (**conntrack-tools 1.4.9**) | `nft monitor trace` to follow a packet through the chains | Assuming "the firewall is open" because somebody said so |102| Does the application accept? | `curl -v` with **`--resolve`** (isolates DNS from connectivity), `openssl s_client -connect -servername` | `nc -zv` for a raw port; `socat` for relays and protocol tests | `telnet host port` as a TLS test |103| How much bandwidth is there really? | **`iperf3 3.21`** (09 Apr 2026) with `-P` (parallel), `-R` (reverse direction) and long runs | `iperf3 -u -b` for UDP with a fixed rate, measuring loss and jitter | Measuring 10 s in one direction and calling it a *baseline* |104| Where does the packet die **inside** the kernel? | **`pwru` v1.0.12** (13 Jul 2026, kernel ≥5.3; `--output-skb` ≥5.9) | `bpftrace 0.26.1` (02 Jun 2026) and `bcc 0.37.0` tools (02 Jul 2026) for bespoke cases | Guessing between `nftables`, routing and the driver |105| And in Kubernetes? | `hubble observe --verdict DROPPED` (Cilium **1.20.0**, 29 Jul 2026); `kubectl debug --image=nicolaka/netshoot` with `--target` | `kubectl debug node/<n>` for the node's netns; Retina where the CNI is not Cilium | `kubectl exec` into a *distroless* container and giving up |106| And in the cloud? | The provider's **flow logs** + its reachability analyser | A capture on the instance if the provider does not offer a managed one | Concluding "it is the cloud" without looking at the flow logs |107108**On eBPF (`pwru`, `bpftrace`, `bcc`, Hubble): recommended, but not as the first step.**109They are the answer to "the packet enters the machine and does not come out, and no classic tool tells me110where" — a real and frequent problem on hosts with containers, and where classic capture does not111reach because the drop happens between capture points. Requirements and cost to accept112beforehand: a modern kernel with **BTF** and `CONFIG_KPROBES`/`CONFIG_BPF`, high privileges on the host113(a security decision, see `container-runtime-security-standards`), and non-zero overhead in114production. **Criteria**: layer 1-2, `ss`, capture at both ends and firewall counters115first; eBPF when those four do not close the case.116117## 3. The method118119### 3.1 Before typing anything: four questions1201211. **What changed?** — The cause is the last change until proven otherwise. The122 diagnosis **starts in change control**: deployments, firewall or routing changes,123 patching, certificate renewal, change of provider, firmware update, the expiry of124 something. If nobody knows what changed, that is the first finding (and a governance problem, not a125 network one).1262. **What exactly is "it does not work"?** — Reproduce the symptom precisely: which source, which127 destination, which port, which protocol, which client, at what time, with which literal error message.128 "The network is bad" is not a symptom; "from host A, `curl` to B:443 hangs after the129 TLS handshake, in 30% of attempts, since 09:14" is.1303. **What is the scope?** — One user or all of them? One destination or all of them? One protocol or all of them?131 One VLAN, one node, one availability zone? Scope **discards more hypotheses in 30132 seconds than an hour of captures**: a failure that affects a single client is not in the133 server, and one that affects every destination is not in the destination.1344. **Did it ever work?** — "It never worked" is a **configuration or design** problem;135 "it worked and stopped working" is a **change or resource exhaustion** problem. They are136 two different investigations and confusing them costs hours.137138### 3.2 Bisection: divide the path, do not walk it139140- **You do not walk the path hop by hop.** You **split it in half**: pick an intermediate point141 with visibility (a router, a load balancer, a node) and determine whether the problem is before or142 after. Repeat. With 8 hops, bisection is 3 tests; walking it linearly, 8 — and with more143 opportunities to get it wrong.144- **Bisection also applies to non-spatial dimensions**: two clients (one that fails,145 one that does not) → what differs?; two destinations; two protocols; two moments; IPv4 vs. IPv6. The cut146 is by *variable*, not just by *place*.147- **One variable at a time.** Changing two things and having it work is not a diagnosis: it is a148 coincidence you will have to deal with again. If pressure forces changing several at149 once to mitigate, it is **noted** and reverted one by one afterwards to identify which one it was.150- **A falsifiable hypothesis before typing**: write down "if the cause is X, then doing Y I will see Z;151 if I see W, X is ruled out". A test that cannot refute your hypothesis is not a test, it is152 a ceremony. This is what separates diagnosis from "firing off commands".153- **Confirmation bias is the main enemy**: as soon as you have a suspect, every154 test seems to confirm it. Antidote: define in advance **what result would make you abandon** that155 hypothesis, and look for it explicitly.156157### 3.3 The canonical sequence158159Four questions, in order. Each one has a binary answer and eliminates half the universe.160161**1) Does the name resolve — and to the right thing?**162- Isolate DNS from the rest **from the first minute**: `curl -v --resolve host:443:<ip>` directly163 compares "does not resolve" against "does not connect". If it works with `--resolve`, the problem is164 resolution and not the network.165- Ask **each** resolver separately and compare them, and **compare that with what the process166 actually uses**: the system may have a local stub, a cache, a `search` that completes the name,167 a forgotten `/etc/hosts` or a library that does not even go through the OS resolver.168 Resolution "from the shell" and "from the application" **are not the same**.169- Negative caching and TTL explain the classic "it works for me and not for you" and "it took an hour to170 fix itself". Zone, TTL and resolver design, in `dns-standards`.171172**2) Does the packet reach the destination?**173- Capture **at the destination**, filtering by source and port. If it does not appear, the packet dies on the174 way: firewall, route, NAT, VLAN, layer 1-2.175- Check at the source that it **leaves** and through which interface: `ip route get <dst> from <src>` answers176 the real question (it includes policy routing and the effective table), not the main table.177- If it leaves and does not arrive, bisect the path and check the counters of the filtering rules at each178 hop: a `counter` incrementing on the drop rule is a proof, not a suspicion.179180**3) Does the response come back? — the forgotten half of the problem**181- **The response gets lost as much as the outbound, and almost nobody looks at it.** If at the destination you see the `SYN`182 and the `SYN/ACK` leaves, but the source does not receive it, the problem is the **return path** and everything183 you were looking at was irrelevant.184- Typical causes of asymmetric failure: the return takes a different path (multihoming, specific185 routes, partial VPN, VRF); a stateful firewall that only sees one direction and drops the other186 for lack of state —**an unmistakable symptom: it works for a while and then cuts off, or it fails187 intermittently and non-reproducibly**—; NAT in one direction and not the other; uRPF dropping because of an188 invalid reverse path.189- **Operational rule**: in every hard case, simultaneous capture at both ends **with synchronised190 clocks**, and compare. This solves 90% of what looks impossible: it tells you in which half191 of the path the packet disappears and with that the investigation is reduced to half the network. If192 the path also has an intermediary (proxy, load balancer, NAT), capture on both of its sides.193- Correcting asymmetry is a design (`networking-standards`) or policy194 (`firewall-policy-standards`) matter: it is **never** fixed by adding a broad `accept`.195196**4) Does the application accept it?**197- The packet arrives, the `SYN/ACK` comes back, and the failure persists: check whether the process **listens** (`ss198 -tanp`), on which address (`0.0.0.0` vs `127.0.0.1` vs `::` — a service on loopback is199 unreachable from outside and looks like a firewall), and whether the accept queue is full200 (`ss -lt` shows `Send-Q` as the backlog and `Recv-Q` as pending: if it saturates, the kernel201 drops `SYN`s and the client sees *timeouts* with the server "alive").202- Failures that **look** like network ones and are the application's or its edge's: an immediate `RST` (nobody listens203 or the proxy rejects), a TLS handshake failing because of SNI, certificate, version or chain204 (`openssl s_client -servername`), redirects, authentication, or the application's own *timeouts*205 shorter than its retry.206- **`RST` vs. *timeout* is the trade's most informative distinction**: `RST` = something answered and207 rejected (live host, closed port, a proxy or firewall that **rejects**); silence = something dropped208 (a firewall that **drops**, a missing route, a downed host). They are not the same problem and they are not209 diagnosed the same way.210211### 3.4 Which lie each tool tells212213No tool lies out of malice: they all answer a narrower question than the one you think you214are asking.215216- **`ping` / ICMP** — it measures *that the destination answers ICMP*, not that the service works. ICMP is usually217 **deprioritised** in the control plane of routers and switches, or filtered by policy. A218 `ping` with 200 ms and loss towards the network device may be perfectly normal while the219 data traffic runs impeccably; and a perfect `ping` says nothing about port 443. **Never use220 `ping` as the only quality measurement or as a proof of service.**221- **`traceroute`/`mtr`** — the trade's most widespread lie: **loss at intermediate hops222 means nothing**. Routers generate `TTL exceeded` responses in their CPU and rate-limit them;223 seeing 40% loss at hop 5 and 0% at the destination means **that router prioritises its224 work, not that there is a problem**. Only two things count: (a) loss at the **last hop** (the225 destination), and (b) loss that **persists** from one hop to the end. Besides, ECMP makes226 each probe take a different path (use fixed-flow mode if your tool supports it),227 the return path is invisible, and MPLS can hide hops entirely. When ICMP/UDP is228 filtered, `traceroute -T` over the application's real port is what reflects the path that229 matters.230- **`ss`** — it tells you the socket's **local** state. `ESTABLISHED` at one end does not imply the other231 is still there: a connection whose peer disappeared without a `FIN` keeps showing as established until a232 keepalive or a write discovers it. `ss -tin` does give gold: RTT, `cwnd`, retransmissions — which233 distinguish "the network is losing" from "the application is slow".234- **`tcpdump`/`tshark`** — they capture where you are and **after** the kernel has decided235 some things and **before** others: a packet dropped by the filter may or may not appear in the236 capture depending on the hook point, and offload (GRO/GSO/TSO/LRO) shows giant "packets" that237 do not exist on the wire (disable it if you are going to analyse sizes or MTU). With sampling or a badly238 written filter, your "it does not show up" may be yours, not the network's. And an unfiltered capture on a loaded239 link loses itself: use a capture filter (BPF) for what you want, and a display240 filter to analyse.241- **`ip route`** — it shows tables; **`ip route get`** shows the decision. With policy rules,242 VRF or several tables, reading the main one and assuming is a classic mistake.243- **`ip neigh`/ARP** — a `STALE` or `FAILED` entry tells you more than a `REACHABLE` one; and a correct entry244 with the **wrong MAC** (duplicate IP, unexpected proxy ARP) is a silent failure245 that no layer 3 test gives away. `arping` from the same segment reveals **duplicate IPs**246 (two answers, two MACs) in one second, which is what no other tool does.247- **`ethtool`** — the only one that sees layer 1-2: negotiation (duplex/speed; a badly negotiated248 *half duplex* shows up as "slow and intermittent" under load), CRC errors, drops due to249 lack of buffer, and per-queue counters. **Cumulative counters**: what matters is the250 **delta** during the failure, not the total since boot.251- **`iperf3`** — it measures what you ask it to, and by default that is not what you think. Classic mistakes: measuring252 10 s (all *slow start*), a single flow (limited by RTT and window, not by the link), only in253 one direction (`-R` measures the other, which may be the broken one), in UDP without setting the rate (`-b`) or setting it254 above capacity and calling your own saturation "network loss", with `iperf3` itself255 as the CPU bottleneck, or measuring against a shared public server.256 **`iperf3` measures one path between two points at one instant; it does not measure "the network".**257- **`curl -v`** — without `--resolve` it mixes DNS, connection, TLS and HTTP into a single result; with258 `--resolve` it separates the first. Its per-phase timings (`-w`) are a diagnosis in themselves: if the259 time goes on the `connect`, it is the network; on the `appconnect`, it is TLS; on the `starttransfer`, it is260 the application.261- **The application logs** — they say "connection timeout" for half a dozen mutually incompatible262 causes. Useful for the exact time and the scope; useless as a diagnosis.263264### 3.5 The usual suspects and their characteristic symptom265266A recognition table. The symptom is what makes you suspect; the proof is what demonstrates it.267268| Suspect | Characteristic symptom | Proof that demonstrates it |269|---|---|---|270| **MTU / PMTU black hole** | **The connection opens and hangs while transferring**: SSH connects but `scp` stalls; the web page loads the HTML and not the images; "the VPN connects but I cannot browse". Typically after a tunnel or a change of encapsulation | `ping` with a large packet and the DF bit, increasing until you find the cut-off; a capture showing retransmissions of the same large segment with no ACK; absence of ICMP *fragmentation needed* coming back |271| **Filtered ICMP that breaks PMTUD** | Identical to the previous one, and **it never fixes itself** | The intermediary blocking ICMP type 3 code 4 (or ICMPv6 *packet-too-big*): it is seen by its absence in the capture on the side that should receive it |272| **DNS** | Intermittency that looks like the network's; "sometimes it takes exactly 5 seconds" (resolver timeout); it works by IP and not by name | `curl --resolve` versus a normal `curl`; querying each resolver separately |273| **Ephemeral port / NAT exhaustion** | Failures that increase with load, on the side that **initiates** many connections (proxy, NAT, API client) | Socket count against `net.ipv4.ip_local_port_range`; on the NAT, active sessions against its capacity |274| **Full conntrack table** | **Silent** drops under load, with a line in `dmesg` nobody looks at | `nf_conntrack_count` against `nf_conntrack_max`; `dmesg` with `table full` |275| **Different return path** | "It works for a while and then cuts off"; intermittent and irreproducible; it works in one direction | Simultaneous capture at both ends: you see the outbound and the response that does not arrive |276| **`TIME_WAIT` / exhausted backlog** | Connection *timeouts* with the server alive and CPU low; it worsens at peaks | `ss -s` (count by state), `ss -lt` with `Recv-Q` growing, the dropped-`SYN` counter |277| **Keepalive vs. an intermediary's *idle timeout*** | **"The session drops exactly after N minutes"** of inactivity. A firewall, NAT, load balancer or cloud with an idle timeout shorter than the client's keepalive | Reproduce with an idle session and a stopwatch; compare the intermediary's timeout with the TCP/application keepalive. It is fixed by lowering the keepalive, not by raising everybody's timeout |278| **Duplicate IP** | Unexplained intermittency that changes with the ARP cache lifetime; "sometimes it lands on a different server" | `arping` from the segment: two answers with different MACs |279| **Duplicate MAC / *MAC flapping*** | Massive loss on a segment; the switch logs learning the same MAC on different ports | Switch log; address table |280| **Layer 2 loop / broadcast storm** | The whole VLAN goes down or crawls; switch CPU at 100%; it starts right after plugging something in | Broadcast/multicast counters per port shooting up; STP with constant topology changes. It is an emergency: the port is isolated first |281| **Aged or incomplete ARP/ND** | A host unreachable from its own segment while everything else is fine | `ip neigh` in `FAILED`/`INCOMPLETE` state |282| **IPv6 enabled and failing while IPv4 works** | "It is slow" with delays of exact seconds at the start; it works with `-4`; it fails only on some clients | Compare `curl -4` and `curl -6`; route and ND on IPv6. Usual cause: `AAAA` published without real IPv6 connectivity, or an IPv6 firewall without parity with IPv4 |283| **Happy Eyeballs masking the failure** | The IPv6 failure is **almost** unnoticeable (the client retries over IPv4 after a short delay), so nobody fixes it and the initial latency is worse for everyone | A capture showing the abandoned IPv6 attempt. **RFC 8305** is the current specification; v3 is still a **draft** (§8) |284| **A proxy or TLS inspection in the middle** | An unexpected certificate, a forced TLS version, rewritten ALPN, HTTP/3 that does not work, mTLS that fails, a `Server` different from the expected one | `openssl s_client -servername` and compare the certificate issuer with the expected one |285| **Load balancer with one bad backend** | A **constant fraction** of requests fails (1 in N) | Repeat the test N+ times recording which backend each one goes to |286| **Expired certificate or incomplete chain** | It fails at an exact time, for everybody at once, with nobody having touched anything | `openssl s_client` showing the chain and the dates |287| **Layer 1** | Growing CRC errors, slow only under load, badly negotiated duplex, degraded optics | `ethtool -S` (delta during the failure), `ethtool` (negotiation), optical power on the device |288| **Saturation / bufferbloat** | Latency that shoots up **only when there is traffic**; the `ping` goes from 10 ms to 300 ms when a download starts | Latency under load versus at rest; link utilisation by percentiles |289290### 3.6 Latency, loss and jitter: measuring them properly291292- **Percentiles, never averages.** The average hides exactly what breaks the experience. Look at293 p50, p95, p99 and **the maximum**; and compare with the at-rest state, not with an absolute number. A p50 of 20 ms294 with a p99 of 2 s is a broken system that averages well.295- **Distinguish the three causes of "it is slow"**, because they have opposite fixes:296 - **Latency (RTT)**: it limits a TCP flow's throughput through the window. If the RTT is high, more297 bandwidth fixes nothing; more parallel flows or a larger window do.298 - **Loss**: it sinks TCP throughput disproportionately (1% loss can cost299 most of the performance on a high-RTT link). It is seen in retransmissions (`ss -tin`),300 not in the `ping`.301 - **Jitter**: irrelevant for a download, lethal for voice and video. It is measured with UDP at a fixed302 rate, not with TCP.303- **The measurement must reproduce the real case**: same source-destination pair, same protocol, same304 transfer size, same time of day. A test at rest does not reproduce a saturation problem,305 and a 10-second test does not reproduce a 20-minute problem.306- **If "it is slow" turns out to be I/O, CPU or the database, say so and close the case there**: the network is307 blamed by default, and proving that it is **not** the network is as valid a result as any308 other (with evidence, not with a denial).309310### 3.7 Modern environments: containers, Kubernetes, clouds and overlays311312- **The packet crosses several *network namespaces*.** Before capturing, decide **which one** you are313 capturing in: inside the container, on the host side of the `veth`, on the bridge, in the node's netns,314 on the physical interface, on the overlay interface. "It does not show up in the capture" almost always315 means "you captured in the wrong namespace".316- **Kubernetes: a checking order by the cluster's own layers** — does the `Service` name317 resolve? → does the `Service` have `Endpoints`/`EndpointSlice` (a `Service` with no endpoints from a badly318 written selector is the most common failure and does not look like a network one)? → is there a `NetworkPolicy` that319 denies it (the drop happens in the datapath, **before** reaching the pod)? → does the `readinessProbe`320 take the pod out of rotation? → do the CNI or `kube-proxy` have the rule? → does the node route?321 `hubble observe --verdict DROPPED` gives the verdict and the reason for the drop in one step; beware of322 **monitoring aggregation**, which can hide individual events from you.323- ***Distroless* containers with no tools**: `kubectl debug` with an ephemeral container324 (**GA since Kubernetes 1.25**) and a diagnostic image (`nicolaka/netshoot`), sharing the325 pod's namespace, and `kubectl debug node/<node>` for the node's netns. Two operational warnings:326 an ephemeral container **cannot be removed** until the pod is deleted and **has no resource327 limits**; and to extract a capture, **stream it over standard output** instead of writing the328 file inside (copying from an ephemeral container does not work).329- **Clouds**: the VPC/VNet **flow logs** are the first stop —they say whether the packet was330 accepted or rejected and by which rule— followed by the provider's reachability analyser. Their331 limits, which must be known: aggregation into windows (you do not see the packet, you see the flow), possible332 sampling, a delay of minutes, and no payload. The specific configuration, in333 `aws-standards`/`azure-standards`/`gcp-standards`.334- **Overlays and tunnels** (VXLAN, GENEVE, IPsec, WireGuard): capture **inside** the tunnel and335 **outside** —they are two different questions: "does the traffic enter the tunnel?" and "does the tunnel reach the other336 side?". And as soon as there is encapsulation, **MTU is the first suspect, always**.337- **Service mesh / sidecar**: the proxy may terminate TLS, rewrite headers, apply its338 own timeouts and retries and return errors that look like the application's. Its access logs339 are the source, not the capture.340341## 4. Quality of the diagnosis (gates)342343A diagnosis is accepted when it meets **all** of this. It is not bureaucracy: it is what prevents344the same incident from coming back in three weeks.3453461. **Reproducibility**: there is an exact command or procedure that produces the symptom at will347 (or, if it is intermittent, a documented condition that triggers it and a measured rate). Without348 reproduction, the fix cannot be validated.3492. **Hypotheses recorded with their result**: what was tried, what was expected, what was seen and which350 hypothesis was ruled out. A record of exclusions is worth as much as the finding, and it prevents the351 next shift from repeating the same tests.3523. **Evidence saved, not described**: captures (`.pcap`) from **both** ends with a timestamp,353 complete command outputs, counters before/after, screenshots of graphs with their354 time range. "We saw packets being lost" is not evidence. Save the evidence **before**355 mitigating: mitigation destroys the state that proves it.3564. **Root cause proven, not inferred** — the central gate. "Proven" means you can357 explain the complete mechanism from the change or the condition to the symptom, **and** that you can358 reproduce the failure by activating the cause and make it disappear by deactivating it. Temporal correlation359 is not causation: "we rebooted and it was fixed" is a data point, not a conclusion.3605. **Fix validated by the test that was failing**, and additionally with the corresponding361 negative test (what was supposed to stay blocked is still blocked). And validated from the **real362 affected source**, not from the engineer's bastion.3636. **Absence of side effects verified**: what was touched did not break anything else. Temporary364 diagnostic changes (open rules, offload disabled, `tcpdump` running, raised timeouts,365 logs at debug) **are explicitly reverted** and the reversion is verified.3667. **Derived findings with an owner**: the configuration drift, the missing rule, the metric367 that did not exist, the runbook that was useless, the alert that did not fire. Each one is opened as work368 with an owner and a date — towards `observability-standards`, `firewall-policy-standards`,369 `networking-standards` or whoever is relevant.3708. **If the case is closed without a root cause** (it happens, and it is legitimate), it is closed **saying so**: what was371 ruled out, what instrumentation is missing to diagnose it next time, and what trigger is left372 armed to capture evidence when it comes back. That is a result; "it fixed itself" is not.373374## 5. Security of the diagnosis375376- **A traffic capture contains personal data, credentials and business content.** It is not377 an innocuous technical file. Treat it as classified data: controlled storage, restricted378 access, minimum retention and deletion when the case closes. If you are going to share it, anonymise it or379 trim to headers (`-s` to limit the capture) and remove the payload. Legal basis and380 minimisation, in `privacy-engineering-standards`.381- **Capturing in production is an action with impact**: it consumes CPU and disk and can fill a382 filesystem. Use a capture filter, a size limit and rotation, and give it a time limit383 from the start. A forgotten `tcpdump` on a server is a future incident.384- **Distinguish a fault from a compromise from the first minute.** If there is any sign of intrusion,385 the objective stops being "restore the service" and becomes "preserve the evidence": the procedure changes,386 do not reboot, do not delete, and escalate to387 `incident-response-forensics-standards` (order of volatility and chain of custody).388- **Port sweeping, scanning or injecting traffic into networks that are not yours —or without internal389 authorisation— is not diagnosis**: it is offensive activity and it is governed in390 `offensive-security-standards`. Inside your network, warn whoever is monitoring so that your test does not391 show up as an attack.392- **Privileges**: capturing requires `CAP_NET_RAW`/`CAP_NET_ADMIN` and eBPF requires more. Grant them393 **temporarily and by name**, not as a permanent configuration nor with a privileged container394 that stays there (`container-runtime-security-standards`). Withdraw them when the case closes.395- **Do not weaken controls to diagnose and then leave it like that**: opening a rule "to test" is the396 fastest route to a permanent `any/any` (`firewall-policy-standards`). If you open, you open with397 an automatic expiry.398399## 6. Operability: preparing before it fails400401- **Diagnosis is prepared, not improvised.** What must exist **before** the incident:402 an up-to-date inventory and SoT (`networking-standards`), a diagram of the traffic's real path,403 OOB access, telemetry with sufficient retention (`observability-standards`), flows, and **baselines**:404 normal RTT, normal throughput, normal error rate. **Without a baseline, "it is high" is an405 opinion.**406- **A synchronised clock on everything that logs or captures** (healthy NTP/chrony). Without it, correlating407 two captures or two logs is impossible, and that is precisely the method that solves the hard cases.408- **An available observation point**: SPAN/mirror, TAP, or at least a host with access to the segment and409 tools installed. If the first step of your diagnosis is "install `tcpdump` on the production410 server", you are already too late.411- **A minimum kit preinstalled or a diagnostic image ready**: iproute2, capture, `mtr`, `curl`,412 `nc`/`socat`, `ethtool`, `iperf3`, and a diagnostic container image for environments with no413 tools.414- **Runbooks by symptom, not by tool**: "it does not resolve", "it connects and hangs while transferring",415 "intermittent", "slow only under load", "it drops every N minutes", "1 in N fails", "it works416 from one host and not from another". Each one with the canonical sequence adapted and its escalation417 criterion.418- **A rescue window open when touching remote access, routes or the firewall** — an inherited and non-419 negotiable rule (`firewall-policy-standards`, `vpn-standards`): OOB console, a second path, or420 a timed rollback. Locking yourself out during a diagnosis is the most predictable and most421 avoidable incident there is.422- **When the diagnosis happens inside a declared incident**: command and communication belong423 to `incident-management-standards`. You give **hypotheses with a confidence level and an estimated time**,424 not premature certainties; and if the IC decides to mitigate before understanding, you mitigate — but **you capture425 the evidence first** and leave the pending diagnosis written down. Mitigating does not close the root cause.426- **Toil**: the same diagnosis repeated three times is an instrumentation or design failure, not427 bad luck. It becomes an alert, an automatic check or a structural fix428 (`sre-practice-standards`).429430## 7. Sustainability and prohibitions431432- **The postmortem feeds the method**: every hard case leaves either a new runbook, or a new433 metric, or a design change. If it leaves nothing, it will happen again.434- **Tool migration**: `net-tools` (`ifconfig`, `netstat`, `route`, `arp`) has been435 unmaintained for years and is absent by default in modern distributions; scripts and436 runbooks that still use it are migrated to **iproute2** with a date. It is not purism: it **hides** namespaces,437 policy routing and state that today determines the diagnosis.438- **Cadence**: review the version and CVEs of the capture and analysis tools (Wireshark/`tshark`,439 `libpcap`/`tcpdump`) along with the rest of the stack — **they are parsers that process hostile input by440 definition**, and 2026 brings a flood of LLM-assisted findings in that family. Analysing an441 untrusted capture with an unpatched Wireshark is exposing yourself.442- **Training with real cases**: keeping captures and timelines of solved cases as443 team training material is worth more than any course.444445**FORBIDDEN**446- ❌ **Rebooting as the first step.** It destroys the state you need (sockets, conntrack,447 counters, ARP cache, in-memory logs) and turns the problem into something irreproducible. It is the last resort,448 and with the evidence already collected.449- ❌ Changing **several things at once** and declaring victory when it works.450- ❌ Touching production blind: diagnostic changes with no hypothesis, no record and no rollback451 plan.452- ❌ **Blaming the network without evidence** — and also absolving it without evidence. Both are the same453 failing.454- ❌ **Capturing at only one end** in a hard case, or capturing without clock synchronisation.455- ❌ Concluding from loss at **intermediate hops** in `traceroute`/`mtr`.456- ❌ Using `ping` as proof that a service works, or as the only quality measurement.457- ❌ `ifconfig`, `netstat`, `route`, `arp` (`net-tools`) in diagnosis or in new runbooks.458- ❌ Reading `ip route` and assuming, instead of asking with `ip route get`.459- ❌ Measuring with `iperf3` for 10 s, a single flow, a single direction, and calling it a *baseline*.460- ❌ Reporting latency averages instead of percentiles.461- ❌ Analysing packet sizes or MTU without disabling the interface's offload.462- ❌ Diagnosing layer 3 without having looked at layer 1-2 errors and negotiation.463- ❌ Closing the case with "it fixed itself", "it was a network thing" or "we rebooted and it works now".464- ❌ Leaving the diagnosis's temporary changes in place (open rules, captures running,465 offload disabled, logs at debug, elevated privileges, a privileged debugging pod).466- ❌ 467468…(truncated)