Netops

NetOps — Ultimate Network Troubleshooting Skill

You are a CCIE- and RHCE-level network engineer. This skill resolves any network-related issue — connectivity, bandwidth, latency, packet loss, MTU, DNS, TLS, routing, switching, security/IDS — by picking the best tool + architecture for the specific symptom rather than reaching for one tool out of habit.

The skill is organized as an index plus focused subfiles. Read the Decision Framework first, then jump to the relevant scenario or reference.

When to Use This Skill

Trigger on any of:

• "X can't reach Y" / "intermittent drops" / "slow" / "high latency" / "packet loss"
• "is this link saturated" / "QoS" / "throughput" / "jumbo frames / MTU black hole"
• "DNS failing" / "TLS handshake failing" / "cert / SNI / cipher" / "QUIC"
• "BGP flapping" / "OSPF adjacency down" / "MAC flapping" / "ARP cache"
• "suspect malware / C2 beaconing / port scan / DNS tunneling"
• "I need to put a sensor on this link" / "monitor this Cisco device"
• "plan a fleet monitoring architecture" / "scale from one box to 500 devices"

The Two-Layer Mental Model (read this first)

Every network problem is solved by combining two layers. Get the layer wrong and you waste hours.

Rule: Telemetry tells you which device/link is misbehaving. Packets tell you why. You almost never start with packets — you start with telemetry (or a user symptom), narrow to a suspect, then escalate to packets on that one suspect. The hybrid is the only production answer.

Decision Framework — pick the tool + architecture

Step 1 — Classify the symptom (which tool)

Step 2 — Pick the capture architecture (how the sensor sees traffic)

The right architecture depends on where the traffic is and what you can change. See reference/architectures.md for full details.

Step 3 — Confirm the platform constraints

Step 4 — Run the scenario playbook

Jump to the matching file in scenarios/:

• scenarios/connectivity.md — "can't reach X", intermittent failures, SYN_SENT hangs, CLOSE_WAIT leaks, ARP/MAC issues, routing black holes
• scenarios/bandwidth-latency.md — saturation, throughput, jitter, RTT spikes, retransmits, QoS, bufferbloat
• scenarios/mtu-path.md — PMTUD black holes, MSS clamping, jumbo frames, fragmentation
• scenarios/dns.md — NXDOMAIN, SERVFAIL, timeout, hijack, split-horizon, DNS tunneling
• scenarios/tls-https.md — handshake failures, cert/SNI/cipher, OCSP, HTTP/2 & QUIC, SSLKEYLOGFILE decryption
• scenarios/security-ids.md — C2 beaconing, port scans, DNS tunneling, Suricata deployment, evidence capture
• scenarios/cisco-fleet.md — BGP/OSPF flaps, interface errors, VLAN/VRF, IOS-XE/NX-OS specifics, pyATS change validation

Skill File Index

Reference (deep command references — accuracy-verified)

• reference/tools.md — Exact syntax for NetWatch, RustNet, tshark, ngrep, Pktmon (every flag you'll use)
• reference/architectures.md — SPAN / RSPAN / ERSPAN / Linux bridge / L3 gateway / one-armed / vSwitch — when, how, configs, trade-offs
• reference/sensor-builds.md — Step-by-step sensor build recipes: Linux bridge sensor, Windows SPAN sensor, Windows L3 gateway sensor, headless streaming sensor, bypass-tap wiring

Cisco native telemetry (the wide/cheap layer)

• cisco/telemetry.md — SNMP, gNMI streaming, NETCONF/RESTCONF, CLI show reference, pyATS/Genie testbed + diff
• cisco/span-erspan.md — Authoritative SPAN/RSPAN/ERSPAN configs for IOS-XE (Catalyst 9300, Cat 8K) and NX-OS (Nexus), with verification commands

Scenarios (worked incidents)

• scenarios/connectivity.md
• scenarios/bandwidth-latency.md
• scenarios/mtu-path.md
• scenarios/dns.md
• scenarios/tls-https.md
• scenarios/security-ids.md
• scenarios/cisco-fleet.md

Scripts (ready to run)

• scripts/bridge-setup.sh — Persistent transparent br0 bridge on Ubuntu/Debian (netplan) and RHEL (NetworkManager)
• scripts/win-l3-gateway.ps1 — Turn a Windows box into an L3 gateway sensor (IP-Forward + NAT optional)
• scripts/tshark-rotate.sh — 24/7 rotating ring-buffer capture to disk
• scripts/pktmon-sensor.ps1 — Headless Pktmon capture-to-ETL with rotation + pcapng conversion
• scripts/erspan-decap.sh — Decapsulate ERSPAN/GRE so netwatch/tshark see the inner packets
• scripts/symptom-router.sh — Interactive: symptom → recommended tool + architecture (the decision framework as a script)

The 30-Second Triage (use this first)

Before touching any tool, gather these 5 facts. 80% of "mysterious" problems are solved here.

1. WHO reported it, WHEN did it start, WHAT changed recently?
2. WHAT is the exact symptom? (timeout? refused? slow? intermittent? which app?)
3. SCOPE: one host? one subnet? one app? everyone? time-of-day pattern?
4. LAYER: is it DNS? (nslookup/dig) routing? (traceroute/mtr) transport? (ping/tcp) app? (curl)
5. FROM WHERE? reproduce from the affected host AND from a known-good host — compare.

Then:

If triage points at a specific device/link, escalate to the matching scenario file. If triage points at "the whole path is fine but the app still fails," you need packets — go to scenarios/connectivity.md or tls-https.md.

Golden Rules (read these; they save careers)

1. Measure from both ends. A problem that looks like "server is slow" is often a client-side DNS or MTU issue. Always reproduce from a known-good host too.
2. Telemetry before packets. Don't SPAN a port until SNMP/gNMI/show interface tells you which port. Packets are expensive and narrow; telemetry is cheap and wide.
3. Capture on the bridge, not the NIC. On a Linux bridge sensor, capture on br0 (sees both directions). Capturing on eth0 sees only one direction. (Reference: reference/sensor-builds.md.)
4. Disable offloads on sensors. ethtool -K br0 gro off tso off lro off gso off so the sensor sees real frames, not re-segmented blobs. Linux only.
5. Bypass tap or don't go inline in production. A sensor crash without fail-open = an outage. Use SPAN where you can't tolerate the SPOF.
6. SSLKEYLOGFILE only for traffic you own. It decrypts your client's TLS. It cannot decrypt third-party or malware TLS — that's JA4 fingerprinting territory.
7. One change at a time. Especially on Cisco gear. Then pyATS-diff to prove what moved.
8. Capture evidence before you fix. NetWatch Flight Recorder (Shift+R→Shift+F), tshark ring buffer, Pktmon ETL. The RCA is worthless without the packet that proved it.
9. Never bridge inside a LACP bundle. Bridge before the bundle splits, or use a tap that handles bundles — otherwise you lose half the traffic and break the hash.
10. Know your platform limits. Windows = great host/SPAN/L3 sensor, never a transparent inline bridge. Linux = everything.

Quick Tool Cheat-Sheet

NetWatch (live packet TUI — Linux/macOS primarily)

netwatch                 # unprivileged: dashboard, connections, interfaces, processes
sudo netwatch            # full capture; select interface in Interfaces tab (press 3)
# No -i flag! Pin interface via config: capture_interface = "br0" in ~/.config/netwatch/config.toml
SSLKEYLOGFILE=/tmp/k.txt sudo netwatch   # + live TLS 1.3 decrypt (Packets tab)
# Tabs: 1 Dashboard 2 Connections 3 Interfaces 4 Packets 5 Stats 6 Topology
#       7 Timeline 8 Processes 9 Insights 0 Egress
# Shift+R arm Flight Recorder, Shift+F freeze, Shift+E export bundle

RustNet (cross-platform process-attribution TUI — Linux/macOS/Windows/FreeBSD)

sudo rustnet -i eth0                 # Linux/macOS
rustnet --show-localhost             # include loopback
sudo rustnet --pcap-export cap.pcap  # capture with process-attribute sidecar
# Tabs: 1 Overview 2 Details 3 Interfaces 4 Graph 5 Help
# Filter: port:443 sni:github.com process:chrome state:established /regex/

# Windows — requires Npcap with WinPcap API-compatible mode
choco install rustnet
rustnet -i "Ethernet"                # run as Administrator

tshark / dumpcap (scriptable capture — all platforms)

sudo tshark -i eth0 -f "tcp port 443" -w cap.pcapng        # live → file
sudo dumpcap -i eth0 -w cap.pcapng -b filesize:100000 -b files:50   # ring buffer
tshark -r cap.pcapng -Y "tcp.analysis.retransmission"        # post-hoc: retransmits
tshark -r cap.pcapng -o "tls.keylog_file:/tmp/k.txt" -Y "tls"  # decrypt TLS post-hoc

ngrep (regex on live payload)

sudo ngrep -d eth0 -W byline 'GET|POST' tcp port 80          # HTTP methods
sudo ngrep -d eth0 -i 'error|exception' port 5432            # app error strings
sudo ngrep -d eth0 -X 'cafebabe'                              # hex pattern
sudo ngrep -I cap.pcap 'password'                             # search a pcap

Pktmon (Windows built-in)

pktmon list                              # list interfaces
pktmon filter add F1 -t TCP -p 443        # add a filter
pktmon start --capture --comp nics -m real-time     # live to console
pktmon start --capture --comp nics --pkt-size 0 -f C:\cap.etl  # capture to ETL
pktmon stop
pktmon etl2pcap C:\cap.etl -o C:\cap.pcapng                # convert to pcapng
pktmon counters                                            # high-level counters

Full references in reference/tools.md.

Escalation Flow (how the layers hand off)

┌─────────────────────────────────────────────────────────────┐
│ 1. USER SYMPTOM / SNMP-gNMI alert                           │
│    "app slow to DB"  /  "Gi1/0/24 input errors climbing"    │
└──────────────────────────┬──────────────────────────────────┘
                           │ (triage: mtr, dig, openssl, curl)
                           ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. TELEMETRY — narrow the suspect                           │
│    gNMI: which interface/BGP session? pyATS diff: changed?  │
│    CLI show interface / show ip route on the Cisco device   │
└──────────────────────────┬──────────────────────────────────┘
                           │ (need bytes → pick architecture)
                           ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. PACKETS — prove the why                                  │
│    SPAN the port → sensor: netwatch/rustnet/tshark          │
│    or inline bridge sensor on the uplink                    │
│    or host-based on the Linux app server                    │
└──────────────────────────┬──────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│ 4. EVIDENCE — Flight Recorder / pcap / ETL + audit          │
│    netwatch Shift+F  ·  tshark ring buffer  ·  pktmon etl   │
└─────────────────────────────────────────────────────────────┘

This flow is the single most important habit. Telemetry scales and tells you where; packets are deep and tell you why. Reach for packets only after telemetry (or a clear user symptom) points at a specific suspect.

Every command in this skill has been verified against official docs (Cisco IOS-XE 17.x config guides, Microsoft Pktmon reference, Wireshark/tshark man pages, ngrep 8 man page, NetWatch v0.26.1, RustNet README/ARCHITECTURE). See verification notes at the bottom of each reference file.