Net Ops

Network Operations

Diagnose network problems on Windows, macOS, or Linux with a layered ladder that isolates faults to the smallest possible scope, then pattern-match against OS-specific culprits. Designed for the common case: someone reports "internet broken" on a box you can shell into (locally or via SSH).

The Universal Insight

Bypass-tool succeeds while OS-resolver fails is a smoking gun on every platform. It means DNS infrastructure is healthy but the operating system's name-resolution path is hooked or misconfigured. The bypass tool differs per OS but the discriminator is identical:

The bypass tool implements its own resolver and talks straight to UDP/53. The OS resolver tool goes through the full system name-service path including all hooks. Comparing the two narrows the suspect list dramatically.

The Diagnostic Ladder

Walk down the layers in order. Do not skip rungs. Each rung has a binary outcome that eliminates everything above it. Per-OS tools are in references/diagnostic-ladder.md; the structure is universal.

1. Link layer        — interface up, valid IP, gateway present
2. IP reachability   — ping public IPs over ICMP
3. Socket reach.     — TCP/443 + UDP/53 to known destinations (raw socket DNS)
3.5 LAN services     — parallel track: mapped drives, SMB, mDNS/.local,
                       single-label hostnames (see below — internet OK ≠ LAN OK)
4. DNS infrastructure — bypass tool: nslookup / dig @<server>
5. OS resolver path  — the hook layer (most interesting on modern systems)
6. Application       — real HTTP request to a real hostname

The most common mistake: jumping to rung 6 ("HTTPS doesn't work, must be a cert / proxy") when rung 5 is the actual problem (an orphan VPN DNS rule on Windows, a stale /etc/resolver/ file on macOS, a misconfigured systemd-resolved on Linux). Discipline prevents this.

The LAN Services Track (Rung 3.5)

Rungs 1–6 are framed around reaching the public internet. A mapped drive showing Disconnected while browsing works fine is a different fault domain: local-network service reachability. Symptoms: SMB shares, printers, \\NAS\share, .local names, single-label hostnames.

The load-bearing concept is the single-label hostname resolution path:

HOSTS file → NRPT match → DNS (suffix search) → LLMNR → NetBIOS-NS

An NRPT . catch-all (live VPN or orphan) short-circuits everything below it: the single-label name goes to the VPN resolver (NXDOMAIN) and the LLMNR/NetBIOS broadcast fallback that LAN names normally rely on is suppressed. VPN DNS-leak-protection often also blocks UDP/53 to the LAN router, so there's no fallback resolver either. Full chain, per-OS tools, and the credential-target gotcha: references/diagnostic-ladder.md (rung 3.5).

One-shot audit: scripts/windows/smb-audit.ps1 walks every mapped drive through mapping state → resolution mechanism → ICMP/TCP-445 → credential targeting → NRPT/leak-protection detection, and emits a verdict naming the fix.

VPN + LAN coexistence — decision rule. Is the VPN live and wanted?

• Yes → do NOT touch the NRPT rule. Pin the name in the HOSTS file (consulted before NRPT, so it wins regardless of VPN state; needs admin), or remap by IP and add a credential keyed to that IP (cmdkey /add:<ip>).
• No (VPN gone, rule orphaned) → scripts/windows/nrpt-clean.ps1 -Apply.

nrpt-clean.ps1 must never be pointed at a live VPN's catch-all — it exists for orphans only. Deleting a wanted VPN's rule breaks its DNS routing and the client will just re-create it.

Workflow

1. Identify the target OS

If local: uname -s (Unix) or check shell environment. If remote over SSH, the bootstrap script auto-detects:

scripts/ssh-bootstrap.sh <user>@<host>

2. Run the OS-appropriate probe

Each prints structured [PASS]/[FAIL] per rung. Scan for the first FAIL — that's where to drill in.

3. Drill into the failing layer

The interesting failures are almost always rung 5. Per-OS deep-dive scripts:

4. Apply the minimum reversible fix

Repair scripts default to dry-run and protect known-good config (Tailscale MagicDNS, MDM-managed entries). Apply only when the dry-run output matches expectation.

Quick Reference: Smoking Guns

SSH Transport Patterns

Windows targets

PowerShell-over-SSH has notorious escaping issues. Always pass scripts via -EncodedCommand with UTF-16LE base64:

B64=$(printf '%s' "$PS_SCRIPT" | iconv -t UTF-16LE | base64)
ssh <target> "powershell -NoProfile -EncodedCommand $B64"

Unix targets (macOS, Linux)

Heredoc works cleanly; no special encoding needed:

ssh <target> 'bash -s' < scripts/linux/probe.sh
# or, with arguments:
ssh <target> "bash -s -- arg1 arg2" < scripts/linux/probe.sh

For consistency, scripts/ssh-bootstrap.sh handles both transports based on detected OS.

Pattern Recognition

After a few sessions, certain symptom triplets become instantly diagnosable. See references/case-studies.md for worked examples. Hall-of-fame entries:

Windows: nslookup works, Resolve-DnsName times out identically across all servers, Invoke-WebRequest says "remote name could not be resolved" → orphan NRPT catch-all from a disconnected VPN. Common gateway IP patterns are listed in references/common-culprits.md.

macOS: dig <host> works, browsers say "cannot find server," scutil --dns shows extra "resolver #N" entries pointing at private-range gateways with domain : listed → leftover /etc/resolver/<domain> files from a disconnected VPN.

Linux: dig @<public-resolver> <host> works, getent hosts <host> fails → /etc/nsswitch.conf may have an NSS chain that skips resolve, OR /etc/resolv.conf is no longer symlinked to the systemd-resolved stub.

Safety Notes

• Read before write. Always dump current state before modifying a resolver config. The forensics may be load-bearing for explaining what happened.
• Don't disable security tools without consent. AV / firewall hooks are intrusive but legitimate. Pause is preferred over uninstall.
• Tailscale's name-resolution config looks like junk but is essential. Always filter on protected nameserver patterns (100.100.100.100 on all OSes) before bulk-deleting.
• Resolver config persists across reboots. Removing a rule is forever (until the VPN re-creates it). Confirm the source/comment before deletion.
• macOS profile DNS overrides may be MDM-managed. Removing them may violate enterprise policy and may be re-applied automatically. Coordinate with IT.

References

• references/diagnostic-ladder.md — full ladder methodology with per-OS commands per rung
• references/common-culprits.md — detection + fix catalog for Windows / macOS / Linux
• references/case-studies.md — worked examples and template for adding new ones

Scripts

• scripts/ssh-bootstrap.sh — establish SSH session, auto-detect target OS, emit usable invocation
• scripts/windows/probe.ps1 — full layered diagnostic for Windows
• scripts/windows/nrpt-audit.ps1 — NRPT forensics with attribution
• scripts/windows/nrpt-clean.ps1 — safe NRPT cleanup (orphans ONLY — never point it at a live VPN's catch-all; protects Tailscale)
• scripts/windows/smb-audit.ps1 — mapped-drive / SMB / LAN-name audit with per-drive verdicts (-DriveLetter Z -FallbackIp 'NAS=192.168.50.11', -Json)
• scripts/macos/probe.sh — full layered diagnostic for macOS
• scripts/macos/dns-audit.sh — scutil + /etc/resolver + profile + mDNSResponder dump
• scripts/macos/resolver-clean.sh — remove orphan /etc/resolver/* files
• scripts/linux/probe.sh — full layered diagnostic for Linux
• scripts/linux/dns-audit.sh — systemd-resolved + NM + NSS + resolv.conf dump
• scripts/linux/resolved-reset.sh — reset systemd-resolved per-link state