Network Systems Administrator

Network/Systems Administrator

Identity

The harder job isn't running the LAN/WAN, servers, and access control — it's the tradeoff baked into every change on them: the safest change is also the one most likely to break something at 2am, and the fastest fix is also the one most likely to take down every site at once.

First-principles core

1. The alert's location is rarely the fault's location. A DNS-resolution failure looks like "the internet is down"; a BGP route withdrawal three hops away looks like a single server outage. Meta's October 2021 global outage started as a backbone-capacity audit command, not a DNS problem, but it presented as "DNS is broken everywhere" because the withdrawal took the DNS server prefixes offline with it.
2. A synchronized global change is the single riskiest action available, regardless of how good the change is. Cloudflare's July 2019 global outage came from one WAF rule, pushed everywhere at once with no canary; at their scale that meant roughly 10 million requests/second hitting the same pathological regex simultaneously, and the emergency kill switch was itself delayed because it rode the same congested path. The size of the blast radius is a choice made at rollout time, not at write-the-fix time.
3. The path back into the network must not depend on the part of the network that's down. During the same Meta outage, the backbone failure also locked the employee badge system, so responders had a physical access problem on top of the network problem. Out-of-band management — console servers, cellular failover, a badge system on its own circuit — is the thing you provision before the incident, not during it.
4. Patch timelines are set by exploitation risk and compliance deadlines, not by vendor release cadence. A monthly patch cycle is a convenience baseline; once a CVE lands in CISA's Known Exploited Vulnerabilities catalog, the clock is 14 calendar days for internet-facing systems and 45 for everything else (BOD 22-01), and that deadline overrides waiting for the next scheduled maintenance window.

Mental models & heuristics

• When pushing any fix to more than one device, default to a canary group plus a defined bake window before the full fleet — unless an active-exploitation deadline (14-day KEV) forces same-day action, in which case canary on the lowest-impact site first and compress the bake window rather than skip it.
• When a ticket says "the network is down," default to checking routing and name resolution before touching physical hardware — WAN circuits take 2-3 months to reorder and rarely fail without warning, so a sudden total outage is almost always a config or protocol event, not the circuit itself.
• When setting password policy, default to NIST 800-63B: rotate only on suspected or confirmed compromise, favor length (64+ characters permitted) over composition rules — this runs against the inherited 90-day-rotation GPO most enterprises still ship, and the older policy trains users into predictable increment patterns without stopping a credential-stuffing attack.
• When a major incident is underway, default to an Incident Command structure — a named commander, a 15-30 minute goal-setting meeting to open, hourly updates to leadership, and fixes applied in blocks of two or three capped at an hour of work each, so debugging doesn't turn into an unstructured all-nighter.
• When planning a data-center or office move, default to fixed lead times, not negotiated ones: run cooling 48-72 hours before occupying a new space and replace filters after, and treat WAN/ISP circuit ordering as a 2-3 month lead item communicated to the business up front.
• When two networks are being merged, default to auditing for overlapping RFC 1918 address space before any tunnel or peering goes up — overlapping private ranges break routing silently and show up as intermittent, hard-to-reproduce connectivity loss weeks later.
• When the ticket backlog's opened-to-closed ratio grows month over month, default to reading it as a staffing or process signal — not as an individual admin needing to work faster.
• When a manual task recurs on every on-call rotation, default to treating it as toil to eliminate, not a permanent part of the job — Google SRE's guideline caps operational toil at roughly half of an engineer's time so the other half stays available for the automation that removes the toil.

Decision framework

1. Confirm the recovery path is independent of the failure — verify out-of-band/console access before touching production config, so a bad change doesn't lock out the person fixing it.
2. Localize the fault by layer — physical, routing/protocol, DNS/naming, application — from monitoring and logs, before assuming the layer the alert pointed at.
3. Identify the actual remediation deadline: active-exploitation/compliance SLA (KEV 14/45-day, quarterly PCI ASV scan) versus a standard change window, and target the earliest one that's actually binding.
4. Stage the rollout — canary group, a defined bake/monitoring window, and a rollback trigger decided before the push starts, never during it.
5. Run a fixed communication cadence for the duration of the incident — hourly updates, work in capped blocks — and keep a timeline as it happens rather than reconstructing it afterward.
6. Close the loop on the tool or process that caused or missed the failure — the audit script, the missing canary step, the alert that didn't fire — so the same failure mode can't repeat undetected.
7. Feed the incident into monthly ticket and backlog metrics so a pattern of recurring incidents shows up as a staffing or automation gap, not twelve unrelated one-offs.

Tools & methods

• Monitoring/NOC stack: SNMP polling, NetFlow/sFlow, a platform like LibreNMS/PRTG/SolarWinds for hardware, link, and server integrity; BGP/route monitors for peering-session state.
• Network automation and config management (Ansible, NAPALM) for staged, auditable pushes instead of hand-typed changes on individual devices.
• IPAM/documentation of record (NetBox or equivalent) kept current, not reconstructed after an incident.
• Vulnerability scanning: an ASV-accredited scanner for the quarterly PCI-scope scan (internal-only or unaccredited tools do not satisfy the control); Tenable/Qualys-class tooling for the rest of the fleet.
• Secrets/credential management (Vault or equivalent) instead of shared static passwords on network gear.
• Change management/CAB ticket as the artifact of record for any fleet-wide push, with the canary/bake/rollback plan written into it before approval.

Communication style

To peers and other admins: precise and log-referenced — cites the specific interface, session, or log line rather than describing symptoms in general terms. To leadership during an incident: states current impact, next update time, and the single next action, on the hourly cadence, not a technical narrative. To other functions requesting a change (security, app teams): states the deadline that's actually driving the timeline (compliance SLA vs. convenience) so the request doesn't get compressed by an assumed urgency that isn't real, and states what the canary/bake plan costs in elapsed time up front rather than promising the fastest possible number.

Common failure modes

• Trusting automation's own success signal instead of an independent health check — Ansible/NAPALM reports "changed" or exits 0 across the fleet, but a subset of devices never actually came back up with the service running; the tool ran, it just didn't verify the outcome it was supposed to produce.
• Watching the bake window only for the metrics named in the rollback trigger — BGP/OSPF reconvergence and VPN reconnect rate both hold clean, but the firmware push regressed something the two named signals don't cover (e.g., a split-tunnel routing behavior), so a clean bake gets read as a clean change outside the metrics actually being watched.
• Hand-patching one device during an emergency without updating the config-management source of truth — the live device and the Ansible/NetBox record of it diverge, so the next fleet-wide push either reverts the emergency fix or fails against a device that no longer matches its recorded state.
• Rolling back a bad stage correctly, then letting the deferred remediation quietly drop off the backlog — instead of re-scheduling the fixed version with an owner and a date, so the underlying vulnerability or deadline reopens with no one visibly on the hook for it.
• Confusing a passing quarterly scan with current safety — treating the last ASV scan as sufficient coverage for a CVE that landed in the KEV catalog the following week.

Worked example

Setup. The quarterly PCI ASV scan flags CVE-2025-1974 (illustrative CVE ID) on the DMZ-facing VPN concentrator firmware, CVSS 9.8. Three days later the CVE is added to the CISA KEV catalog, which sets a 14-calendar-day remediation deadline because the device is internet-facing. The vendor's next scheduled firmware release is 5 weeks out. 40 regional-office VPN concentrators run the same vulnerable firmware version.

Naive read. CVSS 9.8, on the KEV list, deadline running — patch all 40 concentrators in one maintenance window tonight to close the exposure as fast as possible.

Expert reasoning. A synchronous fleet-wide firmware push has no canary and no rollback signal before commitment; if the new image has a routing or VPN-negotiation defect, every regional office loses its WAN/VPN path at the same time, and the admin VPN used to reach the concentrators for remote recovery runs over the same devices being flashed — the same out-of-band dependency problem that trapped Meta's responders behind their own badge system. 14 days is enough runway to stage this instead of rushing it:

• Day 1: flash the 2 lowest-business-impact regional offices (canary). Confirm a separate console-server path into those two sites that doesn't depend on the concentrator's VPN service.
• Day 1-2: 4-hour bake window checking BGP/OSPF neighbor stability and VPN client reconnect success rate on the canary pair. Clean.
• Day 3: batch 2 — 19 concentrators. 24-hour bake window. Clean.
• Day 5: batch 3 — the remaining 19 concentrators (2 + 19 + 19 = 40).

Total elapsed time: 5 days, leaving 9 days of margin inside the 14-day KEV deadline for a reschedule if any batch shows a problem, instead of zero margin from a single all-at-once push.

Deliverable — change ticket, filed before batch 1:

CHG-4471: VPN concentrator firmware remediation — CVE-2025-1974 (KEV, deadline day 14) Scope: 40 DMZ VPN concentrators, firmware X.Y.Z → X.Y.Z+1. Rollout: Day 1 canary (2 lowest-impact sites, console-server access confirmed independent of VPN path) → 4hr bake → Day 3 batch of 19 → 24hr bake → Day 5 batch of remaining 19. Rollback trigger: any canary or batch site failing BGP/OSPF reconvergence within the bake window, or VPN reconnect success rate dropping below 98% of the pre-change baseline. Deadline: Day 14 (CISA BOD 22-01, internet-facing). Current plan completes Day 5, 9 days of margin retained.

Going deeper

• Playbook — staged-rollout and incident-response sequencing, backup/monitoring review checklist, and circuit/capacity planning lead times.
• Red flags — smell tests for network and access-control incidents: what each usually means, the first question to ask, the data to pull.
• Vocabulary — terms generalists misuse, with practitioner usage and the common misuse spelled out.

Sources

• Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup, The Practice of System and Network Administration, 2nd ed. (Addison-Wesley, 2007) — data-center move cooling/lead-time practice, Incident Command-style outage response, ticket opened:closed ratio as a staffing signal, RFC 1918 overlap check during mergers.
• Meta Engineering, "More details about the October 4 outage" (engineering.fb.com, Oct 5, 2021), and Cloudflare, "Understanding how Facebook disappeared from the Internet" (blog.cloudflare.com, Oct 2021) — backbone-audit-tool root cause, BGP withdrawal timeline, badge-system lockout.
• Cloudflare, "Details of the Cloudflare outage on July 2, 2019" (blog.cloudflare.com) — global synchronous WAF rollout, catastrophic-backtracking regex, 27-minute outage, delayed kill switch.
• Google, Site Reliability Engineering and The Site Reliability Workbook (sre.google) — toil cap, error-budget mechanic.
• NIST Special Publication 800-63B, Digital Identity Guidelines (Rev. 4, pages.nist.gov/800-63-4) — password rotation on compromise only, 64-character minimum, no composition rules.
• PCI Security Standards Council, PCI DSS Requirement 11.3.2 — quarterly ASV-scan requirement for internet-facing cardholder-data-environment systems.
• CISA Binding Operational Directive 22-01 (Known Exploited Vulnerabilities Catalog) — 14-day (internet-facing) / 45-day remediation deadlines.
• O*NET-SOC 15-1244.00 task list used only as a coverage skeleton, not as source text.
• Pass 0 research complete as of 2026; no direct practitioner sign-off yet — flag via PR if you can confirm, correct, or add a citation. The CVE ID in the worked example is illustrative, not a claim about a real vulnerability.