Remote Systems Administration

Remote Systems Administration

Use this as an operating decision layer, not a bag of remote commands. Unix-like systems share a vocabulary but not an implementation. Identify the target's platform and active control plane before choosing a command.

Operating contract

1. Discover before changing. Record target identity, production status, OS and release, service manager, package manager, firewall, access route, privilege path, and current state. Do not infer them from hostname, memory, or inventory labels.
2. Use the smallest valid control plane. Native SSH for a bounded one-host task; Ansible for repeatable desired state across a fleet; Paramiko only when a Python program genuinely needs SSH protocol control that the first two cannot provide.
3. Protect access first. Before changing SSH, routing, DNS, a firewall, privilege escalation, or a network interface, establish a tested rollback and a second recovery path. Keep the current session alive until the new path works.
4. Preview, constrain, verify. Limit the target set; use native validation, dry-run, diff, or a canary when available; then verify the affected service and its user-visible boundary. A zero exit code proves only that command ran.
5. Report evidence, not a story. Preserve bounded per-host results: target, command category, before/after evidence, failures, rollback state, and the remaining uncertainty. Never paste secrets, keys, full configuration files, or unbounded logs into the response.

Run the shared classification probe in references/portable-operations.md, then load the matching OS/family overlay and run its command preflight. Select a mutation command only after the required preflight and safety gate.

Read-only discovery handoff

Use this compact format after a preflight. Fill a field only from observed evidence; otherwise write unknown or not supplied.

• Target and scope:
• Observed platform and control planes:
• Bounded evidence:
• Unknown or blocked:
• Next safe action:

First response: classify the job

Required preflight for every mutation

Before the first state-changing command, confirm:

• exact host(s), environment, and authorized scope;
• OS/release and applicable platform overlay;
• service manager, package manager, firewall implementation, and configuration owner;
• access identity, elevation method, and whether the connection traverses a bastion;
• intended state, expected blast radius, rollback command or artifact, and stop condition;
• validation at both the component layer and the relevant external boundary.

Connectivity/access pre-execution gate: explicitly name authorization, retained session, independent recovery path, tested rollback, stop condition, and component plus external-boundary verification. If any is unknown, stop before execution.

Read-only discovery may proceed without confirmation. Read-only means no persistent state: do not create rollback scripts or captures, stage update metadata, alter files, or call a state-changing operation preflight. Stop once the needed platform and control-plane evidence is established. Destructive actions, privilege changes, firewall/remote-access changes, package removals, storage operations, and reboot/shutdown require an explicit directive after this preflight.

Routing references

Platform boundary

Do not flatten platform differences:

• systemctl is not a BSD or macOS service manager.
• rcctl is OpenBSD-specific; FreeBSD uses rc scripts and service.
• launchctl domains and labels are not systemd units.
• apt, dnf, pacman, pkg, pkg_add, softwareupdate, and brew have different update, rollback, and package-origin semantics.
• Linux nftables, BSD PF, and the macOS Application Firewall are separate control planes. Never translate rules mechanically.

Scope boundaries

This skill covers host-level Unix operations. Route containers to docker-compose, clusters to kubernetes, encrypted tailnet policy to tailscale, detection/remediation components to crowdsec, and reliability process design to site-reliability-engineering.

It does not authorize credential recovery, security-boundary bypass, production deletion, provider-console operations, or an unreviewed operating-system upgrade.

Common pitfalls

1. Running the right command on the wrong platform. Discover the service/package/firewall manager before acting.
2. Treating an SSH connection as proof of authority. Connection success does not confirm sudo policy, host identity, or authorization for the change.
3. Using StrictHostKeyChecking=no or Paramiko AutoAddPolicy. An unknown host key is an identity event, not a convenience prompt.
4. Parallelizing first. Establish the procedure on one canary, then roll out in bounded batches with a stop condition.
5. Calling a service healthy because it started. Check process/unit state, logs, listening endpoint, and the relevant dependent or external boundary.
6. Confusing package upgrades with OS upgrades. Platform release lifecycle, kernel/base-system updates, and third-party packages have different procedures.
7. Repairing a remote host through an access-path change without a rollback. Preserve a live session and independent recovery channel.

Verification checklist

• Target and platform were discovered from the live host.
• Mutation had explicit scope, rollback, and recovery path.
• The control plane matched the target platform.
• Fleet work used an inventory, bounded concurrency, and a stop condition.
• Sensitive values and unbounded output were excluded from evidence.
• Verification covered the changed component and its relevant external boundary.
• Per-host success, failure, and rollback status are explicit.