# Linux Config Management

> Use when managing repeatable server state with Ansible, etckeeper, check mode, or drift remediation across Debian/Ubuntu and RHEL families. Use linux-server-provisioning for one-time bootstrap and linux-cloud-init for first boot.

- Skill: `peterbamuhigire/linux-config-management` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add peterbamuhigire/linux-config-management`
- Raw SKILL.md: https://api.skillmd.com/api/skills/peterbamuhigire/linux-config-management/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- License: MIT
- Author: peterbamuhigire (https://skillmd.com/u/peterbamuhigire)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/peterbamuhigire/linux-config-management

---


# Linux Configuration Management

## Distro support

Ansible and `/etc`-tracking are **cross-platform**; write playbooks that work
on both families. Body uses Debian/Ubuntu; the **RHEL family** (Fedora, RHEL,
CentOS Stream, Rocky, Alma, Oracle) equivalents are in the matrix.

| Concept | Debian/Ubuntu | RHEL family |
|---|---|---|
| Ansible package module | `ansible.builtin.apt` | `ansible.builtin.dnf` |
| Portable package module | `ansible.builtin.package` (works on both) | same |
| Mandatory access control | **AppArmor** | **SELinux** (enforcing by default) |
| /etc change tracking | `etckeeper` | `etckeeper` (same) |
| Service manager | systemd | systemd (identical) |
| Gather pkg-mgr fact | `ansible_facts.pkg_mgr` = `apt` | = `dnf` |

**RHEL-family gotchas:** prefer `ansible.builtin.package` (or branch on
`ansible_facts['os_family']` / `pkg_mgr`) so one playbook runs on both. RHEL
enforces **SELinux** — Ansible config that writes files into service paths must
also manage SELinux contexts/booleans (`ansible.posix.seboolean`,
`community.general.sefcontext`); the equivalent on Debian is AppArmor profiles.
Deep SELinux coverage is in `linux-server-hardening` (Phase 2).

In `sk-*` scripts use the `common.sh` primitives instead of hardcoding apt/dnf.
See [`linux-bash-scripting`](../../10-automation-and-scripting/linux-bash-scripting/SKILL.md) and
[`docs/multi-distro/plan.md`](../../docs/multi-distro/plan.md).

<!-- dual-compat-start -->
## Use when

- Converting manual server state into repeatable Ansible or git-tracked configuration.
- Checking for configuration drift or validating idempotency.
- Establishing a safer operating model for `/etc` and other managed assets.

## Do not use when

- The task is a one-off emergency fix where full automation is unnecessary.
- The task is container or virtualization lifecycle work; use `linux-virtualization`.

## Required inputs

| Artefact | Source | Required? | If absent |
|---|---|---|---|
| Inventory and target scope | Operator or automation repository | required | Stop before running a playbook; report the missing host boundary. |
| Playbook, role, or tracked `/etc` baseline | Version-controlled source of truth | required | Produce an adoption plan, not a drift verdict. |
| Change authority, maintenance constraints, and secret source | Change owner | required for mutation | Stay read-only and limit work to check mode and diffs. |

## Workflow

1. Establish the declared state source: playbooks, inventory, or tracked config.
2. Inspect the current live state and compare it to the declared baseline.
3. Run the matching workflow below for adoption, drift checks, or remediation.
4. Verify idempotency and capture what changed versus what remains unmanaged.
5. Stop if inventory scope, authority, secrets handling, check-mode risk, or rollback is unresolved.
6. Recover a partial run by restoring the recorded prior state, limiting the next run to the failed cohort, and rechecking idempotency.

## Quality standards

- Prefer declarative, reviewable state over manual snowflake fixes.
- Make drift visible before changing production.
- Leave a clear path for repeatable re-application.

## Anti-patterns

- Using Ansible as a wrapper for ad-hoc shell. Fix: use idempotent modules and declare the end state.
- Applying changes without check mode and diff. Fix: preview supported tasks and explain unavoidable check-mode gaps.
- Tracking `/etc` without exclusions. Fix: exclude secrets, machine identity, caches, and generated state.
- Hard-coding `apt`, `apache2`, or the `sudo` group. Fix: use facts, portable modules, and family mappings.
- Calling one clean run idempotent. Fix: run again and require zero unintended changes.

## Outputs

| Artefact | Consumer | Acceptance condition |
|---|---|---|
| Declared-state change or drift report | Infrastructure maintainer | Every difference maps to a managed resource, accepted exception, or remediation. |
| Execution evidence | Reviewer | Includes syntax check, check-mode/diff result, applied task recap, and second-run idempotency result. |
| Recovery note | On-call operator | Names the rollback source and manual recovery for a failed partial run. |

## References

- [`references/ansible-patterns.md`](references/ansible-patterns.md)
- [`references/drift-detection.md`](references/drift-detection.md)
- [`references/idempotency-guide.md`](references/idempotency-guide.md)

## Evidence Produced

| Artefact | Acceptance condition |
|---|---|
| Configuration-management evidence | Includes inventory scope, syntax result, check-mode diff, task recap, second-run change count, and accepted drift exceptions. |

## Capability contract

Read/search access to inventory and automation is required. Execute check mode before mutation when supported. Editing playbooks or changing hosts requires explicit authority; secret values must remain redacted and production fan-out must respect the approved batch size.

## Degraded mode

Without host execution, review the automation statically and mark drift and idempotency `not assessed`. Without write authority, return a patch plan and safe validation sequence.

## Decision rules

| Choice | Action | Failure or risk avoided |
|---|---|---|
| Repeated fleet state | Manage with Ansible roles/playbooks | Snowflake hosts. |
| Emergency one-host repair | Apply the narrow fix, then back-port declared state | Permanent unmanaged drift. |
| Unknown live-versus-declared difference | Run check mode and diff first | Overwriting legitimate local state. |

## Worked example

When Apache is manually tuned on one AlmaLinux host, capture the intended directive in the role, run syntax and check mode against one canary, apply it, confirm `httpd` health, and run the playbook again expecting no change before widening the batch.

<!-- dual-compat-end -->

**This skill is self-contained.** Every command below is a standard tool
(`ansible`, `ansible-playbook`, `git`, `etckeeper`, `dpkg`, `diff`). The
`sk-*` scripts in the **Optional fast path** section are convenience
wrappers — never required.

This skill owns the **modern sysadmin practice** of keeping a server's
actual state aligned with a declared state that lives in git. It
complements `linux-server-provisioning` (initial build) with ongoing
continuous alignment.

Informed by *Linux System Administration for the 2020s*: move from "pet
servers we hand-tune" to "cattle we rebuild from code." This skill
starts that journey without requiring a full immutable-infrastructure
rewrite.

It does **not** own:

- **Initial provisioning** — `linux-server-provisioning`.
- **First-boot from YAML** — `linux-cloud-init`.
- **Application deployment** — `linux-site-deployment`.

---

## When to use

- Adopting Ansible for a server that was hand-configured.
- Detecting configuration drift between declared state and actual state.
- Dry-running a playbook against localhost before committing.
- Tracking `/etc` in git for change auditing.
- Deciding "should we rebuild or patch this server?" (this skill is the
  answer).

## When NOT to use

- A one-off manual config change on a throwaway machine.
- Tasks owned by another skill (backups, firewall, deployment).

---

## Standing rules

1. **Every config change is recorded.** If it's not in a git-tracked
   playbook or a committed `/etc` snapshot, it didn't happen.
2. **Idempotency is the ultimate test.** Every Ansible task must be safe
   to run twice. Second run = zero changes.
3. **Drift is a first-class alert.** Run a drift check on weekly cron;
   any drift emails the operator. Unexpected drift means either the
   playbook is incomplete or someone edited the server directly — both
   are bugs.
4. **No manual edits on production.** If you SSH in and `nano`, you
   create a snowflake. Make the change in the playbook, test in
   staging, roll forward.
5. **Secrets never live in plain-text Ansible.** Use Ansible Vault or
   `sops` — see `linux-secrets`.
6. **Every playbook has a rollback plan.** At minimum, the previous git
   commit. For risky changes, a pre-run `/etc` snapshot.

---

## Quick reference — manual commands

### Install Ansible

```bash
# Preferred on both families: pipx (isolates pinned version)
sudo apt install pipx          # Debian/Ubuntu
sudo dnf install pipx          # RHEL family (Fedora, RHEL, Rocky, Alma, ...)
pipx install --include-deps ansible
pipx ensurepath

# Alternative: distro package (older version but packaged)
sudo apt install ansible       # Debian/Ubuntu
sudo dnf install ansible       # RHEL family

# Verify
ansible --version
```

### Local-only Ansible (managed host is localhost)

```bash
# Inventory-less run
ansible-playbook -c local -i 'localhost,' playbook.yml

# Or a minimal inventory file
cat > inventory.ini <<'EOF'
[local]
localhost ansible_connection=local
EOF

# Dry run with diff
ansible-playbook -i inventory.ini -c local playbook.yml --check --diff

# Real run
ansible-playbook -i inventory.ini -c local playbook.yml

# Idempotency test — run twice; second must show 0 changes
ansible-playbook -i inventory.ini -c local playbook.yml
ansible-playbook -i inventory.ini -c local playbook.yml   # MUST show "changed=0"
```

### Check mode tags and facts

```bash
# Run only tagged tasks
ansible-playbook playbook.yml --tags ssh --check --diff

# Skip specific tags
ansible-playbook playbook.yml --skip-tags slow --check

# Gather facts only (no changes)
ansible localhost -c local -m setup
ansible localhost -c local -m setup -a 'filter=ansible_distribution*'
```

### Tracking /etc in git (etckeeper)

```bash
sudo apt install etckeeper
# On install, etckeeper inits a git repo at /etc/.git and commits
sudo etckeeper vcs log | head
sudo etckeeper vcs status

# After a planned change
sudo etckeeper commit "hardening: set PermitRootLogin no"

# Show what changed between two points
sudo etckeeper vcs diff HEAD~5..HEAD
```

Manual git approach (without etckeeper):

```bash
sudo git -C /etc init
sudo git -C /etc add .
sudo git -C /etc commit -m "baseline snapshot $(date -Iseconds)"

# Later — see uncommitted drift
sudo git -C /etc status
sudo git -C /etc diff
```

### Comparing declared vs actual

```bash
# Package list drift
dpkg --get-selections > /tmp/pkgs-actual.txt
diff /root/declared-packages.txt /tmp/pkgs-actual.txt

# Config file drift — diff against a committed baseline
sudo git -C /etc diff HEAD -- ssh/sshd_config
sudo git -C /etc diff HEAD -- sysctl.conf sysctl.d/

# Ansible check mode as drift detection
ansible-playbook -i inventory.ini -c local site.yml --check --diff
# Any "changed: <n>" with n > 0 = drift
```

Full Ansible idioms (idempotent module usage, `creates:`/`removes:`,
`changed_when:`, handlers, roles, templates, Jinja2 filters, 3 complete
playbook examples for linux-skills bases) — see
[`references/ansible-patterns.md`](references/ansible-patterns.md).

Full drift detection strategy (etckeeper vs raw git, Ansible check mode,
AIDE integration, alerting, remediation workflow) — see
[`references/drift-detection.md`](references/drift-detection.md).

Full idempotency guide (common mistakes, how to fix them, two-run test,
CI enforcement) — see
[`references/idempotency-guide.md`](references/idempotency-guide.md).

---

## Typical workflows

### Workflow: Adopting Ansible on an existing server

```bash
# 1. Baseline /etc
sudo apt install etckeeper
sudo etckeeper init
sudo etckeeper commit "baseline for Ansible adoption"

# 2. Write the first playbook — pick something simple (SSH hardening,
#    sysctl, unattended-upgrades). Keep it idempotent.
mkdir -p ~/ansible/playbooks
nano ~/ansible/playbooks/ssh-hardening.yml

# 3. Dry run
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml --check --diff

# 4. If the diff matches intent, apply for real
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml

# 5. Run AGAIN to prove idempotency
ansible-playbook -c local -i 'localhost,' \
    ~/ansible/playbooks/ssh-hardening.yml
# Expected: "changed=0"

# 6. Commit to git and schedule drift check
cd ~/ansible && git init && git add . && git commit -m "initial"
```

### Workflow: Weekly drift check

```bash
# Via etckeeper
sudo etckeeper vcs status                     # anything uncommitted?
sudo etckeeper vcs diff                       # what's drifted

# Via Ansible check mode
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml \
    --check --diff --one-line | tee /tmp/drift.log

# If changed>0, investigate
grep "changed=" /tmp/drift.log
```

### Workflow: "Is my playbook idempotent?"

```bash
# First run: expect changes
ansible-playbook -c local -i 'localhost,' playbook.yml
# Output: "changed=7"

# Immediate second run: MUST be zero
ansible-playbook -c local -i 'localhost,' playbook.yml
# Output: "changed=0"

# If the second run shows changes, the playbook is buggy.
# Common causes: see references/idempotency-guide.md
```

### Workflow: Drift remediation

```bash
# 1. Drift detected on /etc/ssh/sshd_config
sudo git -C /etc diff HEAD -- ssh/sshd_config

# 2. Decide: is the drift legitimate?
#    - Yes → update the Ansible playbook to match new state, commit
#    - No  → re-run the playbook to restore, commit a note

# 3a. If legitimate, update playbook and re-test
vim ~/ansible/roles/ssh/tasks/main.yml
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml --check --diff

# 3b. If unwanted, restore from declared state
ansible-playbook -c local -i 'localhost,' ~/ansible/site.yml --tags ssh

# 4. Commit the resolution
sudo etckeeper commit "drift remediation: ssh config restored to baseline"
```

---

## Troubleshooting / gotchas

- **Playbook is "working" but second run shows changes.** Almost always
  one of: `shell:`/`command:` without `creates:` or `changed_when:`,
  `lineinfile` with a regex that matches multiple lines, `copy` with
  content that renders differently each time (timestamps). See
  `references/idempotency-guide.md`.
- **`ansible-playbook --check` doesn't catch everything.** Some modules
  (notably `command`/`shell`) are conservative in check mode. They
  report "would run" regardless of actual state. Use `creates:` /
  `removes:` for stronger check-mode accuracy.
- **`etckeeper` fails on a large commit.** The initial commit can be
  huge. If git complains, configure `http.postBuffer` or commit in
  stages.
- **`local_action` with `become: yes` prompts for a password.** Use
  `ansible_become_password` from Vault, or run the playbook with
  `--ask-become-pass`.
- **Ansible installed via `apt` is too old for a module you need.**
  Switch to `pipx install ansible` for the latest, or enable the
  Ansible PPA.
- **Handlers don't fire on a failed task.** If a task fails
  mid-playbook, its `notify:` handlers are skipped. Use `force_handlers:
  yes` in the play if you need them to run anyway.

---

## References

- [`references/ansible-patterns.md`](references/ansible-patterns.md) —
  full Ansible reference: idempotent modules, roles, templates, Jinja2,
  3 complete playbook examples.
- [`references/drift-detection.md`](references/drift-detection.md) —
  etckeeper, Ansible check mode, AIDE, cron-scheduled checks,
  remediation workflow.
- [`references/idempotency-guide.md`](references/idempotency-guide.md) —
  common idempotency mistakes, how to fix them, CI enforcement.
- Book: *Linux System Administration for the 2020s* — idempotency,
  drift, cattle not pets.
- Ansible docs: https://docs.ansible.com/

---

## Optional fast path (when sk-* scripts are installed)

Running `sudo install-skills-bin linux-config-management` installs:

| Task | Fast-path script |
|---|---|
| Compare key configs/packages vs git-tracked state | `sudo sk-drift-check` |
| Run an Ansible playbook in check mode with clean summary | `sudo sk-ansible-dry-run --playbook <file>` |
| Initialize and verify /etc tracking, stage + commit | `sudo sk-etc-track [--commit]` |

These are optional wrappers around `ansible-playbook`, `etckeeper`, and
`git`.

## Scripts

This skill installs the following scripts to `/usr/local/bin/`. To install:

```bash
sudo install-skills-bin linux-config-management
```

| Script | Source | Core? | Purpose |
|---|---|---|---|
| sk-drift-check | scripts/sk-drift-check.sh | no | Compare key config files and package list against git-tracked declared state; report drift with diffs. |
| sk-ansible-dry-run | scripts/sk-ansible-dry-run.sh | no | Run an Ansible playbook in `--check --diff` mode against localhost with a clean summary of would-be changes. |
| sk-etc-track | scripts/sk-etc-track.sh | no | Initialize git tracking for `/etc`, verify it's clean, optionally auto-stage + commit. |