DevOps Storage and Backup
Agent Workflow (MANDATORY)
Before ANY storage or backup change, spawn in parallel, one spawn_agent call each:
- explore-codebase - find existing backup scripts, timers, pool/dataset layout files, Velero manifests, inventories
- research-expert - verify the flag or option for the installed version at openzfs.github.io, docs.ceph.com, restic.readthedocs.io, borgbackup.readthedocs.io, velero.io
- mcp__context7__query-docs -
/websites/openzfs_github_io_openzfs-docs,/ceph/ceph,/restic/restic,/websites/borgbackup_readthedocs_io_en_stable,/velero-io/velero
Then read the system read-only (zpool status, zfs list, ceph -s, restic snapshots, velero backup get). After file changes run sniper.
Versions (checked 2026-11-10)
OpenZFS zfs-2.4.4, Ceph 21.2.0 / 20.2.5 / 19.2.7, restic 0.19.1, Borg 1.4.5, borgmatic 2.2.0, Velero 1.18.4, rclone 1.76.0, Rook 1.21.2, Longhorn 1.13.0 (GitHub release pages and docs.ceph.com/releases, read today). Proxmox VE 9.2 ships Ceph Squid and Tentacle, not Umbrella: do not recommend Umbrella for PVE (devops-proxmox). Always compare with the installed version before using a flag.
Pinned versions:
openzfs: "2.4.4 (GitHub release zfs-2.4.4, 2026-08-21)"
ceph: "21.2.0 Umbrella (2026-10-28), 20.2.5 Tentacle, 19.2.7 Squid (EOL 2026-12-02)"
restic: "0.19.1 (2026-07-05)"
borgbackup: "1.4.5 stable (Borg 2 is in beta)"
borgmatic: "2.2.0"
velero: "1.18.4 (repository velero-io/velero)"
rclone: "1.76.0"
rook: "1.21.2"
longhorn: "1.13.0"
seaweedfs: "4.53 (Apache-2.0)"
minio: "community repository no longer maintained"
Overview
| Area | Decision in one line |
|---|---|
| Type | Block for one VM/DB, file for shared POSIX, object for blobs and backups (storage-selection.md) |
| ZFS | Mirrors or raidz2 vdevs, ashift fixed at vdev creation, scrub monthly, snapshots are not backups |
| Ceph | 3 MON, replicated size 3 / min_size 2, failure domain host, never run above 85 percent |
| Backup | 3-2-1-1-0 policy, one immutable or offline copy, encryption on, restore test on a schedule |
| Tools | restic or Borg for hosts, Velero for Kubernetes objects, ZFS send for datasets, PBS for Proxmox guests |
| Proof | A backup that was never restored is a hope, not a backup (restore-testing-dr.md) |
Critical Rules
- Read-only first -
zpool status,zfs list -t all,ceph -s,restic snapshots,velero backup getbefore any change - Dry-run before every destroy -
zfs destroy -nv,restic forget --dry-run,borg prune --dry-run,ceph orch apply ... --dry-run - Destructive commands are proposed, not run - ACTION / EFFECT / PROOF / WAIT-FOR-GO (zfs-destructive-ops.md, ceph-destructive-ops.md, backup-destructive-ops.md)
- A snapshot on the same pool is not a backup - it dies with the pool; the second copy sits on separate hardware
- Never print repository passwords, keys or S3 secrets - files with mode 0600,
--password-file, systemd credentials; names only in output - Restore test before trust - log the evidence (date, source, target, checksum, duration) per run
- Never invent a flag - cite the page; unknown means "not verified"
Architecture
/etc/backup/
├── restic.env # RESTIC_REPOSITORY, RESTIC_PASSWORD_FILE (mode 0600, root)
├── restic.pass # repository password file, never in Git
└── excludes.txt
/usr/local/sbin/restic-backup.sh # nightly backup + check subset
/usr/local/sbin/restic-maintenance.sh # weekly forget --dry-run, prune after go
/etc/systemd/system/restic-backup.{service,timer}
Reference Guide
Concepts
| Topic | Reference | When to Consult |
|---|---|---|
| Storage type | storage-selection.md | Block/file/object, local/shared/distributed |
| NFS and iSCSI | nfs-iscsi.md | Exports, mount options, LUN basics |
| Object storage | object-storage-s3.md | MinIO status, Garage, SeaweedFS, Ceph RGW |
| Kubernetes storage | kubernetes-storage-pointers.md | CSI, Rook-Ceph, Longhorn |
| ZFS pools | zfs-pools-datasets.md | vdev layouts, ashift, ARC, datasets |
| ZFS snapshots | zfs-snapshots-replication.md | Snapshots, holds, send/receive, scrub |
| ZFS destructive | zfs-destructive-ops.md | destroy, rollback, receive -F, redundancy changes |
| Ceph architecture | ceph-architecture-deploy.md | Daemons, pools, CRUSH, cephadm |
| Ceph operations | ceph-operations-health.md | Health states, capacity, OSD replacement |
| Ceph destructive | ceph-destructive-ops.md | osd purge/destroy, pool delete |
| Strategy | backup-strategy.md | 3-2-1-1-0, RPO/RTO, retention |
| restic | restic.md | Host and S3/SFTP backups |
| Borg | borg-borgmatic.md | Borg repos, borgmatic config |
| Velero | velero-kubernetes-backup.md | Cluster object and volume backup |
| rclone / PBS | rclone-pbs-pointers.md | Transport, Proxmox Backup Server |
| Backup destructive | backup-destructive-ops.md | forget --prune, borg prune/compact/delete, velero delete |
| Restore and DR | restore-testing-dr.md | Restore tests, DR drill |
| Capacity and SMART | capacity-smart-monitoring.md | Fullness, SMART, alerts |
Templates
| Template | When to Use |
|---|---|
| restic-backup.sh | Nightly restic backup with check subset |
| restic-backup.service / .timer | systemd units for the script |
| restic-maintenance.sh | Retention preview, prune only with --apply |
| borgmatic-config.yaml | Borg via borgmatic with retention and checks |
| zfs-snapshot-replicate.sh | Snapshot + incremental send with dry-run |
| zfs-snapshot-policy.md | Snapshot/retention policy per dataset class |
| velero-schedule.yaml | Scheduled Kubernetes backup |
| restore-test-checklist.md | Evidence sheet for every restore test |
| dr-runbook.md | Disaster-recovery runbook skeleton |
| backup-policy-table.md | Per-system RPO/RTO/retention table |
Quick Reference
zpool status -v; zfs list -t all -o name,used,avail,refer # ZFS state, read-only
zfs destroy -nv pool/ds@snap # dry run, deletes nothing
ceph -s; ceph health detail; ceph osd df # Ceph state, read-only
restic -r "$REPO" snapshots; restic -r "$REPO" check # repository state
restic -r "$REPO" forget --keep-daily 7 --dry-run # preview, removes nothing
borg prune -v --list --dry-run --keep-daily=7 "$REPO" # preview, removes nothing
velero backup get; velero restore create --from-backup NAME # Kubernetes
Best Practices
DO
- Keep at least one copy off-site and one copy the production credentials cannot delete (append-only, object lock or offline)
- Alert on "no successful backup in N hours", not only on job failure
- Pin tool versions in the backup host image and test restores after every upgrade
- Record every restore test in the checklist template
DON'T
- Treat RAID, ZFS mirrors, Ceph replicas or snapshots as backups
- Run
restic forget --prune,borg pruneorzfs destroyfrom cron without a retention review - Fill a pool or Ceph cluster above 80-85 percent
- Put repository passwords on a command line or in Git
Related skills
- devops-linux
- devops-databases
- devops-proxmox
- devops-kubernetes
- devops-k8s-cluster
- devops-openstack
- devops-observability
- devops-secrets
- devops-sre-ops