# Rwo Volume Maintenance

> Use when a job does maintenance on a single-writer (ReadWriteOnce) volume or live database that a running service still owns — pruning, compaction, retention, vacuum. Prefer snapshot/offline processing and bound the job generously so it cannot starve itself.

- Skill: `selamy-labs/rwo-volume-maintenance` (Agent Skill)
- Install (CLI): `npx skillmds@latest add selamy-labs/rwo-volume-maintenance`
- Raw SKILL.md: https://api.skillmd.com/api/skills/selamy-labs/rwo-volume-maintenance/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: selamy-labs (https://skillmd.com/u/selamy-labs)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/selamy-labs/rwo-volume-maintenance

---


# RWO Volume Maintenance

Maintenance jobs that shrink or tidy state — log/retention pruning, DB
compaction/VACUUM, snapshot trimming — often must run against a volume the live
service still owns. On a `ReadWriteOnce` PersistentVolume or a single-writer
database, the maintenance job and the live writer **contend** for the same
resource. That contention creates a self-reinforcing trap.

## The self-defeating deadline trap

Give such a job a **too-tight** time bound (`activeDeadlineSeconds`, lock
timeout, statement timeout) and you get a vicious cycle:

1. The store grows because maintenance hasn't run.
2. The larger the store, the longer maintenance takes.
3. It now exceeds its tight deadline and is killed before finishing.
4. The store keeps growing → next run is even slower → killed again.

**The job that is supposed to shrink the store is the one that times out** — and
it gets worse every cycle. The store grows unbounded while the maintenance job
shows a steady stream of "deadline exceeded".

## Two fixes, apply both

### Bound generously, sized to the worst case

Set the deadline to the realistic worst-case runtime on the **largest** the
store is allowed to get, with margin — not the time on a fresh, small store.
A maintenance deadline is a budget for the heaviest run, not the lightest. Add
an alert if a run approaches the budget, so you tune it before it starts failing.

### Prefer offline / snapshot processing — don't contend on the live volume

Better than racing the live writer: don't race it.

- **Snapshot, then process the copy.** Take a volume snapshot or clone, mount it
  to the maintenance job, do the heavy work off the live path. The live service
  is never blocked; the job has no deadline pressure from contention.
- **Compact offline** where the engine supports it (e.g. dump/reload, or an
  offline compaction tool) instead of online vacuum that fights live traffic.
- **Stream/export, transform elsewhere, swap back** when the store can tolerate
  a brief swap.

Offline/snapshot processing removes the contention that made the tight deadline
fatal in the first place — which is why it is the real fix, and a generous
deadline is the safety net.

## Checklist

- [ ] Deadline sized to the worst-case store size, with margin and an alert.
- [ ] Heavy maintenance runs against a **snapshot/clone**, not the live volume.
- [ ] If online is unavoidable, the job is idempotent and resumable so a kill
      mid-run doesn't corrupt or lose progress.
- [ ] Store growth is monitored, so maintenance falling behind is visible before
      it becomes unrecoverable.

## SQLite on agent state volumes

For SQLite state on an RWO volume, treat compaction as offline maintenance:

1. Capture `PRAGMA integrity_check`, `page_count`, `freelist_count`, and
   `page_size` before changing the database.
2. Stop the sole writer before `VACUUM`. Scaling a StatefulSet to zero for a
   short maintenance window is safer than compacting while the service runs.
3. Run `PRAGMA wal_checkpoint(TRUNCATE)` when the database uses WAL.
4. Run `VACUUM` only when free disk is comfortably larger than the expected
   compacted database size.
5. Run `PRAGMA integrity_check` again and record the before/after size.
6. Bring the writer back and run its health check.

If the volume shares a root filesystem with container images, compaction may be
the difference between a pod starting and a startup-health gate refusing writes.