Checkpoint Resume

Checkpoint Resume

Inputs

• $target: For a NEW run, a short label (used to build the run id). For a RESUME, the run id or the path to an existing .ulpi/runs/<id>.json.

Goal

Turn a long, multi-unit task into one that can be stopped and restarted at will — from any session, after any interruption — without losing finished work or redoing it. The status file is the durable- primary progress view; git-integrating pipelines reconcile it from their reachable trailer log.

Step 0: New run or resume? (decide FIRST)

• Resume if $target names an existing run id/file, or the user says "resume / continue". Do NOT re-initialize. Read the file, reconcile any git-backed integration state, and proceed to Step 3.
• New run otherwise. Create a fresh status file (Step 1) and proceed.

Getting this wrong is the whole failure mode: a "resume" that writes a fresh pending document throws away the checkpoint. When in doubt, check for the file first.

Success criteria: mode determined; on resume, the existing file is loaded, not clobbered.

Step 0.5: Use the bundled CLI — don't hand-roll the file operations

This skill ships scripts/checkpoint.mjs (zero-dependency Node). It implements the whole contract — atomic writes, refusal to clobber a live checkpoint, refusal to demote a done unit, fail-closed finalize, and the resume-set computation — so USE IT instead of hand-rolling jq:

node <skill-dir>/scripts/checkpoint.mjs init  <file> --task "<desc>" [--units "a,b,c"] [--id <id>] [--launch '<json>']
node <skill-dir>/scripts/checkpoint.mjs unit  <file> <unit> <pending|in_progress|done|blocked|dep_blocked> [--note "…"] [--deps "x,y"]
node <skill-dir>/scripts/checkpoint.mjs phase <file> <phase> <pending|running|done|blocked|skipped>
node <skill-dir>/scripts/checkpoint.mjs get   <file> --summary
node <skill-dir>/scripts/checkpoint.mjs resume <file>     # → { skip, eligible, dep_blocked }
node <skill-dir>/scripts/checkpoint.mjs item  <file> --json '<object-or-array>'   # append durable openItems
node <skill-dir>/scripts/checkpoint.mjs finalize <file> <done|needs_attention|aborted> [--result "…"]
node <skill-dir>/scripts/checkpoint.mjs gc    <runs-dir> [--keep-days 7]  # archive old TERMINAL runs

Append || true at call sites — status writes are non-fatal. The CLI exits 2 (refuses) on the contract-violating operations: re-init over a live checkpoint, demoting a done unit, and finalize done while units are open. Those refusals are the guardrails, enforced in code.

Large-run opt-in. Keep the single atomic JSON file as the default. For plans where rewriting the snapshot on every transition is too costly, import scripts/lib/event-log.mjs, call initializeEventLog(file, doc, { enabled: true }), then appendTransition(...). It fsyncs one JSONL transition before atomically replacing the same reader-compatible .json snapshot; rebuildSnapshot replays after a crash and discards only a torn final non-newline fragment. The event path is explicitly opt-in and does not change run-status.mjs or the ordinary checkpoint CLI contract.

Everything is timestamped (ISO-8601 UTC): the doc (createdAt/updatedAt/finishedAt), each unit (createdAt/updatedAt + startedAt/finishedAt), each phase (startedAt/updatedAt/finishedAt), and each register item (at) — so a reader can show real durations and "updated 3m ago", not guesses.

The typed launch descriptor. Pass init --launch '<json>' to persist the exact relaunch recipe in the file, so the run can be resumed from the status file alone (session-independent). It is a typed object { scriptPath, args? } — init validates it before touching disk (scriptPath must be a non-empty string; args, when present, a JSON object) and refuses an array/string/missing-scriptPath descriptor. To make a run RUNNABLE under the Codex coordinator, persist the coordinator recipe — a scriptPath whose basename is pipeline.mjs and args:{ command:"resume", run:"<id>" }:

node <skill-dir>/scripts/checkpoint.mjs init <file> --task "…" --units "a,b,c" --id my-run \
  --launch '{"scriptPath":"autonomous-pipeline/scripts/pipeline.mjs","args":{"command":"resume","run":"my-run"}}'

The pipeline coordinator's own approve stamps exactly this descriptor for you, so its runs are runnable out of the box. Any other descriptor (a Claude Workflow() script such as pipeline-workflow.js, or an absent/malformed one) is treated as MIGRATION-ONLY / non-runnable — see Step 0.6.

Step 0.6: Query a run the easy way — run-status.mjs (READ-ONLY)

To SEE where a run is (yours or one a pipeline left behind), use the bundled legible reader — it never writes, so it can't disturb a run in flight:

node <skill-dir>/scripts/run-status.mjs                 # newest run for this project, rendered
node <skill-dir>/scripts/run-status.mjs <id>            # a specific run (id prefix is enough)
node <skill-dir>/scripts/run-status.mjs --list          # every run, one line each, newest first
node <skill-dir>/scripts/run-status.mjs --json [id]     # the raw durable doc
node <skill-dir>/scripts/run-status.mjs --resume [id]   # print the Codex resume command for this run
node <skill-dir>/scripts/run-status.mjs --resume --json  # emit ONLY the typed resume descriptor (argv, no shell)

It auto-discovers .ulpi/runs/ by walking up from the cwd, renders phases + a per-task progress bar + the open findings register + a resume affordance. The default render is durable-primary, then adds a best-effort Live workflow overlay and a visible live-agent-vs-durable-unit divergence line. The overlay reads only external Claude journal.jsonl started/result envelopes; absence or format drift prints an honest no live workflow … use /workflows note and never changes durable status. The reader never opens agent transcripts, spawns Git, or writes. --resume classifies the persisted launch descriptor into exactly one of three resume recipes (it never fabricates a command):

• Runnable (codex-cli) — the persisted launch is the coordinator recipe (basename pipeline.mjs, args.command==="resume"). --resume prints the shell-safe Codex command node pipeline.mjs resume --run <id> (the id is a discrete, quoted argv token — never string- interpolated into a shell); --resume --json adds the typed { runnable:true, kind:"codex-cli", argv:[…], shell, resumeSet } descriptor.
• Legacy Workflow (legacy-workflow) — the launch is a Claude Workflow() script (e.g. pipeline-workflow.js). It is labeled MIGRATION-ONLY / non-runnable and is NEVER printed as a runnable shell command; the reader echoes the persisted {scriptPath,args} (with statusFile re-pinned to this run) for migration reference only. --resume --json → { runnable:false, migrationOnly:true, kind:"legacy-workflow" }.
• No / malformed launch (no-launch) — nothing runnable was persisted (pre-coordinator, hand-rolled, or a malformed descriptor): non-runnable, with the computed skip/eligible resumeSet so a human can relaunch via pipeline.mjs approve → pipeline.mjs start --run <id>.

Step 1: Initialize the status file (new run only)

Prefer checkpoint.mjs init. If you must write it by hand, pick a stable id: <label>-<UTC-timestamp> (get the timestamp from date -u +%Y%m%dT%H%M%SZ — never invent one). Write .ulpi/runs/<id>.json:

{
  "schemaVersion": 1,
  "id": "<id>",
  "task": "<one-line description>",
  "status": "running",
  "createdAt": "<UTC now>",
  "updatedAt": "<UTC now>",
  "units": {
    "<unit-id>": { "status": "pending", "dependsOn": [], "note": "", "createdAt": "<UTC now>", "updatedAt": "<UTC now>" }
  },
  "openItems": [],
  "result": null
}

(The CLI stamps createdAt/updatedAt on every unit and phase automatically — the fields above are what init writes; you never hand-maintain them.)

Enumerate the units up front when they're known (DAG tasks, files to migrate); for a discovery-driven run, start with units: {} and add them as they're found. Each unit's status moves pending → in_progress → done (or blocked / dep_blocked).

If the orchestrator that runs the work has no filesystem access (e.g. a sandboxed Workflow), the CALLER creates and updates this file — pass the absolute statusFile path in.

Success criteria: the file exists before any work starts, so a watcher sees the run immediately.

Step 2: Update as work lands (running)

After each unit reaches a terminal state, patch the file with the CLI (Step 0.5): node <skill-dir>/scripts/checkpoint.mjs unit <file> <unit> done || true — it owns locking, atomicity, and the refusals. The jq recipe below is the FALLBACK for environments without Node only; never prefer it when the CLI is available:

jq --arg u "<unit-id>" --arg s "done" --arg t "$(date -u +%Y%m%dT%H%M%SZ)" \
   '.units[$u].status=$s | .updatedAt=$t' .ulpi/runs/<id>.json > .ulpi/runs/<id>.json.tmp \
   && mv .ulpi/runs/<id>.json.tmp .ulpi/runs/<id>.json

Wrap writes so a failure is swallowed (|| true) — a status write must never abort the work. Update openItems with anything the run should carry forward (blocked units, findings), and status at the top level as phases progress.

Success criteria: at any instant, cat-ing the file shows the true current state; a crash here loses at most the in-flight unit.

Step 3: Resume — skip done, rebuild the rest

On resume, reconcile durable integration evidence, then compute the work set:

0. For a git-integrating pipeline, scan Task-Id: <id> trailers reachable from the integration ref (coordinator: pipeline.integrationRef; legacy Workflow: workingBranch). A reachable trailer recovers a lost status write into done; the coordinator stamps reconciled-from-trailer:<sha> and blocks a stale checkpoint done with no reachable commit. A commit only on task/<id> did not integrate and stays eligible. Use the existing done state — never invent a separate integrated state or claim the merge and status write form one transaction.

1. Load units. A unit is done → skip it entirely.

2. ANY unit not done — pending / in_progress / blocked / dep_blocked — whose dependsOn are all done → it's eligible; rebuild it. (in_progress was interrupted mid-flight — redo it. A dep_blocked unit whose dependency has since landed is eligible again — done is the ONLY state that skips.)

3. A unit whose dependency is NOT done → dep_blocked, pointing at the missing root; do not build it on a partial base.

Then run only the eligible set through the same machinery as the original run, writing back to the SAME file (same id/path). The durable skip-done makes this independent of any runtime cache — a template edit or a fresh session doesn't force a full rebuild.

For a coordinator-owned run you don't recompute this by hand — the Codex-native path drives it for you:

node autonomous-pipeline/scripts/pipeline.mjs resume --run <id>   # reads the checkpoint, skips done units

run-status.mjs --resume <id> prints exactly this command when the run's persisted launch is the coordinator recipe; the coordinator re-reads the durable checkpoint and re-runs only the eligible set.

Resume behavior — the five cases (make these explicit)

v1 → v2 migration (add-only, non-breaking)

New runs are written at schemaVersion: 2 (stable-id findings + durable resolvedItems + the typed launch). A schemaVersion: 1 run is fully supported: it loads, resumes and finalizes byte-for-byte unchanged on a READ. The ONLY change ever applied is on a mutating write, where the store does an idempotent, ADD-ONLY in-place upgrade — it adds the missing resolvedItems: [] and bumps schemaVersion — and NEVER rewrites existing units/phases/openItems/launch. An in-flight v1 run is therefore never broken by an upgrade; its finished units stay skipped and its open work stays open. A v1 run with no persisted launch simply resumes via the no-launch path above.

Success criteria: finished units are provably not redone; only the remaining/eligible set runs; a legacy/v1 record resumes without loss and without being shown a fabricated command.

Step 4: Finalize

When the work set is exhausted, write the terminal state: status = done (all units done) / needs_attention (some blocked/open) / aborted; populate result and the final openItems. This file is the run's durable record — tell the user where it lives.

If a run died without a final write, reconstruct the terminal state from the artifacts (git log, the units' own checks) rather than trusting a stale running — see references/status-schema.md.

Success criteria: the file reflects the real end state and enumerates anything still open.

Guardrails

• Never overwrite an existing run's file with a fresh pending document on resume.
• For git-integrating pipelines, reconcile reachable Task-Id trailers before computing the resume set; never make the live session journal a resume dependency.
• Never mark a unit done before it is complete AND verified.
• Never skip a unit whose dependency didn't actually land.
• Never let a status-write failure block, abort, or fail the underlying work.
• Never fabricate a timestamp — read the clock (date -u).
• Never treat a stale/running file as ground truth after a crash — reconstruct from artifacts.
• Keep writes atomic (tmp + mv) and incremental (jq patch, not full rewrite) to survive races.

When To Load References

• scripts/checkpoint.mjs The runnable implementation of this contract — init/unit/phase/get/resume/finalize with atomic writes, timestamped mutations, and code-enforced refusals. Prefer it over hand-rolled file operations, always.
• scripts/run-status.mjs The READ-ONLY legible reader (Step 0.6): renders the newest run (or <id>/--list/--json), and --resume classifies the persisted launch and prints the Codex-native coordinator resume command (node pipeline.mjs resume --run <id>) for a runnable run, or a migration-only/non-runnable pointer otherwise — never a fabricated command. Never writes — safe to run against a live run.
• references/status-schema.md The full status-file schema (per-unit states, dependsOn, openItems, phase blocks), the three verbs (status / stop / resume), atomic-write recipes, and how to rebuild the file from artifacts after a crash. Load when defining a run's schema or writing the resume logic.

Output Contract

Report:

1. run id + status-file path
2. new run vs resume; on resume, how many units were skipped-done vs rebuilt
3. final status (done / needs_attention / aborted) and where the durable record lives
4. any open/blocked units carried in openItems