Checkpoint Long Runs
A long job that keeps all its state in the conversation loses everything when the call times out. Write progress to disk after every unit, keyed by a resume marker, so work already done is never redone.
Procedure
- Design the resume marker before starting: a monotonically increasing index, a set of completed ids, or a cursor. The marker must let you reconstruct "what is left" from disk alone.
- Prefer append-only durability: after each item, append one line to
data/done.jsonl ({"id": "...", "ok": true}) rather than rewriting a big blob each time.
- On start (or restart), load the marker and skip completed work:
done=$(jq -r .id data/done.jsonl | sort -u).
- Process in bounded batches (e.g. 50 items) and flush after each batch, so at most one batch is at risk:
json.dump(state, open("data/state.json","w")) at the batch boundary.
- Make each item's processing idempotent (see
idempotent-operations) so a crash mid-item and a re-run cannot double the effect.
- Run the job as a tracked background process when it can outlive a single tool call, and poll its progress from the checkpoint file rather than blocking.
- On completion, verify the count:
wc -l data/done.jsonl should equal the expected total; a short count means a silent stop — investigate before declaring done.
- Keep the checkpoint and the log together so a resume is: load checkpoint, resume from marker, continue appending.
- Pick a checkpoint granularity that bounds loss: flush every 50 items or 30 seconds, whichever comes first.
- Store the expected total alongside the marker (
echo 5000 > data/total.txt) so completion is a comparison, not an assumption.
- Make the writer atomic: write to
data/state.json.tmp then mv over the target, so a crash mid-write cannot corrupt the checkpoint.
Pitfalls
- Accumulating all results in a Python list in memory, so a timeout loses the whole run.
- Rewriting the entire state file per item, which is slow and can corrupt on interruption.
- A resume that reprocesses everything because the marker was never written to disk.
- Marking an item done before its work succeeded, so a resume skips a failed item.
- Assuming the job finished because the loop ended, without counting the outputs.
- A checkpoint recording only the index but not the item, so a resume cannot tell whether the last item finished.
- Running two instances of the same resumable job, both appending, corrupting the done list.
Verification
total=$(wc -l < data/items.txt); done=$(wc -l < data/done.jsonl)
[ "$total" = "$done" ] && echo COMPLETE || echo "resume from $done"
# passes when done == total at the end, and a mid-run kill resumes at the marker
Report to the user: the checkpoint path, total vs completed counts, and evidence that a kill-and-resume continued rather than restarted.
1---2name: "0xklen-checkpoint-long-runs"3description: "Use when a job may exceed the step, time, or context budget: batch processing, migrations, or multi-page crawls. Persists progress after each unit so a timeout or crash resumes instead of restarting."4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/0xklen-checkpoint-long-runs6---78# Checkpoint Long Runs910A long job that keeps all its state in the conversation loses everything when the call times out. Write progress to disk after every unit, keyed by a resume marker, so work already done is never redone.1112## Procedure13141. Design the resume marker before starting: a monotonically increasing index, a set of completed ids, or a cursor. The marker must let you reconstruct "what is left" from disk alone.152. Prefer append-only durability: after each item, append one line to `data/done.jsonl` (`{"id": "...", "ok": true}`) rather than rewriting a big blob each time.163. On start (or restart), load the marker and skip completed work: `done=$(jq -r .id data/done.jsonl | sort -u)`.174. Process in bounded batches (e.g. 50 items) and flush after each batch, so at most one batch is at risk: `json.dump(state, open("data/state.json","w"))` at the batch boundary.185. Make each item's processing idempotent (see `idempotent-operations`) so a crash mid-item and a re-run cannot double the effect.196. Run the job as a tracked background process when it can outlive a single tool call, and poll its progress from the checkpoint file rather than blocking.207. On completion, verify the count: `wc -l data/done.jsonl` should equal the expected total; a short count means a silent stop — investigate before declaring done.218. Keep the checkpoint and the log together so a resume is: load checkpoint, resume from marker, continue appending.229. Pick a checkpoint granularity that bounds loss: flush every 50 items or 30 seconds, whichever comes first.2310. Store the expected total alongside the marker (`echo 5000 > data/total.txt`) so completion is a comparison, not an assumption.2411. Make the writer atomic: write to `data/state.json.tmp` then `mv` over the target, so a crash mid-write cannot corrupt the checkpoint.2526## Pitfalls2728- Accumulating all results in a Python list in memory, so a timeout loses the whole run.29- Rewriting the entire state file per item, which is slow and can corrupt on interruption.30- A resume that reprocesses everything because the marker was never written to disk.31- Marking an item done before its work succeeded, so a resume skips a failed item.32- Assuming the job finished because the loop ended, without counting the outputs.33- A checkpoint recording only the index but not the item, so a resume cannot tell whether the last item finished.34- Running two instances of the same resumable job, both appending, corrupting the done list.3536## Verification3738 total=$(wc -l < data/items.txt); done=$(wc -l < data/done.jsonl)39 [ "$total" = "$done" ] && echo COMPLETE || echo "resume from $done"40 # passes when done == total at the end, and a mid-run kill resumes at the marker4142Report to the user: the checkpoint path, total vs completed counts, and evidence that a kill-and-resume continued rather than restarted.