Long Running Jobs

long-running-jobs

How an agent should wait for work that outlives a single turn.

The principle

Polling is not the problem. Polling with the model inside the loop is.

until squeue -h -j "$JID" | grep -q .; do sleep 60; done   # 8 hours, ~0 tokens

CronCreate("check training") -> wake -> squeue -> "still epoch 21" -> reschedule
                                                  # 1 full context read PER TICK

Both poll. Only one charges you a context window every 60 seconds. When a session is carrying 400k+ tokens, a scheduled heartbeat is the single most expensive way to learn that nothing has changed.

So the goal is never "avoid polling". It is move the timer out of the context window — into the scheduler, into a blocked shell, or onto disk.

When to use

• A job "takes hours", "runs overnight", or "will finish tomorrow morning".
• The user says "check back when it's done" / "wait for training to finish" / "let me know when the job lands".
• Phase A → Phase B chaining: "run the eval after the training finishes".
• You are about to arm a recurring check-in or repeat a squeue / nvidia-smi / ls checkpoints/ poll to see whether something finished. That impulse is the trigger.

When NOT to use

• The command finishes in seconds or a couple of minutes — just run it.
• You want run-level metrics, curves, or parameter tracking → mlflow-tracking.
• You want to record what the run meant → experiment-knowledge-harness.
• You need the Slurm resource-request or GPU-isolation details themselves → slurm-hpc.

The ladder

Pick the lowest-numbered tier that applies. Do not skip down because a higher tier needs a little setup — the setup is the point.

Tier 0 — let the scheduler own the chain

The agent is not involved in the transition at all. This is almost always right for "run B after A finishes".

# Slurm. Note the ${JID%%;*} — --parsable emits "jobid;cluster" when a
# cluster name is configured, and "afterok:123;mycluster" is not a job id.
JID=$(sbatch --parsable phase_a.sbatch); JID=${JID%%;*}

sbatch --dependency=afterok:"$JID" \
       --kill-on-invalid-dep=yes \
       --mail-type=INVALID_DEPEND,END,FAIL \
       phase_b.sbatch

# pueue (local shell jobs). Use the pueue-job-queue skill's wrapper — it
# already handles defensive id parsing and emits JSON.
P=skills/local/pueue-job-queue/scripts
A=$("$P"/submit.sh --label phase-a -- ./phase_a.sh | jq -r .task_id)
"$P"/submit.sh --label phase-b --after "$A" -- ./phase_b.sh

--kill-on-invalid-dep=yes is not optional. See Gotchas.

Tier 1 — one blocking wait, backgrounded

Launch a command that blocks until the work is done, in the background. The harness re-invokes you when it exits, so you spend nothing while waiting.

sbatch --wait phase_a.sbatch          # blocks; exit code mirrors the job's

scripts/run-and-mark.sh --marker-dir .runs --name v2_full -- python train.py

Launch it once, backgrounded — not in a loop, and not in the foreground. If your harness exposes a background flag on its shell tool, that is the one to use; see references/claude-code-mechanisms.md for the Claude Code specifics, including which mechanism actually wakes you on completion and which does not.

For a job you did not start and hold no handle to, block on a condition in a subshell rather than in your context:

until ! squeue -h -j "$JID" | grep -q .; do sleep 60; done
sacct -j "$JID" --format=State,ExitCode --noheader

Tier 2 — stream filtered events

Only when you need to act before the end. Filter in the shell, never by feeding raw logs into context:

tail -F runs/v2_full.log \
  | stdbuf -oL grep --line-buffered -E \
      'Training complete|Early stopping|CUDA out of memory|Traceback'

A bare tail -F on a training log is an anti-pattern: every epoch line becomes an event and you have reinvented the expensive poll with extra steps.

Tier 3 — scheduled check-in

Legitimate only when no handle exists — a job on a machine you cannot hold a blocking connection to. If you land here, make each tick cheap: check one thing and report one line. Know your scheduler's limits (in Claude Code: session- scoped, and recurring tasks expire after 7 days — see references/claude-code-mechanisms.md).

The invariant: completion must be durable

Orthogonal to all four tiers. Whatever waits, something must survive not waiting.

Sessions die, contexts compact, laptops sleep, SSH drops. If the only record that Phase A succeeded is in the agent's context, that record is one disconnect from gone — and the usual recovery is re-running eight hours of GPU time.

So every long run writes an atomic completion marker:

python train.py; rc=$?
printf '%s\n' "$rc" > runs/v2.exit.tmp && mv runs/v2.exit.tmp runs/v2.exit

mv within one filesystem is atomic, so a reader never sees a half-written marker — it sees either no marker or a complete one. scripts/run-and-mark.sh does this for you; scripts/check-runs.sh reads the markers back and tells a fresh session what finished while it was away.

Absence of a marker means "unknown" — never "failed", never "fine". A run that started and has no marker was probably killed. That is exit code 4 from check-runs.sh, and it is a question for the user, not something to silently re-run.

Choosing

Is there a next step to run after this one?
├─ yes → Tier 0: submit it now with a dependency. Done — you are not in the loop.
└─ no  → Do you need to react before the end (OOM, early stop, threshold)?
         ├─ yes → Tier 2: stream, filtered at the shell.
         └─ no  → Can you hold a blocking handle on the work?
                  ├─ yes → Tier 1: one blocking command, backgrounded.
                  └─ no  → Tier 3: scheduled check-in, cheapest possible tick.

In every branch: the job writes a durable exit-code marker.

Available scripts

• scripts/run-and-mark.sh — run something long as a child process, block until it exits, record completion atomically. Launch once, backgrounded.
  • Flags: --help, --dry-run, --marker-dir DIR, --name NAME, -- <cmd>
• scripts/check-runs.sh — report what finished while you were away; the resume path after a session dies.
  • Flags: --help, --marker-dir DIR, --name NAME, --json

scripts/run-and-mark.sh --marker-dir .runs --name v2_full -- python train.py
scripts/check-runs.sh --marker-dir .runs --json

Bundled assets

• assets/chained.sbatch.template — a Phase A / Phase B pair wired with --parsable, afterok, --kill-on-invalid-dep=yes, a SIGUSR1 checkpoint trap, and the durable marker write. Copy and edit rather than hand-rolling the dependency plumbing.

Reference files

• references/claude-code-mechanisms.md — read when working inside Claude Code and you need to know which mechanism implements each tier: which one wakes you on completion, why foreground sleep is blocked, and the scheduler's limits.
• references/scheduler-chaining.md — read when building a Tier 0 chain: the full Slurm --dependency matrix, array / aftercorr semantics, checkpointing before the wall-clock kill, plus pueue and DVC equivalents.
• references/completion-contracts.md — read when designing the durable record: atomic marker writes, capturing an exit code that outlives the queue, sacct state vocabulary, and shared-filesystem caveats.

Gotchas

• afterany is Slurm's default dependency type. A bare -d 12345 means "after it terminates, success or failure" — which will happily run Phase B on top of a crashed Phase A. Always write the type: -d afterok:12345.
• A failed parent leaves the child pending forever. Slurm's default is "the job stays pending with reason DependencyNeverSatisfied" — not cancellation. It sits in the queue indefinitely, looking like it is merely waiting its turn. Worse: "Once a job dependency fails due to the termination state of a preceding job, the dependent job will never be run, even if the preceding job is requeued" — so fixing and requeueing Phase A does not release Phase B. Always pass --kill-on-invalid-dep=yes, and add --mail-type=INVALID_DEPEND so a hang announces itself.
• --parsable prints jobid;cluster, not a bare id, when a cluster name is configured. Strip it with JID=${JID%%;*}. Skipping this builds --dependency=afterok:123;mycluster — and the ; also terminates the shell command.
• , is AND, ? is OR in a dependency list: "All dependencies must be satisfied if the , separator is used. Any dependency may be satisfied if the ? separator is used."
• sbatch --wait returns 1 for any signal death. "If the job terminated due to a signal rather than a normal exit, the exit code will be set to 1." An OOM kill, a wall-clock TIMEOUT, a scancel, and a plain exit 1 are indistinguishable from the exit code alone. Read sacct -j "$JID" --format=State,ExitCode for the real story. For a job array the recorded code is "the highest value for any task".
• tail --pid=PID is GNU-only. BSD/macOS tail rejects it outright (unrecognized option), and macOS ships no flock and no fswatch by default. Do not build a portable wait on someone else's PID — own the process (so plain wait works) or own a marker file. Shell wait only ever works on children of the current shell.
• inotifywait on a marker file races. If the marker is created before inotifywait starts, it blocks forever. Test for the marker first, then watch. (Linux-only regardless — there is no portable equivalent.)
• On a shared filesystem a marker is not instantly visible. NFS attribute caching means a marker written on a compute node can lag on the login node. Treat "no marker yet" as unknown and keep timeouts generous.

See also

• slurm-hpc — writing the batch scripts these tiers submit.
• pueue-job-queue — a local Tier 0/1 implementation with its own blocking wait.py, group parallelism, and JSON status.
• experiment-knowledge-harness — recording what the run meant once it lands.