Long Task Manager

Long Task Manager

Use this skill for long-running implementation work driven by large documents.

The goal is not to preserve an unlimited chat context. The goal is to make the task durable enough that compaction, resume, or interruption cannot lose progress.

Core Rule

Context can be compacted. Durable files are the source of truth.

• task.md is the progress source of truth.
• state.md is durable working memory.
• verification.md is the completion gate.
• decisions.md stores durable decisions.
• blockers.md stores failed attempts and blocker history.

Do not rely on chat history as the only record of progress, decisions, blockers, test results, or remaining work.

Required Inputs

Find the task directory from the user request. Preferred layout:

specs/<name>/
  spec.md
  plan.md            # or implementation.md for per-line low-level design
  task.md
  verification.md

If the runtime files below are missing, create them before implementation:

specs/<name>/
  state.md
  decisions.md
  blockers.md

Use references/state-template.md for the initial contents.

Document Layering

Keep two design layers separate instead of merging them into one document:

• spec.md — coarse-grained design: background, goal, constraints, risks, approach, impact, ROI. It also indexes the implementation chapters.
• implementation.md (or impl-plan.md) — low-level design: which files change, which lines change, why, and the observability added with the change.

Do not collapse the two. The spec is what you align with the requester and review before work starts; the implementation is what you review line by line before editing code. When asked what the difference is, answer by granularity and audience, not by file name.

If the implementation grows large, split it into chapters and have the spec list them. See the oversized-document policy below.

Startup Protocol

1. Read task.md, state.md, and verification.md.
2. Read targeted sections of plan.md.
3. Do not read full spec.md unless the current task requires it.
4. Identify the first task not marked done.
5. Update state.md with the current task before editing.

If state.md is missing, initialize state.md, decisions.md, and blockers.md first. Do not ask the user to create them manually.

When the request names a directory rather than a single file (for example "the files under implementation/"), enumerate that directory and read every file in it before reporting the plan. Sampling a few and inferring the rest produces a plan with holes, and the user will ask "are you sure you read all of them?" — which means the answer must already be yes. If the set is too large for one context, delegate the reading to subagents per file and merge their summaries; state the file count you covered.

Alignment Gate

When the user asks to initialize a plan, investigate, or design without implementing, stop at the plan. Report the execution plan and wait.

Treat these as explicit alignment requests: "do not implement yet", "let's align first, then take the next step", "don't touch the document yet, tell me how you plan to change it".

Restate the request before proposing anything. The user asks for the same six-part restatement almost every time a new task starts, and asks for it again when a proposal misses the point: requirement, goal, constraints, rough approach, risks, blast radius. Write those six explicitly, in that order, and put what you are not going to do next to the blast radius. This is where a misread surfaces cheaply — a proposal that hangs the change off the wrong layer (a generic middleware instead of the specific path the task is about) reads as plausible until the six parts are written out and the constraint it violates becomes obvious. When the user says the plan is wrong and asks you to re-derive the requirement, redo all six from scratch rather than patching the previous proposal.

Under an alignment request:

• Investigate the chain end to end first: where the code actually does the thing, which branches and bypass paths exist, which caches or invalidation rules are involved. Report unknowns as unknowns.
• Present the options with tradeoffs and a recommendation. Do not start editing the chosen one until the user picks.
• Ask the open questions instead of guessing. The user has repeatedly invited questions at this stage; asking is cheaper than reworking.

Once the user picks an option, implement that option. Do not re-open the comparison.

When the restatement is asked for again in the same session

On a hard task the user will ask for the restatement five or six times in one sitting, each time adding a clause. That is not ritual — each re-ask means the previous one left something unanchored, and the added clause names it. Observed escalation, in the order it arrives: the six parts → the option set per problem with cost/risk/benefit each → the quantitative evidence behind each option → the decision tree and whether it is exhaustive and non-overlapping → blast radius, effort, and ROI per option.

• Answer the escalation, do not re-send the previous restatement with edits. Re-asked means re-derive. If a part is genuinely unchanged, say so in one line and spend the space on the new clause.
• Every decision node needs a quantitative fact under it, or an explicit "not measured yet". A tree whose branches are justified by plausibility reads complete and is not; the user asks "what quantitative data supports each choice" precisely to find those nodes. Name the measurement (what was counted, over what window, how many samples) or mark the node unmeasured and say what would measure it.
• Claim exhaustive/non-overlapping only after enumerating from the code, not from the shape of the tree. "Are the branches complete? Is anything missing?" is answerable only by walking the actual branch points in the implementation. If a branch cannot be distinguished with available data, that is a hole — state it rather than folding it into a neighbour.
• Carry forward everything the user has supplied. Each round they correct a fact or answer an open question; the next restatement must contain those answers, attributed to them. Re-asking a question they already answered, or restating a premise they already overturned, is what triggers the next re-ask.
• When they say a proposal is not OK, take the counter-proposal as the new baseline. They will describe an alternative in their own words and ask "did you understand my approach? restate it". Restate their approach, not a defence of yours, and mark where it changes constraints you had assumed.

An answer that cannot yet be given is a legitimate part of the restatement: list what you still need to investigate and what you need from the user, then continue. Silence on an unknown reads as a covered base.

Execution Loop

For each task:

1. Mark the task in_progress.
2. Inspect relevant code before editing.
3. Delegate context-heavy investigation to subagents when useful.
4. Implement the smallest coherent change.
5. Run task-specific checks from verification.md.
6. Fix failures caused by the change.
7. Update state.md.
8. Mark the task done, done_with_concerns, or blocked.
9. Continue automatically to the next task.

Use references/task-protocol.md for task status rules.

Context Control

Keep the lead agent's context short.

Use subagents for:

• broad codebase search
• reading large files
• test failure analysis
• log analysis
• independent code review
• spec-to-code mapping
• risk review

Ask subagents to return concise structured summaries only. Do not let subagents paste full logs or large file contents into the lead agent context.

Use references/delegation-policy.md before choosing subagents, dynamic workflows, or agent teams.

Dynamic Workflow and Agent Team Choice

Default to the lead agent plus focused subagents.

Choose the execution mode per task:

• Lead agent only for small or sequential tasks.
• Subagents for context-heavy investigation, verification, and review.
• Dynamic workflow for clearly parallelizable work.
• Agent team only for independent workstreams that require coordinated ownership.

Before using a dynamic workflow or agent team:

1. Update state.md.
2. State why that mode is justified.
3. Define non-overlapping work units.
4. Define merge, verification, and conflict-resolution rules.
5. Keep the lead agent responsible for final synthesis and acceptance.

Do not use dynamic workflows or agent teams for small, sequential, same-file, or tightly coupled changes.

Recovery Protocol

After compaction, resume, interruption, or stale context:

1. Read state.md.
2. Read task.md.
3. Read verification.md.
4. Continue from the first non-done task.
5. Read only the necessary sections of plan.md or spec.md.

Use references/recovery-protocol.md for exact recovery steps.

Scope Reduction Policy

Dropping or simplifying part of an agreed plan is a decision, not an implementation detail.

Observed failure: a design element the user had approved quietly disappeared from a later revision, and the user had to ask "why is this gone, and when did I ever decide that?". Silent scope loss is worse than an open disagreement, because it hides from review.

When a task, field, or mechanism from the aligned spec is dropped or downscoped:

1. Record it in decisions.md with attribution: who decided (user or agent), when, and the reason.
2. If the agent decided, say so explicitly. Never present an agent-side simplification as a prior user decision.
3. Flag it in the response that carries the revision, not only in the file.
4. If the drop touches something the user explicitly asked for, ask before revising instead of revising and reporting.

When the user asks "why was X removed", answer with the recorded decision (time, decider, reason). If there is no record, say that plainly — no reconstructed rationale.

When the user proposes the simplification

The opposite direction happens just as often: the user reads a multi-part design and asks "can't we do this the simple way — skip the state machine, just backfill the data?", then follows up with "what's the risk of the simplified version, in detail". Do not answer with a verdict ("that works" / "too risky"). Enumerate, in severity order, what each dropped part was holding up:

• Name the load each removed part was bearing. A design with several layers usually has each layer covering a distinct path. Removing a layer exposes whichever path only that layer covered. Walk them one at a time — which code path is now unguarded, and what reaches production through it.
• Check the ordering constraints the removal creates. The most expensive failure in this shape is a step order that silently undoes itself: the data is rewritten, then the very next request writes the old value back, because the guard that would have stopped it was the layer just dropped. State the corrected step order explicitly, and say which steps must not be merged or reordered.
• Say what the simplification actually saves. Often it removes code, not steps — the rollout is the same number of stages with fewer files touched. Claiming the plan got shorter when only the diff got shorter sets up a wrong expectation.
• Re-check reversibility. Replacing a code-level mapping with a one-off data rewrite can turn rollback from "revert one entry and redeploy" into "restore from an archive query", and a many-to-few mapping is not invertible at all. If an archive step becomes the only source of truth for the original values, say so and make it a hard precondition of the rewrite.
• Flag the parts that must survive the cut, and say in the code why they are load-bearing — a guard that looks useless is the one a later reader deletes, which silently restores the original bug.
• Reject pattern matching on identifiers that carry no guarantee. Bulk rewrites are often expressed as a prefix or substring match on an id. If that id has no enforced relationship to the property being selected on (product-facing names need not contain the vendor string), the match is wrong at the edges and cannot be repaired by refining the pattern. Pull both real inventories, classify each value explicitly, and write the rewrite as an enumerated set.

Then give a recommendation with the conditions attached, and record the decision and its attribution as above.

Handoff Protocol

Work discovered mid-task that is real but out of scope does not belong to the current task. Split it out into a handoff document and let a separate session pick it up.

Observed pattern: the user repeatedly asked for a side issue to be written up as a handoff document to follow up separately, then started a fresh session or worktree from that file alone. The handoff file was the only context that survived.

A handoff document must stand alone, because its reader has none of this session's context:

• background and goal, one paragraph;
• the symptom, and what is already proven versus still hypothesis, each with its evidence source;
• the files, functions, and configuration involved;
• options considered and any decision already made, with attribution;
• the concrete next step;
• how the fix will be verified.

Record the split in decisions.md and say in the response that the item left the current task. Do not silently keep it on the current task list.

When asked to continue from a handoff document, treat it like a spec: read it first, initialize missing runtime files, and re-verify its claims against current code before implementing — a handoff written days ago may describe code that has since changed. Prefer a separate worktree for the split-out work so the two lines of change cannot collide.

A handoff only counts once it is on disk. Write the file, then report its absolute path in the same response, so the next session can start from it. A handoff that exists only as chat text is lost at the session boundary — observed failure: the user opened a fresh session pointed at a handoff path that was never written, and the follow-up work had no context to start from.

If a handoff path you are asked to continue from does not exist, say so and stop. Do not reconstruct a plausible handoff from memory and proceed as if it were the original: locate the real surviving artifacts (design docs, task directories, prior branches), list what you found, and ask which one to continue from. Reconstruction silently substitutes your guess for the previous session's decisions.

Stall and Oversized-Document Policy

Failure modes observed in real long runs:

• Stalled background work. A subagent, workflow, or background task that loops on API retries or network errors is not making progress. Do not wait indefinitely: if the same unit shows repeated retries with no new output, stop it, record the attempt in blockers.md, and restart it from durable state. Durable files make restarts cheap; silent waiting is the expensive option. Liveness is measured by output, not by process state: a unit that is "still running" while its token count, log size, and written artifacts barely move is stalled, not slow — the user has caught this by checking token consumption. When the user asks how it is going, answer with that evidence (elapsed time, output growth, current step, last artifact written), never with "still running".
• Long batch runs with no ETA and no wake-up. When a task fans out over a large work list (per-item model calls, per-file passes, per-user extractions), the user will ask "how is it going, and how much longer?" and then ask you to set a frequent timer that picks the next step up automatically. Both parts are on you to have already done. Report progress as items done / items total, throughput per minute, and an ETA derived from the two — plus what the next step is once it finishes. "Still running" and "almost done" are non-answers; if the total is genuinely unknown, say what bounds it. And do not leave a long batch waiting on the user to come back: schedule a recurring check whose interval is a small fraction of the remaining time, so the run is picked up and the next step started shortly after it finishes rather than at the next time someone happens to ask. State the interval and what the check will do when it fires.
• Stale task-list status. The visible task list is a claim about reality: in_progress asserts a live worker is attached and producing output; an open entry asserts the work genuinely remains. Observed challenges, repeatedly: tasks still displayed open after their work had shipped ("why does it still show one task open?" — asked twice, because the first answer fixed nothing), and tasks sitting in_progress for hours with nothing behind them ("are these two really still running? why are they hanging?", "is that task still alive? is it doing anything?"). Reconcile status the moment verification passes — in the same turn, not at session end. When a delegated worker finishes or dies, update its task immediately; never leave a zombie in_progress. After compaction or resume, reconcile the list against durable state and actual artifacts before continuing — completions from just before the break are the ones most often left unmarked. And when the user questions a status, answer with liveness evidence and fix the display in the same reply. See references/task-protocol.md → Status Hygiene.
• Oversized working documents. If plan.md or an implementation plan grows so large that re-reading it every cycle overflows the context (symptom: forced compaction or API retry loops on every cycle), split it: one file per task plus a short index, and load only the current task's file. Do not keep growing a single monolithic plan document.

Local Verification Cost Policy

Verification steps run on a machine that is often shared and quota-limited. Before running a build, test sweep, or any step that fans out per target, estimate the fan-out and the bytes it will produce, and cap it.

• Count the targets first. A whole-project build in a repo with many binaries links them in parallel, each link spawning its own thread pool and re-reading the same large dependency archives. The observable result is hundreds of threads and processes stuck on IO, load in the hundreds while CPU sits idle, and the machine unusable for everyone on it for tens of minutes. Build the specific target the task needs, or pin build parallelism low, before building everything.
• Do not read high load as high CPU. When a verification step hangs, check the blocked-process count and disk queue depth before concluding the machine is compute-bound. Misreading saturated IO as saturated CPU sends the whole diagnosis the wrong way.
• Reclaim on the way out. A cold full build can consume many gigabytes of intermediate artifacts. Delete the build/link temp directories and any probe directories the task created as soon as the step finishes, and record before/after quota so the reclaim is verified rather than assumed. Do not leave cleanup to a later sweep.
• Report a shared-resource problem with attribution, and do not clean up what you did not create. When a check hits a machine-level limit (quota exceeded, disk full, load unusable), say so in the same response, separated into three parts: what this task consumed, what has already been reclaimed, and what the remainder is (historical accumulation, another session's artifacts, someone else's files). Deleting the remainder is not yours to do — surface it and let the user decide, even when the limit is blocking your own step. An accurate "this is not from my run, here is what is, here is what I cleaned" is more useful than either a silent cleanup or a bare "disk is full".
• Prefer a container or a scratch instance for anything that needs a real service to verify against, and destroy it when the check passes. Record in state.md that the step ran and was cleaned up.

Put the estimate in state.md before running the step, and record the actual cost after. When a verification step has to be skipped because it would not fit the machine, that is a blockers.md entry with the numbers, not a silent skip.

Batch and Backfill Job Policy

A batch or backfill job — a script that fans out per-item work (per-user extraction, per-row rewrite, model calls over a candidate list) — is a long task in its own right, with its own where-to-run and de-risk decisions distinct from a one-shot build.

• Run it where the tools and horsepower are, which is usually local, not the production pod. The default is: anything that can run locally, runs locally. A production pod is weak, its toolchain is incomplete (no editor, missing utilities), and it is slow; running a script there because that is where the data appears to live is a false economy. Check first whether the job can reach the data from a local run (read-only DB access, an export, a data proxy) — the user's standing rule is "whatever can run locally, run locally; never on production." Only run inside the pod when the data genuinely cannot be reached from outside, and say why when you do. And know the coupling cost when you do run in the pod: a long-running foreground process there can block the pod's own redeploy (the new rollout can't take over while the old process holds it), so a deploy that fails during the run may be the run itself — stop the batch to let the deploy through, then resume it from durable state.
• De-risk the real run with a scaled dry-run before committing to it. A dry-run over 5 rows proves the happy path, not the run. Before the real pass, run a dry-run large enough to surface the edge cases that only appear at scale — the input that overflows a conversion, the row that reads back empty, the record that crashes one item mid-list — and enumerate up front what else can be pre-validated (schema of the target table, a probe on the known-bad input, the resume path). "How confident are you it will complete without a mid-run crash, and what can we test before starting?" is a question to have already answered, not to be asked.
• A batch's first live run is a three-stage ramp — dry-run → small batch → full — and the launch pipeline is built around those stages, not hand-edited between them. The user's standing instruction for any new backfill or one-off job is the same three steps ("dry run first; then a small batch; only then the full run"), repeated across several jobs in one week, and it binds the pipeline that launches the job as much as the script: parameterize the stage (dry_run flag, limit, shard or cohort selector) so every stage is a run of the same pipeline with different inputs, rather than a YAML edit between stages that nobody reviewed. The full stage itself starts with one shard or one worker as a canary and releases the rest only after that canary's outcome counters look right (see the alive-is-not-succeeding bullet) — the gate between stages is read from done / failed / rejected counts, never from "it finished". Two corollaries the user has also asked for: (1) a stage that claims to "only verify startup" must be provably unable to execute business logic — a code-level guard the pipeline sets (the dry-run flag short-circuits before any write, and the log shows the short-circuit), not a promise; (2) a one-off pipeline and its deploy manifests live where they can be deleted when the job is done — next to the job they serve, in the repository whose job it is — and removing them (pipeline definition, manifest directory, generated resources) is the last step of the runbook once the full run is verified, with the removal reported. A one-off that is never cleaned up becomes a live trigger someone fires by mistake.
• Confirm idempotent resume before the real run, not after a crash. Long batches get interrupted (pod restart, network, a single bad row). Verify that a re-run skips already-done work rather than redoing or double-writing it, and that it resumes from durable state — test this on real data before the real pass, so an interruption is a resume rather than a restart. Resume state must match the pagination key: a single-column cursor cannot resume a scan keyed on a composite (a, b) — add every key component to the resume parameters, or the re-run silently restarts or skips a slice. Only then launch the real run detached (e.g. nohup), reporting progress as items done / total, throughput, and ETA (see the batch-run bullet under Stall and Oversized-Document Policy).
• Alive is not succeeding: read the outcome counters, not the throughput. A detached batch can look perfectly healthy — the processed counter climbs, downstream calls return 200, the process stays up — while almost every item silently fails a precondition and produces nothing. Observed: a run streamed steady 200s and a rising processed for hours, but the success counter was a rounding error and millions of items were rejected upstream (an admission/dependency check erroring or denying), which the top-line rate hid completely. Every progress line must break the processed total into outcome buckets — done / skipped / empty / rejected / dependency-errored — and the first read of a run judges it by the done rate, not by "it's still going." A near-zero done rate with a healthy processed rate is a stalled run wearing a green light: stop it, find why the precondition fails (wrong key, dependency unreachable, an upstream toggle turned the whole cohort off), fix it, and resume from durable state rather than letting it burn the full list producing nothing. When the user asks how it's going, lead with done/total and the reject breakdown, never with "still running, 200s flowing."
• Enforce the tool's own limits at parse time, and don't silently clamp. A batch knob the operator will turn up under time pressure (concurrency, batch size, page limit) needs its accepted range validated where it is parsed, with the bound in the error — not discovered by a mid-run crash after the operator passed a value the script accepts but a downstream layer rejects. If a value is out of range, fail fast before any work starts; never silently clamp it to a different number, because the operator will report the run at the value they typed.
• Merging the fix is not deploying it: a scheduled batch runs whatever image its manifest pins, and an instance already spawned does not swap code mid-run. When a recurring job is delivered by a "deploy" pipeline (build image → apply the schedule manifest), a code merge alone changes nothing on the running side until that pipeline rebuilds the image and re-applies the manifest — and even then, the instance the scheduler already created for this cycle keeps running the old image to completion. The user's recurring question is exactly this: "I merged it — will tonight's scheduled run use the new code?" Answer it by tracing the actual delivery path, not by pointing at the merge: (1) has the deploy pipeline run since the merge, so the pinned image is rebuilt; (2) does the schedule manifest reference a moving tag or a pinned digest; (3) has an instance for the current cycle already been created off the old image. To force the new code onto the current cycle you usually have to kill the in-flight instance and re-trigger the deploy so the scheduler spawns a fresh one — say that explicitly rather than implying the merge took effect. And when you edit the manifest, check whether the same change belongs in every environment's copy (a fix applied only to the manual/test manifest silently misses the scheduled prod one) — the user has caught exactly this ("you only changed the manual YAML, shouldn't prod change too?").
• A recurring batch must scope its candidate set to what changed, not re-scan the whole population every run. A one-shot backfill is allowed to walk the full list once; a job that fires on a schedule (daily cron) is not — re-scanning the entire population each run does not scale and eventually one run does not finish before the next fires. Observed: a daily cron drafted to regenerate over the full user base would take days per run, and the user's pushback was blunt — "how much does one run cover? a full scan of every user takes days, is that reasonable — did you think it through?". Before wiring a scheduled batch, answer where the candidate set comes from each run and how large it is; the default for a recurring job is incremental — only the entities that changed since the last run's watermark. If incremental scoping needs support the upstream does not yet have (a "changed-since" column or index to select on), that is a direction decision for the user and a separate handoff, not something to paper over by full-scanning. And before launching work that depends on the cron, reason explicitly about what breaks if the cron does not run or has not been deployed yet — "won't it be a problem if the cron doesn't fire?" is a question to have answered before the experiment goes live, not after.
• A scheduled batch's start time is a production decision: check it against the serving-traffic peak curve, and budget the whole occupied window — start time plus measured duration — not just the trigger instant. A recurring data-refresh or backfill job hits the same downstreams live traffic uses, so when it runs matters as much as how hard it pulls. Observed failure, both facets on one job: a refresh cron turned out to fire at a serving peak hour — the user's reaction was blunt ("that's rush hour — running a data refresh then is asking for trouble; what were you thinking?") — and the run, scaled up from a small pilot to a 20× candidate limit, took ~8 hours, plowing straight through the peak regardless of the start instant; the second surprise was that the run was cron-triggered at all, when the user believed they were doing a manual one-off. Before wiring or re-triggering the schedule: (1) establish where the serving peak actually is, and in whose timezone — the scheduler's clock, the host's clock, and the user base's peak rarely agree, and a slot that reads "off-hours" in one zone is morning peak in another; (2) measure duration on a real slice and place the whole [start, start + ETA] window inside the trough; (3) redo this check every time the per-run limit or candidate count is raised — the slot chosen for a 100-item pilot is not automatically safe for the 2,000-item real pass, because duration scales with the limit and the run window creeps into hours it was never meant to touch.
• A batch that runs in an ephemeral container has a bounded lifetime, and the ETA must fit inside it — a run that outlives its window does not resume, it dies with the container. Scheduled batches often run in a job container that is reclaimed after a fixed lifetime (observed: a job kept only ~8 hours). "Resume from durable state after an interruption" assumes the container comes back; when the lifetime cap is the interruption, there is nothing to resume into. So the runtime budget is a hard constraint, not a nice-to-have: measure throughput on a real slice, project the full run against the container's lifetime, and if the projection overruns the window (observed: a run projected at ~20h in an ~8h window), that is a launch blocker to resolve by making the run faster or splitting it across windows — not something to discover when the container disappears mid-run. Report the ETA against the lifetime cap explicitly, and treat closing the gap as P0 before the real pass.
• To select or filter candidates on a cheap predicate, read a denormalized metadata column, not a COUNT or scan of the authoritative store. A per-run candidate filter ("has any messages", "count > 0", "changed since X") does not need the heavy store: the authoritative data usually has a denormalized counterpart — a count column on a metadata table, a summary row, a covered-watermark table you own — that answers the predicate with a point read instead of a fan-out COUNT over tens of millions of entities. Observed: a stage filtering the full population by message count was scanning the heavy chat store when a metadata table carried a message_count column that answered the same question far cheaper, with a fallback to the store only when the metadata row is missing. Before trusting the cheap source, verify its freshness against the authoritative one — how it is populated, whether it lags (eventual consistency, binlog sync, transactional), and by how much — because a stale count silently drops or includes the wrong candidates. Read the actual caller code to find the existing cheap path rather than inventing a new query.
• A batch that shares credentials, connection pools, or rate-limit quota with the live serving path can take production down — isolate the shared resource before the batch goes live, and give it an abort switch. A scheduled or async batch usually reaches the same downstreams the online path uses, and "it's just a background job" hides that it competes for the same finite budget. Two failure shapes observed, both on a nightly cron that ran alongside live traffic: (1) shared API key → rate-limit storm. The batch and the online request path drew on the same provider key; under load the batch's calls burned the quota and the live path started taking 429s. The fix was to split the credential — a dedicated key for the batch, selected by its own env var — so the two cannot starve each other; before wiring that, confirm from the code which key each path actually uses rather than assuming they differ. (2) shared store → connection exhaustion. The batch opened connections to a connection-capped backend with no pool, and under a burst got Connection refused in tight clusters (several at the same instant) as a wave of concurrent workers each built a new connection and hit the cap at once — clustered refusals at connection-build time are the signature of pool-less fan-out against a capped store, not a timeout or an auth failure. The root fix is a bounded connection pool sized like the live service's, not a retry. (3) per-candidate RPC to a live control-plane service → the service itself is the shared resource. A batch that filters its candidate list by calling an online service per item (checking each entity against a live experiment/config/flag service) turns that service into a downstream it can overload — the fan-out that makes the batch fast is exactly what floods the RPC. Before launching, cap the batch's concurrency against that service to a level the service is known to tolerate, and watch the service's own QPS and CPU/memory dashboard during the run, not just the batch's success rate — arrange an escalating alert so the run can be stopped the moment the shared service starts to strain. Whether the filter should call the live service at all is the prior question: if the same membership can be resolved from an export or a snapshot, prefer that over hammering the serving path. Before launching a batch that touches live downstreams, enumerate every shared resource (credentials, pools, DB/quota, control-plane RPCs) and either isolate it or bound the batch's draw on it. And because a bad batch run can need stopping now, build the stop path before the first live run: a one-command way to halt the job and drop its queued work (a switch that deletes the cron-generated tasks), not a scramble to find one after it starts hurting production. "Won't the batch's key collide with the live path's and trigger 429s?" is a question to answer before it fires, not after the first alert.
• The batch's budget on a shared resource is live path + every other offline job on it right now; a fan-out proxy caps per node; "at the ceiling" is not a steady state; and the live path's own health metric is the batch's hard stop-line — enforced by the watcher, not by a human. The shared-resource bullet above says isolate or bound the draw. Four ways that still went wrong, on consecutive days, across two different shared resources: (1) Wrong denominator. A backfill was sized against the live service's connection count alone; a second backfill someone else had been running for two days on the same connection-capped store was never counted, a warm-up job's tail landed on top, and live + backfill + backfill + warm-up hit the cap — the live service threw thousands of 5xx in two waves. Before launching, ask who else is on this store / channel right now (other people's jobs included) and budget against the sum, not against your own job. (2) Fan-out proxies do not multiply capacity. "4 nodes × 400 connections" was read as 1,600; behind a connection proxy every node has to accommodate every connection, so the cap is the per-node figure. (3) Ceiling-minus-thirty is not stable. The advice given at the time was "keep the backfill running and watch it to completion" — wrong. When a shared resource sits within tens of units of its hard cap, the only correct recommendation is to stop or throttle the offline job now; the live path's users must never pay for the backfill. (4) Saturation can arrive as degraded output, not as errors. On a shared model channel the upstream returned HTTP 200 throughout while output quality collapsed — format-check failures and retry amplification jumped, and the live refresh path's failure rate went from well under 1% to 20–30% for the better part of a day — and the channel's tolerance dropped day over day (roughly 3,000 → 2,800 → 800 requests/min before the live path degraded), so yesterday's safe concurrency is not today's. The discipline: wire the live path's own failure-rate / latency metric into the batch watcher as a hard stop-line that acts by itself — freeze new batches and lower concurrency the moment it crosses (e.g. "live failure rate above N% on two consecutive readings"), with the threshold written down before the first shard starts; re-probe the safe concurrency at the start of every run rather than reusing last run's; and print the live path's number in every progress line next to the batch's own. A watcher that reports only the batch's success rate is watching the wrong side of the shared resource.
• Switching one path to a new provider must not repoint a credential or config key that other paths still consume — enumerate every consumer of the key first, then give the migrating path its own independently named key. A provider switch reads as "change the key and base URL", but secret keys and env vars are shared surfaces: changing the value of an existing key silently re-routes every other path that reads it. Observed, one day apart: a key swap meant for one pipeline flipped an unrelated experiment's traffic to the new vendor and was only caught the next day ("yesterday's key change affected the other experiment"); the redesigned wiring that populated one path's variable by referencing another path's key drew the blunt correction "changing one key can affect another — that's wrong"; and even the final version got "these keys are still coupled — fix it". The discipline, in order: (1) before touching any key, list every consumer of it — online main path, side paths, scheduled jobs, one-off scripts — from the manifests and code, not from memory; (2) the migrating path gets a new key with its own name whose value only it reads, and the old key's value is left untouched, so "change A" provably cannot move B; (3) in manifests, inject same-name (env var name equals the secret/config key name) instead of aliasing one name onto another — an alias layer hides the blast radius of a key change and is exactly where the coupling hides; (4) keep switch-type knobs (which provider a path routes to) in the central config store, not scattered across per-deployment envs, so the current routing is readable in one place; (5) before changing how a whole bundle is injecte

…(truncated)