Migrating In-Process Background Jobs
Overview
ASP.NET Framework apps run background work in three shapes, and none of them survives the move to
ASP.NET Core unchanged:
- WebBackgrounder —
JobManager running IJob instances on a timer inside the web app.
HostingEnvironment.QueueBackgroundWorkItem (QBWI) and IRegisteredObject — work the ASP.NET
runtime tracks so shutdown can wait for it.
- Console job runners — a
Main that loops, sleeps, and exits with a code, run by an external
scheduler or left running.
Porting the code is the smaller problem. During a side-by-side migration the Framework host and the
Core host are both live, so a job ported to Core while it still runs in Framework executes twice.
For a cache refresh that is harmless. For a job that sends mail, deletes rows, or charges a card, it is a
data-integrity incident. Step 6 is the part of this skill that prevents it.
Scope
In scope: jobs hosted inside an ASP.NET Framework web app, QBWI and IRegisteredObject work, and
console job executables, moved to ASP.NET Core or the Generic Host — including while both hosts run.
Out of scope — keep the existing scheduler: Hangfire, Quartz.NET, the Azure WebJobs SDK, and Azure
Functions timer triggers can coordinate across hosts, but only when configured to. Do not replace them
with a hand-written BackgroundService; port them with their own migration guidance. Do not assume they
are safe, either: verify that both hosts use persistent (not in-memory) storage, the same storage, a
schema version both package versions accept, clustering or singleton settings that are actually enabled,
stable scheduler and job identifiers, and retry semantics the jobs tolerate. Anything unverified makes
each host schedule its own copy, and Step 6 applies.
Related skills:
managing-shared-database-schema — governs the schedule table this skill adds (Step 6).
migrating-global-asax — Application_Start and Application_End, where many jobs are started.
migrating-aspnet-framework-to-core — the side-by-side task structure that creates the window.
migrating-mvc-dependency-injection — resolving job dependencies from the new container.
Step 1: Inventory every job
Produce a written table in task.md. Search the web project, its class libraries, and every console
project and deployment script:
| Search for |
Indicates |
WebBackgrounder, JobManager, : Job, IJob, IJobCoordinator, WebFarmJobCoordinator |
WebBackgrounder |
QueueBackgroundWorkItem |
QBWI |
IRegisteredObject, HostingEnvironment.RegisterObject |
shutdown-aware background work |
new Timer(, System.Timers.Timer, Task.Run(, new Thread(, ThreadPool.QueueUserWorkItem in startup code |
hand-rolled job |
Application_Start, PreApplicationStartMethod, PostApplicationStartMethod, ApplicationShutdownMethod, OWIN Configuration(IAppBuilder) |
where jobs are started and stopped |
static void Main / static async Task Main with a loop and Thread.Sleep or Task.Delay |
continuous console job |
settings.job, App_Data/jobs/, Task Scheduler XML, cron entries, container job definitions |
how a console job is scheduled |
Hangfire, Quartz, Microsoft.Azure.WebJobs |
existing scheduler — see Scope |
Record one row per job, each citing path:line:
| Job |
Pattern |
Runs in |
Trigger and cadence |
State it changes |
Current coordination |
Evidence |
ExpireStaleRecords |
WebBackgrounder |
web host, every instance |
every 60 min after the previous run |
deletes rows in the app database |
none |
src/Web/Jobs.cs:42 |
"State it changes" decides everything after this step. Read the job body; do not infer it from the name.
Step 2: Classify each job
Answer four questions per job and add the answers to the inventory:
| Question |
Answers |
Decides |
| What does a run change? |
process-local state only, shared state, or both |
how many copies may run |
| How many copies may run one occurrence? |
every instance, or one executor across all hosts |
whether Step 6 applies |
| What if an occurrence is lost or repeated? |
loss is acceptable, or it must eventually run once its effects are idempotent |
delivery guarantee |
| What triggers it? |
timer, startup, request, queue, or an external scheduler |
the port target (Steps 3–5) |
Apply these rules:
- Process-local only — an in-memory cache, a warm-up — runs in every instance of both hosts, with
no guard. Gating it on a shared lock leaves every non-holder serving a stale cache.
- Machine-local files — a local index, a disk cache — are shared by every process on that machine.
When both hosts, or two IIS worker processes, run on one machine, give each its own path or make one
of them the only writer there.
- Any shared write — database, blob, queue, mail, external API, cleanup or deletion — needs a
single executor per occurrence across all hosts and instances. Step 6 is mandatory for it.
- Both — split it into a per-instance refresh and a single-executor job. A write keyed by host
(a heartbeat row per instance) is per-instance; make it an idempotent upsert on the host key.
- Request-triggered work runs on the host that served the request. Classify its delivery:
best effort (an in-memory queue is fine), must eventually complete (durable queue or outbox), or must
complete before the response (keep it in the request).
- Cannot tell whether a write is shared — treat it as shared.
A worked inventory and classification of a real application, including which of its jobs must not
be guarded: ref/worked-example.md.
Step 3: Port WebBackgrounder jobs
WebBackgrounder depends on System.Web — its JobHost registers with HostingEnvironment — so it cannot
be referenced from ASP.NET Core. Rewrite each job as a BackgroundService, preserving these behaviours:
| WebBackgrounder behaviour |
What the port must do |
Execute() often returns an unstarted new Task(...); JobHost starts it |
Never await job.Execute() on legacy code — an unstarted task never completes. Rewrite the body as async Task RunAsync(CancellationToken). |
new Task(async () => ...) is an async void lambda: the task completes at the first await |
The real work was untracked and its exceptions escaped the scheduler. Await the work directly in the port. |
Start() runs the shortest-interval job of each JobManager straight away; the others first run one interval later; after that each interval runs from the end of the previous run |
Run first, then delay, for the job that ran at Start() — a delay-first loop leaves its cache cold after every restart. Delay first for the others, unless the legacy code also ran them at startup through a QBWI call or a constructor. Delay after each run. |
One JobManager runs all its jobs serially on one timer |
Separate services run concurrently. Keep jobs in one service only if they share state that is not thread-safe. |
Fail(...) with RestartSchedulerOnFailure = true, or a coordinator that logs and swallows |
A failed run was logged and the schedule continued. Keep that policy explicitly — see Step 7. |
RestartSchedulerOnFailure left at its default, false, with a coordinator that lets faults through |
One faulted run stopped the JobManager's timer for good: every job in it stopped until the next recycle. Do not reproduce that. When adding the guard to the Framework copy, set RestartSchedulerOnFailure = true with a logging Fail(...) handler first — a claim surfaces store errors and lost claims as faults. |
SingleServerJobCoordinator, or a custom coordinator that calls Execute() |
No cross-instance coordination existed: every instance already ran the job. |
WebFarmJobCoordinator with a WorkItems table |
A cross-instance mutex, not once-per-interval: N servers each run it once per interval, without overlapping. See Step 6. |
Timeout |
Only expires a WorkItems reservation. It never cancelled a run. |
Code for the port, including a scope per run: ref/porting-patterns.md.
Step 4: Port QueueBackgroundWorkItem and IRegisteredObject
| Legacy shape |
Port to |
QBWI from startup, often a while (!token.IsCancellationRequested) refresh loop |
a BackgroundService |
| QBWI from a request, best effort |
a bounded Channel<T> drained by a BackgroundService, with a DI scope per item |
| QBWI from a request whose work must eventually complete |
a durable queue or an outbox table, drained by a single-executor job |
| One-off, loss-tolerant work with nothing to drain |
Task.Run under ExecutionContext.SuppressFlow(), with its exception observed and IHostApplicationLifetime.ApplicationStopping as its token |
IRegisteredObject.Stop(bool immediate) |
IHostedService.StopAsync, or ApplicationStopping |
Semantics that change:
- ASP.NET tracks QBWI items and tries to delay AppDomain shutdown until they finish. The Generic Host
waits only for hosted services'
StopAsync, up to HostOptions.ShutdownTimeout (30 seconds by
default). Task.Run work is not waited for, and an in-memory Channel<T> loses whatever is queued
when the process stops. Neither is a substitute for durable delivery.
- QBWI does not flow
ExecutionContext. Neither should the port: capture the values the work needs
when it is queued. Never capture HttpContext or a scoped service — both are gone once the response
completes.
- Bound every queue and decide what happens when it is full: wait, drop, or fail the request.
Code: ref/porting-patterns.md.
Step 5: Port console jobs to the Generic Host
- Keep the invocation contract. A job that an external scheduler (Task Scheduler, cron, a triggered
WebJob, a container job) starts once per trigger stays one-shot: run, set the exit code, exit. Do not
turn it into a permanent loop — the scheduler is already the loop.
- A continuous runner — loop, sleep, repeat — becomes a
BackgroundService whose delay takes the
stopping token. A legacy Task.Delay(sleep) without a token blocks shutdown.
- Map the runner's own switches — run-once, delay, period — to configuration explicitly. The command-line
provider ignores single-dash switches unless mapped, and a bare flag is not a value.
- Return a non-zero exit code whenever the occurrence was not completed — a failed run or a lost claim —
and set it explicitly. Do not rely on an unhandled exception to produce one.
- While the legacy executable is still scheduled, its port is a second copy of the same job. Step 6
applies to it exactly as it does to a web-hosted job, with aligned windows as the schedule model.
Code: ref/porting-patterns.md.
Step 6: Guard against duplicate execution
Applies to every single-executor job (Step 2) that can run in more than one place at once: in both
hosts during side-by-side, in several instances of one host, in two IIS worker processes during an
overlapped recycle, in old and new versions during a rolling deployment, or as a legacy console job
still scheduled next to its port.
Invariants
- Every copy takes the same guard — Framework and Core, every instance. A guard only the new host
takes is not a guard. The Framework job changes too, or is verifiably stopped first.
- Mutual exclusion is not one run per occurrence. A lock stops two runs overlapping; it does not
stop the Framework host running at 10:00 and the Core host running again at 10:00:05. The next-due
time must live in the shared store, decided in the same atomic step as the claim.
- Decide with the store's clock, never a host clock. Hosts drift.
- Fail closed. If the store cannot be reached, the job does not run.
- Renew while running, and cancel the work as soon as a renewal fails or cannot be confirmed — and
when the lease runs out before one is confirmed, even if the renewal request is still stalled.
- A claim reduces duplicates; it cannot eliminate them. A holder that crashes after its side
effects but before recording the occurrence is retried, and so is one that finishes after its lease
expired, so delivery is at-least-once. Make every protected effect idempotent — a natural key, an
upsert, an idempotency key the receiver honours — or fenced: in the same transaction as the write, under
an update lock, check the claim's generation, an active holder, and a live lease — see
Fencing a write. A fence only protects writes that
run it, and a generation check alone is not a fence: Fail, Complete, and an expired but unclaimed lease
all keep the generation. Keep each fenced transaction well under the renewal interval, or it blocks the
holder's own renewal; split long work into short fenced batches.
- Key idempotency on the occurrence, fence on the attempt. The generation changes on every claim,
retries included, so an idempotency key built from it lets the retry repeat the effect. Build keys from
the job name and the occurrence's due time, which retries share. Mail and third-party calls cannot be
fenced; give them that key, or record that the user accepts a rare duplicate.
- Decide what a failed run means, per job. Either retry the same occurrence after a delay the store
enforces — so hosts do not retry hot — or end it and wait for the next interval, as WebBackgrounder did.
Neither is a safe default for every job.
- Pick the schedule model per job. Fixed delay (next run one interval after the last ended) matches
in-process loops. A job an external scheduler starts on a calendar needs aligned windows, or a long run
makes the next trigger find nothing due.
- Record the delivery guarantee, failure policy, and schedule model per job in
task.md.
At-least-once is this recipe; at-most-once means completing before acting and accepting a lost
occurrence when a holder crashes.
Choose a guard
| Guard |
Stops a recorded occurrence running again |
Fencing token |
Needs |
Use when |
| SQL schedule row |
yes — next-due time in the row, fixed delay or aligned windows |
Generation column |
one new table, SQL Server 2012+ or Azure SQL |
both hosts reach a shared SQL Server database (default) |
| Azure blob lease with metadata |
yes — approximate fixed delay in blob metadata |
generation in metadata |
a storage account, one blob per job, and a Framework claim you write and test yourself |
no shared SQL database |
sp_getapplock |
no — mutex only |
none |
nothing |
only alongside a next-due row, or where the job's own data records progress |
| Dedicated single worker process |
no |
none |
a deployment change |
moving the job out of the web hosts — and still take a claim: a replica count of one is not a mutex during rolling deployments or restarts |
| Configuration switch per job |
no |
none |
configuration |
a kill switch only; per-instance configuration is not atomic during a rollout |
Recipes, the Framework-side code, and worked examples: ref/duplicate-execution-guard.md.
The Core-side reference is ref/scheduled-singleton-job.cs.
An existing WebFarmJobCoordinator is not reusable as-is. Two hosts using two different guards have
no guard. Either the Core port reserves through the same WorkItems table with the same JobName,
the same transaction, and the same timeout — which keeps the old mutex-only behaviour — or both hosts move
to the schedule row, Framework first. Moving is two Framework deployments: add the claim and keep the
coordinator, then remove the coordinator once every instance takes the claim. Removing it in the same
deployment leaves old and new instances holding different guards for the length of the rollout.
The schedule table is a schema change. Load managing-shared-database-schema before creating it:
get_instructions(kind='skill', query='managing-shared-database-schema'). The table is an additive new
table created and seeded by the single schema owner's migration. Application logins hold DML only during
the window, so the application never runs CREATE TABLE at startup.
Handover
A rolling deployment of the guard is itself a window in which guarded and unguarded processes coexist, so
the guard is activated only once no unguarded process remains:
- The schema owner creates the schedule table and seeds one row per job — interval and schedule model
included — inactive:
NextDueUtc far in the future, so no guarded copy runs yet.
- Deploy the guard into the Framework job, on every instance. Wait until no unguarded process
remains: the rollout has finished, old IIS worker processes have exited, and no unguarded console copy
is still scheduled.
- Activate: one
UPDATE sets each row's NextDueUtc to its first occurrence. The job does not run between
the last unguarded process stopping and this step, so keep that gap short. Confirm each run then
advances Generation and that no run happens without a claim.
- Deploy the Core port under the same job name. It reads the interval from the row, so the two hosts
cannot disagree about it.
- Watch several occurrences: one holder per generation, no overlap while leases are live, and no recorded
occurrence run again.
- Retire the Framework copy whenever convenient. Never remove the guard while two copies can run.
IIS keeps the Framework side moving under you. With default settings an application pool stops after
20 idle minutes and starts on demand, so as traffic drains to Core, the Framework host's in-process jobs
silently stop — the guard lets Core claim the work, but per-instance jobs there simply stop. Overlapped
recycling briefly runs two worker processes, so a single-instance Framework host overlaps itself.
Step 7: Shutdown, startup, and failure policy
- Pass the stopping token through every
await. Raise HostOptions.ShutdownTimeout if a run needs
longer than 30 seconds to wind down; work that ignores cancellation is abandoned at the timeout.
- Since .NET 6 an exception escaping
ExecuteAsync stops the host — in a web host, the whole site.
Preserve the legacy failure policy deliberately: if the old scheduler logged a failed run and carried
on, catch per run, log, back off, and report consecutive failures to a health check. Let configuration
and invariant errors at startup fail fast. Do not set BackgroundServiceExceptionBehavior.Ignore
globally as a substitute.
- From .NET 10 all of
ExecuteAsync runs off the startup path. On earlier targets its synchronous prefix
blocks startup, so reach the first await immediately.
- A legacy run at startup — a constructor that did the work, or a QBWI kick — becomes the first
iteration of
ExecuteAsync, not constructor or StartAsync code.
- A
BackgroundService is a singleton. Create a DI scope per run with IServiceScopeFactory; never
inject a scoped DbContext into it.
Step 8: Verify
- Per-instance jobs log runs from both hosts.
- Single-executor jobs, with both hosts against one store: runs never overlap while leases are live, and
once an occurrence is recorded neither host runs it again; kill the holder mid-run and another host retries the same
occurrence after the lease expires — the expected at-least-once repeat; a run that outlives its lease is
not recorded; make the store unreachable and nothing runs while both hosts stay up; stop a host and its
claim is released.
- The fence, for jobs that use one: suspend the holder past its lease (debugger or process suspend), let
another host claim, resume, and confirm the stale attempt's fenced write is rejected with error 50002.
- A failing run leaves the web host serving, shows in the health check, and is retried or ended as the
job's failure policy says.
- Request queues are bounded and behave as decided when full.
- One-shot console jobs return non-zero whenever the occurrence was not completed.
Decomposition Rules
(Read by the task-breaker when this skill is attached to a task being decomposed. Both task-breaker
and task-executor scan for this exact heading — do not rename it.)
Restate the pin on every job subtask. A subtask's task.md is written from the breakdown alone, so a
parent's #skill: pin does not reach it. Copy #skill:migrating-inprocess-background-jobs into every
subtask that ports, enables, schedules, guards, or retires a background job.
Pin the schema skill where the store is created. A subtask that creates or seeds the schedule table
also carries #skill:managing-shared-database-schema.
Order the work. Inventory and classification first; the guard in the Framework host before any
single-executor job is enabled in the Core host; the Core port next; retiring the legacy copy last.
Never create a subtask that enables a single-executor job in the Core host while any copy runs unguarded.
Success criteria
- Inventory and classification written to
task.md, every row citing path:line
- Per-instance jobs run in every instance of both hosts, unguarded
- Every single-executor job takes the same guard in every copy, with its next-due time in the shared store
- Delivery guarantee, failure policy, and schedule model recorded per job; protected effects idempotent,
keyed by the occurrence, or fenced on generation, active holder, and live lease
- The schedule table created by the schema owner's migration, never by the application
- No legacy
Task started unawaited, no async void job body, no scoped service captured by a job
- A failed run does not stop the web host; shutdown cancels runs and releases claims
- One-shot console jobs keep their scheduler and their exit codes
1---2name: migrating-inprocess-background-jobs3description: >4 Ports background jobs from ASP.NET Framework to ASP.NET Core and the .NET Generic Host, and stops a5 job running twice while the legacy and new hosts are both live. Use when an app uses WebBackgrounder6 (JobManager, IJob, IJobCoordinator, WebFarmJobCoordinator), HostingEnvironment.QueueBackgroundWorkItem,7 IRegisteredObject or HostingEnvironment.RegisterObject, timers or Task.Run loops started from8 Application_Start, WebActivatorEx or OWIN Startup, or console job runners whose Main loops and sleeps9 or runs once per external trigger. Covers BackgroundService, IHostedService, Channel<T> work queues, Generic Host and Worker10 Service console jobs, shutdown and failure behaviour, and a duplicate-execution guard for side-by-side11 migration: a shared SQL schedule row or an Azure blob lease that both hosts take, fencing tokens, and12 handover. Triggers for "background job", "scheduled job", "hosted service", "job runs twice", "double13 fire", "leader election", "distributed lock", or "singleton job".14metadata:15 source: https://skillmd.com/skills/microsoft/migrating-inprocess-background-jobs16 traits: .NET|CSharp|VisualBasic|DotNetCore17 discovery: lazy18---1920# Migrating In-Process Background Jobs2122## Overview2324ASP.NET Framework apps run background work in three shapes, and none of them survives the move to25ASP.NET Core unchanged:26271. **WebBackgrounder** — `JobManager` running `IJob` instances on a timer inside the web app.282. **`HostingEnvironment.QueueBackgroundWorkItem`** (QBWI) and `IRegisteredObject` — work the ASP.NET29 runtime tracks so shutdown can wait for it.303. **Console job runners** — a `Main` that loops, sleeps, and exits with a code, run by an external31 scheduler or left running.3233Porting the code is the smaller problem. During a side-by-side migration the Framework host and the34Core host are **both live**, so a job ported to Core while it still runs in Framework executes twice.35For a cache refresh that is harmless. For a job that sends mail, deletes rows, or charges a card, it is a36data-integrity incident. Step 6 is the part of this skill that prevents it.3738## Scope3940**In scope:** jobs hosted inside an ASP.NET Framework web app, QBWI and `IRegisteredObject` work, and41console job executables, moved to ASP.NET Core or the Generic Host — including while both hosts run.4243**Out of scope — keep the existing scheduler:** Hangfire, Quartz.NET, the Azure WebJobs SDK, and Azure44Functions timer triggers **can** coordinate across hosts, but only when configured to. Do not replace them45with a hand-written `BackgroundService`; port them with their own migration guidance. Do not assume they46are safe, either: verify that both hosts use persistent (not in-memory) storage, the same storage, a47schema version both package versions accept, clustering or singleton settings that are actually enabled,48stable scheduler and job identifiers, and retry semantics the jobs tolerate. Anything unverified makes49each host schedule its own copy, and Step 6 applies.5051**Related skills:**52- `managing-shared-database-schema` — governs the schedule table this skill adds (Step 6).53- `migrating-global-asax` — `Application_Start` and `Application_End`, where many jobs are started.54- `migrating-aspnet-framework-to-core` — the side-by-side task structure that creates the window.55- `migrating-mvc-dependency-injection` — resolving job dependencies from the new container.5657## Step 1: Inventory every job5859Produce a **written table in `task.md`**. Search the web project, its class libraries, and every console60project and deployment script:6162| Search for | Indicates |63|---|---|64| `WebBackgrounder`, `JobManager`, `: Job`, `IJob`, `IJobCoordinator`, `WebFarmJobCoordinator` | WebBackgrounder |65| `QueueBackgroundWorkItem` | QBWI |66| `IRegisteredObject`, `HostingEnvironment.RegisterObject` | shutdown-aware background work |67| `new Timer(`, `System.Timers.Timer`, `Task.Run(`, `new Thread(`, `ThreadPool.QueueUserWorkItem` in startup code | hand-rolled job |68| `Application_Start`, `PreApplicationStartMethod`, `PostApplicationStartMethod`, `ApplicationShutdownMethod`, OWIN `Configuration(IAppBuilder)` | where jobs are started and stopped |69| `static void Main` / `static async Task Main` with a loop and `Thread.Sleep` or `Task.Delay` | continuous console job |70| `settings.job`, `App_Data/jobs/`, Task Scheduler XML, cron entries, container job definitions | how a console job is scheduled |71| `Hangfire`, `Quartz`, `Microsoft.Azure.WebJobs` | existing scheduler — see Scope |7273Record one row per job, each citing `path:line`:7475| Job | Pattern | Runs in | Trigger and cadence | State it changes | Current coordination | Evidence |76|---|---|---|---|---|---|---|77| `ExpireStaleRecords` | WebBackgrounder | web host, every instance | every 60 min after the previous run | deletes rows in the app database | none | `src/Web/Jobs.cs:42` |7879"State it changes" decides everything after this step. Read the job body; do not infer it from the name.8081## Step 2: Classify each job8283Answer four questions per job and add the answers to the inventory:8485| Question | Answers | Decides |86|---|---|---|87| What does a run change? | process-local state only, shared state, or both | how many copies may run |88| How many copies may run one occurrence? | every instance, or one executor across all hosts | whether Step 6 applies |89| What if an occurrence is lost or repeated? | loss is acceptable, or it must eventually run once its effects are idempotent | delivery guarantee |90| What triggers it? | timer, startup, request, queue, or an external scheduler | the port target (Steps 3–5) |9192Apply these rules:9394- **Process-local only** — an in-memory cache, a warm-up — runs in **every instance of both hosts**, with95 no guard. Gating it on a shared lock leaves every non-holder serving a stale cache.96- **Machine-local files** — a local index, a disk cache — are shared by every process on that machine.97 When both hosts, or two IIS worker processes, run on one machine, give each its own path or make one98 of them the only writer there.99- **Any shared write** — database, blob, queue, mail, external API, cleanup or deletion — needs a100 **single executor per occurrence** across all hosts and instances. Step 6 is mandatory for it.101- **Both** — split it into a per-instance refresh and a single-executor job. A write keyed by host102 (a heartbeat row per instance) is per-instance; make it an idempotent upsert on the host key.103- **Request-triggered** work runs on the host that served the request. Classify its delivery:104 best effort (an in-memory queue is fine), must eventually complete (durable queue or outbox), or must105 complete before the response (keep it in the request).106- **Cannot tell whether a write is shared** — treat it as shared.107108A worked inventory and classification of a real application, including which of its jobs must **not**109be guarded: [ref/worked-example.md](ref/worked-example.md).110111## Step 3: Port WebBackgrounder jobs112113WebBackgrounder depends on `System.Web` — its `JobHost` registers with `HostingEnvironment` — so it cannot114be referenced from ASP.NET Core. Rewrite each job as a `BackgroundService`, preserving these behaviours:115116| WebBackgrounder behaviour | What the port must do |117|---|---|118| `Execute()` often returns an **unstarted** `new Task(...)`; `JobHost` starts it | Never `await job.Execute()` on legacy code — an unstarted task never completes. Rewrite the body as `async Task RunAsync(CancellationToken)`. |119| `new Task(async () => ...)` is an `async void` lambda: the task completes at the first `await` | The real work was untracked and its exceptions escaped the scheduler. Await the work directly in the port. |120| `Start()` runs the **shortest-interval** job of each `JobManager` straight away; the others first run one interval later; after that each interval runs from the **end** of the previous run | Run first, then delay, for the job that ran at `Start()` — a delay-first loop leaves its cache cold after every restart. Delay first for the others, unless the legacy code also ran them at startup through a QBWI call or a constructor. Delay after each run. |121| One `JobManager` runs all its jobs **serially** on one timer | Separate services run concurrently. Keep jobs in one service only if they share state that is not thread-safe. |122| `Fail(...)` with `RestartSchedulerOnFailure = true`, or a coordinator that logs and swallows | A failed run was logged and the schedule continued. Keep that policy explicitly — see Step 7. |123| `RestartSchedulerOnFailure` left at its default, `false`, with a coordinator that lets faults through | One faulted run stopped the `JobManager`'s timer for good: every job in it stopped until the next recycle. Do not reproduce that. When adding the guard to the Framework copy, set `RestartSchedulerOnFailure = true` with a logging `Fail(...)` handler first — a claim surfaces store errors and lost claims as faults. |124| `SingleServerJobCoordinator`, or a custom coordinator that calls `Execute()` | No cross-instance coordination existed: every instance already ran the job. |125| `WebFarmJobCoordinator` with a `WorkItems` table | A cross-instance **mutex**, not once-per-interval: N servers each run it once per interval, without overlapping. See Step 6. |126| `Timeout` | Only expires a `WorkItems` reservation. It never cancelled a run. |127128Code for the port, including a scope per run: [ref/porting-patterns.md](ref/porting-patterns.md).129130## Step 4: Port QueueBackgroundWorkItem and IRegisteredObject131132| Legacy shape | Port to |133|---|---|134| QBWI from startup, often a `while (!token.IsCancellationRequested)` refresh loop | a `BackgroundService` |135| QBWI from a request, best effort | a bounded `Channel<T>` drained by a `BackgroundService`, with a DI scope per item |136| QBWI from a request whose work must eventually complete | a durable queue or an outbox table, drained by a single-executor job |137| One-off, loss-tolerant work with nothing to drain | `Task.Run` under `ExecutionContext.SuppressFlow()`, with its exception observed and `IHostApplicationLifetime.ApplicationStopping` as its token |138| `IRegisteredObject.Stop(bool immediate)` | `IHostedService.StopAsync`, or `ApplicationStopping` |139140Semantics that change:141142- ASP.NET tracks QBWI items and tries to delay AppDomain shutdown until they finish. The Generic Host143 waits only for hosted services' `StopAsync`, up to `HostOptions.ShutdownTimeout` (30 seconds by144 default). `Task.Run` work is **not** waited for, and an in-memory `Channel<T>` loses whatever is queued145 when the process stops. Neither is a substitute for durable delivery.146- QBWI does not flow `ExecutionContext`. Neither should the port: capture the values the work needs147 when it is queued. Never capture `HttpContext` or a scoped service — both are gone once the response148 completes.149- Bound every queue and decide what happens when it is full: wait, drop, or fail the request.150151Code: [ref/porting-patterns.md](ref/porting-patterns.md).152153## Step 5: Port console jobs to the Generic Host154155- **Keep the invocation contract.** A job that an external scheduler (Task Scheduler, cron, a triggered156 WebJob, a container job) starts once per trigger stays one-shot: run, set the exit code, exit. Do not157 turn it into a permanent loop — the scheduler is already the loop.158- A **continuous** runner — loop, sleep, repeat — becomes a `BackgroundService` whose delay takes the159 stopping token. A legacy `Task.Delay(sleep)` without a token blocks shutdown.160- Map the runner's own switches — run-once, delay, period — to configuration explicitly. The command-line161 provider ignores single-dash switches unless mapped, and a bare flag is not a value.162- Return a non-zero exit code whenever the occurrence was not completed — a failed run or a lost claim —163 and set it explicitly. Do not rely on an unhandled exception to produce one.164- While the legacy executable is still scheduled, its port is a second copy of the same job. Step 6165 applies to it exactly as it does to a web-hosted job, with **aligned windows** as the schedule model.166167Code: [ref/porting-patterns.md](ref/porting-patterns.md).168169## Step 6: Guard against duplicate execution170171Applies to every **single-executor** job (Step 2) that can run in more than one place at once: in both172hosts during side-by-side, in several instances of one host, in two IIS worker processes during an173overlapped recycle, in old and new versions during a rolling deployment, or as a legacy console job174still scheduled next to its port.175176### Invariants1771781. **Every copy takes the same guard** — Framework and Core, every instance. A guard only the new host179 takes is not a guard. The Framework job changes too, or is verifiably stopped first.1802. **Mutual exclusion is not one run per occurrence.** A lock stops two runs overlapping; it does not181 stop the Framework host running at 10:00 and the Core host running again at 10:00:05. The next-due182 time must live in the shared store, decided in the same atomic step as the claim.1833. **Decide with the store's clock**, never a host clock. Hosts drift.1844. **Fail closed.** If the store cannot be reached, the job does not run.1855. **Renew while running**, and cancel the work as soon as a renewal fails or cannot be confirmed — and186 when the lease runs out before one is confirmed, even if the renewal request is still stalled.1876. **A claim reduces duplicates; it cannot eliminate them.** A holder that crashes after its side188 effects but before recording the occurrence is retried, and so is one that finishes after its lease189 expired, so delivery is **at-least-once**. Make every protected effect idempotent — a natural key, an190 upsert, an idempotency key the receiver honours — or fenced: in the same transaction as the write, under191 an update lock, check the claim's generation, an active holder, and a live lease — see192 [Fencing a write](ref/duplicate-execution-guard.md#fencing-a-write). A fence only protects writes that193 run it, and a generation check alone is not a fence: Fail, Complete, and an expired but unclaimed lease194 all keep the generation. Keep each fenced transaction well under the renewal interval, or it blocks the195 holder's own renewal; split long work into short fenced batches.1967. **Key idempotency on the occurrence, fence on the attempt.** The generation changes on every claim,197 retries included, so an idempotency key built from it lets the retry repeat the effect. Build keys from198 the job name and the occurrence's due time, which retries share. Mail and third-party calls cannot be199 fenced; give them that key, or record that the user accepts a rare duplicate.2008. **Decide what a failed run means, per job.** Either retry the same occurrence after a delay the store201 enforces — so hosts do not retry hot — or end it and wait for the next interval, as WebBackgrounder did.202 Neither is a safe default for every job.2039. **Pick the schedule model per job.** Fixed delay (next run one interval after the last ended) matches204 in-process loops. A job an external scheduler starts on a calendar needs aligned windows, or a long run205 makes the next trigger find nothing due.20610. **Record the delivery guarantee, failure policy, and schedule model per job in `task.md`.**207 At-least-once is this recipe; at-most-once means completing before acting and accepting a lost208 occurrence when a holder crashes.209210### Choose a guard211212| Guard | Stops a recorded occurrence running again | Fencing token | Needs | Use when |213|---|---|---|---|---|214| SQL schedule row | yes — next-due time in the row, fixed delay or aligned windows | `Generation` column | one new table, SQL Server 2012+ or Azure SQL | both hosts reach a shared SQL Server database (default) |215| Azure blob lease with metadata | yes — approximate fixed delay in blob metadata | generation in metadata | a storage account, one blob per job, and a Framework claim you write and test yourself | no shared SQL database |216| `sp_getapplock` | no — mutex only | none | nothing | only alongside a next-due row, or where the job's own data records progress |217| Dedicated single worker process | no | none | a deployment change | moving the job out of the web hosts — and still take a claim: a replica count of one is not a mutex during rolling deployments or restarts |218| Configuration switch per job | no | none | configuration | a kill switch only; per-instance configuration is not atomic during a rollout |219220Recipes, the Framework-side code, and worked examples: [ref/duplicate-execution-guard.md](ref/duplicate-execution-guard.md).221The Core-side reference is [ref/scheduled-singleton-job.cs](ref/scheduled-singleton-job.cs).222223**An existing `WebFarmJobCoordinator` is not reusable as-is.** Two hosts using two different guards have224no guard. Either the Core port reserves through the **same** `WorkItems` table with the same `JobName`,225the same transaction, and the same timeout — which keeps the old mutex-only behaviour — or both hosts move226to the schedule row, Framework first. Moving is two Framework deployments: add the claim and **keep** the227coordinator, then remove the coordinator once every instance takes the claim. Removing it in the same228deployment leaves old and new instances holding different guards for the length of the rollout.229230**The schedule table is a schema change.** Load `managing-shared-database-schema` before creating it:231`get_instructions(kind='skill', query='managing-shared-database-schema')`. The table is an additive new232table created and seeded by the single schema owner's migration. Application logins hold DML only during233the window, so the application never runs `CREATE TABLE` at startup.234235### Handover236237A rolling deployment of the guard is itself a window in which guarded and unguarded processes coexist, so238the guard is activated only once no unguarded process remains:2392401. The schema owner creates the schedule table and seeds one row per job — interval and schedule model241 included — **inactive**: `NextDueUtc` far in the future, so no guarded copy runs yet.2422. Deploy the guard into the **Framework** job, on every instance. Wait until no unguarded process243 remains: the rollout has finished, old IIS worker processes have exited, and no unguarded console copy244 is still scheduled.2453. Activate: one `UPDATE` sets each row's `NextDueUtc` to its first occurrence. The job does not run between246 the last unguarded process stopping and this step, so keep that gap short. Confirm each run then247 advances `Generation` and that no run happens without a claim.2484. Deploy the Core port under the same job name. It reads the interval from the row, so the two hosts249 cannot disagree about it.2505. Watch several occurrences: one holder per generation, no overlap while leases are live, and no recorded251 occurrence run again.2526. Retire the Framework copy whenever convenient. Never remove the guard while two copies can run.253254**IIS keeps the Framework side moving under you.** With default settings an application pool stops after25520 idle minutes and starts on demand, so as traffic drains to Core, the Framework host's in-process jobs256silently stop — the guard lets Core claim the work, but per-instance jobs there simply stop. Overlapped257recycling briefly runs two worker processes, so a single-instance Framework host overlaps itself.258259## Step 7: Shutdown, startup, and failure policy260261- Pass the stopping token through every `await`. Raise `HostOptions.ShutdownTimeout` if a run needs262 longer than 30 seconds to wind down; work that ignores cancellation is abandoned at the timeout.263- Since .NET 6 an exception escaping `ExecuteAsync` **stops the host** — in a web host, the whole site.264 Preserve the legacy failure policy deliberately: if the old scheduler logged a failed run and carried265 on, catch per run, log, back off, and report consecutive failures to a health check. Let configuration266 and invariant errors at startup fail fast. Do not set `BackgroundServiceExceptionBehavior.Ignore`267 globally as a substitute.268- From .NET 10 all of `ExecuteAsync` runs off the startup path. On earlier targets its synchronous prefix269 blocks startup, so reach the first `await` immediately.270- A legacy run at startup — a constructor that did the work, or a QBWI kick — becomes the first271 iteration of `ExecuteAsync`, not constructor or `StartAsync` code.272- A `BackgroundService` is a singleton. Create a DI scope per run with `IServiceScopeFactory`; never273 inject a scoped `DbContext` into it.274275## Step 8: Verify276277- Per-instance jobs log runs from **both** hosts.278- Single-executor jobs, with both hosts against one store: runs never overlap while leases are live, and279 once an occurrence is recorded neither host runs it again; kill the holder mid-run and another host retries the same280 occurrence after the lease expires — the expected at-least-once repeat; a run that outlives its lease is281 not recorded; make the store unreachable and nothing runs while both hosts stay up; stop a host and its282 claim is released.283- The fence, for jobs that use one: suspend the holder past its lease (debugger or process suspend), let284 another host claim, resume, and confirm the stale attempt's fenced write is rejected with error 50002.285- A failing run leaves the web host serving, shows in the health check, and is retried or ended as the286 job's failure policy says.287- Request queues are bounded and behave as decided when full.288- One-shot console jobs return non-zero whenever the occurrence was not completed.289290## Decomposition Rules291292*(Read by the task-breaker when this skill is attached to a task being decomposed. Both `task-breaker`293and `task-executor` scan for this exact heading — do not rename it.)*294295**Restate the pin on every job subtask.** A subtask's `task.md` is written from the breakdown alone, so a296parent's `#skill:` pin does not reach it. Copy `#skill:migrating-inprocess-background-jobs` into every297subtask that ports, enables, schedules, guards, or retires a background job.298299**Pin the schema skill where the store is created.** A subtask that creates or seeds the schedule table300also carries `#skill:managing-shared-database-schema`.301302**Order the work.** Inventory and classification first; the guard in the Framework host before any303single-executor job is enabled in the Core host; the Core port next; retiring the legacy copy last.304Never create a subtask that enables a single-executor job in the Core host while any copy runs unguarded.305306## Success criteria307308- Inventory and classification written to `task.md`, every row citing `path:line`309- Per-instance jobs run in every instance of both hosts, unguarded310- Every single-executor job takes the same guard in every copy, with its next-due time in the shared store311- Delivery guarantee, failure policy, and schedule model recorded per job; protected effects idempotent,312 keyed by the occurrence, or fenced on generation, active holder, and live lease313- The schedule table created by the schema owner's migration, never by the application314- No legacy `Task` started unawaited, no `async void` job body, no scoped service captured by a job315- A failed run does not stop the web host; shutdown cancels runs and releases claims316- One-shot console jobs keep their scheduler and their exit codes