# Background Work And Hosted Services > Review .NET background processing - BackgroundService loops, scoped dependency resolution, graceful shutdown, timers, queue consumption, and outbox patterns. Use when reviewing IHostedService, BackgroundService, recurring jobs, or queue consumers. - Skill: `sarmkadan/background-work-and-hosted-services` (Agent Skill) - Install (CLI): `npx skillmds@latest add sarmkadan/background-work-and-hosted-services` - Raw SKILL.md: https://api.skillmd.com/api/skills/sarmkadan/background-work-and-hosted-services/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: Sarmkadan (https://skillmd.com/u/sarmkadan) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/sarmkadan/background-work-and-hosted-services --- # Background Work and Hosted Services ## The loop that must not die `ExecuteAsync` is called once. An unhandled exception ends the service silently for the rest of the process lifetime (pre-.NET 8) or tears down the whole host (.NET 8+ default `BackgroundServiceExceptionBehavior.StopHost`). Neither is what a polling loop wants - it wants to log and continue: ```csharp // non-compiling: illustrative // WRONG: first transient DB blip permanently stops processing (or kills the app) protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { await ProcessBatchAsync(stoppingToken); await Task.Delay(_interval, stoppingToken); } } // RIGHT: failure of one iteration is logged, backed off, and survived protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { try { await ProcessBatchAsync(stoppingToken); } catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) { break; } catch (Exception ex) { _logger.LogError(ex, "Batch failed, retrying after backoff"); } await Task.Delay(_interval, stoppingToken); } } ``` This loop is one of the three sanctioned homes of `catch (Exception)`. The `OperationCanceledException` filter matters: cancellation during shutdown exits cleanly instead of logging a spurious error. ## Scoped services in a singleton world Hosted services are singletons; `DbContext` is scoped. Injecting it is a captive dependency - one context instance living for the process lifetime, accumulating tracked entities and breaking on concurrent iterations. One scope per unit of work: ```csharp using var scope = _scopeFactory.CreateScope(); var db = scope.ServiceProvider.GetRequiredService(); ``` Per iteration or per message, not per service lifetime. A scope held for hours is the same bug with extra steps. ## Graceful shutdown - Honor `stoppingToken` everywhere: every await in the loop takes it. A service ignoring it stalls shutdown until `HostOptions.ShutdownTimeout` (default 30s) expires and in-flight work is killed mid-write. - Work that must not be killed mid-transaction: cancellable dequeue phase, non-cancellable commit phase (`CancellationToken.None`, explicitly) - and keep the commit short enough to finish inside the shutdown window. - `StopAsync` overrides that await completion of in-flight work are correct; `StopAsync` doing new work is not. - Kubernetes note: SIGTERM starts the shutdown clock; `terminationGracePeriodSeconds` must exceed `ShutdownTimeout` plus your longest commit, or the pod is SIGKILLed anyway. ## Timers are worse than loops `System.Threading.Timer` firing an async callback is fire-and-forget: overlapping executions when work outlasts the interval, and exceptions vanish. The `while + Task.Delay` loop (or .NET 8+ `PeriodicTimer`) is strictly better - naturally non-overlapping, exception-visible, cancellation-aware: ```csharp using var timer = new PeriodicTimer(_interval); while (await timer.WaitForNextTickAsync(stoppingToken)) { await RunOnceAsync(stoppingToken); } ``` Review flag: any `new Timer(...)` in a hosted service, and any recurring schedule expressed in local time (see datetime skill - DST skips/doubles it). ## Queue consumers - Ack/complete the message only after the work committed. Ack-then-process converts every crash into silent message loss. - Every consumer assumes at-least-once delivery: handlers are idempotent (dedupe table keyed on message id, or naturally idempotent upserts). "The queue delivers exactly once" appearing in a design doc is a review rejection by itself. - Poison messages: bounded retry with backoff, then dead-letter with the exception attached. An unbounded redelivery loop on a permanently failing message pins the consumer at 100% doing nothing. - Concurrency limit is explicit (`MaxConcurrentCalls`, prefetch count) and sized against the downstream dependency, not defaulted. ## Unbounded resource consumption (DoS) from external input Any queue, channel, or background dispatch fed by external input must have a bounded capacity to prevent unbounded memory growth and denial-of-service through resource exhaustion. Untrusted input sources include HTTP endpoints, message queues, or any public API that can enqueue work. ### The unbounded channel vulnerability Feeding an unbounded `Channel` directly from a public endpoint allows an attacker to enqueue arbitrarily large payloads, causing unbounded memory growth and potentially crashing the process: ```csharp // VULNERABLE: unbounded memory growth from untrusted input public class UnsafeController : ControllerBase { private readonly Channel _channel = Channel.CreateUnbounded(); [HttpPost("enqueue")] public async Task Enqueue([FromBody] LargePayload payload) { // No size or rate limiting - attacker can send MBs of data await _channel.Writer.WriteAsync(payload); return Ok(); } } // Hosted service drains the channel public class PayloadProcessor : BackgroundService { private readonly Channel _channel; protected override async Task ExecuteAsync(CancellationToken stoppingToken) { await foreach (var payload in _channel.Reader.ReadAllAsync(stoppingToken)) { await ProcessAsync(payload, stoppingToken); } } } ``` The above allows an attacker to send gigabytes of data, filling memory until the process crashes. The channel has no backpressure mechanism. ### Bounded channel with backpressure Use `Channel.CreateBounded` with a capacity limit and proper error handling: ```csharp // SAFE: bounded capacity with backpressure public class SafeController : ControllerBase { // Bounded channel with capacity limit private readonly Channel _channel = Channel.CreateBounded( new BoundedChannelOptions(1000) // Max 1000 items { FullMode = BoundedChannelFullMode.Wait, // Blocks when full SingleReader = true, SingleWriter = false }); [HttpPost("enqueue")] public async Task Enqueue([FromBody] LargePayload payload, CancellationToken ct) { try { // Will block if channel is full, providing natural backpressure await _channel.Writer.WriteAsync(payload, ct); return Ok(); } catch (OperationCanceledException) { return StatusCode(503, "Service busy - try again later"); } catch (ChannelFullException) { return StatusCode(429, "Too many requests - queue full"); } } } // Hosted service drains the channel public class PayloadProcessor : BackgroundService { private readonly Channel _channel; private readonly ILogger _logger; protected override async Task ExecuteAsync(CancellationToken stoppingToken) { await foreach (var payload in _channel.Reader.ReadAllAsync(stoppingToken)) { try { await ProcessAsync(payload, stoppingToken); } catch (Exception ex) { _logger.LogError(ex, "Failed to process payload"); // Continue processing next item } } } } ``` Key improvements: - **Bounded capacity**: Channel has a fixed maximum size (1000 items) - **Backpressure**: When full, writers block instead of accepting unbounded memory growth - **Error handling**: Proper exception handling prevents crashes from corrupt data - **Rate limiting**: Returns 429 when queue is full instead of accepting more work - **Cancellation support**: Respects cancellation tokens from HTTP layer ### Task.Run fan-out without concurrency cap Similarly, spawning unbounded `Task.Run` calls from a request handler creates a DoS vector: ```csharp // VULNERABLE: unbounded parallelism from untrusted input [HttpPost("process-all")] public IActionResult ProcessAll([FromBody] List ids) { // Attacker sends 100,000 IDs -> 100,000 concurrent tasks foreach (var id in ids) { Task.Run(() => ProcessItemAsync(id)); // No limit! } return Ok(); } ``` Use `Parallel.ForEachAsync` with a bounded concurrency instead: ```csharp // SAFE: bounded concurrency [HttpPost("process-all")] public async Task ProcessAll([FromBody] List ids, CancellationToken ct) { // Max 10 concurrent operations await Parallel.ForEachAsync( ids, new ParallelOptions { MaxDegreeOfParallelism = 10, CancellationToken = ct }, async (id, innerCt) => await ProcessItemAsync(id, innerCt) ); return Ok(); } ``` ### Rules for external input feeds 1. **Bounded capacity required**: Any channel, queue, or collection fed by external input must have a fixed maximum size 2. **Backpressure**: Use bounded channels with `BoundedChannelFullMode.Wait` or throw when full 3. **Rate limiting**: Return 429 (Too Many Requests) when capacity is reached 4. **Size limits**: Validate payload sizes before enqueuing (e.g., max 1MB per item) 5. **Concurrency caps**: Use `Parallel.ForEachAsync` with `MaxDegreeOfParallelism` or explicit semaphores 6. **Timeouts**: Always respect cancellation tokens from external sources 7. **Monitoring**: Track queue depth and reject requests when approaching capacity limits Review flags: - Any `Channel.CreateUnbounded()` fed directly from a controller/action - Any `Task.Run`/`Task.Factory.StartNew` without concurrency limits - Any unbounded collection growth from HTTP input without validation - Missing `CancellationToken` parameters on public endpoints feeding background work ## Scheduling work from requests A request handler that needs work done after the response: do not `Task.Run` (captured scope dies with the request - see async skill). Minimum viable: a singleton `Channel` written by the handler, drained by a hosted service. But if the work must survive a process restart, an in-memory channel is not a queue - use the outbox pattern (work row committed in the same transaction as the business change, relayed by a background service) or a durable queue. The review question is always "what happens if we deploy mid-flight?" - in-memory answers "the work is gone".