Background Processing in .NET: Queues, Workers and Scheduled Jobs

Background is a delivery promise

Task.Run inside an endpoint can outlive the request but not necessarily the process. A deployment, crash or scale-in can erase the task after the API told the user it was accepted. If the work matters, acceptance must be recorded durably before success is returned.

Decide whether work is best effort, at least once or coordinated through a durable job record. “Exactly once” is rarely a property of distributed delivery; reliable systems tolerate repeated delivery and make business effects idempotent.

Choose the mechanism from ownership and durability

Do not introduce a broker for a low-value in-process refresh. Do not use process memory for invoices, payments or customer notifications whose loss would require explanation.

BackgroundService needs a lifetime plan

A hosted service is normally singleton. Create a DI scope per message or unit of work for scoped services such as DbContext, await the work and dispose the scope. Never resolve a scoped handler once and retain it for the worker lifetime.

Respect the stopping token, stop accepting new work during shutdown and give in-flight operations a bounded drain period. Readiness should fail while a worker cannot safely consume; liveness should not restart a healthy process merely because one dependency is temporarily unavailable.

Unhandled exceptions can terminate execution according to host behaviour. Make the top-level loop's failure policy explicit and ensure repeated catastrophic failure becomes visible rather than a silent tight restart loop.

Queue acknowledgement is the transaction boundary

Complete a message only after its required effects are durable. Completing first risks loss; completing after work means a crash can redeliver. That redelivery window is why handlers need idempotency.

A database update and outgoing event cannot usually share a transaction with the broker. Use an outbox: commit business state and an event record together, then publish and mark the outbox separately. Consumers need an inbox or business idempotency rule where repeated effects matter.

Do not hold a message lock during unpredictable human or external workflows. Persist a job state and continue through explicit messages or a workflow mechanism.

Idempotency belongs to the business effect

Checking whether a message ID was seen can suppress transport duplicates, but the same business command may arrive with a different ID. Use a stable operation key where the domain has one: payment reference, report request or import batch.

Store the key and outcome atomically with the side effect. An in-memory set is lost on restart. A separate “processed” write before the business commit can mark failed work as complete; after the commit it leaves a duplicate window.

Idempotency is not always “do nothing.” A repeat may need to return the original outcome, continue an incomplete step or verify that the requested state already exists.

Retries need a budget and classification

Retry transient throttling, network faults and temporary unavailability with bounded exponential backoff and jitter. Do not retry validation, authorisation or unsupported contract failures. A deserialisation error will not heal on attempt twenty.

Account for retries at every layer. SDK, handler, queue and scheduler retries can multiply into a storm. Set one end-to-end attempt and elapsed-time budget, honour provider guidance and expose attempt count in telemetry.

A downstream timeout may have committed work. Use idempotency before retrying non-read operations. A circuit breaker reduces pressure on a failing dependency but should release the message for delayed retry rather than hold worker capacity waiting.

Poison messages are operational work queues

After a bounded number of attempts, move unprocessable work to a dead-letter or failed state with reason, trace, contract version and safe identifiers. Do not include secrets or complete sensitive payloads merely to aid diagnosis.

A dead-letter queue without alerts, ownership and replay tooling is delayed data loss. Operators need to inspect, correct where allowed, replay idempotently and record disposition. Bulk replay must respect downstream capacity and avoid repeating already completed effects.

Ordering is expensive and narrower than it sounds

Global ordering destroys parallelism and is rarely required. Most domains need ordering per aggregate, account or session. Partition on that key and include sequence or version information so consumers can detect gaps and stale events.

Even ordered brokers can redeliver and consumers can observe effects out of order across queues. Make state transitions conditional on expected version rather than assuming arrival order is truth.

Scheduled work has calendar semantics

“Every 24 hours” and “at 09:00 Europe/London every day” are different schedules across daylight-saving changes. Store the schedule's timezone and recurrence rule, calculate each next occurrence explicitly, and decide what happens to missing or duplicated local times.

Multiple scheduler instances can trigger the same job. Use a distributed lease, unique execution record or idempotent job key. Decide whether a missed run after downtime is skipped, coalesced or replayed; catching up every minute of a two-day outage can cause another outage.

Bound concurrency before scaling out

Queue depth is not permission to start unlimited work. Constrain consumers by database connections, provider quotas, CPU and memory. Prefetch should not lock more work than an instance can finish before leases expire.

Scale on queue age and completion rate as well as count. One old message among new traffic can reveal starvation. Backpressure is part of correctness when downstream services cannot absorb burst load.

Observe business completion

Measure accepted, started, completed, retried, dead-lettered and age-at-completion. Correlate HTTP acceptance to job and message identifiers across retries. Queue health alone cannot show that invoices were produced correctly.

A 202 response should return a durable operation identifier and status resource where users need progress. Terminal failure must be visible to the initiating workflow; logging an exception in a worker is not user communication.

Production checklist

• Durably record work before acknowledging acceptance.
• Create and dispose a DI scope per work unit.
• Complete messages only after durable effects.
• Make effects idempotent with atomic keys and outcomes.
• Classify retries and cap the end-to-end budget.
• Give dead-letter work ownership and replay tooling.
• Limit concurrency from downstream capacity.
• Define ordering only at the required scope.
• Model recurrence timezone and missed-run policy.
• Measure business completion and oldest work age.

Related C3 Software guides

• Azure Service Bus Patterns for Reliable Business Workflows
• Dependency Injection in .NET: Practical Patterns and Common Mistakes
• Error Handling in ASP.NET Core APIs That Clients Can Rely On