Designing Resilient .NET Integrations with External APIs

Classify the business operation before choosing a policy

A catalogue lookup, address validation, payment creation and bulk export cannot safely share one generic resilience pipeline. Record whether repetition is safe, whether a timeout can conceal success, the maximum acceptable staleness and the end-to-end business deadline.

HTTP method is useful evidence but not a complete classification. GET should be safe, yet a poorly designed provider may attach side effects. PUT is defined as idempotent at protocol level, but an endpoint can still send a notification on every call. POST can be made safely repeatable when the provider honours a stable idempotency key.

The most important distinction is between failed and unknown. A connection rejected before sending probably failed. A timeout after the request body left the process may hide a successful remote commit. Retrying an unknown write without reconciliation can create the incident the retry policy was meant to prevent.

Every control solves one narrow failure

Combining controls without understanding their interaction can be worse than using none. Retries increase load during failure; long per-attempt timeouts consume the total budget; a queue can move overload from memory into an ever-growing backlog. Resilience is a bounded system, not a bag of middleware.

Start with one end-to-end timeout budget

If a user request must finish in five seconds, three ten-second attempts are fiction. Divide one total budget across local queueing, connection, individual attempts, response reading and parsing. Leave time for the application to return a meaningful result.

Connection timeout and request timeout answer different questions. A service may connect quickly and then stream forever. Reading only response headers can release control earlier for large downloads, but the body still needs cancellation, size limits and disposal.

Propagate cancellation so abandoned work does not consume capacity unnecessarily. Cancellation stops the local wait; it does not reverse a request already processed remotely. Treating OperationCanceledException as proof that nothing happened is a correctness bug.

Retry narrowly-and assume defaults need review

Retry transient connection faults, throttling and selected server failures with a bounded exponential delay and jitter. Honour Retry-After where it fits the business deadline. Do not retry validation, authentication, contract deserialisation or every 500 response blindly.

Current .NET resilience handlers can compose rate limiting, total timeout, retry, circuit breaking and attempt timeout. That convenience does not classify your operation. The standard handler retries all HTTP methods unless unsafe methods are disabled or the strategy is customised. A production team must review policy per client and operation rather than equating a package default with business safety.

Limit attempts and total elapsed time. A retry that completes after the caller has already shown failure can create confusing late side effects. Record every attempt under one logical operation identifier so telemetry distinguishes customer work from retry amplification.

Idempotency belongs to the business boundary

For a create operation, generate a stable client operation identifier before the first attempt and reuse it across retries. The provider should persist the key with the result and return the same outcome for duplicates. A random key generated per attempt provides no protection.

Key scope and expiry matter. It should be unique within the relevant customer and operation domain, survive longer than plausible retries and reject reuse with a materially different payload. Hashing the canonical request can help detect accidental key collision, but canonicalisation must be stable.

If the provider offers no idempotency mechanism, persist a local “outcome unknown” state and query using a business reference before resubmitting. Locks in your own application cannot prevent duplicates inside somebody else's system.

Circuit breakers protect capacity, not correctness

A breaker observes recent failures and temporarily blocks calls when a threshold is exceeded. It prevents every request waiting on a dependency already known to be unhealthy and gives that dependency room to recover. It does not make the customer's operation succeed.

Scope the breaker to a real failure domain. A global breaker can disable healthy endpoints or regions because one operation fails; an unbounded breaker per tenant creates uncontrolled state. Often provider, endpoint class and credential boundary provide a reasonable compromise.

Half-open probes need limited concurrency and observable results. When the breaker opens, choose a business behaviour: queue acceptable work, serve explicitly stale reference data, disable an optional feature or return a clear temporary state. Throwing faster is load protection, not graceful degradation.

Bound concurrency before the dependency consumes the application

IHttpClientFactory manages handlers and helps avoid common connection-pool and DNS-lifetime problems, but it does not guarantee fair use of a dependency. Hundreds of concurrent slow calls can still exhaust connection slots, memory and downstream quota.

Limit concurrent expensive operations and bound the waiting queue. Once the queue is full, reject or defer work deliberately. Unlimited waiting converts overload into latency and memory growth. Partition limits when one tenant or job type could starve interactive traffic.

Provider rate limits and local concurrency limits are related but different. A request-per-minute quota needs pacing; concurrency limits cap simultaneous work. Adaptive throttling can respond to 429s, but it should not oscillate so aggressively that recovery creates another burst.

A 200 response can still be an integration failure

Providers return missing fields, unknown enum values, error objects inside successful responses and semantically impossible values. Validate contracts at the adapter boundary before external data enters the domain. Quarantine invalid payloads and expose a diagnosable status rather than converting them into misleading defaults.

Keep provider DTOs behind an anti-corruption layer. The rest of the application should depend on internal meanings, not a vendor's naming and optionality. This makes contract drift visible in one place and prevents supplier changes spreading across business logic.

Consumer tests and sandboxes are useful, but production is where throttling, unusual data and undocumented size limits appear. Monitor schema and semantic failures separately from transport availability.

Choose synchronous or asynchronous delivery from business latency

If the user cannot continue without an answer, a synchronous call may be necessary. If completion can happen later, persist intent and queue work with a status the user can revisit. Do not keep an HTTP request open through minutes of retries merely because the first implementation was synchronous.

Queued delivery normally provides at-least-once processing. Consumers still need idempotency, retry classification and poison-message handling. A dead-letter queue is not recovery; it is storage for work that now needs diagnosis, correction and controlled replay.

Backpressure must reach the entry point. Accepting unlimited jobs while workers fall behind creates an invisible outage with an impressive queue depth. Set age and size objectives, reject excess demand or communicate delayed completion honestly.

Fallbacks must not invent authoritative truth

Serving cached product descriptions during a catalogue outage may be acceptable. Reusing an old account balance, permission decision or fraud result may not be. Define freshness and provenance per data type, and show when a value is stale if users act on it.

A fallback should reduce capability safely. “We cannot verify this now” can be the correct outcome. Catching every exception and returning an empty list is dangerous because absence becomes indistinguishable from dependency failure.

Cache keys must include tenant and other authorisation context. Resilience that leaks another customer's data is not resilience. Cache invalidation and outage behaviour should be tested together rather than treated as independent concerns.

Reconciliation closes the uncertainty gap

Payments, shipments and submissions need a scheduled process that queries by stable operation reference, compares local and remote state and repairs divergence. Reconciliation is not an incident-only database script; it is part of the integration contract.

Model states such as pending, confirmed, rejected and outcome unknown explicitly. Preserve attempt history without making technical attempt count the business state. Operators need safe actions-query, retry, cancel or mark resolved-with authorisation and audit.

Decide which system is authoritative for each fact. “Last write wins” is not a reconciliation policy when both systems represent different responsibilities. Some conflicts require a person and a reasoned resolution rather than automatic overwriting.

Observe the logical operation, not just HTTP calls

Trace all attempts, queue hops and callbacks using one correlation or operation identifier. Record provider, operation, outcome class, attempt count, latency and breaker state without logging credentials or sensitive bodies. High-cardinality customer identifiers need controlled handling.

Useful metrics include total operation success, unknown outcomes, retry amplification, throttling, queue age and reconciliation lag. A healthy average response time can conceal a growing population of unresolved business records.

Alert on customer impact and exhausted recovery capacity, not every transient retry. Resilient systems expect individual failures; operators need to know when the designed controls are no longer containing them.

Production checklist

• Classify safety, freshness and deadline per operation.
• Apply one end-to-end budget with bounded attempts.
• Review .NET resilience defaults for unsafe methods.
• Use stable idempotency keys for retried writes.
• Scope breakers to meaningful failure domains.
• Bound concurrency, queue size and workload fairness.
• Validate successful response semantics at the adapter.
• Make stale and degraded behaviour explicit.
• Persist uncertain outcomes and reconcile them.
• Trace attempts as one business operation.