Designing Reliable Webhooks in ASP.NET Core

A webhook is a distributed message, not “just an HTTP POST”

The sender and receiver do not share a transaction. A timeout cannot tell the sender whether the receiver committed its work. Networks duplicate, delay and reorder deliveries; endpoints deploy, certificates expire and customers return success before their background work fails.

Design the contract as at-least-once delivery. The sender may transmit the same logical event more than once, and the receiver must make repetition safe. Exactly-once business effect can sometimes be achieved through idempotency and constraints, but exactly-once network delivery is not a useful promise.

Also separate event from attempt. One event has stable identity and occurred at a specific logical time. Each delivery attempt has its own timestamp, response, latency and failure. Confusing them makes audit and support unable to answer whether the business fact duplicated or only its transport did.

Persist intent before attempting delivery

If an application updates business state and then calls a subscriber directly, a crash between those operations loses the webhook. Writing the event to an outbox in the same local database transaction closes that gap. A dispatcher can publish it after commit.

The outbox guarantees durable intent, not successful delivery. Workers still need leases or concurrency control, retry scheduling and idempotent status updates. Publishing before the business transaction commits creates the opposite error: subscribers observe an event for state that later rolls back.

Capture the event payload or a stable versioned representation at creation time. Rebuilding it later from current database state can rewrite history. Avoid storing unnecessary sensitive fields simply because the originating entity contains them.

Give events stable identity and versioned meaning

A useful envelope contains an event ID, event type, schema version, occurrence time, tenant or account context where appropriate and the payload. Delivery attempt IDs and signature timestamps belong to transport metadata rather than changing event identity.

Name events as facts such as invoice.paid, not vague notifications such as record.updated. Consumers need enough context to decide what happened without retrieving mutable state that may already have changed again.

Schema evolution should be additive where possible. Keep old fields through a deprecation period, tolerate unknown fields and distinguish absent from explicit null. Version the event contract when meaning changes, not whenever an internal class is refactored.

Sign the exact bytes that travelled

Signatures usually combine a timestamp, event or delivery identifier and the raw request body under a keyed MAC such as HMAC-SHA-256. The receiver must verify the bytes before model binding or reserialisation. Equivalent JSON can have different whitespace, property order or number formatting and therefore a different signature.

Use constant-time comparison and reject unsupported signature versions. Include a key identifier so rotation can overlap old and new keys. Do not log signing secrets, full signature material or sensitive payloads during troubleshooting.

TLS protects transport to the endpoint; it does not prove the request originated from the expected webhook producer once intermediaries and public endpoints are considered. A shared signing secret authenticates possession, while asymmetric signatures can reduce shared-key distribution at greater operational complexity.

Replay protection needs more than a timestamp

A signed request can be captured and resent. Require the signature timestamp within a bounded tolerance and store the event or delivery identity so duplicates do not repeat the business effect. Timestamp tolerance must allow realistic clock skew without making the replay window unnecessarily large.

Do not mark an event consumed before its local transaction succeeds. Record idempotency and business state atomically, or a crash can cause later legitimate redelivery to be discarded. A unique database constraint is often a stronger deduplication control than a check followed by an insert.

Keep deduplication records at least as long as the sender may retry, including manual replay. Expiring them too early turns an old retry into a new event.

Return success only after the receiver owns the work

A receiver should authenticate, validate basic envelope structure, persist the event durably and then respond quickly. Expensive business processing belongs behind a queue or inbox table. Returning 200 before durable persistence creates acknowledgement without ownership.

Conversely, completing a ten-minute workflow inside the request encourages sender timeouts and duplicate attempts. A 2xx response should mean the receiver has accepted responsibility, not necessarily completed every downstream action.

Choose status codes deliberately. Invalid signatures should not invite infinite retries. Temporary capacity failure may deserve retry. Unknown subscriptions should be disabled or treated as permanent failure. Document how the sender classifies responses rather than relying on every receiver to guess.

Idempotency is a business decision

Recording that an event ID was seen prevents exact duplicate processing, but related events can still arrive out of order. A delayed customer.updated might overwrite newer state. Use source versions, occurrence sequence or current-state queries where ordering matters.

Some effects are naturally idempotent: set order status to Paid if its version advances. Others require an explicit operation key: issue a refund once for event 123. Sending an email may tolerate duplicates differently from moving money.

Do not wrap all processing in a long database transaction while calling external services. Persist local intent and use an outbox for further effects. Idempotency has to continue across each asynchronous boundary.

Retry schedules should protect both organisations

Use bounded exponential backoff with jitter for transient network faults, throttling and selected server errors. Respect Retry-After within the delivery policy. Immediate tight retries turn a receiver outage into additional denial of service.

Set a maximum delivery age or attempt count, then move the event to a failed state. “Retry forever” grows storage, obscures permanent configuration errors and may surprise a customer with a months-old side effect.

Expose manual replay with guardrails. Operators should see payload version, attempts, last response and whether the event may already have succeeded. Replay should retain the event ID so receiver idempotency remains effective.

Subscriber URLs create an SSRF boundary

A SaaS product that sends webhooks to customer-supplied URLs can be abused to request cloud metadata, loopback addresses and internal services. Validating only the original string is insufficient because DNS resolution and redirects can cross the boundary.

Allow intended schemes, resolve and block private, loopback and link-local destinations as policy requires, and revalidate every redirect and connection. Prefer controlled egress and disable redirects unless necessary. Limit response size, connection time and total duration.

Ownership verification, such as a challenge exchange, reduces accidental misconfiguration but does not replace network controls. Re-resolve carefully on later deliveries because DNS can change after registration.

Secrets and subscription lifecycle need product design

Give each subscription an independent signing secret so one compromise does not affect every customer. Show it only through a protected workflow, store it securely and support overlapping rotation. Revocation should stop future delivery promptly without deleting evidence of earlier attempts.

Subscriptions need states such as pending verification, active, paused and disabled. Repeated permanent failures should pause delivery and notify an authorised administrator rather than consuming resources indefinitely.

Event selection is an authorisation decision. A subscriber permitted to receive billing events may not be permitted to receive participant details. Scope topics and payload fields, and re-evaluate access when tenant permissions or ownership changes.

Observability should explain one event's life

Track event creation, availability for dispatch, every attempt, response classification, next retry and final state under stable event and subscription IDs. Metrics should show delivery latency, success by attempt, queue age, disabled subscriptions and failure categories.

A dashboard reporting 99% HTTP success can hide one customer receiving nothing. Partition views by tenant and destination while protecting URLs and payload data. Alert on sustained customer impact and growing backlog, not each expected transient failure.

Give customers a safe delivery history containing times, status, event type and redacted response evidence. Self-service diagnosis reduces support access to sensitive payloads and helps receivers distinguish signature, availability and application-processing faults.

Test failure, not merely the sample request

Contract tests should verify raw-body signature handling, key rotation, unknown fields and time tolerance. Reliability tests should force timeouts after receiver commit, duplicate delivery, reordering, worker crash and outbox replay.

Load tests need slow and failing destinations, because healthy localhost receivers do not reveal connection and queue exhaustion. Confirm per-subscription fairness so one failing endpoint cannot starve all others.

Finally, rehearse recovery from lost credentials, accidental subscription deletion and dispatcher outage. A reliable webhook service is an operated messaging product, not a controller action plus a retry loop.

Production checklist

• Commit webhook intent with business state through an outbox.
• Separate stable event identity from delivery attempts.
• Version domain-oriented event contracts.
• Sign raw bytes and support key rotation.
• Reject replay while keeping processing atomic.
• Acknowledge only after durable acceptance.
• Make every business effect idempotent.
• Bound retries, backlog and delivery age.
• Protect customer destinations against SSRF.
• Expose event-level observability and safe replay.