Architecture case study · open-source library

Reliable Webhook Delivery in Distributed Systems

An e-commerce platform must communicate order events even when receivers time out, return HTTP errors, become unavailable, or process the same delivery more than once.

Open repository ↗Production guide ↗

The e-commerce scenario is illustrative. The guarantees described here match the v1.0.0 API and tests; they are not claims of adoption, benchmarks, or production operation.

Context and problem

A direct POST cannot preserve intent when the network interrupts the conversation.

After recording an order, sending HTTP once couples a business decision to a remote dependency. A timeout does not reveal whether the receiver processed the request; 5xx responses may be temporary; a 4xx normally requires correction rather than infinite retries.

If the process crashes after the receiver accepts the message but before success is persisted, a later attempt may duplicate the event. The solution must make queued work, attempts, temporary ownership, and terminal states observable without turning distributed uncertainty into an exactly-once promise.

Architecture requirements

Reliability is a set of contracts, not merely a retry policy.

Reliability

With a conforming durable store, accepted work can survive the process and resume under an at-least-once model.

Idempotency

Stable IDs make enqueueing idempotent; the receiver still has to deduplicate effects.

Scalability

Bounded concurrency prevents over-claiming; leases coordinate workers that share a store.

Security

HTTPS is the default; HMAC-SHA256 authenticates the envelope when a signer and secret provider are registered.

Observability

Structured logs, ActivitySource, and Meter expose queueing, attempts, outcomes, and duration.

Recovery

An expired lease enables reclaim; retries persist their due time; terminal states stop being claimable.

At-least-onceIdempotencyBounded concurrencySecurityObservabilityEvolvability

Solution architecture

Separating intent, coordination, transport, and policy makes each guarantee explicit.

ReliableWebhooks flow: the application enqueues into a store, the dispatcher acquires a lease, the transport sends to the receiver, and success, retry, permanent failure, or dead letter is persisted with telemetry. — The core defines the protocol. Actual durability comes from the application-provided `IWebhookDeliveryStore`, while effect idempotency belongs to the receiver.

1. EnqueueIWebhookEnqueueService validates limits and uses a stable ID.
2. PersistenceIWebhookDeliveryStore keeps the message, state, and schedule.
3. ClaimWebhookDispatcher claims only due work within available capacity.
4. LeaseA token and expiration identify the active owner and reject stale writes.
5. TransportIWebhookDeliveryTransport represents one HTTP attempt.
6. TransitionSuccess, retry, permanent failure, or dead letter is persisted.

Enqueue and stable identity

The public service validates configured limits before persistence. The ID identifies the event–subscriber pair; enqueueing it again must return the existing record without replacing its payload, destination, or state.

Enqueue implementation ↗

Store as a port

IWebhookDeliveryStore isolates storage technology but requires atomic claims, renewal, stale-owner rejection, and terminal transitions. The package ships no durable adapter.

Persistence contract ↗

Dispatcher and leases

The dispatcher fills only available slots, renews a lease during an attempt, and cancels work after ownership is lost. On shutdown, it stops claiming and waits for the grace period.

Dispatcher source ↗

Replaceable transport

One SendAsync call is one attempt. The HTTP transport classifies 2xx as success, 408, 425, 429, and 5xx as retryable, and other statuses as permanent.

Default classifier ↗

Backoff with jitter

The default policy uses capped exponential backoff and positive jitter. A valid Retry-After may delay beyond the local cap; exhausting the attempt limit dead-letters the delivery.

Default policy ↗

HMAC and telemetry

The HMAC-SHA256 signer covers timestamp, ID, event type, content type, and exact payload bytes. Instrumentation publishes stable trace and metric names without payload, signature, or destination.

HMAC signer ↗

Public API

The application registers persistence, configures dispatch, and enqueues an immutable message.

// Production: the application supplies a conforming durable store.
services.AddSingleton<IWebhookDeliveryStore, MyDurableWebhookStore>();

ReliableWebhooksBuilder webhooks = services.AddReliableWebhooks(options =>
{
    options.Dispatcher.MaxConcurrency = 8;
    options.Dispatcher.LeaseDuration = TimeSpan.FromMinutes(2);
    options.Retry = new WebhookRetryPolicyOptions { MaxAttempts = 5 };
});
webhooks.AddHostedDispatcher();

WebhookMessage message = new(
    id: $"{orderCreated.EventId}:{subscriber.Id}",
    eventType: "order.created",
    destination: subscriber.WebhookUri,
    payload: JsonSerializer.SerializeToUtf8Bytes(orderCreated),
    contentType: "application/json");

await webhookEnqueueService.EnqueueAsync(message, cancellationToken);

AddHostedDispatcher() is opt-in. An application may instead resolve WebhookDispatcher and own its lifecycle directly.

Execution scenarios

The guarantees show up in failure paths, not only in the happy path.

Successful delivery

The dispatcher acquires a lease, the transport receives 2xx, and the store runs MarkSucceededAsync. Succeeded is terminal and is not normally claimed again.

Guarantee: a success transition by the current owner; duplicates remain possible if the endpoint accepted the request but persisting success failed.

Timeout followed by retry

The transport returns a retryable failure. The policy calculates a due time with backoff/jitter, and the store runs ScheduleRetryAsync, releases the lease, and preserves the attempt count.

Guarantee: another attempt after the schedule with a conforming store; a timeout does not prove the receiver did not process the request.

Permanent failure

A status such as 400 is permanent by default. The dispatcher calls MarkPermanentlyFailedAsync, and the item is not retried automatically.

Guarantee: a traceable terminal state. Exhausted retryable failures become DeadLettered, also without automatic replay.

Worker restart

With a durable store, pending work remains available, and an abandoned lease can be reclaimed after expiration. The former token cannot persist a late transition.

Limit: with InMemoryWebhookDeliveryStore, a restart loses the queue; there is no durable recovery in that mode.

Decisions and trade-offs

The library controls the protocol; the application retains domain- and infrastructure-dependent decisions.

At-least-once, not exactly-once

Retrying after an uncertain remote outcome preserves the chance of delivery but accepts duplicates. Exactly-once over HTTP would require transaction coordination the library and receiver do not share.

Leases, not uncoordinated execution

Temporary ownership supports multiple workers and abandoned-work recovery. The store must provide atomic claims, an authoritative clock, and token compare-and-set semantics.

Retry with backoff

Spacing attempts reduces pressure on an unavailable receiver; jitter reduces synchronized retry storms. Recovery latency increases, and each policy must match the remote contract.

Extensible persistence

Keeping the core independent of databases and ORMs makes the library reusable. The explicit cost is an application-owned conforming durable adapter for production.

Receiver idempotency

A stable ID helps deduplication, but the library cannot control the remote effect. The receiver chooses an appropriate authenticated key and persists its own idempotency decision.

Separated ports

Store, transport, classifier, retry policy, signer, and providers can evolve independently. Composition gains flexibility, while custom implementations must preserve documented contracts.

Responsibility split between ReliableWebhooks and the consuming application
Library responsibility	Application responsibility
Enqueue, claim, lease, attempt, and transition protocol.	Supply durable, atomic persistence appropriate for production.
Default HTTP transport, classification, and configurable retries.	Choose policy, limits, quotas, dead-letter operation, and any replay.
HMAC-SHA256 signer and secret/header provider hooks.	Manage secrets, rotation, destination authorization, and receiver verification.
Logs, traces, and metrics with stable public names.	Collect, export, alert, define SLOs, and protect persisted data.
Stable identity and idempotent enqueue in the store contract.	Make the receiver's business effects idempotent.

Limitations and evolution

Reliability only extends as far as the implementation truly supports.

The library does not guarantee exactly-once delivery, a distributed transaction with the receiver, receiver-side idempotency, automatic dead-letter replay, or a secret-rotation policy. The receiver must tolerate duplicate messages.

InMemoryWebhookDeliveryStore is process-local and not durable. The package ships no production store: the application must implement a conforming IWebhookDeliveryStore, including atomicity, leasing, coordination time, state persistence, and backend-specific tests.

PublicNetworkWebhookDestinationPolicy reduces SSRF risk for untrusted URLs but does not replace change authorization, allow-lists, egress controls, or connection-time DNS-rebinding protection. HTTPS provides confidentiality; HMAC authenticates content but does not encrypt the payload.

Outcomes and lessons

The verifiable benefit is making failure, ownership, and recovery part of the model.

Instead of hiding failures inside an HTTP loop, the design records when work exists, who may change it, when another attempt becomes eligible, and which states finish the flow. Concurrency and recovery can then be reasoned about through executable evidence.

The central lesson is about boundaries: retries do not create exactly-once, leases do not create durability, and HMAC does not create confidentiality. The architectural outcome depends on the library protocol, persistence adapter, observable operation, and an idempotent receiver working together.

Technical evidence

Documentation, implementation, tests, and a runnable sample available for inspection.

READMEScope, guarantees, quick start, and library boundaries. Production guideDI, retries, leases, HMAC, destination security, telemetry, and troubleshooting. Persistence contractDurability, atomic claiming, ownership, clocks, and required transitions. Conformance testsIdempotent enqueue, concurrency, reclaim, renewal, retry, and terminal states. Dispatcher testsSuccess, failure, concurrency, lease renewal, shutdown, and error isolation. Observability testsPublic names, logs, traces, metrics, and sensitive-data exclusion. Runnable sampleSuccess, retry, dead letter, signing, and diagnostics with the in-memory store. RepositorySource, documentation, tests, release, and project history.