Webhooks and event delivery from first principles — the engineering discipline for the callbacks the SaaS economy synchronizes on, for both receivers and providers. Why webhooks are hard (the real contract: at-least-once, unordered, bounded retries, latency-sensitive, point-in-time payloads — loose by necessity; webhooks as state synchronization over an unreliable channel), delivery semantics (the two-generals shape of a POST; exactly-once delivery as myth, exactly-once effect as receiver-built; backoff, jitter, bounded horizons, per-endpoint isolation, ack-on-durable-acceptance), idempotent consumption (atomic event-ID claims, idempotent effects via same-transaction record-and-effect, version-gated applies, fetch-on-event, bounded parking, per-resource serialization), security (HMAC over raw bytes with constant-time comparison and secret rotation, signed-timestamp replay windows, fail-closed — and the provider side as SSRF-by-design: resolve-validate-pin, egress isolation, bounded responses), the durable intake architecture (thin verifying endpoint → queue → idempotent workers, DLQs that are monitored/diagnosable/replayable, poison-event quarantine), thin vs fat events (stale-by-construction fat payloads vs doorbell-and-fetch convergence; fat for facts, thin for state; events plus reconciliation polling), building the provider platform (outbox capture, per-endpoint filtered fan-out, actuarial endpoint health, suspension that warns and retains, delivery logs and bulk redelivery), and reconciliation and durable sync (modified-since sweeps counting missed deliveries, deletion detection, rehearsed backfill — events for latency, reconciliation for truth, replay for recovery). Grounded in Wikipedia (webhook, EDA, exponential backoff, two generals, idempotence, race conditions, HMAC, SSRF, message queues, DLQs, pub-sub, polling, rate limiting, multitenancy, CDC, eventual consistency).
Webhooks look like the easiest integration pattern in software: the provider POSTs you JSON when something happens. Then production arrives — events show up twice, out of order, late, or not at all; a deploy drops an hour of deliveries; a signature check fails silently — and the team discovers that webhooks are a distributed system whose other half they don't control. This series is the missing engineering discipline for both sides of that contract.
The real webhook contract: at-least-once, unordered, bounded retries, latency-sensitive, point-in-time payloads — loose by necessity, not negligence. The provider delivers possibilities; the receiver establishes truth. Webhooks are state synchronization over an unreliable channel, and this series is the missing discipline for both sides.
Every delivery guarantee is a bet about failure, and webhook systems inherit the oldest result in distributed messaging: between two parties over an unreliable channel, you can deliver at-most-once or at-least-once, but exactly-once delivery is not on the menu — only exactly-once effect, manufactured by the receiver. This post builds that result from first principles and derives the retry machinery both sides need: schedules, backoff, jitter, and the acknowledgment discipline that makes them work.
A webhook POST is the two-generals problem in miniature: the ack can always be lost, so the choices are at-most-once (silent loss) or at-least-once (loud duplicates) — and providers rightly choose loud. The retry machinery that makes it livable: exponential backoff, jitter, bounded published horizons, per-endpoint isolation.
Post 2 ended with the only honest promise available: at-least-once delivery, duplicates included. This post builds the receiver that makes duplicates harmless and disorder survivable — the dedup store, the effect-level idempotency that outranks it, and the versioned-state techniques that let events arrive in any order and still converge on the truth. This is where "exactly-once" actually gets manufactured.
Three layers make duplicates harmless and disorder survivable: atomic event-ID claims with honest retention, idempotent effects (absolute operations, record-and-effect in one transaction, derived keys on downstream calls), and disorder recovery — version-gated applies, fetch-on-event, bounded parking, per-resource serialization.
A webhook endpoint is a public URL that accepts POSTs and changes state in your systems — which is to say, it's an unauthenticated write API unless you make it otherwise. The security model is well understood and regularly skipped: verify signatures correctly (several popular ways to do it wrong), bound the replay window, and remember that on the provider side, delivering webhooks means making HTTP requests to arbitrary customer URLs — an SSRF machine by design. This post covers both directions of the attack surface.
Receivers: HMAC over raw bytes, constant-time, scheme-aware, rotatable secret lists; signed-timestamp replay windows; fail closed loudly — and schema-validate anyway. Providers: webhook delivery is SSRF by design — resolve-validate-pin destinations, egress-isolate workers, bound and never echo responses.
The naive webhook handler does everything inline: receive, verify, run business logic, call three services, return 200. It works in the demo and fails in every interesting way in production — timeouts under load, lost events on deploy, one bad payload blocking the stream. The production architecture is almost boringly uniform across serious teams: a thin endpoint that durably enqueues and immediately acks, workers that process from the queue, and explicit machinery for the events that refuse to process. This post builds that architecture piece by piece.
The production receiver: a thin verifying endpoint that durably enqueues and acks in milliseconds, workers processing idempotently partitioned by resource, bounded retries for transients, and a monitored, diagnosable, replayable DLQ so one poison payload never dams the stream. The dashboard falls out of the design.
Every webhook program eventually argues about payload shape: send the whole resource ("fat"), or just an ID and a hint ("thin")? The argument feels stylistic and is actually architectural — it decides your staleness semantics, your security exposure, your versioning burden, and whether consumers converge on truth or drift from it. This post makes the trade explicit, explains why providers have been drifting thin, and locates the pattern most receivers should default to: the event as doorbell, the API as truth.
Fat events are self-contained but stale-by-construction, data-exposing, and a second API to version; thin events are doorbells whose fetch returns authorized current truth and dissolves ordering problems. Default: doorbell, queued fetch, version-gated upsert — fat for immutable facts, thin for mutable state.
The moment your product offers webhooks, you've signed up to run a delivery platform: fan-out to thousands of endpoints you don't control, operated by teams of wildly varying competence, every one of them capable of being slow, down, or misconfigured — without letting any of them degrade the others or your own system. This post is the provider's view: the architecture, the per-endpoint health machinery, and the policy decisions that separate a webhook feature from a webhook platform.
The provider side: outbox-captured events, per-endpoint filtered fan-out, signed deliveries through contained workers — and per-endpoint isolation so one slow stranger costs only themselves. Endpoint health run actuarially, suspension that warns early and retains for redelivery, and the trust-earning platform short-list.
Everything in this series hardened the event path — and post 1's contract still stands: delivery is not guaranteed, eventually. Endpoints stay down past retry horizons, subscriptions get misconfigured, providers have incidents too, and events from before your integration existed never arrive at all. The final discipline accepts this and closes the loop: periodic reconciliation against the source of truth, convergence-oriented sync design, and the backfill machinery that makes "we missed a week" a procedure instead of a crisis. This is what turns a webhook integration into a synchronization system.
Non-delivery is in the contract, so build the backstop: modified-since sweeps through the same version-gated upserts (healing and counting missed deliveries), deliberate deletion detection, and rehearsed resumable backfill. The end state is a sync engine: events for latency, reconciliation for truth, replay for recovery.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.