Reliable webhook ingestion kit

Payments, messaging and telephony providers all deliver state changes through webhooks, and they all behave the same way: they retry, they deliver out of order, they send duplicates, and they time out if the handler is slow. Integrations that do the work inside the HTTP handler eventually lose or double-apply an event. This kit isolates that concern once so each new integration only writes its handler.

• Processing inside the request means a slow downstream call becomes a provider timeout and a retry storm.
• Providers guarantee at-least-once delivery, so handlers see duplicates unless something removes them.
• When a handler bug ships, the affected events are gone unless the raw payloads were kept.

The HTTP endpoint does three things and nothing else: verify the provider signature against the raw body, insert the raw event into a PostgreSQL inbox table keyed by provider and event ID, and return 200. The unique key makes a duplicate delivery a no-op. A relay picks up new inbox rows and enqueues them in BullMQ; workers run the integration-specific handler with exponential backoff and jitter. A job that exhausts its attempts moves to a dead-letter queue with the error attached, and the inbox row records the status of every attempt. Because the raw payload and headers are kept, a CLI can replay a single event, everything in the dead-letter queue, or a time range — which is also how a fixed handler gets applied to events it previously failed on. Handlers are written to be idempotent, with the event ID available as the natural key.

What this design does not solve is ordering: two events for the same entity can be processed out of sequence. The next step is per-entity ordering keys in the queue and a chaos test — killed workers, duplicated and reordered deliveries — to put numbers on throughput and confirm nothing is lost.