Handling Webhooks Reliably: Signatures, Idempotency, and Retries

The happy path for a webhook handler is ten lines: parse the JSON, update the database, return 200. The production path has to survive forged requests, duplicate deliveries, out-of-order events, slow handlers that trigger retries, and outages that queue up hours of events. (What Is a Webhook? covers the basics.)

The delivery guarantees you actually get

Nearly every provider (Stripe, GitHub, Shopify, Twilio, and so on) offers at-least-once delivery:

An event may be delivered more than once — retries after timeouts, network blips, or provider-side replays.
Events may arrive out of order.
Delivery may be delayed, sometimes by hours during incidents.
If your endpoint keeps failing, the provider retries with backoff for a limited period, and may eventually disable the endpoint.

Design for those, not for the happy path.

1. Verify the signature — against the raw body

Anyone can POST JSON to /webhooks/stripe. Providers sign each request, typically with an HMAC of the payload and a timestamp using a shared secret. Verify it before trusting anything.

The classic bug: verifying against a re-serialised body. If a JSON middleware parses the body first, JSON.stringify(req.body) won't be byte-identical to what was signed, and verification fails (or, worse, someone "fixes" it by skipping verification). Use the raw bytes:

// Express: raw body for this route only
app.post("/webhooks/stripe", express.raw({ type: "application/json" }), (req, res) => {
  let event
  try {
    event = stripe.webhooks.constructEvent(
      req.body,                                  // Buffer, untouched
      req.headers["stripe-signature"],
      process.env.STRIPE_WEBHOOK_SECRET,
    )
  } catch {
    return res.status(400).send("Invalid signature")
  }
  // …
})

In Next.js route handlers, read await request.text() before any parsing. Also:

Check the timestamp tolerance (the official SDKs do) to limit replay attacks.
Use constant-time comparison if you implement HMAC checks yourself.
Keep the webhook secret server-side and separate per environment. (Secrets Management Beyond .env Files.)

2. Acknowledge fast, process later

Providers time out webhook requests, often within seconds to tens of seconds. If your handler sends emails, calls other APIs, or runs a slow query before responding, a timeout triggers a retry — while the first attempt may still complete. Now you're processing it twice, concurrently.

The robust pattern:

Verify the signature.
Persist the raw event (with its provider event ID) to a table or queue.
Return 2xx immediately.
Process asynchronously in a worker.

CREATE TABLE webhook_events (
  provider     text        NOT NULL,
  event_id     text        NOT NULL,
  type         text        NOT NULL,
  payload      jsonb       NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz,
  PRIMARY KEY (provider, event_id)
);

This table doubles as an audit log and a replay mechanism. (Your App Needs Background Jobs.)

3. Make processing idempotent

At-least-once delivery means your handler must produce the same result whether an event is processed once or five times.

Deduplicate on the provider's event ID. The primary key above does it at ingestion:

INSERT INTO webhook_events (provider, event_id, type, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (provider, event_id) DO NOTHING;

If zero rows were inserted, it's a duplicate: acknowledge and stop.

Make the effects idempotent too, because a worker can crash after doing the work but before marking the event processed:

Prefer state assignments over increments: SET status = 'paid' is idempotent; SET credits = credits + 100 is not.
Where you must increment, record the event ID in the same transaction as the effect (e.g. a credit_grants row with a unique constraint on event_id). (Database Transactions Explained.)
For outbound side effects (emails, third-party calls), use idempotency keys where the downstream API supports them, or record "sent" markers keyed by event ID.

4. Don't trust event order — or the payload's freshness

customer.subscription.updated can arrive before customer.subscription.created. An old "past_due" event can arrive after a newer "active" one.

Strategies:

Re-fetch current state from the provider's API when handling an event, and treat the webhook as a notification that something changed. Simple and robust, at the cost of an API call.
Compare timestamps or versions: ignore events older than the state you've already applied (store last_event_at per object).
Model state transitions so invalid ones are rejected rather than applied blindly.

5. Respond with the right status codes

2xx — received (even if it's a duplicate, or an event type you ignore).
4xx — for invalid signatures. Many providers still retry 4xx, so don't rely on them to stop retries for a bug.
5xx — only for genuine transient failures you want retried.

Return 2xx for event types you don't handle; otherwise they'll be retried until the endpoint is disabled.

6. Monitor and replay

Alert on signature failures (misconfigured secret or an attack) and on processing lag (oldest unprocessed event).
Check the provider's dashboard for failed deliveries after incidents.
Keep a way to reprocess stored events, and a way to backfill from the provider's API for events you never received.

Testing locally

Stripe CLI: stripe listen --forward-to localhost:3000/webhooks/stripe forwards real test-mode events, and stripe trigger payment_intent.succeeded fires one.
Other providers offer similar CLIs, or "redeliver" buttons in their dashboards.
A tunnel (for providers without a CLI) exposes your local server temporarily.
Write tests that deliver the same event twice and events out of order, and assert the final state is correct. (How to Write Tests With AI.)

Checklist

Signature verified against the raw body; timestamp tolerance enforced
Event persisted with a unique (provider, event_id), then 2xx returned quickly
Processing happens in a worker, with retries
Effects are idempotent (state assignment, or effect + event ID in one transaction)
Ordering handled by re-fetching or version checks
Unhandled event types return 2xx
Alerts on signature failures and processing lag; replay path exists

EasySpawn runs your app as an always-on server with a managed Postgres database and room for background workers — the pieces a reliable webhook pipeline needs — on your own domain with SSL. See how it works for AI-built apps or join the waitlist.