Building a Webhook Delivery System That Actually Retries

The problem with naive webhook delivery

Most webhook implementations fire an HTTP request and forget. If your customer's server is down, or returns a 500, the event is lost forever. We've been on the receiving end of that, and it's a terrible developer experience. Here's how we avoided it.

Delivery guarantees we target

We aim for at-least-once delivery with a maximum of 5 attempts over 24 hours. We intentionally avoid exactly-once delivery because it requires distributed transactions that add too much complexity for the value gained at our scale.

The retry schedule

Retries follow exponential backoff with jitter to avoid thundering herds:

• Attempt 1: immediate
• Attempt 2: 30 seconds
• Attempt 3: 5 minutes
• Attempt 4: 30 minutes
• Attempt 5: 6 hours

1function nextRetryDelay(attempt: number): number {
2  const base = Math.pow(2, attempt) * 1000; // exponential
3  const jitter = Math.random() * 1000;       // ±1s jitter
4  return Math.min(base + jitter, 6 * 60 * 60 * 1000);
5}

Signature verification

Every webhook payload is signed with HMAC-SHA256 using a per-endpoint secret. This lets customers verify that requests genuinely come from Reloop.

The replay UI

Failed deliveries are surfaced in the dashboard with full request/response details. One click re-queues the event. This turns webhook debugging from a black box into a transparent, self-service experience.

Lessons learned

• Log everything: store the raw outbound payload and the response body, not just the status code.
• Separate delivery from processing: webhook workers should not share a process with API workers.
• Circuit break slow endpoints: if an endpoint consistently times out, back off aggressively before burning through retries.