Notifier delivery backlog

Webhook delivery fell eleven hours behind. The retry ceiling was the proximate cause; the dead-letter path is why nobody noticed.

2026-08-29 · notifier · action items open

Timeline

Time (UTC)	Event
04:12	Downstream endpoint for the largest destination begins returning 502. Retries start.
04:41	First deliveries exhaust the four-attempt ceiling and move to the dead-letter queue.
06:00	Queue depth alert does not fire: drained items count as delivered, and dead-lettering drains them.
11:20	Destination recovers on its own. Nothing replays.
15:08	Customer report of missing webhooks. Investigation starts here, eleven hours in.
16:30	Dead-letter queue replayed by hand.

What actually happened

Four attempts with the backoff we had covers about nine minutes. The outage was seven hours. Every delivery to that destination for the duration was dead-lettered, and the dead-letter queue is not on any dashboard — it was built as a debugging aid and quietly became the place undelivered traffic goes to be forgotten.

The alert we had watched queue depth. Dead-lettering removes an item from the queue, so the metric we were alerting on went down during the incident. It looked like a quiet morning.

What we are changing

• Retry ceiling raised from four to six, shipped in 2026.9.1. This buys roughly forty minutes, not seven hours. It is a stopgap and should not be read as the fix.
• Alert on dead-letter arrival rate rather than queue depth. Not yet done.
• Per-destination retry budgets so one bad endpoint cannot consume the shared ceiling — see the delivery rework design doc. Not yet done.

Service is still marked degraded on the console until the second and third items land.