Notifier delivery rework

Replace the shared fixed retry ceiling with a per-destination delivery budget.

Draft · target 2026.10 · owner: rk

Problem

The retry ceiling is global and fixed. One destination in a long outage consumes attempts that were never shared with anyone else, but the ceiling is reached per-delivery, so the effect is that every message to that destination is dead-lettered at the same shallow depth. The August backlog is the worked example: four attempts covered nine minutes of a seven-hour outage.

Raising the ceiling to six bought forty minutes. Raising it further trades one failure mode for another — a destination that is down for a day would hold retries open long enough to matter to everything else on the worker pool.

Proposal

Each destination gets a delivery budget replenished over time. A delivery consumes budget on failure. When the budget is exhausted the destination is marked failing and its traffic is parked rather than dead-lettered — parked traffic replays automatically when the destination recovers.

Behaviour	Today	Proposed
Retry depth	Fixed, six	Budget-driven, per destination
After exhaustion	Dead-letter	Park, with automatic replay
Blast radius	Shared worker pool	Isolated per destination
Operator action	Manual replay	None in the recovery case

Open questions

• What is the dead-letter contract after this lands? Parking makes dead-lettering rare but not impossible, and callers currently treat the dead-letter queue as the place to look. Changing that is a breaking change to an interface we never documented.
• How long does parked traffic stay parked? A destination that never returns turns the park into an unbounded queue. A TTL reintroduces silent loss, which is the thing this is trying to remove.
• Budget replenishment rate: fixed, or proportional to the destination's historical volume? Proportional is obviously right and obviously harder to reason about during an incident.

Not in scope

Delivery ordering. It is not guaranteed today and this does not change that, though parking makes the lack of a guarantee more visible than it currently is.