Engineering notes

Archiver index divergence

The index of what-went-where disagreed with object storage for rows moved under the 90-day rule.

2026-07-11 · archiver · closed

Timeline

Time (UTC)Event
2026-07-09A restore request for a row archived in May returns not-found. Treated as a one-off.
2026-07-10Second not-found. Audit of a sample of 2,000 archived rows finds 31 with index entries pointing at objects that do not exist.
2026-07-11Index rebuilt from object storage listings.

What actually happened

The archiver wrote its index entry before confirming the object write. A storage-side error after the index write left an entry pointing at nothing. The window is small and the error is rare, which is why it took three months of accumulation to become visible.

No data was lost. The rows were still in the staging tables; the 90-day rule deletes only after a successful archive, and the archive was not successful. The index was the only thing that was wrong, and it was wrong in the direction that produces a confusing error rather than a quiet loss.

What we changed