The Log That Ate the Storage Budget

The story

Event sourcing had been very good to Magpie. Every movement of money, kept forever, perfectly auditable — exactly what a bank needs. The little bird had been right all along. The finance team, however, had noticed something: the database bill was climbing in a straight line that never bent, and the line was starting to look less like a cost and more like a problem.

The cause was simple arithmetic. Magpie had millions of accounts, each generating events with every coffee, every round-up, every transfer. Nothing was ever deleted, because deleting events was the one thing the whole architecture promised not to do. After three years, the event store held billions of rows, the vast majority of them ancient — events from accounts that hadn’t moved in years, or detail so old that no living query ever touched it. And it was all sitting on the same expensive, high-performance storage as last night’s transactions.

There was a performance tax, too. Replaying an aggregate with a hundred thousand historical events was slow, and the sheer table size made everything a little more sluggish than it needed to be. Ada, the founder, put it bluntly: “Keeping everything forever is our superpower and apparently also our biggest line item. Surely we don’t have to choose between ‘auditable’ and ‘affordable.’”

Why this is hard the traditional way

An append-only log only grows. That’s the point — and the bill. The naive way to control cost is to delete old data, but in an event-sourced system deletion is exactly what you can’t do without destroying the history and audit guarantees you adopted it for. So teams feel trapped: pay forever for storage you mostly never read, or compromise the immutability that made the whole thing trustworthy.

Hot, fast, expensive storage is also the wrong home for cold data. Years-old events that are needed maybe once for an audit don’t belong on the same tier as today’s transactions — but separating them by hand, while keeping them readable and keeping replay correct, is fiddly and easy to get wrong.

How Relay changes the ending

Relay supports event archiving and compaction: moving old, cold events to cheaper storage and compacting streams (often using snapshots, so an aggregate can be rebuilt from a recent snapshot plus the events since, instead of replaying from the dawn of time) — all without breaking event-sourcing guarantees. The history isn’t deleted; it’s relocated and made efficient, still there when an audit comes asking.

// Cold events move to cheap storage; recent state stays hot and fast.
// Replay uses a snapshot + recent events instead of millions of ancient rows.
await archiver.ArchiveBefore(olderThan: DateTime.UtcNow.AddYears(-2));
await compactor.CompactStreams(snapshotAndKeepTail: true);
// Auditable: archived events are still readable. Affordable: they're not on the hot tier.

Now Magpie keeps its superpower and bends the cost curve. Old events live on storage that costs a fraction as much but can still be read when a regulator or a dispute requires them. Replay is fast because aggregates rebuild from a recent snapshot plus a short tail, not a hundred thousand rows. “Auditable” and “affordable” stop being a choice.

What it costs you to ignore this

• Costs grow without bound. An append-only log on hot storage is a bill that only ever increases, in a straight line.
• Performance degrades with age. Ever-larger tables and ever-longer replays make a healthy system slowly sluggish.
• The false dilemma traps you. Without archiving, “control cost” looks like “delete history” — sacrificing the audit trail you built on.
• Cold data hogs hot storage. Paying premium rates to store events nobody reads is money set on fire monthly.