The Transaction Feed That Fell Behind

The story

Magpie’s most-used screen was the transaction feed — the running list of “you spent £3.40 at a café” that landed, ideally, the instant the card was tapped. That feed was powered by a projection: every payment event flowed through one process that updated the feed read model.

One process. For everyone. On a normal day, fine. On the last Friday of the month — payday, when what felt like the entire customer base simultaneously bought lunch, paid rent, and treated themselves — the firehose of payment events arrived faster than the single projection could process them. The feed fell behind. Then further behind. At peak it was ten minutes stale, which for a “real-time spending” app is an eternity: customers tapped their cards, saw nothing, and assumed the payment had failed.

Kemi’s instinct was the usual one: throw hardware at it. But the projection processed events strictly one after another to keep them in order, so a bigger server didn’t help — the bottleneck wasn’t CPU, it was sequentiality. One lane, no matter how fast the car, can only carry so much.

“We need more lanes,” Kemi said, “but we still need each customer’s events in order within their own lane.” That tension — parallelism and per-customer ordering — is exactly the thing naive parallelization gets wrong.

Why this is hard the traditional way

A projection usually processes events in order, single-threaded, because order matters: you can’t show “refund of £3.40” before “spent £3.40.” That ordering guarantee is also a throughput ceiling. When the event rate exceeds what one sequential processor can handle, lag grows without bound, and no vertical scaling fixes it — you can’t parallelize a strictly ordered queue by adding cores.

The obvious fix — “just run several copies” — breaks ordering: two workers processing the same customer’s events can apply them out of sequence, producing a feed that’s fast and wrong. So teams are stuck choosing between a feed that’s correct-but-slow and one that’s fast-but-scrambled.

How Relay changes the ending

Relay supports intra-projection sharding: splitting one hot projection into several parallel lanes, partitioned by a key (here, the account), so each lane processes its own slice independently — while every event for a given account always lands in the same lane, preserving per-account ordering.

// One projection, many lanes. Partition by account so each customer's
// events stay ordered within their lane, but different customers run in parallel.
public sealed class TransactionFeedProjection : IProjection
{
    public int ShardCount => 16;                       // sixteen lanes
    public string PartitionKey(PaymentEvent e) => e.AccountId.ToString();
}

Now payday’s firehose is spread across sixteen lanes instead of crammed through one. Throughput scales with the lane count, the feed keeps up, and — crucially — each customer’s transactions still arrive in the right order, because all of their events share a lane. The coffee shows up the instant the card is tapped, even when the whole country is buying lunch.

What it costs you to ignore this

• Success becomes lag. The busiest, most important moments are exactly when a single-lane projection falls behind.
• Vertical scaling hits a wall. A strictly ordered processor can’t be sped up by a bigger box; the limit is structural.
• Naive parallelism corrupts order. Splitting work without a partition key produces a feed that’s fast and wrong.
• Stale feeds read as failures. Customers who don’t see their action assume it broke — and act again.