Stories · Season 2 — Frontrow · Episode 05

The Payout That Ran Twice

In which Frontrow fixes its single point of failure by running everywhere — and immediately pays everyone twice.

The story

After the Nightly Job Incident (see 014), Mara did the responsible thing: she made the settlement job redundant. Now it ran on all the application servers, so no single box dying could ever take it down again. Resilient. Bulletproof. Ship it.

The next morning, every venue had been paid twice.

All six servers had woken up at 2 a.m., all six had seen the day’s unsettled sales, and all six had cheerfully initiated payouts. Frontrow had solved “the job might not run” by guaranteeing “the job runs six times.” Clawing back the duplicate transfers took a week of awkward phone calls, and a couple of venues had already spent the money.

“So we can’t run it on one box,” Tobi said slowly, “and we can’t run it on all of them.” Mara nodded. “We need exactly one of them to do it. Every time. Even though they’re identical and none of them knows about the others.”

That last sentence is one of the genuinely hard problems in distributed systems, and it had just cost them a week.

Why this is hard the traditional way

“Run this on exactly one node, even though all the nodes are identical and any of them might die” is the leader-election problem, and naive solutions are a graveyard.

A “designated primary” config setting means a manual scramble when that node dies. A “first one to write a row wins” flag works until two nodes write at the same instant, or until the winner crashes holding the flag and nobody can take over. A lock with a timeout introduces the nastiest bug of all: the old leader pauses (GC, slow disk), its lease expires, a new leader takes over, and then the old one wakes up still believing it’s in charge — and now you have two leaders again, both paying out.

How Relay changes the ending

Relay provides distributed coordination as infrastructure: distributed locks and leader election, backed by the database, so exactly one instance does the work — and leadership moves automatically when a node dies.

// Only the elected leader runs the payout. The other five stand by.
if (await leaderElector.IsLeaderAsync("nightly-settlement"))
    await RunSettlementAsync();
// Leader dies? Another instance is elected. No config edits, no double payout.

And for the “old leader wakes up still thinking it’s boss” problem, Relay supports fencing: leadership is stamped with a token, and stale leaders are detected and rejected. The result is what Mara actually wanted from the start — redundancy and exactly-once execution: six servers ready to do the job, exactly one doing it, and a clean automatic handover when things fail.

What it costs you to ignore this

  • Redundancy without coordination doubles your work. More servers becomes more duplicate side effects, not more safety.
  • Money moved twice is hard to unmove. Duplicate payouts, charges, and notifications all turn into manual cleanup.
  • Manual failover is a 3 a.m. ritual. “Which box is the primary now?” is not a question you want to answer half-asleep.
  • Split-brain is the silent killer. Two leaders who each think they’re alone is the bug that survives your first three fixes.