Stories · Season 1 — Pemberton & Crumb · Episode 03

The Payment That Published But Never Happened

In which two systems disagree about reality, and the customer is caught in the middle.

The story

By now Pemberton & Crumb had a second service: a warehouse system that packed and shipped orders. The store would take a payment, save the order, and then publish an OrderPaid message so the warehouse knew to start packing. Simple. Two lines of code, right next to each other:

await _db.SaveOrderAsync(order);          // 1. write to our database
await _broker.PublishAsync(orderPaid);    // 2. tell the warehouse

It worked thousands of times. Then, one Tuesday afternoon, the message broker hiccupped for about ninety seconds. During that window, several orders saved successfully to the database — line 1 succeeded — and then line 2 threw. The warehouse never heard about them.

The customers had paid. The store said “confirmed!”. And the pickles sat on a shelf, unpacked, because the one system that needed to know never got the memo. Dana found out three days later from a support ticket titled, in all caps, “WHERE ARE MY PICKLES.”

The reverse also happens, and it’s worse: the broker publishes successfully, then the database transaction rolls back. Now the warehouse is packing an order that, as far as the store is concerned, never existed.

Why this is hard the traditional way

This is the dual-write problem, and it’s one of the most common ways distributed systems quietly lie. You’re writing to two places — your database and your message broker — and there is no way to make both happen atomically. They’re different systems. One can succeed while the other fails.

The naive fixes all leak:

  • Publish first, then save? Now you can notify the world about something that didn’t happen.
  • Save first, then publish? Now you can save something the world never hears about (the pickle situation).
  • Wrap it in a try/catch and retry? The retry can fail too, and now you’re hand-rolling a reliability system at 2 a.m. with the broker still flapping.

There is no ordering of two independent writes that’s safe. The problem is the two writes.

How Relay changes the ending

Relay uses the transactional outbox. The trick is beautifully boring: instead of publishing to the broker directly, you write the message into an outbox table — in the same database transaction as the order. One transaction. It either all commits or all rolls back. There’s no in-between for reality to leak through.

// The order and the "tell the warehouse" message commit together, atomically.
// If the transaction rolls back, the message was never staged. Period.
order.RaiseIntegrationEvent(new OrderPaid(order.Id));
await _repository.SaveAsync(order);   // state + outbox message: one commit

A separate relay process then reads the outbox and pushes messages to the broker, retrying safely until each one is delivered. If the broker is down for ninety seconds, the messages wait patiently in the outbox and go out the moment it recovers. Nothing is lost, nothing is invented, and Dana sleeps through the broker’s next hiccup.

The store’s database and the warehouse can never again disagree about whether an order was paid.

What it costs you to ignore this

  • Money and goods get out of sync. Customers pay for things that never ship, or receive things that were never charged.
  • The failures are invisible until a human complains. Nothing errors loudly; two systems just quietly hold different versions of the truth.
  • “Just retry it” becomes a second system. You end up building a worse, buggier outbox by hand, under pressure.
  • Reconciliation becomes a recurring chore. Someone spends every Monday morning diffing two databases.