Stories · Season 1 — Pemberton & Crumb · Episode 08

The Order That Got Stuck Forever

In which a five-step process dies on step three, and a customer's money goes to live in limbo.

The story

A wholesale order at Pemberton & Crumb was now a proper little journey. Place the order, charge the card, reserve the stock, book a courier, send the confirmation. Five steps, across three services. When all five worked, it was a thing of beauty.

Step three did not always work.

One morning, a large order charged the customer’s card successfully (step two), then tried to reserve the stock (step three) — and the warehouse service was mid-deploy and didn’t answer. The process simply… stopped. The card was charged. No stock was reserved. No courier was booked. No confirmation went out. The order sat in a state best described as Schrödinger’s pickle: paid for, but not real.

The customer had paid $1,800 and received silence. Dana found the order days later, stuck halfway through its journey, and faced the genuinely hard question: what do you even do now? You can’t just re-run the whole thing — that would charge the card a second time. You have to carefully figure out which steps happened, which didn’t, and either finish the job or unwind it. By hand. For every stuck order. And there were more than one.

The worst part: there was no record of intent. Nothing in the system said “this order is trying to get from A to B and is currently stuck at C.” There was just a half-mutated row and a charged card, and a human left to reverse-engineer what should happen next.

Why this is hard the traditional way

A multi-step business process that spans services is a distributed transaction, and the database’s tidy little BEGIN/COMMIT/ROLLBACK does not stretch across service boundaries. You cannot wrap “charge the card” (Stripe) and “reserve the stock” (your warehouse) in one transaction that rolls back together. If step three fails, step two has already really, truly happened. The money is gone.

So teams hand-code the orchestration: a tangle of if/try/catch that tries to remember where it is, retry the right step, and undo the earlier ones if it gives up. This code is famously, brutally hard to get right, because it has to be correct for every combination of which-steps-succeeded — and it has to survive the process crashing while it’s in the middle of recovering. Most teams discover the gaps the way Dana did: one stuck order at a time, in production, with a customer waiting.

How Relay changes the ending

Relay models long-running processes as sagas — and a saga is a first-class, durable thing, not a pile of if statements. It has explicit state, it’s saved after every step, and it survives crashes and restarts.

  • It remembers where it is. The saga’s state is persisted, so “this order is paid, stock not yet reserved” is a real, queryable fact — not something a human reconstructs from a half-finished row.
  • It can wait. When the warehouse is mid-deploy, the saga waits (with a timeout) and resumes when the answer comes, instead of dying on the spot.
  • It knows how to undo. Relay’s routing slips let each step record a compensation — the action that reverses it. If the process can’t complete, Relay walks the completed steps backwards (last-in, first-out) and runs each compensation: release the stock, refund the card, cancel the courier. The distributed transaction unwinds cleanly, automatically.
// Each step records how to undo itself. If a later step fails,
// Relay runs the compensations in reverse — including refunding the charge.
itinerary.AddActivity("ChargeCard",    charge,  compensation: RefundCard);
itinerary.AddActivity("ReserveStock",  reserve, compensation: ReleaseStock);
itinerary.AddActivity("BookCourier",   book,    compensation: CancelCourier);
// Booking fails? -> CancelCourier (n/a) -> ReleaseStock -> RefundCard. Customer made whole.

Replayed with Relay, the failed order doesn’t get stuck. Step three can’t reserve stock, the saga times out waiting, decides it can’t complete, and automatically refunds the $1,800 charge and notifies the customer — all without a human noticing at the time. There is no Schrödinger’s pickle. There is only an order that either fully happened or fully didn’t.

What it costs you to ignore this

  • Customers’ money gets stranded. A half-completed process can leave real charges with nothing delivered — the fastest way to a chargeback and a furious review.
  • Recovery is manual, risky, and per-incident. Every stuck process is a hand-investigation where a wrong move (a careless retry) double-charges.
  • The orchestration code is a bug farm. Hand-rolled saga logic has to handle every partial-failure permutation and survive crashing mid-recovery — almost nobody gets it fully right.
  • It doesn’t scale with the business. Every new multi-step workflow multiplies the surface area of “what happens when step N fails?”