Stories · Season 1 — Pemberton & Crumb · Episode 05

The Black Friday That Melted

In which one slow service takes down five healthy ones, and the retries finish the job.

The story

Pemberton & Crumb ran their first Black Friday sale. “Pickle Friday,” Crumb insisted on calling it. The marketing worked too well. Traffic was ten times normal, which was thrilling for about four minutes.

The tax-calculation service — a third-party API Dana had wired in months ago and forgotten about — started responding slowly under the load. Not failing. Just… slow. Two seconds. Then five. Then ten.

Here’s where it got ugly. Every order placement waited on that tax call. As tax got slower, order requests piled up, each holding a thread and a database connection while it waited. Within minutes the store ran out of threads — not because the store was slow, but because it was all standing in line behind one sluggish dependency. The product pages, the account pages, the “track my order” page: all dead, all because of a tax API nobody was even using on those pages.

And then the retries kicked in. The store had been configured to retry failed tax calls — three times, immediately. So a struggling tax service suddenly received four times the traffic it was already drowning in. The retries didn’t heal anything. They were gasoline.

Pickle Friday lasted eleven minutes before the whole thing fell over.

Why this is hard the traditional way

Distributed systems fail in a specific, vicious way: slow is worse than down. A service that’s down fails fast and frees up resources. A service that’s slow holds every caller hostage, and the backpressure travels upstream until something unrelated runs out of threads and dies. This is a cascading failure, and it’s how one bad dependency takes down a whole system.

The usual defenses are easy to get wrong:

  • Naive retries make it worse. Retrying a failing service immediately, with no backoff, multiplies the load on the exact thing that’s already on fire.
  • No timeout means no escape. If a call can wait forever, then under load, everything waits forever.
  • No isolation means no containment. If the tax calls and the product-page reads share the same thread pool, the tax problem becomes everyone’s problem.

Building these protections by hand, correctly, for every external call is a lot of fiddly code that you only find out you got wrong during your biggest sale.

How Relay changes the ending

Relay ships the resilience patterns as configurable behaviors you wrap around operations — no hand-rolled retry loops:

  • Retries with backoff and jitter — wait longer between attempts, with a little randomness, so a struggling service gets breathing room instead of a stampede.
  • A circuit breaker — after enough failures, stop calling the broken service entirely for a while. Fail fast, free the threads, and give the dependency time to recover instead of pounding it while it’s down.
  • Rate and concurrency limits — cap how many calls can be in flight, so a slow dependency can never consume every thread in the process.
  • A kill switch — auto-pause an operation when the error rate spikes, instead of melting down in public.
// The tax call gets a timeout, capped concurrency, and a circuit breaker.
// When tax goes slow, the breaker trips, those calls fail fast,
// and the product pages stay up because they were never sharing the same lane.
[Resilient(MaxConcurrency = 20, Retry = RetryPolicy.ExponentialBackoff, CircuitBreaker = true)]
public sealed record CalculateTaxQuery(Guid OrderId) : IQuery<TaxResult>;

Replayed with Relay, Pickle Friday looks different. The tax service slows down, the circuit breaker trips after a handful of failures, tax calls start failing fast (Dana had a sensible fallback rate ready), and crucially — the rest of the store keeps serving. The dependency that’s struggling gets less load, not more, so it recovers on its own. One feature degrades. The business stays open.

What it costs you to ignore this

  • One weak dependency can take down everything. Without isolation, the blast radius of any single failure is your entire system.
  • Your busiest day is when it happens. Cascading failures are load-triggered, so they wait for your highest-traffic, highest-stakes moment to appear.
  • Retries become an attack on yourself. Misconfigured retries turn a small outage into a self-inflicted denial-of-service.
  • Recovery is slow and manual. Without circuit breakers, a system under cascade can’t heal until a human intervenes — usually by turning things off.