Stories · Season 1 — Pemberton & Crumb · Episode 07

The Tenant Who Saw Someone Else's Data

In which a single missing line of SQL becomes the worst email a SaaS founder can send.

The story

Pemberton & Crumb had grown up. They now sold their store as a platform — other pickle entrepreneurs could run their own shops on it. “Pickle-as-a-Service,” Crumb beamed. Each shop was a tenant, and the cardinal rule, the one rule above all rules, was simple: a tenant must never, ever see another tenant’s data.

For months, the rule held. The way it held was that every database query Dana wrote ended with WHERE tenant_id = @currentTenant. Every. Single. One. It was a rule enforced by vigilance and code review and remembering.

Then a new engineer, three weeks in, added a perfectly reasonable feature: a “recent orders” widget. They wrote the query. They tested it — it worked great on their test account. They shipped it. It passed review, because the reviewer was busy and the query looked fine.

It was missing the WHERE tenant_id clause.

For six hours, the recent-orders widget on every shop showed recent orders from every shop on the platform. A competitor saw their rival’s order volume. Someone saw a customer’s name and address that wasn’t theirs. The bug was caught when a tenant emailed, more confused than angry: “Who is Bob’s Brine Emporium and why are their orders on my dashboard?”

That email led to the one Crumb had to write next, to every customer, with the word “incident” in it.

Why this is hard the traditional way

Hand-rolled multi-tenancy is secure by default until someone forgets — and someone always, eventually, forgets. The isolation depends on a human writing WHERE tenant_id = @t on every query, configuring every cache key with the tenant, and scoping every background job correctly, forever, with no misses. It’s a security model held together by discipline, which is to say, held together by the worst day of your most tired engineer.

And the failure mode isn’t a crash. It’s silent. The query runs fine, returns data, looks healthy — it just returns the wrong tenant’s data. Tests written against a single tenant can’t catch it. The first signal you get is a customer seeing data they should never have seen, and by then it’s not a bug, it’s a breach: the single most damaging thing a SaaS can ship.

How Relay changes the ending

Relay makes tenant isolation fail-closed — the system’s default is deny, and you have to explicitly say when something is allowed to cross tenant lines.

  • It refuses to guess. Every message must declare its posture: [TenantScoped] (operates within one tenant) or [GlobalOperation] (a deliberate, reviewed exception). A message that declares neither doesn’t run — there’s no silent default that leaks.
  • It resolves the tenant for you. From a header, a claim, a subdomain, or a route — pick a strategy, and the current tenant is established once, at the edge, not re-derived in every handler.
  • It enforces isolation in the database itself. Relay can set the tenant context on the connection (PostgreSQL row-level security), so the database filters by tenant even if a query forgets to. The safety net lives below the application code, where a forgotten WHERE clause can’t reach.
  • Caches and background jobs are tenant-aware too, so yesterday’s leak vector — a cached result built for one tenant, served to another — is closed by default.
// This won't silently leak. It can't run without declaring its tenant posture,
// and the tenant filter lives below it, in the data layer.
[TenantScoped]
public sealed record GetRecentOrdersQuery(int Count) : IQuery<IReadOnlyList<OrderSummary>>;

Replayed with Relay, the new engineer ships the recent-orders widget, forgets the tenant filter exactly as before — and nothing leaks. The row-level isolation returns only the current tenant’s rows regardless, and the missing declaration would have been caught before the message could even run. The worst email never gets written.

What it costs you to ignore this

  • A cross-tenant leak is a breach, not a bug. It triggers disclosure obligations, lost contracts, and the kind of headline you don’t recover from.
  • It’s silent and untestable by accident. Single-tenant tests pass; the leak shows up only with real, multiple tenants in production.
  • The risk grows with every new query. Discipline-based isolation gets less safe as your team and codebase grow — exactly backwards from what you want.
  • Trust is the product. For a platform, “your data is isolated from your competitors’” isn’t a feature. It’s the entire reason customers pay you.