The Nightly Job That Quietly Stopped

The story

Every night at 2 a.m., a job rolled up the day’s ticket sales into a settlement report — the numbers Frontrow used to pay out venues and reconcile with their payment processor. It had run, uncomplaining, for years. Nobody thought about it. Which is exactly the problem.

One night it didn’t run. Not with an error, not with a page — it just didn’t. The cron entry was on a single box, that box had been quietly cycled out during an infrastructure migration, and the job went with it. No box, no cron, no report, no noise.

Nobody noticed for eleven days. They noticed when a venue called asking where their payout was, and Mara went looking for the settlement reports and found a two-week hole. Eleven nights of sales had never been rolled up. Reconstructing them meant replaying card transactions by hand and praying the totals matched.

“How long has this been our single point of failure?” Tobi asked. The honest answer was “since the beginning — we just got lucky until now.”

Why this is hard the traditional way

A crontab line on one server is the most fragile kind of important. It has no redundancy (one box dies, it’s gone), no memory (miss a run and there’s no record it should have happened), and no catch-up (the missed night is simply lost rather than made up). And it’s silent — cron’s idea of error handling is an email to a mailbox nobody reads.

The deeper trap is that these jobs are invisible because they’re reliable. The ones that have worked forever are the ones nobody monitors, which makes them the ones whose failure you discover from a customer instead of a dashboard.

How Relay changes the ending

Relay treats recurring work as a first-class, durable concept. You define a job with a cron schedule, and the framework owns the schedule, the execution, and — crucially — the bookkeeping of what ran and what didn’t.

// A durable, monitored recurring job — not a line in someone's crontab.
public sealed class NightlySettlement : IJob
{
    public string CronExpression => "0 2 * * *";   // 02:00 daily, DST-aware
}
// Missed a run because the service was down? A catch-up policy can backfill it
// instead of pretending the night never happened.

Because the schedule is durable and not tied to one box, retiring a server doesn’t retire your settlement run. Because runs are tracked, a missed or backlogged job is something you can alert on — Relay exposes backlog and health, so “the nightly job hasn’t run” becomes a page at 2:15 a.m., not a phone call from a venue eleven days later. And catch-up policies mean a missed night can be backfilled rather than lost.

What it costs you to ignore this

• Silent failure is the worst failure. The job that stops without complaining is the one that costs you weeks.
• One box is one outage from a gap in your books. Critical work shouldn’t depend on a server nobody remembers.
• Reconstruction is expensive and error-prone. Rebuilding missed runs by hand is slow, and the numbers might not even match.
• Trust is the real casualty. Late payouts and “we lost your numbers” are how partners start shopping for a competitor.