Articles · Scheduling & Time ·019

Recurring Jobs

Runnable sample on GitHub

Sample: samples/019-recurring-jobs — cron occurrence planning and durable jobs. dotnet test. No database.

Prerequisite: 018 — Scheduling.

Overview

Article 018 scheduled one-shot future work. This article covers the recurring case — “every night at 2am,” “every Monday,” “every 15 minutes” — and the job as the unit of recurring/background work. Two pieces combine: a cron occurrence planner that, given a cron expression and a time zone, computes when a schedule should next fire and (after downtime) which missed runs to enqueue (Skip vs Backfill); and the job abstraction (IJob / IJobHandler<TJob> / IJobScheduler) for the durable unit of work the schedule enqueues. Together they give you reliable, multi-instance-safe, DST-correct recurring execution.

Why this exists

“Run nightly” sounds trivial until production. A naive Timer firing every 24h drifts, double-fires across instances, loses its place on restart, and gets daylight-saving wrong (does “2am daily” fire twice on the fall-back night, or skip on spring-forward?). And “what happens to the 2am run if the service was down at 2am?” — silently skip it, or run it late? These are real decisions, and a hand-rolled timer makes them implicitly and usually wrongly. The recurring scheduler makes them explicit and correct: cron expressions with named time zones (so 2am means 2am there, DST-aware), a durable schedule (survives restarts, claimed once across instances), and a catch-up policy that decides what to do about missed occurrences (Skip the backlog and resume, or Backfill every missed run, bounded). The job abstraction gives the enqueued work a typed handler and durable delivery (it rides the scheduler/outbox), so a recurring job is reliable end to end, not a fire-and-pray timer.

When to use this

  • Scheduled maintenance: nightly cleanup, hourly aggregation, weekly reports, periodic cache warming.
  • Polling / reconciliation: “every 5 minutes, reconcile with the external system.”
  • Billing / lifecycle cadence: “on the 1st of each month, generate invoices”; “daily, expire trials.”
  • Any “do this on a cron schedule, reliably, once across the fleet, correctly across time zones / DST.”

And use jobs (vs scheduling a command) when the recurring/background work is naturally a unit of work with a handler rather than a domain command — cleanup, report generation, an integration sync.

When not to use this

  • One-shot future work. That’s the plain scheduler (article 018), not a recurring schedule.
  • Sub-minute / high-frequency cadence. Cron is minute-grained (or second, with 6-field expressions), and the processor polls; it’s not for “every 100ms.” Use an in-process loop for tight intervals.
  • Infrastructure-level cron. If the cadence is an ops concern (rotate logs, run a batch container), a k8s CronJob or cloud scheduler may fit better than an in-app recurring schedule — unless the work needs your domain/types/transaction.
  • Best-effort, single-instance, loss-tolerable. A PeriodicTimer is simpler if durability and exactly-once-across-instances don’t matter.

The costs: a recurring-schedule table and a processor; the catch-up policy decision (and its bounds); cron expressions to get right (a notorious footgun); and the same payload-versioning/retention concerns as the scheduler.

Concepts

Cron + time zone. A standard cron expression (0 2 * * * = 02:00 daily; 6-field includes seconds) evaluated in a named IANA/Windows time zone, so occurrences are DST-correct.

IRecurringOccurrencePlanner (CronRecurringOccurrencePlanner). Pure, no infrastructure: GetFirstOccurrence(cron, tz, after) computes the next run; Plan(schedule, now) returns the occurrences to enqueue now (per catch-up policy) and the next future fire time.

Catch-up policy. After downtime, missed occurrences are handled by RecurringCatchUpPolicy:

  • Skip — fire the due occurrence once, drop the rest, resume from the next future run.
  • Backfill — enqueue every missed occurrence in order, bounded by MaxBackfill.

RecurringSchedule. The persisted schedule (name, cron, time zone, payload, catch-up policy, NextFireAt). The recurring processor claims due schedules, enqueues their occurrences as one-shot scheduled messages, and advances NextFireAt.

Jobs. IJob (a serialisable unit of work), IJobHandler<TJob> (its handler), IJobScheduler (EnqueueAsync / ScheduleAsync(runAt|delay)). A job is staged as a Kind=Job scheduled message; JobScheduledMessageDispatcher resolves and invokes the handler when it fires.

Architecture

graph LR
    Cron[RecurringSchedule + cron + tz] --> Planner[CronRecurringOccurrencePlanner]
    Planner -->|Plan: occurrences + NextFireAt| Proc[RecurringScheduleProcessor]
    Proc -->|enqueue one-shot| Sched[(relay_scheduled_messages)]
    Sched --> SP[SchedulerProcessor]
    SP --> Disp[JobScheduledMessageDispatcher]
    Disp --> H[IJobHandler&lt;TJob&gt;]

Building it step by step

The sample is samples/019-recurring-jobs.

1. Plan cron occurrences (pure)

var planner = new CronRecurringOccurrencePlanner();
planner.GetFirstOccurrence("0 2 * * *", "UTC", now);          // next 02:00

var schedule = new RecurringSchedule { CronExpression = "0 2 * * *", TimeZoneId = "UTC",
                                       CatchUpPolicy = RecurringCatchUpPolicy.Backfill, MaxBackfill = 10,
                                       NextFireAt = /* last known */ };
var plan = planner.Plan(schedule, now);   // plan.Occurrences (to enqueue), plan.NextFireAt

2. Define a job and its handler

public sealed class CleanupJob : IJob { public string Target { get; set; } = ""; }
public sealed class CleanupJobHandler : IJobHandler<CleanupJob>
{
    public Task HandleAsync(CleanupJob job, CancellationToken ct) { /* do the work */ return Task.CompletedTask; }
}

3. Enqueue and dispatch a job

await jobScheduler.EnqueueAsync(new CleanupJob { Target = "temp-files" }); // Kind=Job scheduled message
// when it fires, the dispatcher resolves IJobHandler<CleanupJob> and invokes it
await new JobScheduledMessageDispatcher(serializer).DispatchAsync(message, provider, default);

4. Register (production)

services.AddRelayScheduling();                          // the one-shot scheduler the occurrences ride
services.AddRelaySchedulerEfCore<TContext>();
services.AddRelayRecurringScheduling();                 // the recurring planner + processor
services.AddRelayRecurringSchedulingEfCore<TContext>();
services.AddRelayJobs();
services.AddJobHandler<CleanupJob, CleanupJobHandler>();
// then: await recurringScheduler.UpsertRecurringAsync("nightly-cleanup", "0 2 * * *", new CleanupJob { Target = "temp" });

Complete source code

File Contents
CronTests.cs First occurrence; Skip vs Backfill planning
Jobs.cs The job, its handler, a capturing scheduler
JobTests.cs Enqueue stages a job; dispatch invokes the handler

Running the example

dotnet test samples/019-recurring-jobs/Recurring.Sample.Tests

Testing

The planner is pure, so its logic is tested directly and deterministically:

planner.GetFirstOccurrence("0 2 * * *", "UTC", Utc(2026,1,1,3)).Should().Be(Utc(2026,1,2,2));

// Skip: one occurrence, resume in the future
plan.Occurrences.Should().ContainSingle().Which.Should().Be(Utc(2026,1,1,2));
plan.NextFireAt.Should().Be(Utc(2026,1,6,2));

// Backfill: every missed occurrence, in order
plan.Occurrences.Should().Equal(Utc(2026,1,1,2), Utc(2026,1,2,2), Utc(2026,1,3,2), Utc(2026,1,4,2));

And the job round-trip — enqueue stages a Kind=Job message; dispatch resolves and invokes the handler:

await jobScheduler.EnqueueAsync(new CleanupJob { Target = "temp-files" });
scheduler.Captured!.Kind.Should().Be(ScheduledDeliveryKind.Job);
await dispatcher.DispatchAsync(scheduler.Captured!, provider, default);
handler.Handled!.Target.Should().Be("temp-files");

Testing the planner separately from the job dispatch keeps each concern’s correctness independently verifiable — cron math is notoriously bug-prone, so it gets its own focused, pure tests.

Production considerations

  • Choose the catch-up policy per schedule, deliberately. A nightly report probably wants Skip (one report, not five, after a five-day outage); a billing/ledger job may want Backfill (every missed run must execute). MaxBackfill bounds the catch-up so a long outage doesn’t unleash a flood.
  • Always specify the time zone. “2am” is ambiguous without one; set TimeZoneId so DST is handled and the run happens at the local 2am, not UTC 2am. Spring-forward/fall-back edge cases are handled by the planner, but only if you tell it the zone.
  • Idempotent jobs. A recurring job can, under failure/retry, run twice for one occurrence. Make handlers idempotent (especially under Backfill, which intentionally enqueues many).
  • Run the recurring processor and the scheduler. Recurring schedules enqueue one-shot scheduled messages; both processors must run (and be monitored) for jobs to fire.
  • Watch cron expressions. A wrong cron (* * * * * = every minute, not “once”) is the classic self-DoS. Validate expressions and review them; the planner is only as correct as the cron you give it.
  • Pause/resume and TriggerNow. Use the recurring scheduler’s pause/resume/trigger-now for operational control (disable a job during an incident, run one immediately to test).

Common mistakes

  • Hand-rolled Timer firing “every 24h.” Drifts, double-fires across instances, ignores DST, loses its place on restart. The recurring scheduler exists precisely to avoid this.
  • Ignoring the missed-run question. Not choosing a catch-up policy means accepting whatever the default does after downtime — which may silently skip a run that had to happen, or backfill a flood that shouldn’t. Decide.
  • Cron in the wrong (or no) time zone. “2am daily” in UTC for a local business is an hour off and DST- broken. Set the time zone.
  • Non-idempotent recurring jobs. Retries and backfill mean handlers run more than once per logical occurrence; non-idempotent ones corrupt or duplicate.
  • A runaway cron expression. MaxBackfill unbounded plus a frequent cron after a long outage = a flood. Bound backfill; sanity-check frequency.

Tradeoffs

Benefits. Durable, multi-instance-safe, DST-correct recurring execution; an explicit catch-up policy for missed runs; typed jobs with reliable delivery; and pause/resume/trigger-now operability.

Costs. A recurring-schedule table and processor; the catch-up decision and its bounds; cron-expression risk; and job idempotency requirements.

Alternatives

  • In-process periodic timers (PeriodicTimer, hosted loops). Simple and precise for tight, ephemeral, single-instance intervals; lost on restart and wrong under multi-instance for anything durable.
  • OS / infrastructure cron (k8s CronJob, systemd timers, cloud schedulers). Durable and managed, ideal for infrastructure cadence that triggers a process/endpoint. They don’t carry your domain types or transaction; Relay’s recurring scheduler runs inside the app with your jobs/commands. Use infra cron for infra, Relay’s for domain-level recurring work.
  • Hangfire / Quartz.NET. Mature recurring-job libraries with their own storage and dashboards. Great standalone; on Relay, the built-in recurring scheduler integrates with the same scheduler, bus, and outbox rather than adding a parallel system.

Objectively: Relay’s recurring scheduler wins for domain-level recurring work needing durability, multi-instance safety, DST correctness, and catch-up control; infra cron wins for infrastructure cadence; in-process timers win for tight ephemeral intervals.

Lessons from production systems

  • DST and time zones cause the embarrassing bugs. “The nightly job ran at 1am for half the year” or “ran twice on the fall-back night” are real incidents from UTC-only timers. Naming the time zone and letting the planner handle DST eliminates them.
  • The missed-run policy is discovered after the first outage. Teams find out they wanted Backfill (the billing run had to happen) or Skip (five days of reports flooded in) the hard way. Decide the policy when you create the schedule, not during the post-mortem.
  • Cron-expression mistakes are the self-inflicted outages. * * * * * instead of a daily expression has DoS’d more than one system. Treat cron strings as code: review, test, validate.
  • Idempotency is non-negotiable under backfill. A backfilling job that isn’t idempotent turns a catch- up into a corruption event. The mature pattern pairs Backfill with idempotent handlers, always.

Should you use this?

Situation Recommendation
Recurring domain work (nightly cleanup, reports, reconciliation, billing cadence) Strong fit
Cron schedules needing DST correctness + catch-up control Strong fit
Durable background jobs with typed handlers Strong fit (jobs)
One-shot future work Use the scheduler (article 018)
Sub-minute / high-frequency cadence No — use an in-process loop
Pure infrastructure cadence Consider OS/cloud cron
Best-effort, single-instance, loss-tolerable Lean against — a PeriodicTimer is simpler

Next steps

You’ve now covered the asynchronous, time-based, and workflow capabilities. The remaining articles address the production track — running all of this safely at scale: multi-tenancy (isolating tenants across the event store, projections, sagas, and cache), observability, resiliency, distributed coordination, schema evolution, and a capstone reference architecture. The next one is multi-tenancy.

➡️ 020 — Multi-Tenancy (planned — see the coverage plan)