Articles · Scheduling & Time ·018

Scheduling

Runnable sample on GitHub

Sample: samples/018-scheduling — schedule and dispatch a delayed command. dotnet test. No database.

Prerequisites: 001 — Getting Started. Underpins 015 — Sagas.

Overview

Some work shouldn’t happen now — it should happen later: send a reminder in 30 minutes, expire an unpaid order in an hour, retry a failed call after a backoff. Relay’s scheduler makes that durable. IScheduler.ScheduleCommandAsync(command, delay) stores a scheduled message (in PostgreSQL); a background SchedulerProcessor claims due rows (with FOR UPDATE SKIP LOCKED, so multiple instances never double-fire) and a dispatcher deserialises and executes the command through the bus. It is the durable, multi-instance-safe alternative to in-memory timers — and the mechanism behind saga timeouts (article 015).

Why this exists

The naive “later” is an in-memory timer (Task.Delay, a Timer, a BackgroundService loop). It has three fatal flaws for anything that matters: it’s lost on restart (the process dies, the reminder never fires), it doesn’t scale (each instance has its own timers, so a multi-instance deployment either fires N times or, worse, fires zero times if the instance holding the timer died), and it’s invisible (no record of what’s pending). Durable scheduling fixes all three: the scheduled work is a row in the database, so it survives restarts; a single processor claims each due row exactly once across all instances; and the pending work is queryable. “Run this command at this time, reliably, exactly once, across a fleet” is the guarantee, and you can’t get it from an in-process timer.

When to use this

Use the scheduler for durable, delayed, exactly-once execution of a command or message:

  • Reminders / deadlines: “remind the user in 24h,” “expire the cart in 30 min,” “escalate the ticket if unacknowledged in 4h.”
  • Saga timeouts: the engine behind RequestTimeout — act when something doesn’t happen (article 015).
  • Delayed / deferred actions: “publish this post at 9am,” “deactivate the trial at end of period.”
  • Backoff retries: schedule a retry of a failed operation after a delay (the scheduler also powers consumer second-level redelivery).

The signal: “do X at time T (or after delay D), and it must not be lost on restart, and it must run once even with many instances.”

When not to use this

  • Immediate work. If it should happen now, just do it — scheduling adds a round-trip through the store for no reason.
  • Sub-second precision / high-frequency timers. The scheduler polls (and wakes on notify); it’s for seconds-to-days, not microsecond-precise or thousands-per-second timers. For those, an in-memory timer or a specialized system is appropriate.
  • Best-effort, loss-tolerable delays. If losing the occasional delayed action on restart is fine, an in-memory timer is simpler. The scheduler’s durability is overhead you don’t need.
  • Recurring schedules. “Every night at 2am” is the recurring scheduler (article 019), which plans occurrences from a cron expression; the plain scheduler is for one-shot future work.

The costs: a scheduled-messages table and a processor to run/monitor; a slight imprecision (poll interval

  • claim latency, not exact-to-the-millisecond); and the need to keep payloads serialisable and their types resolvable when they fire.

Concepts

IScheduler. ScheduleCommandAsync(command, delay|fireAt) stages a one-shot scheduled command; ScheduleAsync(ScheduledMessage) stages an arbitrary scheduled message; CancelAsync / CancelByCorrelationAsync / CancelByKeyAsync cancel pending ones. Staging happens on the ambient transaction (atomic with the work that requested it).

ScheduledMessage. The stored row: MessageType + Payload (the serialised command/job/envelope), Kind (Command, SagaTimeout, Job, IntegrationEvent, ConsumerRedelivery), ScheduledFor (when it’s due), status, retry/lease bookkeeping, optional correlation/cancellation keys and tenant.

SchedulerProcessor. A background service that claims due rows (ScheduledFor <= now) with FOR UPDATE SKIP LOCKED, dispatches each to the dispatcher for its Kind, marks it executed, and retries/dead-letters failures. Multi-instance-safe by construction.

Dispatchers. One per Kind. CommandScheduledMessageDispatcher deserialises the command and executes it via ICommandBus; the saga-timeout, job, and redelivery kinds have their own (articles 015/019/014).

Architecture

sequenceDiagram
    participant A as Caller (or saga)
    participant Sch as IScheduler
    participant DB as relay_scheduled_messages
    participant P as SchedulerProcessor
    participant D as CommandScheduledMessageDispatcher
    A->>Sch: ScheduleCommandAsync(cmd, 30m)
    Sch->>DB: stage row (Kind=Command, ScheduledFor=now+30m)
    Note over P,DB: later, when due…
    P->>DB: claim due rows (FOR UPDATE SKIP LOCKED)
    P->>D: dispatch
    D->>D: deserialise → ICommandBus.Execute(cmd) → mark Executed

Building it step by step

The sample is samples/018-scheduling.

1. A command to run later

[SkipTransaction] // no DB in this sample
public sealed record SendReminderCommand(string Message) : ICommand;

2. Schedule it

var scheduler = new Scheduler(repository, new SystemTextJsonScheduledMessageSerializer());
await scheduler.ScheduleCommandAsync(new SendReminderCommand("renew subscription"), TimeSpan.FromMinutes(30));
// → a stored ScheduledMessage, Kind=Command, ScheduledFor ≈ now + 30 min

3. When due, the processor claims and the dispatcher executes

The sample drives the claim/dispatch by hand (production runs the SchedulerProcessor continuously):

var due = await repository.ClaimDueAsync();
await new CommandScheduledMessageDispatcher(serializer).DispatchAsync(due.Single(), provider, default);
// → the command runs through ICommandBus → its handler executes

4. Register (production)

services.AddRelayScheduling();                 // IScheduler, the command dispatcher, the processor (hosted)
services.AddRelaySchedulerEfCore<TContext>();  // PostgreSQL storage (relay_scheduled_messages)

Complete source code

File Contents
Reminders.cs The command, handler, and an in-memory scheduler repository
SchedulingTests.cs Scheduling stages a message; dispatching runs the command

Running the example

dotnet test samples/018-scheduling/Scheduling.Sample.Tests

Testing

The two halves — schedule and fire — are tested separately:

// scheduling stages a Command message due at the right time
await scheduler.ScheduleCommandAsync(new SendReminderCommand("renew subscription"), TimeSpan.FromMinutes(30));
message.Kind.Should().Be(ScheduledDeliveryKind.Command);
message.ScheduledFor.Should().BeCloseTo(DateTimeOffset.UtcNow.AddMinutes(30), TimeSpan.FromSeconds(5));

// dispatching a due scheduled command actually runs it
await dispatcher.DispatchAsync(due.Single(), provider, default);
provider.GetRequiredService<ReminderLog>().Messages.Should().Contain("pay invoice");

Testing the dispatch through the real CommandScheduledMessageDispatcher + ICommandBus proves the round-trip: a serialised command, claimed when due, deserialises and executes its handler.

Production considerations

  • Run the processor and monitor the backlog. A scheduler with nothing claiming due rows is just a table. Run SchedulerProcessor; alert on the scheduler backlog / oldest-due age (rows overdue but not firing means the processor is stuck or down).
  • Keep payloads serialisable and types resolvable. A scheduled command fires later — its type must still be resolvable then (don’t rename/remove a scheduled command’s type while rows referencing it are pending). Version carefully.
  • Expect approximate, not exact, timing. Firing is due-time + poll/claim latency. For “in ~30 minutes” that’s fine; for hard real-time it isn’t the right tool.
  • Use cancellation keys for cancellable schedules. A saga timeout that may be cancelled (payment arrived) uses correlation/cancellation keys so CancelByCorrelation/CancelByKey can drop pending rows cheaply.
  • Prune executed rows. Like outbox/inbox, executed scheduled messages accumulate; DeleteCompletedAsync / maintenance keeps the table bounded.
  • It’s multi-instance-safe — lean on that. FOR UPDATE SKIP LOCKED means many instances can run the processor; each due row fires once. A stale lease (crashed processor) is reclaimed.

Common mistakes

  • Using in-memory timers for durable work. Lost on restart, wrong under multi-instance. If “later” must survive a deploy or run once across a fleet, use the scheduler.
  • Not running/monitoring the processor. Scheduled rows that never fire are a silent failure. The processor and its backlog alert are mandatory.
  • Renaming/removing scheduled command types. A pending row referencing a now-missing type can’t be dispatched. Treat scheduled payload types as a contract until their rows drain.
  • Expecting millisecond precision. It polls; it’s seconds-grained. Don’t build tight real-time logic on it.
  • Forgetting to cancel. A scheduled action that’s no longer needed (the order was paid) should be cancelled, or it fires anyway. Use cancellation keys.

Tradeoffs

Benefits. Durable (survives restarts), exactly-once across instances (SKIP LOCKED), queryable pending work, and the same mechanism powers saga timeouts and second-level retries.

Costs. A table and a processor to run/monitor; approximate timing; payload-type versioning concerns; and retention to manage.

Alternatives

  • In-memory timers (Timer, Task.Delay, hosted-service loops). Simple and precise, but lost on restart and wrong under multi-instance. Fine only for ephemeral, single-instance, loss-tolerable delays.
  • Cloud schedulers (Azure Functions timers, AWS EventBridge Scheduler, cron in k8s). Durable, managed, good for infrastructure-level or coarse scheduling. They live outside your app and typically trigger an endpoint; Relay’s scheduler is in your domain (schedules a command/saga timeout with your types and transaction). Use cloud schedulers for infra cadence, Relay’s for domain-level delayed work.
  • A dedicated job/queue system (Hangfire, Quartz.NET). Mature delayed/recurring job libraries. If you’re not on Relay’s stack they’re great; on Relay, the built-in scheduler integrates natively with the bus, sagas, and outbox.

Objectively: Relay’s scheduler wins for durable, domain-level, multi-instance-safe delayed execution integrated with your commands and sagas; in-memory timers win for ephemeral precision; cloud schedulers win for infrastructure cadence.

Lessons from production systems

  • “Lost on restart” is discovered at the worst time. Teams that schedule reminders/expiries with in-memory timers find out during a deploy that a day’s worth of pending actions vanished. Durable scheduling makes restarts a non-event.
  • The multi-instance double-fire (or zero-fire) is the other classic. In-memory timers across N instances either fire N times or, if the holder dies, never. FOR UPDATE SKIP LOCKED makes “exactly once across the fleet” automatic — a property teams reinvent badly without it.
  • Backlog monitoring catches stuck processors early. An overdue-but-not-firing row means the processor is wedged; teams that alert on oldest-due age catch it before users notice missing reminders.
  • Scheduled-payload versioning bites months later. A command type renamed long after rows were scheduled can’t fire. The durable lesson: scheduled payload types are a long-lived contract.

Should you use this?

Situation Recommendation
Durable delayed action (reminder, expiry, deferral) Strong fit
Saga timeouts / “act if X doesn’t happen” Strong fit (the engine behind it)
Backoff retries of failed work Strong fit
Immediate work No — just do it
Sub-second / high-frequency timers No — use an in-memory timer
Recurring (cron) schedules Use the recurring scheduler (article 019)
Best-effort, loss-tolerable delay, single instance Lean against — a timer is simpler

Next steps

The scheduler runs one-shot future work. The next article covers the recurring case — work that runs on a cron schedule, with time zones and catch-up policies for missed runs — plus durable jobs as the unit of recurring/background work.

➡️ 019 — Recurring Jobs