Articles · Projections & Read Models ·032

Automated Blue/Green Projection Rebuilds

Runnable sample on GitHub

Sample: samples/032-automated-blue-green-rebuilds — a pure, in-memory test of the auto-swap policy (fakes the rebuild manager; no Docker). dotnet test.

Prerequisite: 008 — Projection Operations — introduces IRebuildableProjection, IProjectionRebuildManager, and the operator-driven shadow swap.

Overview

Article 008 gave you the shadow (blue/green) rebuild: a projection implements IRebuildableProjection, IProjectionRebuildManager builds a fresh shadow read model from history while the live one keeps serving, and once the shadow has caught up an operator calls SwapAsync to atomically promote it. That last step is manual — someone watches the progress gauge and pushes the button at the right moment.

This article automates that button. IAutomaticProjectionSwapper / AutomaticProjectionSwapper (in Nuvora.Nexus.Relay.Projections.Rebuild) starts the shadow rebuild, polls progress, and the moment the rebuild reports Completed it performs the swap itself — zero-touch blue/green. It swaps under no other outcome: a rebuild that is still Running, or that ends Failed/Cancelled, leaves the live read model exactly where it was.

Why this exists

The manual swap from 008 is correct but it has a human in the hot path. For a large, always-on, user-facing read model that’s a problem in two directions:

  • The window to swap is unattended. A shadow rebuild of billions of events can finish at 3 a.m. or mid-deploy. Waiting for an operator to notice “it’s caught up” and run SwapAsync wastes the very zero-downtime property the shadow gave you — the new model is ready but the old one keeps serving stale results until someone looks.
  • Manual swaps are error-prone. A human swapping the wrong projection, or swapping a shadow that hasn’t actually caught up, is exactly the kind of mistake that turns a routine reproject into an incident. A policy that swaps only on a Completed status, and never otherwise, removes the judgement call.

The automatic swapper is a thin, well-defined policy on top of the existing rebuild manager: start a shadow → poll → swap on Completed, stop on anything terminal. It is deliberately small so the safety properties are obvious.

When to use this

  • Large, always-on read models where you can’t tolerate the in-place rebuild’s stale window and don’t want a human babysitting the swap.
  • Routine, repeatable reprojections — a scheduled rebuild after a deploy, a nightly reshape — where the swap criterion is simply “the shadow caught up,” with no judgement needed.
  • Hands-off zero-downtime reprojection as a standard operation: kick it off (or wire it into a background worker / leader-elected job) and trust it to promote the new model when, and only when, it’s ready.

When not to use this

  • When you want a human gate. If swapping requires sign-off (a QA pass against the shadow, a manual data spot-check), keep the operator-driven SwapAsync from 008. The automatic swapper swaps as soon as the status is Completed — that’s the point, and it’s wrong if you wanted to look first.
  • For in-place rebuilds. There is nothing to swap in an in-place rebuild (it rewinds the live projection). The swapper defaults to RebuildMode.Shadow precisely because the swap is only meaningful for a shadow. Don’t reach for it on a small read model where an in-place reset is fine.
  • When the projection isn’t idempotent or has side effects. Same precondition as every rebuild (008): a rebuild replays all of history. If that’s unsafe, automating the swap doesn’t fix it — it just promotes a bad model faster.

Concepts

Shadow rebuild (recap from 008). The projection implements IRebuildableProjection so the manager can build a shadow copy of the read model from history while the live one keeps serving. The two flavours are RebuildMode.InPlace (rewind the live projection) and RebuildMode.Shadow (build-then-swap). Blue/green is the shadow flavour.

Progress polling. IProjectionRebuildManager.GetProgressAsync returns a RebuildProgress (Status, CurrentPosition, TargetPosition, PercentComplete) — or null if no rebuild is recorded for that (projection, tenant). Progress is derived from the projection’s checkpoint versus the head the manager captured when the rebuild started.

Swap-when-caught-up. The swapper polls on a configurable interval (default 2s). It does not swap on a position heuristic — it swaps when the manager declares the rebuild Completed. The manager owns the “caught up” decision (its RebuildOptions.SwapThreshold governs how close to the live head the shadow must get before it reports complete); the swapper trusts that status. This keeps the policy trivially correct: one terminal state means swap, the rest mean stop.

Statuses (ProjectionRebuildStatus). Running → keep polling. Completed → SwapAsync, then return. Failed / Cancelled → return without swapping. A null progress (no rebuild row) → return without swapping. Cancellation of the passed CancellationToken also returns without swapping.

Architecture

flowchart TD
    Run[RunAsync projection] --> Start[StartRebuildAsync<br/>RebuildMode.Shadow]
    Start --> Poll[GetProgressAsync]
    Poll --> S{Status?}
    S -->|Running| Wait[Task.Delay pollInterval] --> Poll
    S -->|Completed| Swap[SwapAsync<br/>atomic blue → green] --> Done[return]
    S -->|Failed / Cancelled / null| Stop[return — live model untouched]
    Poll -.cancellation requested.-> Stop

Building it step by step

The sample is samples/032-automated-blue-green-rebuilds. It is a pure test: it fakes the rebuild manager, so there is no database, no shadow table, no Docker — just the swap policy.

1. The policy (the real AutomaticProjectionSwapper)

The framework type is small enough to read in full. It defaults to a shadow rebuild, polls, and swaps on Completed:

options ??= new RebuildOptions { Mode = RebuildMode.Shadow };   // blue/green needs a shadow to swap
await _manager.StartRebuildAsync(projectionName, options, tenant, ct);

while (!ct.IsCancellationRequested)
{
    var progress = await _manager.GetProgressAsync(projectionName, tenant, ct);
    switch (progress?.Status)
    {
        case ProjectionRebuildStatus.Completed:
            await _manager.SwapAsync(projectionName, tenant, ct);   // automatic swap
            return;
        case ProjectionRebuildStatus.Cancelled:
        case ProjectionRebuildStatus.Failed:
        case null:
            return;                                                 // do NOT swap
    }
    await Task.Delay(_pollInterval, _time, ct);                     // still Running → poll again
}

2. A scripted fake rebuild manager (pure)

To test the policy we don’t need a real shadow rebuild — we need a manager that reports a chosen sequence of statuses. The fake clamps on the last status so an overshooting poll keeps seeing the terminal state:

internal sealed class FakeRebuildManager(params ProjectionRebuildStatus[] statuses) : IProjectionRebuildManager
{
    public int StartCalls { get; private set; }
    public int SwapCalls { get; private set; }

    public Task StartRebuildAsync(string n, RebuildOptions? o = null, Guid? t = null, CancellationToken c = default)
    { StartCalls++; return Task.CompletedTask; }

    public Task<RebuildProgress?> GetProgressAsync(string n, Guid? t = null, CancellationToken c = default)
        => Task.FromResult<RebuildProgress?>(new RebuildProgress(n, t, Next(), 50, 100, 50));

    public Task SwapAsync(string n, Guid? t = null, CancellationToken c = default)
    { SwapCalls++; return Task.CompletedTask; }
    // CancelRebuildAsync omitted for brevity
}

3. Drive it on a tiny poll interval

var swapper = new AutomaticProjectionSwapper(manager, timeProvider: null, pollInterval: TimeSpan.FromMilliseconds(10));
await swapper.RunAsync("account-activity").WaitAsync(TimeSpan.FromSeconds(5));   // bounded so a hang fails loudly

Complete source code

File Shows DB?
AutomaticProjectionSwapTests.cs starts + auto-swaps on Completed; never swaps while Running / Failed / Cancelled; defaults to Shadow No
FakeRebuildManager.cs scripted in-memory IProjectionRebuildManager No

Running the example

dotnet test samples/Relay.Samples.slnx

Or just this sample:

dotnet test samples/032-automated-blue-green-rebuilds/Projections.AutoSwap.Tests

Needs only the .NET 10 SDK — no Docker, no database.

Testing

The split mirrors the rest of Relay: push the logic into a pure, fast policy and fake the database-backed dependency. Four assertions pin the contract:

It swaps once the rebuild has caught up. Script Running then Completed; the swapper starts the rebuild and swaps exactly once:

var manager = new FakeRebuildManager(ProjectionRebuildStatus.Running, ProjectionRebuildStatus.Completed);
await Swapper(manager).RunAsync("account-activity").WaitAsync(TimeSpan.FromSeconds(5));
manager.StartCalls.Should().Be(1);
manager.SwapCalls.Should().Be(1);   // Completed ⇒ automatic swap

It does NOT swap while the rebuild is still running. Script only Running, cancel after the loop has polled, and assert it never swapped a half-built shadow:

var manager = new FakeRebuildManager(ProjectionRebuildStatus.Running);
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(200));
await Swapper(manager).RunAsync("account-activity", cancellationToken: cts.Token).WaitAsync(TimeSpan.FromSeconds(5));
manager.SwapCalls.Should().Be(0);   // a running rebuild must never swap a partial read model

A failed or cancelled rebuild does not swap. Script Running then Failed (and again with Cancelled); SwapCalls stays 0 — the live read model is untouched.

These run with zero infrastructure because the behaviour under test is a policy over statuses, not a database operation. The genuinely database-backed parts (the shadow build, the atomic swap) are covered by 008’s integration tests against real PostgreSQL — a mock there would prove nothing.

Production considerations

  • DI: AddRelayAutomaticProjectionSwap(). Register the swapper alongside the rebuild manager: services.AddRelayProjectionRebuild(); then services.AddRelayAutomaticProjectionSwap();. The latter registers IAutomaticProjectionSwapper; it requires the rebuild manager (and, for durable progress, the EF Core rebuild store via AddRelayProjectionRebuildEfCore<TContext>).
  • Idempotent projections are still the precondition. Automating the swap does not relax 008’s rule: a rebuild replays all history, so the projection must be idempotent and side-effect-free. Automation just promotes the result faster — including a bad one if the projection isn’t safe to replay.
  • Mind the lag during the swap. The shadow catches up to the head captured at start; live writes keep arriving. After the swap the freshly-promoted model has a brief tail of events to apply. Keep the SwapThreshold tight enough that the catch-up tail is small, and watch the lag gauge so you know the promoted model has converged.
  • Run it under coordination at scale-out. Like any singleton rebuild, two instances auto-swapping the same projection would race. Drive the swapper from a leader-elected/locked worker (article 023) so one node owns the rebuild-and-swap.
  • Keep the work cancellation-aware. Pass and honour a stoppingToken; on cancellation the swapper returns without swapping, leaving the live model intact for a clean shutdown mid-rebuild.

Common mistakes

  • Automating the swap on a non-idempotent projection. The 008 rebuild hazard, now hands-off: a non-idempotent replay corrupts the shadow and the swapper promotes the corruption. Fix idempotency first.
  • Using it for an in-place rebuild. There’s nothing to swap in-place. The swapper defaults to RebuildMode.Shadow; don’t override it to InPlace and expect a swap.
  • Expecting a human gate. If you needed to inspect the shadow before promoting, the automatic swapper is the wrong tool — it swaps the instant the status is Completed. Use the manual SwapAsync.
  • Racing swaps across instances. Running the swapper on every replica double-rebuilds and races on the swap. Coordinate it (leader election / distributed lock).
  • Polling too slowly for a fast rebuild. The default 2s interval is fine for a long rebuild; if your rebuild finishes in seconds you’ll add seconds of stale serving after it’s actually caught up. Tune pollInterval to the rebuild’s scale.

Tradeoffs

Benefits. Zero-touch blue/green: the new read model is promoted the moment it’s ready, with no operator in the hot path and no judgement call about when to swap. The policy is tiny and its safety property is obvious — it swaps on exactly one status and stops on all the others.

Costs. You give up the human inspection point — the swap happens automatically, so a shadow that is technically Completed but semantically wrong (a projection bug you’d have caught by eyeballing it) gets promoted. And it inherits every rebuild precondition (idempotency, no side effects) with the added risk that automation applies them faster and unattended.

Alternatives

  • Operator-driven swap (008). Watch progress, call SwapAsync by hand. The right choice when you want a human gate or sign-off before promotion; the automatic swapper is for the routine, unattended case.
  • In-place rebuild (ProjectionManager.ResetAsync). Rewind the live projection and replay; no swap, no shadow. Simpler, but serves stale/partial data while it runs — fine for a small read model or a maintenance window, not a large user-facing one.
  • Database-level blue/green (swap whole read databases). Build a new read store and cut over at the infrastructure layer. Heavier and outside the framework; warranted for a whole-store migration, overkill for one read model.

Lessons from production systems

  • The unattended finish is where manual swaps bleed value. Teams that shadow-rebuild but swap by hand routinely leave a ready model unserved for hours because the rebuild finished off-hours. Automating the swap recovers exactly the zero-downtime property the shadow was for.
  • A swap policy with one rule is one you can trust at 3 a.m. “Swap on Completed, stop on everything else” is small enough to reason about completely — which is why it’s safe to let it run unattended, whereas a clever multi-condition heuristic is the thing that swaps the wrong model.
  • Idempotency gets tested the first time an automated rebuild replays history. Just like 008’s manual rebuild — only now nobody’s watching when the doubled rows appear. Make idempotency a test, not a hope, before you automate the swap.

Should you use this?

Situation Recommendation
Large, always-on read model, routine reproject Strong fit — hands-off zero-downtime swap
Scheduled rebuild after deploy / nightly reshape Strong fit — wire it into a leader-elected job
You need to inspect the shadow before promoting Use manual SwapAsync (008) — keep the human gate
Small read model / maintenance window In-place rebuild — no shadow needed
Projection isn’t idempotent / has side effects Fix that first — automation promotes corruption faster

Next steps