Projection Operations

Sample: samples/008-projection-operations — poison handling and a rebuild over real PostgreSQL. dotnet test (needs Docker).

Prerequisite: 007 — Projections.

Overview

Article 007’s projection host works while every event is well-behaved. Production guarantees two things that break that assumption. First, some event will be poison — a bug in the projection, a payload it can’t handle, an unexpected null — and it will throw every time the host retries it. Without a release valve, that one event blocks the stream forever and the read model freezes at that point (head-of-line blocking). Second, read models change — a logic fix, a new column, a corrupted table, a brand-new view — and you need to rebuild them from history rather than migrate data. This article covers both: dead-lettering poison events so the stream keeps moving, and rebuilding read models from the event log.

Why this exists

Poison events. A projection that throws on event N stops at N. The host retries — it throws again. Every subsequent event (N+1, N+2, …) is stuck behind it, so the read model is frozen at N-1 while the write side races ahead. One bad event has taken down a whole read model. You cannot edit the event (it’s immutable history), and you cannot just ignore the failure silently (you’d lose the event). The correct behavior is: retry a bounded number of times (transient failures recover), then set the event aside in a dead-letter store and advance past it so the stream keeps flowing — turning a total outage into one missing row you can investigate and replay.

Rebuilds. Read models are derived, so when their definition changes you don’t migrate them — you reproject them. A logic bug means the read model has been wrong; fix the projection and rebuild from history and it’s correct. A new requirement (add a column, a new view) means reprojecting history into the new shape. A corrupted table means rebuilding from the authoritative events. This is the superpower event sourcing gives the read side: because the events are the truth, any read model can be reconstructed, any time, with no data migration — if the projection is idempotent and the operation is supported. Relay supports it in-place (rewind and replay) and, for zero downtime, via a shadow read model that you build then atomically swap.

When to use this

Poison handling is not optional — it is on for every projection (the host dead-letters after MaxEventRetries). The decisions are how many retries and how you respond to a dead-letter (alert, fix, replay). You always want it; a projection without a poison release valve is a future outage.

Rebuilds are an operational tool you reach for when:

• A projection had a bug. The read model has been computing the wrong balance / status / total. Fix the projection, rebuild, and the read model is correct from history — no manual data surgery.
• A read model’s shape must change. New column, new grouping, new index — reproject history into the new shape instead of writing a fragile migration.
• You add a new read model. A new query need over existing data: register the projection, rebuild it from history, and it’s populated as if it had always existed.
• A read model is corrupted. Bad deploy, manual UPDATE gone wrong, partial write — rebuild from the authoritative events and discard the corruption.

Concretely: a payments dashboard that miscounted (fix + rebuild), a new “high-value accounts” view (add

• rebuild), an order read model that needs a new column (reshape + rebuild).

When not to use this

• Don’t rebuild to fix a write-side bug. Rebuilds reconstruct read models from events. If the events are wrong (a bug wrote bad events), a rebuild faithfully reprojects the bad events — it fixes read-model logic, not corrupt history. Wrong events are a schema-evolution / corrective-event problem (article 024), not a rebuild.
• Don’t dead-letter what you should fix. Dead-lettering is for genuinely poison events, not a way to paper over a projection that throws on normal data. If everything is dead-lettering, the projection is broken — fix it and replay, don’t raise MaxEventRetries to hide it.
• Don’t rebuild casually on huge streams without a plan. Reprojecting billions of events is real work (time, load). For very large read models, prefer the shadow/blue-green rebuild and schedule it, rather than an in-place rebuild that serves stale/partial data for hours.

The costs: poison handling means accepting that a dead-lettered event leaves a gap in the read model until you replay it (eventual, manual repair); and rebuilds cost time and load proportional to history, and demand genuinely idempotent, side-effect-free projections (a rebuild replays everything).

Concepts

Poison event & retries. When ProjectAsync throws, the host rolls back the batch (no partial write) and retries on the next cycle. After MaxEventRetries failures for that specific event, the host records it in the dead-letter store and advances the checkpoint past it, so the stream continues. Failure tallies are durable (they survive restarts and are shared across instances), so the retry budget is honored across crashes.

Dead-letter store. IProjectionDeadLetterStore keeps the skipped events (projection name, global position, event type, payload, error). You query it to see what’s broken, and — after fixing the projection — replay them. A dead-lettered event is not lost; it is set aside.

In-place rebuild. ProjectionManager.ResetAsync(name, fromPosition: 0) rewinds the projection’s checkpoint (and clears its durable failure tallies, and runs the projection’s ResetAsync clear hook) so the running host replays history from the start. The read model serves stale/partial data while rebuilding, then converges — simple, with a brief consistency window.

Shadow (blue/green) rebuild. For zero downtime, a projection implements IRebuildableProjection (prepare a shadow read model, build it from history while the live one keeps serving, then atomically swap). IProjectionRebuildManager drives it with progress tracking. Use it when you cannot tolerate the in-place rebuild’s stale window.

Idempotency is the precondition. A rebuild replays every event. If the projection isn’t idempotent (blind inserts, non-atomic increments without the checkpoint guarantee), a rebuild corrupts the read model. Idempotent projections (guarded upserts) rebuild cleanly.

Architecture

graph TD
    H[Projection host] -->|apply event N| P[Projection]
    P -->|throws| R{retries < MaxEventRetries?}
    R -->|yes| H
    R -->|no| DL[(dead-letter store)]
    DL --> A[advance checkpoint past N]
    A -->|stream continues| H
    Fix[fix projection] -.->|replay| DL
    subgraph Rebuild
        Reset[ResetAsync from 0] --> Rewind[checkpoint → 0, clear failures]
        Rewind --> Replay[host replays history, idempotently]
    end

Building it step by step

The sample is samples/008-projection-operations.

1. A projection that can fail — idempotently

public sealed class AccountActivityProjection(IRelayDbContextAccessor accessor, IEventSerializer serializer)
    : IProjection
{
    public string Name => "account-activity";
    public bool Handles(string eventType) => eventType == typeof(AccountOpened).FullName;

    public async Task ProjectAsync(EventData @event, CancellationToken ct = default)
    {
        var opened = (AccountOpened)serializer.DeserializeEvent(@event);
        if (opened.Owner == "POISON")                              // stand-in for a real poison event
            throw new InvalidOperationException("poison event");

        if (!await Set().AnyAsync(r => r.Id == opened.AccountId, ct)) // idempotent: safe to replay
            Set().Add(new AccountActivityReadModel { Id = opened.AccountId, Owner = opened.Owner });
    }
}

The idempotent insert is what makes both dead-letter replay and rebuild safe — applying the same event twice is harmless.

2. Configure the retry budget

new ProjectionOptions { BatchSize = 100, PollingInterval = TimeSpan.FromMilliseconds(100), MaxEventRetries = 2 }

MaxEventRetries is the dial: high enough to ride out transient failures (a brief DB hiccup), low enough that a genuinely poison event is set aside quickly instead of retried forever.

3. Inspect and replay dead-letters

var deadLetters = await deadLetterStore.GetAsync("account-activity"); // what got skipped, and why
// … fix the projection, deploy, then replay the dead-lettered events (ProjectionManager) …

4. Rebuild from history

var manager = new ProjectionManager(scopeFactory, failureCache, NullLogger<ProjectionManager>.Instance);
await manager.ResetAsync("account-activity", fromPosition: 0); // rewind; the running host replays history

For zero-downtime, implement IRebuildableProjection and drive IProjectionRebuildManager (prepare shadow → catch up → swap) instead.

Complete source code

File	Contents
Projections/AccountActivityProjection.cs	A projection that poisons on "POISON", idempotently
ProjectionOperationsTests.cs	The poison-skip + the reset/rebuild
LedgerDbContext.cs	ApplyRelayProjectionCheckpoints() (incl. dead-letters/failures)

Running the example

dotnet test samples/008-projection-operations/Ledger.ProjectionOps.Tests   # needs Docker

Testing

The poison event is skipped, not fatal. The proof is that an account after the poison still projects — if the stream were wedged, it never would:

await bus.Execute(new OpenAccountCommand(before, "Ada"));
await bus.Execute(new OpenAccountCommand(poison,  "POISON")); // throws in the projection
await bus.Execute(new OpenAccountCommand(after,  "Grace"));

await host.StartAsync(default);
var unwedged = await WaitUntilAsync(() => activity.AnyAsync(r => r.Id == after), 60s);
unwedged.Should().BeTrue();                                   // 'after' projected ⇒ poison was skipped
(await activity.AnyAsync(r => r.Id == poison)).Should().BeFalse();
(await deadLetters.GetAsync("account-activity")).Should().Contain(d => d.Payload.Contains("POISON"));

A reset rebuilds from history. Corrupt the read model, reset the checkpoint, and the running host reconstructs it:

await activity.Where(r => r.Id == id).ExecuteDeleteAsync();     // simulate corruption
await manager.ResetAsync("account-activity", fromPosition: 0);  // rewind
(await WaitUntilAsync(() => activity.AnyAsync(r => r.Id == id), 90s)).Should().BeTrue(); // rebuilt

These run against real PostgreSQL because the dead-letter commit, the checkpoint rewind, and the gap-safe replay are all database behaviors — the very things a mock would fail to verify.

Production considerations

• Alert on dead-letters. A dead-lettered event is a gap in the read model and a signal of a bug or bad data. Wire an alert on dead-letter count; a silent dead-letter is a silently-wrong read model.
• Replay after you fix the projection — don’t just lower the bar. The lifecycle is: poison detected → dead-lettered → you fix the projection → deploy → replay the dead-letters. Raising MaxEventRetries to stop the dead-lettering is treating the symptom; the event is still unprojectable.
• Tune MaxEventRetries to distinguish transient from poison. Too low and a momentary DB blip dead-letters good events; too high and a truly poison event burns cycles retrying. A handful of retries is usually right; pair it with backoff if your transient failures are slow to clear.
• Prefer shadow rebuilds for large, always-on read models. An in-place rebuild serves stale/partial data while it runs — fine for a small read model or a maintenance window, painful for a large, user-facing one. IRebuildableProjection + the rebuild manager build a shadow and swap atomically, so users never see a half-built model.
• Rebuilds must be side-effect-free. A rebuild replays all history. A projection that sent an email or called an API per event will re-send/re-call thousands of times on rebuild. Keep projections pure (read-model writes only); side effects go on the integration-event path (articles 011–014).
• Monitor lag, especially during a rebuild. A rebuild is a big catch-up; watch the lag gauge and health check so you know when it has converged and whether it is keeping up with live writes.
• Scale-out needs coordination. Multiple host instances rebuilding the same projection would race; AddRelayProjectionCoordinationPostgres() (an advisory lock) gives each projection one active worker.

Common mistakes

• No poison release valve (rolling your own host). If you ever drive projections yourself, a projection that throws forever blocks the stream. The dead-letter-after-retries behavior is the whole point; don’t defeat it.
• Non-idempotent projections + rebuild. Blind inserts or unguarded increments turn a rebuild into corruption (double rows, doubled balances). Idempotency is the precondition for the rebuild superpower; without it, rebuilds are dangerous.
• Side effects in a projection. The rebuild disaster: replay history, re-send two years of notifications. Projections write read models and nothing else.
• Ignoring the dead-letter store. Dead-lettered events that nobody looks at are permanent gaps. Treat the dead-letter store as an alertable, actionable queue, not a black hole.
• Rebuilding to fix bad events. A rebuild reprojects whatever the events say. If the events are wrong, you need corrective events / upcasting (article 024), not a rebuild.
• In-place rebuilding a large user-facing read model in business hours. Users see stale/partial data for the duration. Use a shadow rebuild or a maintenance window.

Tradeoffs

Benefits. One poison event can’t take down a read model; skipped events are preserved for repair, not lost; read models can be reconstructed from history for bug fixes, reshapes, new views, or corruption — no data migration; and shadow rebuilds make even that zero-downtime.

Costs. A dead-lettered event is a gap until manually replayed (eventual, hands-on repair); rebuilds cost time and load proportional to history; and both demand idempotent, side-effect-free projections — a discipline that, if violated, turns the safety nets into hazards.

Alternatives

• Stop-the-world on failure (no dead-lettering). Halt the projection when an event fails and page a human. Maximally safe against silently skipping data, maximally fragile operationally — one bad event is a read-model outage. Acceptable only where any skipped event is unacceptable and you have 24/7 operators; for most systems, dead-letter-and-continue is the better default.
• Migrate read models in place (no rebuild). Write a SQL migration to reshape the read model rather than reproject. Fine for trivial additive changes; fragile and lossy for anything logic-bearing, and it can’t fix a projection bug (the existing rows are already wrong). Rebuilds derive correctness from the events; migrations preserve whatever (possibly wrong) state exists.
• Blue/green at the database level (swap whole databases). Build a new read database and cut over. Heavier-weight than a shadow projection and outside the framework; sometimes warranted for a total read-store migration, overkill for one read model.

Objectively: dead-letter-and-continue plus from-history rebuilds is the right operational model for event-sourced read sides. The alternatives fit narrow cases (zero-skip tolerance, trivial reshapes, whole-store migrations).

Lessons from production systems

• The first poison event is a rite of passage. Teams that built the dead-letter release valve shrug it off (one row missing, alert fires, fix-and-replay). Teams that didn’t discover head-of-line blocking at 2 a.m. when the dashboard froze. Build the valve before you need it — Relay’s is on by default; just don’t defeat it.
• Rebuilds change how fearlessly teams evolve read models. When any read model can be rebuilt from events, “we need a new column / a new view / to fix this calculation” is a routine reproject, not a scary migration. The teams that internalize this iterate on the read side constantly; the ones that don’t treat read models as precious and ossify.
• Idempotency is tested by the first rebuild, painfully. Projections that “worked” because each event arrived once reveal their non-idempotency the first time a rebuild replays history — doubled totals, duplicate rows. Make idempotency a design rule and a test, not a hope.
• Dead-letters need an owner. A dead-letter store nobody watches is a silently-incomplete read model. The successful pattern is an alert plus a runbook (inspect → fix projection → deploy → replay), so poison events get repaired rather than accumulating as invisible gaps.

Should you use this?

Situation	Recommendation
Any production projection (poison handling)	Strong fit — it’s on; tune retries and alert on dead-letters
Fixing a projection bug / reshaping a read model	Strong fit (rebuild) — reproject from history
Adding a new read model over existing data	Strong fit (rebuild)
Large, always-on, user-facing read model	Use shadow rebuild — avoid the in-place stale window
The events are wrong	Not a rebuild — use corrective events / upcasting (article 024)
Zero tolerance for any skipped event, 24/7 ops	Consider stop-the-world over dead-lettering

Next steps

You now have a durable, fast, observable, operable event-sourced write-and-read system. The next article steps off the write path entirely to address the read path’s other lever: caching queries so the hot ones don’t even hit the read model — with automatic invalidation and stampede protection.

➡️ 009 — Caching Queries