Articles · Event Sourcing ·034

Event Archiving & Compaction

Runnable sample on GitHub

Sample: samples/034-event-archiving-and-compaction · Subsystem: 2.3 · Prerequisites: 005 — Event Sourcing Basics, 024 — Schema Evolution

Overview

An event store is append-only: events are never updated and never deleted, which is exactly what makes it a trustworthy source of truth. But “never deleted” has a cost — the hot relay_events table, the one every append writes to and every replay reads from, only ever grows. For a handful of very long-lived streams that can become the dominant cost on the write path: bloated indexes, slower vacuums, a working set that no longer fits in cache.

Archiving (a.k.a. compaction) addresses this without violating immutability. Instead of deleting old events, it moves them: a stream’s early versions are copied into a separate relay_events_archive table and removed from the live one. History is fully preserved — the rows still exist, just not in the table that backs the live workload. Relay ships one primitive for this:

  • IEventArchiver / EfCoreEventArchiver<TContext> — ArchiveStreamAsync(aggregateId, upToVersionInclusive) moves a stream’s events with version <= N out of relay_events and into relay_events_archive, in a single copy-then-delete transaction, and returns the number moved.

This sample demonstrates it against a real PostgreSQL: append a stream of several versions, archive a prefix, and prove the live table now holds only the un-archived tail.

Why this exists

The event store grows forever (article 005 says so plainly — “the event store only grows”). For most aggregates that is a non-problem: a stream is a few dozen events and storage is cheap. But some streams are pathological — a long-running account, a device that emits telemetry events for years, an aggregate that is written to thousands of times a day. Those streams turn the hot table into a place where 1% of the aggregates own 99% of the rows, and every append, every replay, and every autovacuum pays for that imbalance.

The naive fixes are all wrong. Deleting old events destroys history — the entire point of event sourcing — and breaks any future reprojection. “Just truncate the stream” loses the ability to audit or rebuild. Archiving threads the needle: the events stay, fully intact, in a table sized for cold access; the hot table is sized for the live workload. It is the storage analogue of what snapshots do for replay cost (article 006) — both bound the cost of a long stream, one for storage, one for read.

When to use this

  • A few streams dominate storage. When a small set of aggregates own most of the relay_events rows and that imbalance is measurably hurting the write path (index size, vacuum time, cache hit rate).
  • Streams already covered by a snapshot. Archiving is safe precisely when replay never needs the archived prefix — which is true when a snapshot at or after upToVersionInclusive exists, because the repository replays from the snapshot forward (article 006). Snapshot first, then archive the prefix the snapshot subsumes.
  • Retention / tiering policies. When compliance or cost says “events older than N must move to cold storage but still be retrievable,” the archive table is that cold tier.

When not to use this

  • Don’t archive hot or short streams. If a stream is still being actively appended to and replayed from its head, or if it is short, archiving buys you nothing and adds a moving part. Archive the long, cold tails — not everything.
  • Don’t archive a prefix that isn’t covered by a snapshot. If you archive versions 0–100 but the aggregate has no snapshot, the next GetByIdAsync replays from version 0, finds the prefix gone in the live table, and rebuilds the wrong state. Snapshots are the safety interlock — see Production considerations.
  • Don’t use it to erase data. Archived events still exist; this is relocation, not deletion. For GDPR “right to erasure” of personal data inside events you need true, irreversible removal — 035 — Crypto-Shredding (encrypt per-subject, then destroy the key) is the pattern for that, and it composes with archiving rather than replacing it.
  • Don’t treat the archive as a query surface. Like the live store, it answers “give me this stream,” not “give me all accounts over $1000” — that is still a projection’s job (article 007).

Concepts

The archive table. relay_events_archive mirrors relay_events column-for-column (global position, tx id, event id, stream name, aggregate id, aggregate type, version, event type, payload, metadata, occurred-at, tenant id), so an archived row is a faithful copy — same positions, same payloads. You map it with ApplyRelayEventArchive() in OnModelCreating, exactly as you map the live store with ApplyRelayEventStore().

upToVersionInclusive. ArchiveStreamAsync(aggregateId, upToVersionInclusive: N) moves every event of that aggregate whose per-stream version <= N. It is inclusive: passing 2 archives versions 0, 1, and 2. It returns the count actually moved, so you can assert or log how much was compacted.

Copy-then-delete, atomically. The archiver runs INSERT … SELECT into the archive (with ON CONFLICT (global_position) DO NOTHING, so a re-run is idempotent) and then DELETE from the live table, both in one transaction on its own connection. Either both happen or neither does — there is no window where an event has been deleted from the live table but not yet written to the archive.

What stays queryable. After archiving, IEventStore.GetEventsAsync(aggregateId) returns only the events still in the live table — the un-archived tail. The archived events are not gone; they are in relay_events_archive, retrievable for audit or a restore, but they no longer participate in the live append/replay path.

Architecture

flowchart LR
    App[append path / replay] -->|reads & writes| Live[(relay_events<br/>hot, bounded)]
    Arc[IEventArchiver<br/>ArchiveStreamAsync id, N] -->|1 INSERT…SELECT version ≤ N| Archive[(relay_events_archive<br/>cold, faithful copy)]
    Arc -->|2 DELETE version ≤ N| Live
    Snap[(relay_snapshots)] -.replay starts after snapshot,<br/>so archived prefix is never read.-> Live
    subgraph One transaction
        Arc
    end

The live table backs the live workload; the archive holds the cold tail. A snapshot at/after the archived version is what makes the move safe — replay starts after it, so the archived prefix is never needed.

Building it step by step

The full sample is in samples/034-event-archiving-and-compaction.

1. Map both tables onto your DbContext

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    base.OnModelCreating(modelBuilder);
    modelBuilder.ApplyRelayEventStore();     // relay_events (the hot, live log)
    modelBuilder.ApplyRelaySnapshots();      // relay_snapshots (the safety interlock; article 006)
    modelBuilder.ApplyRelayEventArchive();   // relay_events_archive (the compaction target)
}

2. Wire the event store, unit of work, and the archiver

// A factory is required: the archiver owns its own transaction, independent of any request scope.
services.AddDbContextFactory<ArchiveDbContext>(o => o.UseNpgsql(connectionString));
services.AddDbContext<ArchiveDbContext>((sp, o) => o.UseNpgsql(connectionString),
    contextLifetime: ServiceLifetime.Scoped, optionsLifetime: ServiceLifetime.Singleton);

services.AddRelayEventStoreEfCore<ArchiveDbContext>();      // IEventStore + serializer
services.AddRelayUnitOfWorkEfCore<ArchiveDbContext>();      // the transaction the append commits in
services.AddRelayEventArchivePostgres<ArchiveDbContext>();  // IEventArchiver (stream compaction)

EfCoreEventArchiver<TContext> is constructed from an IDbContextFactory<TContext> — registering only AddDbContext (and not AddDbContextFactory) leaves the archiver with nothing to resolve.

3. Archive a stream’s old events

var archiver = serviceProvider.GetRequiredService<IEventArchiver>();
int moved = await archiver.ArchiveStreamAsync(aggregateId, upToVersionInclusive: 2);
// versions 0,1,2 are now in relay_events_archive; the live table keeps only version 3 onward.

In production this call lives behind a scheduled job (article 018/019) or a leader-elected background worker (article 023): pick a stream that has a snapshot, archive the prefix the snapshot covers, repeat.

Complete source code

File Shows DB?
ArchiveDbContext.cs ApplyRelayEventStore() + ApplyRelaySnapshots() + ApplyRelayEventArchive() Yes
ArchiveFixture.cs Testcontainers Postgres + the production DI wiring (incl. the DbContext factory) Yes
ArchivingTests.cs append a stream, archive a prefix, assert the moved count and the live tail Yes

Running the example

dotnet test samples/034-event-archiving-and-compaction/Ledger.Archiving.Tests

This needs Docker — the fixture starts a real postgres:16 via Testcontainers, applies the schema with EnsureCreatedAsync(), and runs the archiver against it. That is deliberate: archiving is a move between two database tables, so the only honest demonstration is against a real store. In production you replace EnsureCreatedAsync() with EF migrations (or the baseline SQL in libraries/nuvora-nexus-relay/docs/migrations/).

Testing

The sample appends a stream of five versions, calls ArchiveStreamAsync(id, upToVersionInclusive: 2), and asserts both halves of the contract:

var moved = await archiver.ArchiveStreamAsync(aggregateId, upToVersionInclusive: 2);
moved.Should().Be(3);                                                  // 0,1,2 moved out

var remaining = await store.GetEventsAsync(aggregateId);
remaining.Select(e => e.Version).Should().Equal(3, 4);                 // only the live tail remains

Further cases prove the edges: archiving the whole stream empties the live table for that aggregate, and an inclusive bound of 0 moves exactly one event. These are integration tests against real PostgreSQL, not mocks, because the guarantee being verified — an atomic copy-then-delete across two tables — only exists at the database level. Mocking the store would test the mock.

Production considerations

  • Snapshot first; archive only what the snapshot covers. The archived prefix must never be needed by replay. Take (or confirm) a snapshot at version S, then archive upToVersionInclusive: S (or lower). Archiving past the snapshot, or with no snapshot at all, corrupts replay — the aggregate rebuilds from a stream with a hole. Treat “has a snapshot at ≥ N” as a precondition the job checks before it archives.
  • Archive cadence. Run it as a scheduled job (article 018/019) or leader-elected worker (article 023), not on the request path. A common policy: nightly, snapshot the longest streams, then archive the versions the new snapshot subsumes. Size the batch so the transaction stays short.
  • Restoring from the archive. The archive is a faithful copy (positions preserved), so a restore is the inverse move — INSERT … SELECT back into relay_events. Keep a runbook for it; you will want it the first time someone needs to reproject the full history of an archived stream.
  • Reprojection still works — through the archive. A from-scratch projection rebuild (article 008) that needs pre-archive history reads the archive, not just the live table. Make sure your rebuild tooling knows the archive exists before you rely on archiving heavily.
  • It bounds storage, not replay. Snapshots bound replay cost; archiving bounds live-table size. They are complementary — and archiving depends on snapshots for safety. Don’t reach for archiving until a stream’s storage footprint actually hurts.

Common mistakes

  • Archiving without a snapshot. The number-one footgun: the prefix is gone from the live table, no snapshot covers it, and the next load replays a stream with a hole. Always snapshot first.
  • Treating archiving as deletion / erasure. Archived events still exist. If the requirement is “make this personal data unrecoverable,” archiving does not satisfy it — use crypto-shredding (article 035).
  • Archiving hot or short streams. Pure overhead. Target the long, cold tails that actually dominate storage; leave everything else alone.
  • Forgetting ApplyRelayEventArchive() (or the DbContext factory). Without the table mapped, the copy fails; without AddDbContextFactory, AddRelayEventArchivePostgres has no factory to resolve.
  • Running it on the request path. Archiving is a bulk maintenance move. Put it behind a scheduler or a leader-elected worker, not in a command handler.

Tradeoffs

  • Bounded live table vs. an extra moving part. Archiving keeps the hot table small (faster appends, smaller indexes, cheaper vacuums) at the cost of a maintenance job, a second table, and a snapshot-before-archive discipline. Worth it only when a few long streams measurably dominate — premature archiving is just complexity.
  • Two-table history vs. one. History is still complete, but now it lives in two places, and any tool that walks full history (reprojection, audit export) must read both. A clean conceptual model (the archive is a faithful copy) keeps this manageable, but it is more surface area.
  • Relocation vs. erasure. Archiving deliberately preserves data — which is exactly why it cannot satisfy “delete this forever.” That is a feature for audit and a non-feature for GDPR erasure; pair it with crypto-shredding (article 035) when you need both bounded growth and true deletion.

Alternatives

  • Snapshots alone (article 006). Bound replay cost without moving any rows. If your problem is “long streams replay slowly” rather than “the hot table is too big,” snapshots are the whole answer — archiving adds nothing. Often you want snapshots and archiving for different reasons.
  • Partition / table inheritance at the database level. PostgreSQL can range-partition relay_events by position or time so old partitions are physically separate and cheap to detach. More operationally involved (and DB-specific) than the application-level archiver, but it keeps everything in one logical table. Reach for it at very large scale.
  • Cold-storage export (S3/Parquet) + delete. Stream old events to object storage and delete the rows. Cheapest storage, but the events leave the database entirely — reprojection and audit now need a separate pipeline. The archive table is the middle ground: cold, but still in the database and still SQL-queryable.
  • Crypto-shredding (article 035). Solves a different problem — irreversible erasure of personal data — by destroying encryption keys, not by moving rows. Complementary to archiving, not a substitute.

Lessons from production systems

  • The teams that got burned archived a prefix with no snapshot behind it and discovered the corruption only when an aggregate replayed to the wrong state weeks later. The fix is a hard precondition in the job: never archive past the newest snapshot. Make it impossible to call ArchiveStreamAsync otherwise.
  • Almost nobody needs archiving on day one — the event store grows slowly until one or two streams go pathological. The right time to add it is when storage metrics, not anxiety, say so. Until then, snapshots cover the replay-cost half of the problem.
  • The first restore is always harder than expected because no one wrote the inverse move down. The teams that sleep well kept a tested “restore from archive” runbook from the moment they shipped archiving.

Should you use this?

If you… Then…
have a few long, cold streams dominating relay_events yes — snapshot them, then archive the covered prefix
have only short streams, or storage isn’t hurting no — it’s pure overhead; revisit when metrics say so
need replay to be faster (not storage smaller) snapshots (article 006), not archiving
must irreversibly erase personal data crypto-shredding (article 035), not archiving
archive without a snapshot stop — replay will rebuild the wrong state

Next steps

  • 035 — Crypto-Shredding: the other half of “events live forever” — irreversible erasure of personal data by destroying keys, which composes with archiving.
  • 006 — Concurrency & Snapshots: snapshots are the interlock that makes archiving safe — replay starts after the snapshot, so the archived prefix is never read.
  • 008 — Projection Operations: rebuilding read models from full history — which, once you archive, means reading the archive too.