Articles · Event Sourcing ·028

Pessimistic Per-Aggregate Write Lock

Runnable sample on GitHub

Sample: samples/028-pessimistic-aggregate-locking · Subsystem: 2.13 · Prerequisite: 006 — Concurrency & Snapshots

Overview

Article 006 established Relay’s default for concurrent writes to an event-sourced aggregate: optimistic concurrency. Two writers both load version k, both try to append k+1, the unique index on (aggregate_id, version) lets exactly one through, and the loser gets a ConcurrencyConflictException and retries (reload, re-apply, re-append). This is free, lock-free, and the right answer almost always.

But “retry the loser” has a failure mode: a hot aggregate. When many writers hammer the same aggregate id at once — a flash-sale SKU everyone reserves, a single counter the whole fleet increments — the conflict rate climbs, retries stack on retries, and you burn CPU re-doing work that keeps colliding. For that narrow case Relay ships a pessimistic alternative: IAggregateWriteLock (implementation DistributedAggregateWriteLock). Acquire it inside the command before loading the aggregate, and concurrent writers to the same id serialize — the second writer waits for the first instead of racing it and retrying. Writers to every other aggregate are unaffected.

Why this exists

Optimistic concurrency detects conflicts; it does not prevent them. Under low contention that is ideal — no locks held, maximum throughput, a rare retry. Under high contention on one aggregate it inverts: the work done before the conflict (load the stream, replay, evaluate the invariant, attempt the append) is thrown away on every loser, and the losers immediately try again into the same storm. A bounded retry loop with backoff softens it, but past a certain contention level you are paying to discover the conflict over and over.

Pessimistic locking pays a different price up front. Take a lock keyed to this aggregate id, and the second writer blocks until the first commits — so it does its expensive work once, against fresh state, with no wasted attempt. You have traded “everyone tries, most fail and retry” for “they form an orderly queue.” For a genuinely hot aggregate that trade is a win; for everything else it is pure overhead, which is exactly why this is opt-in and per-aggregate, not a global mode.

When to use this

  • A genuinely hot aggregate. One aggregate id takes concurrent writes often enough that the optimistic retry rate is measurably hurting throughput or latency. You have measured this, not guessed it.
  • Conflict-prone, non-commutative commands. When the loser cannot simply re-apply blind intent (a reservation against limited stock, where the available quantity changed underneath it), serializing is cleaner than reload-and-retry.
  • A narrow, identified spot. You are locking one aggregate type’s hot instances, not wrapping your whole write path. Other aggregates keep running optimistic and lock-free.

When not to use this

  • As the default. Optimistic concurrency is the default for event sourcing for good reasons; do not reach for locks because conflicts are possible. Reach for them when conflicts are frequent and measured.
  • For cold or low-contention aggregates. If conflicts are rare, the lock is overhead that only reduces throughput (every writer now serializes even when they would not have collided).
  • To paper over a modelling smell. Constant contention on one aggregate usually means the boundary is too coarse — everyone writes a single god aggregate. Splitting it, or moving coordination into a saga (article 015), often beats both retrying and locking. Lock the hot spot you cannot redesign, not the one you have not redesigned yet.
  • As a transaction or an invariant. The lock serializes writers; it is not a substitute for the expected-version check (which stays on) or for a database transaction.

Concepts

AcquireAsync vs TryAcquireAsync. Two ways to take the per-aggregate lock:

Task       AcquireAsync(string aggregateType, Guid aggregateId, CancellationToken ct = default);
Task<bool> TryAcquireAsync(string aggregateType, Guid aggregateId, CancellationToken ct = default);
  • AcquireAsync blocks until the lock is held for the current transaction. This is the normal pessimistic path: the second writer waits for the first, then proceeds. Honour the CancellationToken so a slow lock does not pin a request forever.
  • TryAcquireAsync takes the lock without waiting and returns false if another writer holds it. Use it when “someone else is already writing this aggregate” is itself a useful answer — fail fast, shed load, or surface a 409 — rather than queueing behind them.

Taken on the unit-of-work transaction. DistributedAggregateWriteLock rides an IDistributedLock’s transaction-scoped flavour (AcquireForCurrentTransactionAsync / TryAcquireForCurrentTransactionAsync). The lock is bound to the command’s ambient unit-of-work transaction — you do not dispose a handle.

Released at commit/rollback. Because the lock is transaction-scoped, it is released automatically when the unit of work commits or rolls back. There is no “forgot to release” path: end the transaction (either way) and the lock is gone, on the same connection that held it. On the PostgreSQL backend this is a transaction-level advisory lock, freed by the server when the transaction ends — so even a crash mid-command releases it.

Per-aggregate-id namespacing. The resource key is relay:aggregate:{type}:{id}. Two consequences: writers to the same (type, id) contend (that is the point), and writers to a different id — or a different aggregate type with the same id — get a different key and never block each other. The lock is exactly as granular as one aggregate instance.

Architecture

sequenceDiagram
    participant A as Writer A (txn A)
    participant B as Writer B (txn B)
    participant L as IAggregateWriteLock<br/>(relay:aggregate:StockItem:{id})
    A->>L: AcquireAsync("StockItem", id)
    L-->>A: held (on txn A)
    B->>L: AcquireAsync("StockItem", id)
    Note over B,L: B waits — same aggregate id
    A->>A: load → reserve → append (no conflict)
    A->>L: COMMIT / ROLLBACK releases the lock
    L-->>B: held (on txn B)
    B->>B: load fresh state → reserve → append
    B->>L: COMMIT / ROLLBACK releases the lock
    Note over A,B: a writer to a DIFFERENT id uses a different key → never waits

Building it step by step

The sample is samples/028-pessimistic-aggregate-locking. It exercises the pure lock contract with an in-memory fake, so there is no database to run.

1. Acquire the lock inside the command (before loading)

Take the lock first, then load and write the aggregate. Because it is transaction-scoped, contending writers serialize on it and it releases when the unit of work ends:

public async Task Handle(ReserveStockCommand cmd, CancellationToken ct)
{
    // Pessimistic: serialize concurrent writers to THIS SKU before doing any work.
    await aggregateWriteLock.AcquireAsync(nameof(StockItem), cmd.SkuId, ct);

    var item = await repository.GetByIdAsync(cmd.SkuId, ct); // loaded under the lock
    item.Reserve(cmd.Quantity);
    await repository.SaveAsync(item, ct);                    // commit releases the lock
}

2. Or fail fast instead of waiting

When queueing is the wrong behaviour, try without waiting:

if (!await aggregateWriteLock.TryAcquireAsync(nameof(StockItem), cmd.SkuId, ct))
    throw new BusyAggregateException(cmd.SkuId); // someone else is writing this SKU right now

3. Wire it in production (DB-backed)

The lock needs a real distributed lock underneath. In production you register both:

// The Postgres advisory-lock backend (needs AddRelayEventStoreEfCore):
services.AddRelayDistributedLockPostgres<InventoryDbContext>();
// The per-aggregate write lock over it (in Persistence.EfCore):
services.AddRelayAggregateWriteLock();

The sample skips DI entirely and constructs new DistributedAggregateWriteLock(fakeLock) directly, which is all you need to prove the contract.

Complete source code

File Shows DB?
AggregateWriteLockTests.cs same-id writers serialize; different ids don’t block; key is namespaced per type+id No (fake lock)
FakeDistributedLock.cs in-memory IDistributedLock tracking held keys; EndTransaction = commit/rollback No
Inventory.Locking.Tests.csproj references Relay.Core only (the lock is pure) No

Running the example

dotnet test samples/028-pessimistic-aggregate-locking/Inventory.Locking.Tests

Needs only the .NET 10 SDK — no Docker, no PostgreSQL.

Testing

DistributedAggregateWriteLock is pure: it builds a stable resource key and delegates to an IDistributedLock. That makes the whole serialization contract testable with a fake, exactly as the distributed-lock primitives are (article 023’s FakeLock). The FakeDistributedLock tracks a set of currently-held resource keys to enforce real mutual exclusion, and exposes an EndTransaction to stand in for the commit/rollback that releases the lock:

// Writer A takes the SKU's lock; B's no-wait try is denied while A holds it.
(await sut.TryAcquireAsync("StockItem", sku)).Should().BeTrue();
(await sut.TryAcquireAsync("StockItem", sku)).Should().BeFalse();

// A blocking acquire WAITS rather than failing — until A's transaction releases the lock.
var writerB = sut.AcquireAsync("StockItem", sku);
(await Task.WhenAny(writerB, Task.Delay(200))).Should().NotBe(writerB);
fake.EndTransaction(ResourceKey(sku)); // A commits/rolls back
(await Task.WhenAny(writerB, Task.Delay(5000))).Should().Be(writerB);

The other tests confirm writers to different ids never wait, and that the resource key is namespaced by aggregate type and id (relay:aggregate:{type}:{id}). The genuine database behaviour — that a Postgres advisory lock is held and freed by the transaction — is the backend’s concern (article 023 covers it against real Postgres); here we prove the contract the command relies on, deterministically and fast.

Production considerations

  • Lock in a consistent order to avoid deadlocks. A command that touches two aggregates and locks them must always lock them in the same order across the codebase (e.g. by sorted id), or two commands can each hold one lock and wait for the other. The single-aggregate case (the common one) cannot deadlock with itself.
  • It costs throughput by design. Serializing writers caps concurrency on that aggregate at one. That is the point for a hot aggregate, but it means the lock is a bottleneck — keep the locked section short (load, decide, append, commit) so the queue drains quickly.
  • Only for genuinely hot aggregates. Applied broadly it makes your write path single-threaded per-aggregate everywhere, trading the (cheap, lock-free) optimistic default for held locks. Measure the conflict rate first; lock the few aggregates that warrant it.
  • Optimistic concurrency stays on. The expected-version check is intrinsic to appending and does not go away. The lock reduces conflicts to (ideally) zero on the hot aggregate; it does not replace the safety net.
  • Wire the real backend. In production register AddRelayAggregateWriteLock (in Nuvora.Nexus.Relay.Persistence.EfCore) over AddRelayDistributedLockPostgres<TContext> (which needs AddRelayEventStoreEfCore). The Postgres transaction-scoped advisory lock is what makes the serialization cluster-wide and crash-safe.

Common mistakes

  • Acquiring the lock after loading the aggregate. Then two writers can both load stale state before either locks, and you have reintroduced the race. Lock first, then load, so the loser loads fresh state after the winner commits.
  • Using it as the default. Wrapping every command in a per-aggregate lock serializes writers that would never have collided — pure overhead. Default to optimistic; lock the measured hot spot.
  • Expecting to release it manually. It is transaction-scoped; there is no handle to dispose. It releases when the unit of work commits or rolls back — do not try to “unlock” it.
  • Locking multiple aggregates in inconsistent orders. That is how you turn “no conflicts” into “occasional deadlock.” Order your acquisitions.
  • Treating the lock as the invariant. It serializes who writes when; the aggregate’s own command methods still enforce the business rule, and the version check still guards the append.

Tradeoffs

Benefits. A hot aggregate’s expensive write work runs once per writer against fresh state instead of being repeatedly attempted-and-discarded; non-commutative commands serialize cleanly instead of reload-retry churn; and because the lock is transaction-scoped, release is automatic and crash-safe (no leaked locks). It is per-aggregate, so the rest of the system stays lock-free.

Costs. Throughput on the locked aggregate is capped at one writer at a time (by design); you take on deadlock-ordering discipline for multi-aggregate commands; and the lock leans on the database’s availability and connection budget (it reuses the Postgres advisory-lock machinery). Applied beyond the genuinely hot aggregates it is a net loss versus the optimistic default.

Alternatives

  • Optimistic concurrency + bounded retry (the default — article 006). Detect conflicts via the expected-version check and retry the loser with backoff/jitter. Correct and lock-free; the right choice for everything but a measured hot aggregate. The pessimistic lock is the targeted exception, not a replacement.
  • Reshape the aggregate. Frequent contention is often a sign the aggregate is too coarse. Splitting it so writers touch different ids removes the contention entirely — better than locking or retrying when it is feasible.
  • A saga / process manager (article 015). When the real need is to coordinate a multi-step workflow rather than serialize a single write, a saga sequences the steps without holding a write lock across them.
  • A bare distributed lock (article 023). IDistributedLock is the general primitive; you could hand-roll per-aggregate locking on it. IAggregateWriteLock is the purpose-built wrapper — stable per-aggregate key, transaction-scoped — so you do not have to.

Lessons from production systems

  • The classic incident is a flash-sale SKU: thousands of reservations hit one aggregate, the optimistic retry rate spikes, and the service spends its CPU losing races. Pessimistic locking on that one aggregate type turns the storm into a queue, and reusing the Postgres advisory lock means it ships the same day.
  • Teams that lock everything to “be safe” quietly serialize their whole write path and wonder why throughput fell. The discipline is the opposite: optimistic by default, lock the few hot aggregates you can prove.
  • The most durable fix is frequently not a lock at all — it is a smaller aggregate boundary. Lock the hot spot you cannot redesign now; put the redesign on the list.

Should you use this?

If you… Then…
have a measured hot aggregate with high optimistic-conflict churn yes — pessimistic per-aggregate lock
have rare conflicts / cold aggregates no — optimistic concurrency (it’s free and lock-free)
can split the aggregate to remove the contention prefer that — no lock or retry needed
need a fail-fast “someone’s already writing this” signal use TryAcquireAsync
are coordinating a multi-step workflow, not one write reach for a saga (article 015)

Next steps