Articles · Production Operations ·023

Distributed Coordination: Locks, Leader Election, Partitioning

Runnable sample on GitHub

Sample: samples/023-distributed-coordination · Subsystem: 2.13 · Prerequisites: 022 — Resiliency, 005 — Event Sourcing Basics

Overview

You scale a service by running more copies of it — but now those copies can step on each other: two instances run the same nightly job, or both try to advance the same projection, or process the same key out of order. Relay provides a handful of primitives to coordinate a cluster without a new dependency (it reuses PostgreSQL):

  • IDistributedLock — mutual exclusion across processes, via PostgreSQL advisory locks.
  • ILeaderElector — run work on one node the simple way: a connection-bound lock that releases when the process dies (at-least-one leadership, no renewal).
  • IFencedLeaderElector — leader election backed by a renewable lease with a fencing token: proactive failover when a leader stalls (not only when it crashes), and a monotonic token so a stale leader can be fenced out.
  • INodeRegistry — heartbeat-based cluster membership: which instances are currently live.
  • IPartitioner — deterministically split a keyed workload across instances.

The partitioner is a pure, DB-free primitive; the lock and leader election use the database as arbiter.

Why this exists

The single-instance assumptions that work in development quietly break when you scale out. A cron-style job that was fine on one box now fires N times. A poller double-processes. A “just use a static flag/lock” reflex only coordinates threads in one process, not processes across machines. The correct tools — a distributed lock, leader election, consistent partitioning — are easy to get subtly wrong (locks that don’t release on crash, elections with no recovery, hashing that’s unstable across processes). Relay ships correct versions and, crucially, backs the lock with the database you already run, so coordination needs no ZooKeeper/etcd/Redis.

When to use this

  • Distributed lock: guard a critical section that must not run concurrently across instances (a schema migration, a one-shot bootstrap, a shared external resource).
  • Leader election: a singleton background worker (a poller, a rebalancer, a scheduler tick) that should run on one node and fail over if that node dies.
  • Partitioning: scale an ordered, keyed consumer — each instance owns a subset of partitions, and every key always maps to the same partition (preserving per-key order).

When not to use this

  • For per-request work. Most command handling is already safe under concurrency (optimistic concurrency on the event store, FOR UPDATE SKIP LOCKED on the outbox/scheduler). Don’t wrap normal request handling in a distributed lock — it serializes your whole service.
  • As a transaction. A distributed lock is not a substitute for a database transaction or optimistic concurrency; it’s coarse-grained mutual exclusion around side effects.
  • When the framework already coordinates it. The outbox processor, scheduler, and projection coordinator already use advisory locks internally. You rarely need to lock those paths yourself.

Concepts

PostgreSQL advisory locks. pg_advisory_lock(key) is a lock keyed by a 64-bit integer, enforced by the database across every connection. Relay namespaces the high 32 bits (so "LOCK" locks never collide with the projection coordinator’s "PROJ" locks) and hashes the resource name into the low 32 bits. Two flavours:

  • Session lock (TryAcquireAsync) — held by a dedicated connection, released on dispose or when the connection drops. This is what the connection-bound ILeaderElector uses: if the leader’s process dies, its connection drops and the lock releases server-side, so another node can take over. There’s no lease renewal on this path, so its elected work should be idempotent; when you need proactive failover and fencing, use IFencedLeaderElector (below).
  • Transaction lock (TryAcquireForCurrentTransactionAsync) — held until the ambient transaction commits/rolls back. Good for “do this once inside this unit of work.”

Leader election. DistributedLockLeaderElector.RunAsync(role, onElected, ct) contends for leader:<role>; while it holds the lock it runs onElected, and when it doesn’t it backs off and re-contends. Leadership ends when the work returns, the process dies, or cancellation is requested.

Fenced leader election (renewable lease). Where connection-drop failover isn’t enough — an alive-but-stalled leader still holds its connection-bound lock — IFencedLeaderElector (LeaseLeaderElector) elects through an ILeaseStore it renews on an interval. If the leader stalls and misses a renewal, the lease lapses after LeaseTtl (default 30s) and another node takes over proactively. Each acquisition carries a monotonically-increasing fencing token, handed to your work — RunAsync(role, (fencingToken, ct) => …) — so a downstream store can reject writes stamped with a stale token, fencing the old leader out. INodeRegistry complements it with heartbeat membership (GetLiveNodesAsync), so a coordinator can see which instances are currently alive.

Partitioning. DefaultPartitioner.GetPartition(key, partitionCount) maps a key to [0, partitionCount) with FNV-1a hashing over the UTF-8 bytes. The critical property is stability: the same key yields the same partition in every process and after every restart — unlike string.GetHashCode(), which is randomized per process and would scatter a key’s events across partitions, destroying per-key ordering. For workloads where you will resize the partition set, ConsistentHashPartitioner (opt in via PartitionOptions.UseConsistentHashing) places partitions on a hash ring so changing the count remaps only a fraction of keys instead of nearly all of them — trading a little load skew for far less rebalancing churn.

Architecture

flowchart TD
    subgraph Cluster
        N1[Instance 1] -->|pg_advisory_lock 'leader:reports'| PG[(PostgreSQL)]
        N2[Instance 2] -->|contend| PG
        N3[Instance 3] -->|contend| PG
    end
    PG -->|granted| Leader[Instance 1 = leader<br/>runs the singleton work]
    PG -->|denied → back off + retry| N2
    N1 -.process dies → connection drops → lock releases.-> Failover[Instance 2 takes over]

    subgraph Partitioning
        Key[key 'order-42'] -->|FNV-1a mod N| P[partition 3]
        P --> Owner[owned by whichever instance holds partition 3]
    end

Building it step by step

1. Partition a key (pure)

var partitioner = new DefaultPartitioner();
partitioner.GetPartition("order-42", partitionCount: 8);   // always the same number, in [0,8)

2. Elect a leader (lock-backed)

var elector = new DistributedLockLeaderElector(distributedLock);   // or services.AddRelayLeaderElection()
await elector.RunAsync("reports", async ct =>
{
    // runs on exactly one node; re-elected elsewhere if this node dies
    while (!ct.IsCancellationRequested) { await GenerateReportsAsync(ct); await Task.Delay(TimeSpan.FromMinutes(5), ct); }
}, stoppingToken);

2b. Elect a leader with a renewable, fenced lease

// DI: services.AddRelayLeaseStorePostgres<MyDbContext>();   // needs ApplyRelayLeases() in OnModelCreating
//     services.AddRelayFencedLeaderElection(o => o.LeaseTtl = TimeSpan.FromSeconds(30));
await fencedElector.RunAsync("reports", async (fencingToken, ct) =>
{
    // renewed while we run; if this node stalls past the TTL, another takes over with a higher token.
    // Stamp downstream writes with fencingToken so a stale leader's writes can be rejected.
    while (!ct.IsCancellationRequested) { await GenerateReportsAsync(fencingToken, ct); await Task.Delay(TimeSpan.FromMinutes(5), ct); }
}, stoppingToken);

3. Take a distributed lock (Postgres)

// DI: services.AddRelayDistributedLockPostgres<MyDbContext>();  (needs AddRelayEventStoreEfCore)
await using var handle = await distributedLock.TryAcquireAsync("nightly-export");
if (handle is not null)
{
    // we hold it cluster-wide; another instance gets null and skips
}

Complete source code

File Shows DB?
PartitioningTests.cs stable, in-range, distributed hashing No
LeaderElectionTests.cs elected work runs; retries until it wins No (FakeLock)
FakeLock.cs scripted in-memory IDistributedLock No
DistributedLockTests.cs exclusivity, release, distinct resources Yes
CoordinationFixture.cs Testcontainers Postgres + lock wiring Yes

Running the example

dotnet test samples/023-distributed-coordination/Coordination.Sample.Tests

The partitioner and leader-election tests need only the .NET 10 SDK; the distributed-lock tests spin up PostgreSQL via Testcontainers (Docker required).

Testing

Note the split: the pure parts (partitioner) and the lock-shaped part (leader election, via a scripted FakeLock) are tested with zero infrastructure, while only the genuinely database-backed lock uses Postgres. This is the pattern throughout Relay — push logic into pure, fast-to-test primitives and keep the irreducible database interaction in one focused integration test. The FakeLock (grant/deny script) lets you assert the elector retries without timing or containers.

Production considerations

  • Leader work should be idempotent (connection-bound election). The ILeaderElector path has no lease renewal: a network partition can briefly produce two leaders (the old one hasn’t noticed its connection dropped). Design the work so a rare double-run is harmless — or use IFencedLeaderElector, whose renewable lease + fencing token give proactive failover and let downstream writes fence a stale leader out.
  • Keep the elected work loop cancellation-aware. It runs until cancelled; honour stoppingToken so shutdown releases leadership promptly instead of waiting for a lock timeout.
  • Pick partition count up front (modulo hashing). With the default FNV-1a modulo partitioner, changing PartitionCount reshuffles key→partition ownership (and thus per-key ordering across the change). Size it for your maximum fan-out and leave it — or use ConsistentHashPartitioner (UseConsistentHashing), which remaps only a fraction of keys when the count changes.
  • Don’t hold a session lock across long idle periods unless that’s the intent — it pins a connection. For “run once” use the transaction lock inside the unit of work instead.
  • Advisory locks are advisory. They coordinate cooperating code that all takes the lock; they don’t stop a rogue connection from ignoring them. They’re for your own cluster’s coordination, not security.

Common mistakes

  • Using string.GetHashCode() to partition — it’s randomized per process, so the same key lands on different partitions in different instances, shredding per-key order. Use IPartitioner.
  • A leader with no idempotency, assuming exactly-one — assume at-least-one and design accordingly.
  • Wrapping normal request handling in a distributed lock — you’ve just made your service single-threaded cluster-wide. Use the per-row concurrency the event store/outbox already provide.
  • Forgetting the lock needs a dedicated connection — the session lock uses a DbContextFactory; wire one (the sample does) or TryAcquireAsync has nowhere to hold the lock.

Tradeoffs

  • Reusing Postgres vs. a dedicated coordinator. Advisory locks mean no new infrastructure (no ZooKeeper/etcd/Redis) — a huge operational win — at the cost of tying coordination to your database’s availability and connection budget. For most Relay services that’s exactly the right trade.
  • Connection-bound vs. lease-based election. The connection-bound ILeaderElector is simple and crash-safe (the server drops the lock when the process dies) but gives at-least-one leadership under partitions and can’t detect an alive-but-stalled leader. IFencedLeaderElector adds a renewable lease + fencing token for proactive failover and stale-leader fencing, at the cost of a relay_leases table and renewal traffic. Default to the simple one with idempotent work; reach for the fenced one when a double-run is genuinely unsafe.
  • Static partition count vs. elasticity. Fixed partitions with modulo hashing keep ordering stable but cap parallelism and reshuffle nearly everything if you resize; ConsistentHashPartitioner (UseConsistentHashing) makes resizing cheap — only a fraction of keys move. Full dynamic repartitioning with live ownership hand-off is far more complex; pick a generous count, and consistent hashing if you expect to grow.

Alternatives

  • ZooKeeper / etcd / Consul — purpose-built coordination with leases, watches, and stronger consistency. Worth it at large scale or when you already run them; overkill if PostgreSQL suffices.
  • Redis (Redlock) — popular distributed locking, but Redlock’s safety is famously debated for correctness-critical locks. Fine for best-effort; advisory locks on your transactional database are a safer default for “must not double-run.”
  • Kubernetes leases (coordination.k8s.io) — great for electing a leader pod for a singleton workload at the platform level; complementary to in-process election for finer-grained roles.

Lessons from production systems

  • The classic scale-out incident is “the nightly job ran five times.” Leader election (or a distributed lock) around singleton work is the fix, and reusing Postgres means you can ship it the same afternoon.
  • Per-process hashing bugs are maddening: everything works on one instance and corrupts ordering the moment you scale to two. A stable partitioner with a “same key → same partition” test prevents it.
  • “Exactly once” is a myth across partitions; the teams that sleep well built idempotent leaders and consumers and stopped chasing it.

Should you use this?

If you… Then…
run >1 instance with a singleton job yes — leader election
must serialize a side effect cluster-wide yes — distributed lock
scale an ordered keyed consumer yes — partitioner (and own partitions per instance)
already run etcd/ZooKeeper at scale consider them; advisory locks are the no-new-dependency default

Next steps