Articles · Production Operations ·030

Fenced Leadership & Cluster Membership

Runnable sample on GitHub

Sample: samples/030-fenced-leadership-and-membership · Subsystem: 2.13 · Prerequisites: 023 — Distributed Coordination, 022 — Resiliency

Overview

Article 023 elects a leader the simple way: a connection-bound ILeaderElector holds a PostgreSQL advisory lock for as long as its connection lives, and the database drops that lock when the process dies. That is crash-safe and dependency-free, but it has a blind spot — it only fails over when a leader dies, not when a leader is alive but stalled (paused in a GC, stuck on a blocked I/O, frozen by a long pause). The lock is still held, so no one else can take over, yet no work is happening.

This article covers the renewable-lease companion: fenced leadership plus cluster membership, both in Nuvora.Nexus.Relay.Core.Coordination.

  • IFencedLeaderElector (LeaseLeaderElector) — leader election backed by a renewable lease that must be re-affirmed before its TTL elapses. A stalled leader misses a renewal, the lease lapses, and another node takes over proactively. Each takeover carries a monotonically-increasing fencing token the work is handed, so a delayed old leader can be fenced out by downstream writers.
  • ILeaseStore (InMemoryLeaseStore) — the renewable lease the elector sits on: acquire-or-renew keyed by role, with a TTL and a fencing token.
  • INodeRegistry (InMemoryNodeRegistry) — heartbeat-based cluster membership: which instances are currently live, for observability and smarter rebalancing.

Where 023’s lock and election use PostgreSQL as the arbiter, this sample uses the in-memory implementations and a controllable clock — no database, no Docker — so the contracts are visible and the time-dependent behaviour (expiry, takeover, liveness) is exact.

Why this exists

The connection-bound election in 023 answers “the leader’s process died” perfectly: the server drops the lock, someone else wins it. But the worse failure in practice is the leader that is technically up and not making progress — a 40-second stop-the-world GC, a thread wedged on a socket that will eventually time out, a container the orchestrator hasn’t killed yet. Its connection is still open, so its lock is still held, so no failover happens. The singleton job just… stops, silently, until someone pages.

A renewable lease turns “still alive” into “still working”: leadership is only valid while the leader keeps proving it by renewing. Miss the renewals and you lose the lease — proactive failover, not death-triggered failover.

That introduces a new hazard, though: the split-brain window. The old leader could resume mid-write just as the new leader takes over, and now two leaders briefly believe they’re in charge. The fencing token closes it. Each ownership change bumps a monotonic counter; the work is handed its token and stamps it on downstream writes; the downstream store remembers the highest token it has seen and rejects anything lower. The resumed old leader’s writes carry a stale token and are refused — it is fenced out. (This is the pattern from Martin Kleppmann’s “How to do distributed locking.”)

Finally, once you have multiple instances coordinating through leases, you usually want to see them — which is what INodeRegistry provides: a cheap heartbeat membership view without a peer-to-peer gossip protocol.

When to use this

  • A singleton background worker where a double-run is genuinely unsafe and you can’t make it idempotent — use fenced election so a stalled leader is taken over and its late writes are fenced.
  • Work that writes to a fencing-aware store (one that can compare and reject by token) — the fencing token only protects you if something downstream actually checks it.
  • Cluster membership/observability — INodeRegistry to answer “which instances are live right now” for dashboards or rebalancing decisions.

When not to use this

  • When idempotent work + connection-bound election is enough. If a rare double-run is harmless, the simpler ILeaderElector from 023 has no renewal traffic and no lease table. Default to it.
  • For per-request work. Same as 023 — normal command handling is already safe under concurrency (optimistic concurrency, FOR UPDATE SKIP LOCKED). Don’t put a lease around it.
  • As a transaction or a general mutex. A lease is coarse-grained single-active-owner coordination, not a substitute for a database transaction.
  • With the in-memory store across processes. InMemoryLeaseStore / InMemoryNodeRegistry are single-process (great for tests and single-node use). Multi-node leadership needs a durable, shared store (e.g. the PostgreSQL-backed one).

Concepts

Lease + TTL + renewal. ILeaseStore.TryAcquireOrRenewAsync(role, ownerId, ttl) is one call that does three things depending on state: it acquires if the lease is free or expired, renews (extends the expiry) if ownerId already holds it and it’s still live, or returns LeaseResult.NotAcquired if a different, non-expired owner holds it. The holder must call it again before ttl elapses; if it stops (stalls or crashes), the lease lapses and is up for grabs. The clock is injected, so “after the TTL it lapses” is a deterministic test, not a sleep.

Fencing-token monotonicity. A LeaseResult carries a FencingToken (a long). The rule the store enforces: the token increases on every change of ownership and never on a same-holder renewal. So the first holder gets 1; renewing keeps it 1; a takeover after expiry gets 2, the next takeover 3, and so on — strictly increasing, monotone. Downstream writers track the highest token they’ve accepted and reject any write stamped lower. A stalled-then-resumed old leader still carries its old (lower) token, so its late writes are rejected — fenced out.

Node-registry heartbeats. INodeRegistry.HeartbeatAsync(nodeId) records “this node is alive as of now”; GetLiveNodesAsync() returns the nodes whose most recent heartbeat is within the liveness window. A node that stops heartbeating drops out once its last heartbeat is older than the TTL. This is membership-by-liveness — no gossip, no consensus — backed by the same injectable clock so the lapse is testable.

Architecture

flowchart TD
    subgraph Cluster
        A[Instance A] -->|TryAcquireOrRenew 'reports'| LS[(Lease store<br/>role → owner, token, expiresAt)]
        B[Instance B] -->|contend / renew| LS
    end
    LS -->|Acquired token=1| Lead[Instance A = leader<br/>runs work fencingToken=1<br/>renews every RenewInterval]
    LS -->|NotAcquired → back off + retry| B
    A -.stalls: misses renewals → lease lapses after LeaseTtl.-> Over[Instance B takes over<br/>token=2 strictly higher]
    Over -->|writes stamped token=2| DS[Downstream store<br/>tracks highest token seen]
    A -.resumes, writes token=1.-> DS
    DS -->|token 1 < 2 → reject| Fenced[stale leader fenced out]

    subgraph Membership
        A -->|HeartbeatAsync| NR[(Node registry)]
        B -->|HeartbeatAsync| NR
        NR -->|GetLiveNodesAsync| View[live nodes within the TTL]
    end

Building it step by step

The sample is a set of focused, pure unit tests over the in-memory implementations — the clearest way to see each contract.

1. A renewable lease (deterministic time)

var clock = new ManualTimeProvider();          // controllable; no wall-clock sleeps
var store = new InMemoryLeaseStore(clock);

var a = await store.TryAcquireOrRenewAsync("reports", "node-a", TimeSpan.FromSeconds(30));
a.Acquired.Should().BeTrue();
a.FencingToken.Should().Be(1);                 // first holder → token 1

// node-a stalls and never renews; advance past the TTL and node-b takes over.
clock.Advance(TimeSpan.FromSeconds(31));
var b = await store.TryAcquireOrRenewAsync("reports", "node-b", TimeSpan.FromSeconds(30));
b.FencingToken.Should().BeGreaterThan(a.FencingToken);   // 2 > 1 → fences a resumed node-a out

TryAcquireOrRenewAsync returns a LeaseResult(bool Acquired, long FencingToken, DateTimeOffset ExpiresAt); a same-holder renewal keeps the token, a takeover after expiry bumps it.

2. Fenced leader election

var elector = new LeaseLeaderElector(store, new LeaseLeaderOptions
{
    OwnerId = "node-a",
    LeaseTtl = TimeSpan.FromSeconds(30),
    RenewInterval = TimeSpan.FromSeconds(10),  // renew well inside the TTL
});

await elector.RunAsync("reports", async (fencingToken, leaderCt) =>
{
    // runs on one node, renewed while it runs; if this node stalls past the TTL another takes over with a
    // higher token. Stamp downstream writes with fencingToken so a stale leader's writes are rejected.
    while (!leaderCt.IsCancellationRequested)
    {
        await GenerateReportsAsync(fencingToken, leaderCt);
        await Task.Delay(TimeSpan.FromMinutes(5), leaderCt);
    }
}, stoppingToken);

A second LeaseLeaderElector contending for the same role over the same store keeps losing while the leader renews — it does not run concurrently.

3. Cluster membership by heartbeat

var registry = new InMemoryNodeRegistry(livenessTtl: TimeSpan.FromSeconds(30), clock);

await registry.HeartbeatAsync("node-a");
(await registry.GetLiveNodesAsync()).Should().Contain("node-a");

clock.Advance(TimeSpan.FromSeconds(31));        // node-a stops heartbeating
(await registry.GetLiveNodesAsync()).Should().BeEmpty();   // dropped once older than the TTL

Complete source code

File Shows DB?
LeaseStoreTests.cs acquire, renew (same token), contention, takeover with a higher fencing token No
FencedLeaderElectionTests.cs elected work runs with a token; a second instance does not run concurrently No
NodeRegistryTests.cs heartbeat liveness, lapse after the TTL, refresh, sorted live set No
ManualTimeProvider.cs controllable clock for deterministic expiry/liveness No

Running the example

dotnet test samples/030-fenced-leadership-and-membership/Coordination.Fenced.Tests

Needs only the .NET 10 SDK — there is no database or broker. (Compare 023, whose distributed-lock tests spin up PostgreSQL via Testcontainers; the fenced primitives here are exercised entirely in-memory.)

Testing

Everything is pure and deterministic. InMemoryLeaseStore and InMemoryNodeRegistry take an optional TimeProvider; the tests inject a ManualTimeProvider and call Advance(...) instead of sleeping, so a “after 30s the lease lapses and node-b takes over with token 2” assertion runs in microseconds and never flakes — the same testability principle as the resiliency primitives in 022: never bake DateTimeOffset.UtcNow into time-dependent code; take a TimeProvider. The fenced-elector tests do run the renew/retry loop on the real clock (it’s a background loop), but with millisecond intervals and bounded waits, so a regression fails fast rather than hanging.

Production considerations

  • Size LeaseTtl against RenewInterval with margin. Renew well inside the TTL (the defaults are a 30s TTL renewed every 10s — three chances to renew before lapse). Too tight and a brief hiccup costs you leadership unnecessarily; too loose and a real stall takes too long to fail over. RenewInterval should be a small fraction of LeaseTtl.
  • Account for clock skew. TTL/expiry are compared against a clock. With a shared durable store, the store’s clock is the arbiter, which avoids per-node skew; if expiry is computed per node, skew can expire a lease early or late. Keep TTLs comfortably larger than your worst expected skew.
  • Idempotent vs. fenced — pick deliberately. Fencing only protects writes that pass through something that checks the token. If your downstream can’t compare-and-reject by token, you don’t actually have fencing — you have a lease, and your work still needs to be idempotent (the at-least-one reality of 023 applies in the brief overlap). Use the fencing token and keep the work safe to repeat.
  • Heartbeat membership is eventually-consistent. GetLiveNodesAsync can briefly list a node that just died (until its TTL) or omit one that just started (until its first heartbeat). Treat it as a hint for rebalancing/observability, not as a strongly-consistent roster.
  • The in-memory store is single-process. It’s for tests and single-node use. Multi-node leadership needs the durable, shared store; the in-memory one will happily let every process think it’s the leader.

Common mistakes

  • RenewInterval too close to LeaseTtl. One missed renewal then drops leadership. Leave headroom.
  • Trusting the fencing token without enforcing it. Handing the work a token and then never checking it downstream is decorative. The protection lives in the downstream compare-and-reject.
  • Assuming the lease makes the work exactly-once. Failover has an overlap window; design for at-least-one and let the fencing token reject the loser’s late writes.
  • Using InMemoryLeaseStore in a multi-node deployment. Each process has its own dictionary, so each one elects itself. Use a shared durable store across processes.
  • Forgetting to honour the leadership token. RunAsync cancels the token you’re handed when the lease is lost; long-running work must observe it (pass it to Task.Delay, loop on IsCancellationRequested) so a fenced leader stops promptly.

Tradeoffs

  • Renewable lease vs. connection-bound lock. The connection-bound ILeaderElector (023) is the simplest correct option — crash-safe, no renewal traffic, no lease table — but blind to a stalled leader and only at-least-one under partitions. The renewable lease adds proactive failover and fencing at the cost of renewal traffic, a lease table, and TTL tuning. Default to the simple one with idempotent work; reach for the lease when a double-run is genuinely unsafe.
  • TTL responsiveness vs. stability. A short TTL fails over fast but is twitchy under transient hiccups; a long TTL is stable but slow to recover. There’s no free lunch — pick for your tolerance.
  • Fencing strength vs. downstream complexity. Fencing tokens are airtight if every protected write path checks them — which means threading a token through your storage layer. That’s real work; weigh it against simply making the work idempotent.
  • Heartbeat membership vs. real consensus. Heartbeat liveness is cheap and dependency-free but only eventually consistent and can’t decide split-brain. A consensus system (Raft/ZooKeeper) gives a stronger roster at far higher operational cost.

Alternatives

  • etcd / ZooKeeper / Consul leases — purpose-built lease + watch primitives with strong consistency and built-in fencing-style revisions (etcd’s mod_revision, ZooKeeper’s zxid). Worth it at large scale or when you already run them; overkill if your database can hold the lease.
  • Kubernetes leases (coordination.k8s.io/Lease) — exactly this pattern at the platform level: renewable, TTL’d, used by controllers to elect a leader pod. Great for electing a pod; complementary to in-process leases for finer-grained roles within a process.
  • Redis (Redlock) leases — popular, but Redlock’s correctness for fencing is famously debated; Kleppmann’s critique is the canonical read. Fine for best-effort; prefer a lease on your transactional store (or a real coordinator) for “must not double-write.”

Lessons from production systems

  • The incident that motivates fencing is the stalled leader, not the dead one: the box is up, the lock is held, and the singleton job has silently stopped for minutes. A renewable lease turns that into an automatic, prompt failover.
  • The split-brain write is real and subtle: the old leader resumes from a long GC pause after the new one took over and writes stale data. Fencing tokens — and a downstream that rejects lower tokens — are the well-understood fix; teams that skip the downstream check think they have fencing and don’t.
  • “Exactly once” remains a myth across a failover boundary. The teams that sleep well combined a fencing token with idempotent work and stopped chasing it — exactly the lesson from 023, one layer up.

Should you use this?

If you… Then…
run a singleton job where a double-run is harmless no — use connection-bound election (023) + idempotent work
run a singleton job where a stalled leader must be taken over yes — fenced election (renewable lease)
write to a store that can reject by token yes — stamp the fencing token and enforce it downstream
just need “which instances are live?” yes — INodeRegistry heartbeat membership
already run etcd/ZooKeeper at scale consider their leases; the DB-backed lease is the no-new-dependency default

Next steps