Articles · Production Operations ·022

Resiliency: Retries, Circuit Breakers, Rate & Concurrency Limits

Runnable sample on GitHub

Sample: samples/022-resiliency · Subsystem: 2.13 (resiliency) · Prerequisites: 014 — Reliable Messaging

Overview

Distributed systems fail in partial, transient ways: a database blips, a downstream API times out, a queue backs up. Relay’s reliability machinery — the outbox, the scheduler, the inbox, the projection host — all lean on a set of small, pure resiliency primitives that live in Nuvora.Nexus.Relay.Core.Resiliency:

  • IRetryPolicy / ConfigurableRetryPolicy — how long to wait before attempt n.
  • ICircuitBreaker / CircuitBreaker — stop calling a dependency that is clearly down.
  • IRateLimiter / TokenBucketRateLimiter — cap the rate of an operation.
  • IConcurrencyLimiter / ConcurrencyLimiter — cap the number in flight at once.
  • ITimeoutPolicy / TimeoutPolicy — bound how long one operation may run, cancelling it and throwing TimeoutRejectedException if it overruns (a Polly-style standalone timeout).
  • IKeyedConcurrencyLimiter / KeyedConcurrencyLimiter — cap the number in flight per key (e.g. per tenant or customer), so one hot key can’t starve the whole bulkhead.

They are deliberately tiny and dependency-free, so you can use them on your own code paths exactly the way the framework uses them on its.

Why this exists

Every team eventually writes a for loop with Thread.Sleep to retry something, then discovers it hammers a struggling dependency into the ground (no backoff), or retries a non-transient error forever, or stampedes when a thousand callers retry on the same schedule. The same team later bolts on an ad-hoc “is the service down?” flag. These are well-understood patterns — exponential backoff with jitter, the circuit breaker, the token bucket — and getting them subtly wrong is easy. Relay ships correct, tested implementations so its own durable components behave well under stress, and exposes them so you do not reinvent them.

When to use this

  • Calling a flaky external dependency (payment gateway, search index, third-party API).
  • Protecting a scarce resource (a connection pool, a rate-limited upstream) from overload.
  • Anywhere you would otherwise write retry/backoff logic by hand.

When not to use this

  • Inside a command handler that already runs in the transactional pipeline. Don’t retry a failing command in place — let it fail, and let the outbox/scheduler/inbox redeliver it. Those already use these primitives with durable state. In-process retries lose their budget on restart.
  • For business-level “try again later” semantics (e.g. “re-attempt this order in 24h”). That’s a scheduled command (article 018), not a retry policy.
  • As a substitute for backpressure at the transport. The messaging layer has dedicated consume filters (rate-limiter, circuit-breaker, concurrency-limiter) wired into the pipeline — prefer those for broker consumers.

Concepts

Primitive Question it answers Key knobs
Retry policy “How long until the next attempt?” strategy (immediate/interval/exponential/incremental), base, max, increment, jitter
Circuit breaker “Is this dependency down — should I even try?” failure threshold, failure ratio, sampling window, break duration
Rate limiter “Am I going too fast?” permits per window, window, burst capacity
Concurrency limiter “Are too many already running?” max concurrency
Timeout policy “Has this taken too long — abandon it?” timeout duration
Keyed concurrency limiter “Are too many already running for this key?” max concurrency per key

Backoff strategies. Exponential (base · 2^(n-1), capped) is the default and the right choice for most transient faults. Jitter spreads retries so callers that failed together don’t retry together (the thundering herd). The schedule is deterministic without jitter, which is why the framework can precompute it into a Postgres array for SQL-side outbox backoff.

Circuit states. Closed (calls flow), Open (calls rejected immediately — fail fast), HalfOpen (after the break duration, a single trial call is allowed; success closes, failure re-opens). This turns a slow cascade of timeouts into a fast, cheap rejection while the dependency recovers.

Token bucket. A bucket holds up to burst tokens and refills at permits/window. Each call takes a token; an empty bucket means “rejected, try later.” It admits bursts but bounds the sustained rate.

Architecture

flowchart LR
    Caller -->|1. may I call?| CB{Circuit breaker}
    CB -->|Open: fail fast| Reject1[reject immediately]
    CB -->|Closed/HalfOpen| RL{Rate limiter}
    RL -->|bucket empty| Reject2[reject: too fast]
    RL -->|token taken| CL{Concurrency limiter}
    CL -->|no permit| Reject3[reject: too many in flight]
    CL -->|lease acquired| Work[call dependency]
    Work -->|success| RecordOk[breaker.RecordSuccess]
    Work -->|failure| RecordFail[breaker.RecordFailure + retry policy decides delay]

The primitives compose but are independent — use one, or layer several. The retry policy decides the delay after a failure; the circuit breaker decides whether to attempt at all.

Building it step by step

The sample is a set of focused unit tests — the clearest way to see each primitive’s contract.

1. A retry curve

var policy = new ConfigurableRetryPolicy(new RetryPolicyOptions
{
    Strategy = RetryStrategy.Exponential,
    BaseDelaySeconds = 2,
    MaxDelaySeconds = 300,
});

policy.BuildDelaySchedule(4);          // [2, 4, 8, 16] seconds
policy.Evaluate(6, maxAttempts: 5, RetryContext.ForAttempt(6)); // RetryDecision.Stop

Evaluate returns a RetryDecision(ShouldRetry, Delay); BuildDelaySchedule materialises the jitter-free curve. Add JitterFraction = 0.1 to spread retries ±10%.

2. A circuit breaker (deterministic time)

var breaker = new CircuitBreaker(
    new CircuitBreakerOptions { FailureThreshold = 3, FailureRatio = 0, BreakDuration = TimeSpan.FromSeconds(15) },
    timeProvider);                     // inject a clock so tests don't sleep

breaker.RecordFailure(); breaker.RecordFailure(); breaker.RecordFailure();
breaker.State;                         // Open
breaker.TryEnter();                    // false → fail fast
// ...later...
clock.Advance(TimeSpan.FromSeconds(15));
breaker.TryEnter();                    // true → half-open trial
breaker.RecordSuccess();               // → Closed

3. Rate and concurrency limits

var rate = new TokenBucketRateLimiter(new RateLimiterOptions { PermitsPerWindow = 10, Window = TimeSpan.FromSeconds(1) }, clock);
rate.TryAcquire();                     // true until the bucket empties, then false until it refills

var gate = new ConcurrencyLimiter(maxConcurrency: 1);
using var lease = await gate.TryAcquireAsync();  // null when full; dispose frees the permit

Complete source code

File Shows
RetryPolicyTests.cs exponential curve, cap, attempt budget
CircuitBreakerTests.cs trip on threshold, half-open recovery
RateLimiterTests.cs burst then refill
ConcurrencyLimiterTests.cs lease cap, idempotent release
ManualTimeProvider.cs controllable clock

Running the example

dotnet test samples/022-resiliency/Resiliency.Sample.Tests

No database or broker — these are pure CPU-and-memory primitives.

Testing

Every primitive is deterministic. The circuit breaker and rate limiter take an optional TimeProvider; the tests inject a ManualTimeProvider and call Advance(...) rather than sleeping, so a “after 15s the breaker half-opens” assertion runs in microseconds and never flakes. This is the single most important testability decision in the design — never bake DateTimeOffset.UtcNow into a time-dependent primitive; take a TimeProvider.

Production considerations

  • Always cap the backoff. Unbounded exponential growth means an event that keeps failing waits hours, then days. MaxDelaySeconds (default 300) bounds it.
  • Add jitter for shared schedules. If many workers retry on the same curve, jitter prevents a synchronized stampede when the dependency recovers.
  • Size the circuit breaker’s window to the traffic. FailureRatio + MinimumThroughput avoid tripping on one failure during a quiet period; FailureThreshold is the absolute-count trip.
  • Leases must be disposed. Wrap ConcurrencyLimiter/lock acquisitions in using so a permit is always returned, even on exception. Release is idempotent, so double-dispose is safe.
  • Prefer durable retries for messages. These in-memory primitives lose their state on restart; the outbox/scheduler persist retry counts and backoff in Postgres. Use the primitives for synchronous calls, the durable components for messages.

Common mistakes

  • Retrying non-transient errors. A ValidationException or a 404 will never succeed on retry — retry only transient faults (timeouts, 503s, deadlocks). Wasted retries amplify load.
  • Retrying inside a transaction. Holding a DB transaction open across backoff delays exhausts the connection pool. Fail the unit of work and let a durable component redeliver.
  • Sleeping in tests. Coupling a test to real time makes it slow and flaky. Inject TimeProvider.
  • A circuit breaker with no half-open trial. Without the trial state it never recovers; it must allow a probe after the break duration.

Tradeoffs

  • Fail fast vs. availability. An open circuit rejects calls that might have succeeded — you trade a few false rejections for not drowning in timeouts. Usually worth it; tune the break duration.
  • In-memory vs. durable. These primitives are fast and simple but per-process and non-persistent. Cluster-wide rate limiting or restart-proof retry budgets need shared state (the durable components, or a distributed limiter).
  • More knobs vs. simplicity. The options give real control (ratio, window, throughput) at the cost of more to reason about. The defaults are sensible; change them with data.

Alternatives

  • Polly is the de-facto .NET resilience library and is excellent for rich, composable policies (bulkhead, hedging, fallback) around HTTP clients. Relay’s primitives are intentionally smaller and shared with its durable internals; use Polly at your HTTP edges and these for Relay-integrated paths. They coexist happily.
  • System.Threading.RateLimiting (the BCL) offers production rate limiters; reach for it when you need its algorithms or ASP.NET Core middleware integration.

Lessons from production systems

  • The first incident is usually a retry storm: a dependency slows, every caller retries, the extra load finishes it off. Backoff + jitter + a circuit breaker turn that doom loop into a brief, self-limiting blip.
  • Teams consistently under-cap backoff and over-trust “transient.” Write down which exceptions are retryable; default to not retrying.
  • Time-dependent code that isn’t TimeProvider-based becomes untested code, because nobody wants a 15-second test. The primitives that take a clock are the ones that stay correct.

Should you use this?

If you… Then…
call a flaky synchronous dependency yes — wrap it with the breaker + retry policy
process messages that can fail use the durable outbox/scheduler/inbox instead
need cluster-wide rate limiting layer a distributed limiter; these are per-process
already use Polly at your HTTP edges keep it; use these for Relay-integrated paths

Next steps

  • 023 — Distributed Coordination: when resilience needs cluster state — locks, leader election, partition ownership.
  • 021 — Observability: the breaker trips and rate-limit rejections are emitted as metrics (relay.circuit_breaker.trips, relay.rate_limiter.rejections) — watch them.