Reliable Messaging (End to End)

Sample: samples/014-reliable-messaging — command → outbox → broker → consumer, over real PostgreSQL + the in-memory transport. dotnet test (needs Docker).

Prerequisites: 011 — The Outbox Pattern, 012 — Messaging & Transports, 013 — The Inbox Pattern.

Overview

The previous three articles built the pieces; this one assembles them into the complete, reliable, cross-service flow:

A command publishes an integration event → staged as a committed outbox row (011).
The outbox processor relays committed rows to the broker (012).
A consumer receives the message; the inbox makes its effect exactly-once (013).

The net guarantee: a state change in one service reliably produces an effect in another — no lost events, no phantom events, no duplicated effects — without distributed transactions. The sample drives the outbox→broker→consumer path end-to-end and verifies both that the message arrives and that the outbox row is marked Processed.

Why this exists

Reliable cross-service messaging is a composition problem: each piece is necessary but insufficient alone. The outbox without a processor stages events that never leave. A transport without the outbox reintroduces the dual-write bug. A consumer without the inbox double-applies at-least-once deliveries. The reliability emerges only when all three combine, and the combination has a precise contract worth naming: exactly-once persistence (the outbox’s atomic commit) → at-least-once delivery (the processor’s retrying relay over the transport) → exactly-once effect (the inbox’s atomic dedup). Each arrow is a guarantee; the chain is end-to-end reliability. This article exists to show the seam where the pieces meet, because that seam — the handoffs between staging, relaying, and consuming — is where real-world reliability is won or lost.

When to use this

Use the full stack whenever a committed change in one service must reliably cause work in another, which is the defining shape of an event-driven microservice system:

Order → fulfillment → shipping → notification: each service reacts to the previous one’s events, and a lost or duplicated event anywhere corrupts the chain.
Payment → ledger + receipt + analytics: one event fans out to several services, each of which must process it exactly once.
Any saga / process manager (article 015): long-running workflows are built on reliable messaging — the saga reacts to events and sends commands, all of which must be delivered reliably.

If your services communicate by events at all, this is the substrate they stand on.

When not to use this

A monolith / single service. If everything is in one process and one database, domain events (article 003) in the command’s transaction give you reliability without any of this. The full stack is for crossing boundaries.
Synchronous, immediate consistency requirements. Reliable messaging is eventually consistent — the consumer’s effect lags the command. If you need the other service’s answer now, that’s an RPC.
Throwaway / best-effort flows. If event loss is genuinely acceptable, the operational weight (outbox table, processor, inbox table, broker) isn’t worth it.

The costs are the union of the parts: a broker to run and monitor; outbox and inbox tables to store, prune, and watch for backlog; processors as hosted services; eventual consistency end-to-end (and the UX/design that implies); and the at-least-once reality that makes the inbox mandatory.

Concepts

The reliability chain. Exactly-once persistence (outbox atomic commit) → at-least-once delivery (outbox processor + transport, with retry/dead-letter) → exactly-once effect (inbox atomic dedup). No link is optional.

The outbox processor. A background service that claims committed Pending outbox rows (FOR UPDATE SKIP LOCKED), publishes them to the broker, and marks them Processed. A crash between publish and mark causes a re-publish — at-least-once — which the inbox absorbs.

The transport. Carries the message (012). In-memory for tests (deterministic delivery), RabbitMQ / Azure Service Bus in production. The outbox and inbox processors are transport-agnostic.

The inbox processor. Subscribes to the broker, dedups, dispatches to handlers, and commits effect + dedup atomically (013).

End-to-end eventual consistency. The consumer’s effect happens after the command commits and the relay runs — milliseconds to seconds, more if the broker is degraded. Design for it.

Architecture

sequenceDiagram
    participant Cmd as Command (Service A)
    participant OB as Outbox (DB, Service A)
    participant OP as Outbox processor
    participant Broker as Broker
    participant IP as Inbox processor (Service B)
    participant IB as Effect + Inbox (DB, Service B)
    Cmd->>OB: publish → staged row (atomic commit)
    OP->>OB: claim Pending (SKIP LOCKED)
    OP->>Broker: publish → mark Processed
    Broker->>IP: deliver (at-least-once)
    IP->>IB: dedup + effect (atomic) → ack
    Note over OB,IB: exactly-once persist → at-least-once deliver → exactly-once effect

Building it step by step

The sample is samples/014-reliable-messaging. It wires the outbox (011) and the transport (012) together and drives the relay explicitly.

1. Register the outbox, a transport, and (in production) the processors

services.AddRelayEventStoreEfCore<SampleDbContext>();
services.AddRelayUnitOfWorkEfCore<SampleDbContext>();
services.AddRelayOutboxEfCore<SampleDbContext>();
services.AddRelayInMemoryTransport(o => o.DeterministicDelivery = true); // or AddRelayRabbitMq(...)
services.AddRelay(typeof(Program).Assembly);

// production also: services.AddOutboxProcessor();  (+ the inbox processor / AddRelayInboxEfCore on the consumer)

2. The command publishes (staged atomically)

await integrationEventBus.Publish(new OrderPlacedIntegrationEvent { OrderId = command.OrderId }, ct);

3. The outbox processor relays to the broker

The sample drives one relay cycle explicitly (production runs it continuously as a hosted service):

var outboxProcessor = new OutboxProcessor(scopeFactory, broker, Options.Create(new OutboxProcessorOptions()), NullLogger<OutboxProcessor>.Instance);
await outboxProcessor.ProcessPendingMessagesAsync(default); // claim Pending → publish → mark Processed
await broker.DrainAsync();                                   // deterministic delivery to the subscribed consumer

The consumer side (subscribe → inbox dedup → handler) is the inbox processor from article 013; this sample focuses on the outbox→broker→consumer delivery and the Processed transition.

Complete source code

File	Contents
`Orders.cs`	The integration event + the publishing command
`ReliableFixture.cs`	Outbox + in-memory transport wiring
`EndToEndTests.cs`	command → outbox → relay → consumer + `Processed`

Running the example

dotnet test samples/014-reliable-messaging/Reliable.Sample.Tests   # needs Docker

Testing

The end-to-end test follows the message from a command to a consumer and verifies both ends of the delivery guarantee:

// a consumer is bound; the command stages a committed outbox row
await consumer.SubscribeAsync("orders", Array.Empty<string>(), (m, _) => { received.Add(m); return Task.CompletedTask; });
await bus.Execute<PlaceOrderCommand, string>(new PlaceOrderCommand(orderId), default);

// the outbox processor relays it to the broker; deterministic delivery reaches the consumer
await outboxProcessor.ProcessPendingMessagesAsync(default);
await broker.DrainAsync();

received.Should().ContainSingle(m => m.EventType == typeof(OrderPlacedIntegrationEvent).FullName && m.Payload.Contains(orderId.ToString()));
rows[0].Status.Should().Be(OutboxMessageStatus.Processed); // published once, durably

This runs against real PostgreSQL (the outbox commit and the Processed transition are database facts) with the in-memory transport’s deterministic delivery (so the relay→consume step is a reliable assertion, not a sleep). Swapping in RabbitMQ changes the fixture, not the flow.

Production considerations

Run both processors and monitor both backlogs. The outbox processor relays; the inbox processor consumes. Alert on outbox backlog/oldest-pending-age (broker down or relay stopped) and on inbox lag / dead-letters (consumer failing). A silent processor is a silent outage.
Prune both tables, with safe retention. Outbox Processed rows and inbox dedup rows both grow. AddRelayMaintenance prunes them; keep inbox retention above your redelivery/DLQ horizon (article 013).
Embrace eventual consistency in the design. The consumer’s effect lags the command. Build UX and downstream logic that tolerate the gap (status = “processing”, idempotent re-checks), and don’t query the other service’s state immediately expecting your event to have landed.
Dead-letter, alert, replay. Messages that fail past the retry cap dead-letter (outbox side and broker DLQ). Treat the DLQ as an alertable queue with a runbook (diagnose → fix → replay), not a void.
Scale out safely. FOR UPDATE SKIP LOCKED on the outbox and the dedup key on the inbox make multi-instance processors safe — no double-publish, no double-effect.
Pick the transport per environment. In-memory for tests, a real broker for production; the command/outbox/inbox code is identical (article 012).

Common mistakes

Using only part of the chain. Outbox without a processor (events never leave), transport without outbox (dual-write returns), consumer without inbox (duplicate effects). Reliability is the whole chain.
Expecting synchronous consistency. Reading the consumer’s effect right after the command will often miss it — it hasn’t been relayed/consumed yet. That’s not a bug; it’s eventual consistency.
Ignoring dead-letters. A message that can’t be processed sits in a DLQ; unwatched, it’s a permanent gap. Alert and replay.
Unbounded backlogs / tables. No monitoring and no retention turns the outbox/inbox into the biggest, slowest tables in the database and hides broker outages.
Assuming the in-memory transport is production-ready. It’s in-process; it ships nothing across a boundary. Use a real broker in production.

Tradeoffs

Benefits. End-to-end reliability across services without distributed transactions; each guarantee (persist/deliver/effect) is explicit and testable; multi-instance-safe; transport-portable; and the foundation for sagas and event-driven architecture.

Costs. Operational weight (a broker, two tables, two processors, monitoring, pruning); end-to-end eventual consistency and the design it demands; and the at-least-once reality threaded through the whole system.

Alternatives

Synchronous calls (HTTP/gRPC) between services. Immediate and simple for request/response, but couples availability (the caller fails if the callee is down) and offers no built-in reliability for “fire and react.” Use it where you need an answer now; use reliable messaging where you need a reliable eventual effect.
A managed event bus / streaming platform (Kafka, EventBridge, etc.). Powerful for high-throughput streaming and fan-out. Often used with an outbox (to get the event into the stream atomically) and an inbox-equivalent (consumer offsets + idempotency). Relay’s stack gives you the same guarantees on a general broker without adopting a streaming platform; choose by scale and ecosystem.
Distributed transactions / sagas-as-2PC. Atomic cross-service in theory, fragile in practice. Reliable messaging + a saga (article 015) is the modern replacement: local transactions plus reliable events plus compensation.

Objectively: for reliable, eventually-consistent cross-service effects, this stack is the standard architecture; synchronous calls and streaming platforms are complements or alternatives at different points on the consistency/scale spectrum.

Lessons from production systems

Reliability is only as strong as the weakest link. The teams that get burned implemented two of the three pieces and assumed reliability — a perfect outbox feeding consumers with no inbox still double-charges. Audit the whole chain, end to end.
Eventual consistency is a product conversation, not just an engineering one. “The order is placed but the confirmation email hasn’t arrived yet” is correct behavior that confuses users if the UX pretends it’s synchronous. The successful teams design for the gap explicitly.
The dead-letter runbook is the difference between resilient and fragile. Every reliable-messaging system eventually parks a message it can’t process. Teams with an alert + diagnose-fix-replay runbook recover routinely; teams without one accumulate silent gaps.
Operational monitoring is the real cost, and the real value. The code is the easy part; the durable investment is dashboards/alerts for both backlogs, the DLQ, and consumer lag. Teams that build that see problems early; teams that don’t learn from customers.

Should you use this?

Situation	Recommendation
Event-driven microservices (committed change → reliable effect elsewhere)	Strong fit — the standard stack
Sagas / long-running cross-service workflows	Strong fit — built on this
Fan-out of one event to several services	Strong fit
A single service / monolith	No — domain events suffice (article 003)
Synchronous, immediate answer needed	No — use RPC for that hop
Throwaway / best-effort, loss acceptable	Lean against — the weight isn’t justified

Next steps

Reliable messaging lets services react to each other’s events. But some workflows are more than a single reaction — they coordinate multiple steps across services, with timeouts and compensation when a step fails. That orchestration is a saga, and it is built directly on the reliable-messaging foundation you now have.

➡️ 015 — Sagas (planned — see the coverage plan)