Articles · Production Operations ·021

Observability: Telemetry, Metrics, Tracing, Health

Runnable sample on GitHub

Sample: samples/021-observability · Subsystem: 2.13 · Prerequisites: 001 — Getting Started

Overview

You cannot operate what you cannot see. Relay is instrumented with OpenTelemetry-native primitives out of the box — no wrappers, no proprietary format. Everything flows through a single ActivitySource and a single Meter, both named Nuvora.Nexus.Relay, exposed as RelayTelemetry.SourceName and RelayTelemetry.MeterName. Dispatching a command or query records a counter, a duration histogram, and a trace span; the outbox, projections, sagas, scheduler and transports each add their own counters and gauges. Forwarding all of it to a backend is one line per signal.

Why this exists

Distributed, event-driven systems are opaque by default: a command returns 200, but did its integration event reach the broker? Is the projection lagging? Is the outbox backing up? Bolting on telemetry after the fact means instrumenting dozens of internal seams by hand and inventing metric names that won’t match the next service’s. Relay instruments those seams once, with a consistent naming scheme (relay.<area>.<thing>), so a dashboard built for one Relay service works for all of them. Because it uses the BCL’s ActivitySource/Meter (the OpenTelemetry data model), it needs no Relay- specific exporter — the standard OTel SDK collects it.

When to use this

  • Always, in any deployed service. Wire AddRelayInstrumentation() into your OpenTelemetry setup and point it at your collector (Tempo/Jaeger for traces, Prometheus/OTLP for metrics).
  • When debugging “where did my event go?” — the trace spans stitch command → outbox → broker → inbox.
  • When alerting on health: outbox backlog, projection lag, dead-letter depth, breaker trips.

When not to use this

  • There is no “don’t” — but don’t build custom telemetry that duplicates RelayTelemetry. If you need a new signal, emit your own Meter/ActivitySource; don’t fork the framework’s.
  • Don’t export high-cardinality tags (e.g. a per-entity id as a metric tag) — that explodes your metrics backend. Relay’s tags are bounded (command/query type names, not instances).

Concepts

Three signals, one source.

Signal Mechanism Example
Traces (spans) RelayTelemetry.ActivitySource command CreateProductCommand
Metrics (counters/histograms/gauges) RelayTelemetry.Meter relay.commands.executed, relay.command.duration
Health IHealthChecksBuilder extensions AddRelayOutbox(), AddRelayProjections()

Counters vs. gauges. Counters monotonically increase (relay.commands.executed, relay.outbox.published) — you rate them in the backend. Observable gauges sample a current value on scrape (relay.outbox.pending_depth, relay.projections.max_lag) and report -1 until the owning component takes its first sample, so “no data yet” is distinguishable from “zero.”

Listeners are the collection mechanism. OpenTelemetry’s SDK is, under the hood, a MeterListener and an ActivityListener that subscribe to named sources. You can attach your own listeners in-process — which is exactly how you test telemetry without standing up a collector.

Architecture

flowchart LR
    subgraph Relay
        Bus[Command/Query Bus] -->|span + counter| T[RelayTelemetry<br/>ActivitySource + Meter<br/>'Nuvora.Nexus.Relay']
        Outbox -->|counters + gauges| T
        Projections -->|lag gauge| T
        Sagas -->|reaction counters| T
        Scheduler -->|delivery counters| T
    end
    T -->|AddRelayInstrumentation| OTel[OpenTelemetry SDK]
    OTel --> Traces[(Tempo / Jaeger)]
    OTel --> Metrics[(Prometheus / OTLP)]
    HC[Health checks] --> Probe[/health endpoint/]

Building it step by step

1. Production wiring (one line per signal)

services.AddOpenTelemetry()
    .WithTracing(t => t.AddRelayInstrumentation())   // Nuvora.Nexus.Relay.Diagnostics
    .WithMetrics(m => m.AddRelayInstrumentation());

// health
services.AddHealthChecks()
    .AddRelayOutbox()
    .AddRelayProjections()
    .AddRelayScheduler();

AddRelayInstrumentation() is a two-line shim: builder.AddSource(RelayTelemetry.SourceName) / builder.AddMeter(RelayTelemetry.MeterName). There is no magic — it subscribes the OTel SDK to Relay’s source and meter.

2. Asserting telemetry without a backend

The sample attaches the same listeners the SDK uses and dispatches a command through the real bus:

using var meterListener = new MeterListener
{
    InstrumentPublished = (i, l) =>
    {
        if (i.Meter.Name == RelayTelemetry.MeterName && i.Name == "relay.commands.executed")
            l.EnableMeasurementEvents(i);
    },
};
meterListener.SetMeasurementEventCallback<long>((_, value, tags, _) => { /* capture value + tags */ });
meterListener.Start();

using var activityListener = new ActivityListener
{
    ShouldListenTo = s => s.Name == RelayTelemetry.SourceName,
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllDataAndRecorded,
    ActivityStopped = a => { /* capture span */ },
};
ActivitySource.AddActivityListener(activityListener);

await commandBus.Execute<PingCommand, string>(new PingCommand("hello"), default);
// → a "relay.commands.executed" measurement {command: "PingCommand", success: true}
// → a span named "command PingCommand"

Complete source code

File Shows
TelemetryTests.cs instrument names; metric + span emitted on dispatch

Running the example

dotnet test samples/021-observability/Observability.Sample.Tests

Testing

The in-process MeterListener/ActivityListener pattern is the canonical way to unit-test telemetry: no collector, no exporter, deterministic. The framework’s own QueryTelemetryTests use exactly this to guarantee the bus records a success/failure measurement. Asserting on telemetry is worthwhile — metric names and tags are a public contract that dashboards and alerts depend on, so a rename is a breaking change you want a test to catch.

Production considerations

  • Sampling. Trace everything in dev; in production use a parent-based sampler so you keep full traces for a fraction of requests. Metrics are cheap — keep them all.
  • Cardinality. Relay tags are type names (bounded). If you add your own spans/metrics, never tag with unbounded values (user ids, order ids) — use them as span attributes you can search, not metric dimensions.
  • The gauges that start at -1 (relay.outbox.pending_depth, relay.projections.max_lag, relay.scheduler.due_depth) are your early-warning system. Alert on trend, not absolute value: steadily rising pending depth means the processor can’t keep up.
  • Health endpoints should be split: a liveness probe (is the process up?) and a readiness probe (AddRelayOutbox/Projections/Scheduler — can it do useful work?). Don’t fail liveness on a backed-up outbox or Kubernetes will pointlessly restart a healthy pod.

Common mistakes

  • Reinventing metric names. Use RelayTelemetry’s; a service-specific scheme means every dashboard is bespoke.
  • High-cardinality tags — the fastest way to blow up (and get billed for) a metrics backend.
  • Forgetting AddRelayInstrumentation() — the instruments exist and record, but nothing exports them, so you see nothing and assume it’s broken.
  • Alerting on a single gauge sample. The -1 sentinel and scrape timing mean point values mislead; alert on rate/trend over a window.

Tradeoffs

  • Built-in vs. flexible. You get a fixed, sensible set of signals. Richer custom telemetry is on you (but composes — your ActivitySource and Relay’s both flow to the same SDK).
  • Overhead. Instrumentation is cheap but not free; histograms and many gauges add CPU/memory. The defaults are tuned for negligible cost; if you export at very high frequency, measure.

Alternatives

  • OpenTelemetry directly is what this is — Relay just pre-instruments its internals. You still bring the OTel SDK and exporters.
  • Vendor agents (Datadog, New Relic, Application Insights) auto-instrument ASP.NET Core and can ingest OTLP; point them at the same source/meter. Relay’s signals show up as first-class spans/metrics.

Lessons from production systems

  • The teams that sleep well wired observability on day one, not after the first incident. The cost is one line per signal; the payoff is seeing the outbox back up before customers do.
  • “Where did my event go?” is answered in seconds when command → outbox → broker → inbox share a trace, and in hours when they don’t. End-to-end tracing is the highest-leverage thing you can turn on.
  • Metric names and tags are an API. Treat a rename like any breaking change — and keep a test (like the sample’s) that pins them.

Should you use this?

If you… Then…
run Relay in production yes — AddRelayInstrumentation() is non-negotiable
debug message flow turn on tracing; follow the span tree
need custom business metrics add your own Meter alongside Relay’s
run only local unit tests the listener pattern lets you assert telemetry with zero infra

Next steps