Articles · Production Operations ·037

Operational Event Queries: A Forensic Scan over the Log

Runnable sample on GitHub

Sample: samples/037-operational-event-queries · Prerequisites: 005 — Event Sourcing Basics

Overview

The event store is an append-only log, and article 005 is emphatic about the rule that follows from that: never query the event store for reads. It answers “give me this aggregate’s stream,” not “give me all accounts over $1000.” Those queries come from projections (article 007). That rule is correct, and this article does not relax it.

But there is a narrow, genuinely useful exception that lives entirely outside your application’s read paths: incident response and forensics. At 3am, with a projection misbehaving and a customer escalation open, you do not want to build a new projection and wait for it to catch up — you want to scan the raw log for a needle: “show me every PaymentFailed event in the last window, in order, so I can see what actually happened.” For that one-off, throwaway scan, Relay exposes IEventStore.QueryEventsAsync(predicate, maxResults, …) — a paged predicate scan over the global log that returns matches in global-position order. It is a flashlight for walking the log during an investigation, explicitly not a general query engine.

Why this exists

When something is wrong in production, the event log is the ground truth — it is, by construction, the one place that cannot have drifted from what really happened (article 005). The temptations during an incident are both bad: dropping into a database console and writing ad-hoc SQL against relay_events (coupling you to the physical schema, and easy to get the gap-safe ordering wrong), or standing up a new projection just to answer a question you will ask exactly once. QueryEventsAsync gives you a supported in-process scan that reuses the same paged global read (GetAllEventsAsync) the framework’s own catch-up machinery uses — so it pages in bounded batches, returns events in global-position order, and stops at a cap you set. It is the right tool for “let me look,” kept deliberately small so nobody mistakes it for the read path.

When to use this

  • Incident investigation. “Did we ever emit FraudFlagged for this account, and when relative to the charge?” A predicate over event type / aggregate id / parsed payload, capped at a sane maxResults, answers it without new infrastructure.
  • Forensics and audit spot-checks. Walk the log for a specific signature after the fact — a one-off scan whose results you read and discard.
  • A throwaway, one-time read-model seed. When you genuinely only need a handful of matching events once (and will not need them again), a bounded scan can be simpler than a projection.

When not to use this

  • Never on an application read path. This is the whole point. If a feature needs to list, filter, or search events, that is a projection (article 007) — an indexed read model kept current as events arrive. QueryEventsAsync is a full scan; using it to serve requests turns every query into a walk of the entire log.
  • Not as a query engine. There is no index behind the predicate. “All events where payload.amount > X” is evaluated by scanning and deserializing, row by row. That is fine for a one-off at 3am; it is a performance catastrophe as a feature.
  • Not in a hot path or a loop. It scans. Calling it per request, per message, or inside a tight loop is the same mistake as querying the event store for reads, just spelled differently.

Concepts

Paged GetAllEventsAsync. The scan is built on the global read GetAllEventsAsync(fromPosition, maxCount, …), where fromPosition is exclusive (only events with Position > fromPosition are returned). QueryEventsAsync pages through the log in fixed-size batches (1,000 at a time), advancing fromPosition to the last position it saw, so the working set stays bounded even on a huge log.

Predicate filter. You pass a Func<EventData, bool> evaluated against each event — on its EventType, AggregateId, AggregateType, Position, or its parsed JSON Data. It is ordinary in-process code; there is no translation to a server-side query (a custom store may override QueryEventsAsync with, say, a PostgreSQL jsonb filter, but the default scans).

maxResults cap. The scan collects matches until it has maxResults of them (default 100), then stops — it does not read the rest of the log. This is the guardrail that keeps a forensic scan from turning into a full-table read. Zero or negative is rejected.

Position order. Matches are returned in ascending global Position — the order events were committed to the log — which is exactly what you want when reconstructing “what happened, in what sequence.”

It is a default interface method. QueryEventsAsync has a default implementation on the interface. In C#, a default interface method is callable only through the interface type, not through a concrete class that implements IEventStore. So you must hold the store as an IEventStore reference to call it — calling it on the concrete type does not even compile. The sample’s StoreWith(…) helper returns IEventStore for exactly this reason.

Architecture

flowchart TD
    Op[On-call engineer / forensic tool] -->|predicate + maxResults| Q[IEventStore.QueryEventsAsync<br/>default interface method]
    Q -->|page: GetAllEventsAsync fromPosition exclusive, batch 1000| Log[(Global event log)]
    Log -->|batch in position order| Q
    Q -->|apply predicate, collect until maxResults| Match{enough matches?}
    Match -->|no, advance fromPosition| Q
    Match -->|yes / log exhausted| Out[matches in global-position order]

Building it step by step

The full sample is in samples/037-operational-event-queries.

1. An in-memory store that implements the paged global read

QueryEventsAsync only needs one member to do its work: the paged GetAllEventsAsync. So the sample’s stub implements that over a List<EventData> (treating fromPosition as exclusive) and lets every other interface member throw NotSupportedException — no database, broker, or container.

public sealed class InMemoryEventStore : IEventStore
{
    public List<EventData> Events { get; } = new();

    public Task<IReadOnlyList<EventData>> GetAllEventsAsync(long fromPosition = 0, int? maxCount = null, CancellationToken ct = default)
    {
        IEnumerable<EventData> q = Events.Where(e => e.Position > fromPosition).OrderBy(e => e.Position);
        if (maxCount.HasValue) q = q.Take(maxCount.Value);
        return Task.FromResult<IReadOnlyList<EventData>>(q.ToList());
    }

    // every other IEventStore member: throw new NotSupportedException();
}

2. Call it through an IEventStore reference

Because QueryEventsAsync is a default interface method, the scan is callable only via the interface type. The helper returns IEventStore, not the concrete InMemoryEventStore:

private static IEventStore StoreWith(params (long Position, string Type)[] events)
{
    var store = new InMemoryEventStore();
    foreach (var (position, type) in events) store.Events.Add(Evt(position, type));
    return store;   // typed as IEventStore so QueryEventsAsync is callable
}

3. Scan for the needle

var store = StoreWith((5, "PaymentFailed"), (1, "PaymentFailed"), (3, "OrderPlaced"), (2, "PaymentFailed"));

var failures = await store.QueryEventsAsync(e => e.EventType == "PaymentFailed", maxResults: 50);
// failures come back in global-position order: 1, 2, 5

Complete source code

File Shows DB?
InMemoryEventStore.cs in-memory IEventStore stub over a List<EventData> No
OperationalEventQueryTests.cs position order, maxResults, empty, argument guards No

Running the example

dotnet test samples/037-operational-event-queries/Forensics.EventQueries.Tests

This needs only the .NET 10 SDK — no Docker, no database. The scan is pure logic over an in-memory list, so it is fully deterministic.

Testing

The sample proves the four properties that make the scan trustworthy as a diagnostic:

Matches come back in global-position order, even when seeded out of order:

var store = StoreWith((5, "PaymentFailed"), (1, "PaymentFailed"), (3, "OrderPlaced"), (2, "PaymentFailed"), (4, "OrderPlaced"));
var result = await store.QueryEventsAsync(e => e.EventType == "PaymentFailed");
result.Select(e => e.Position).Should().Equal(1, 2, 5);

It honours maxResults — collecting exactly that many matches and stopping:

var result = await store.QueryEventsAsync(e => e.EventType == "PaymentFailed", maxResults: 2);
result.Select(e => e.Position).Should().Equal(1, 2);

It is empty when nothing matches (no false end-of-stream), and it guards its arguments — a null predicate throws ArgumentNullException, a zero/negative maxResults throws ArgumentOutOfRangeException. All four are pure unit tests against the in-memory store, because the behaviour being verified (ordering, capping, guards) is in the default method, not in the database.

Production considerations

  • It scans — bound maxResults deliberately. The cap is what separates a forensic peek from a full-log read. Set it to the smallest number that answers your question; the default of 100 is a sensible ceiling for “show me the recent ones.”
  • Run it off-peak, and against a replica if you have one. A scan competes with live traffic for I/O. During an incident on the primary, prefer a read replica or a quiet window.
  • Keep it out of hot paths. This is the cardinal rule restated: no per-request, per-message, or in-loop calls. If you find yourself wanting that, you want a projection.
  • The predicate runs in-process. Filtering on parsed payload means deserializing each scanned event. That cost is acceptable for a one-off and unacceptable as a feature.
  • A store may override it. The default scans; a custom IEventStore can override QueryEventsAsync with a server-side query (e.g. a PostgreSQL jsonb filter). Even then, treat it as a diagnostic, not a read path.

Common mistakes

  • Using it as a query engine. Wiring QueryEventsAsync behind an API endpoint or a list view. There is no index; every call walks the log. Build a projection (article 007) instead — that is the read path, and it is the rule from article 005.
  • Calling it on a concrete type. new InMemoryEventStore().QueryEventsAsync(…) does not compile — default interface methods are invisible on the implementing class. Hold the store as IEventStore.
  • Forgetting fromPosition is exclusive. A custom store whose GetAllEventsAsync treats fromPosition as inclusive would make the pager re-read the same tail forever; the default detects non-advancing batches and throws rather than loop. Implement it as “Position > fromPosition.”
  • Leaving maxResults unbounded in spirit. Passing a huge cap “just in case” turns the scan back into a full read. Pick a real bound.

Tradeoffs

Benefits. A supported, in-process way to scan the authoritative log during an incident, with no new infrastructure and no hand-written SQL against the physical schema; it reuses the same paged, bounded global read the framework’s catch-up readers use, so ordering and batching are correct by default.

Costs. It is a full scan with an in-process predicate — linear in the log it walks and gated only by maxResults. It is a diagnostic, not a feature: there is no index, no caching, and no place for it on a read path. The default-interface-method shape is also a small papercut — you must remember to call it through IEventStore.

Alternatives

  • Projections (article 007). The right answer for any application query over events: an indexed read model, kept current as events arrive, queryable cheaply and repeatedly. If a question will be asked more than once, it belongs in a projection, not a scan.
  • A dedicated search index. For rich, ad-hoc operational search across a large log (full-text, range, faceted), stream events into Elasticsearch/OpenSearch and query that. More infrastructure, but the right tool when forensic search is a recurring need rather than a 3am one-off.
  • A store-specific server-side query. Overriding QueryEventsAsync with a jsonb/SQL filter pushes the predicate into the database — faster than an in-process scan, but still a diagnostic path, and it couples you to that store.

Lessons from production systems

  • The teams that stay calm during incidents have a supported way to look at the log that does not mean “open a SQL console and hope you got the ordering right.” A bounded predicate scan is that tool.
  • The same teams are disciplined about never letting the scan leak onto a read path. The moment “let me look” becomes “the feature lists events by scanning,” latency and load fall off a cliff — and the fix is always a projection that should have existed.
  • “Bound it” is the whole safety story. An unbounded scan over a multi-year log is indistinguishable from a SELECT * with no WHERE; maxResults is the seatbelt, so wear it.

Should you use this?

If you… Then…
need a one-off forensic scan during an incident yes — bounded QueryEventsAsync, off-peak/replica
are serving an application list/search/filter no — build a projection (article 007)
call it per request, per message, or in a loop no — that is querying the event store for reads
need recurring rich search over the log consider a dedicated search index

Next steps

  • 007 — Projections: the read path. Anything you would be tempted to serve with a scan belongs here — an indexed read model kept current as events arrive.
  • 005 — Event Sourcing Basics: the rule this article carves a narrow exception to — the event store is the source of truth, and you query projections, not the log.