Event-driven architecture: what happens when nobody knows the order anymore?

Event-driven architecture promises decoupling. Services publish facts. Other services react. Nobody waits on a synchronous call chain. The diagram looks clean. Operations discovers a harder truth: when nobody owns the order of work, nobody owns the story of what happened.

A payment succeeds. An inventory reservation fails. A notification sends anyway. A customer receives confirmation for an order that was never fulfilled. Each service behaved correctly in isolation. The system failed in composition. The failure mode is not a bug in one handler. It is the absence of a shared model for causality across asynchronous boundaries.

Constrange reviews event-driven programmes that optimised for throughput and forgot that humans still expect sequences: created before shipped, authorised before debited, agreed before recorded. Eventual consistency is a design choice, not an excuse. If you cannot explain what 'eventually' means in minutes, you have a product problem dressed as architecture.

What you lose when you lose order

Synchronous systems hide ordering behind a call stack. Event-driven systems expose it as a design problem. Messages arrive out of order. They arrive twice. They arrive after a compensating action already ran. Consumers infer state from partial history. Without explicit rules, inference becomes guesswork with audit consequences.

Eventual consistency is a contract, not a shrug

Teams say 'eventually consistent' as if it ends the conversation. It should start one. Eventually when? Under what failure conditions? What does the user see in the meantime? What operations can run safely on stale state? What happens if 'eventually' takes longer than the business tolerates?

Strong consistency everywhere is expensive and often unnecessary. Undefined consistency is expensive in a different currency: support tickets, manual reconciliation, reversals, and the quiet work of teams who no longer trust the platform numbers.

• User-visible state: what can be shown while downstream catches up
• Operational state: what back-office may act on before all handlers complete
• Financial state: what may never be wrong even briefly
• Audit state: what must be reconstructable months later
• Recovery state: what replay must restore without double effect

Idempotency is not optional

At-least-once delivery is the practical default for durable messaging. That means handlers will see duplicates. Idempotency keys, natural keys, compare-and-set writes, and deduplication stores are not optimisation. They are the minimum bar for a consumer that mutates state.

Non-idempotent handlers fail quietly at first. A duplicate shipment. A second refund. Two ledger entries for one authorisation. The incident post-mortem blames 'a duplicate message' as if duplication were exceptional. In event-driven systems, duplication is weather.

1. Define the idempotency key at the domain boundary — not inside each handler ad hoc
2. Persist processed keys with the same durability as the side effect
3. Make side effects safe under retry: upsert, not blind insert
4. Test duplicate delivery in CI, not only happy path
5. Log idempotency decisions — 'ignored duplicate' is operational signal

Ordering strategies that actually help

Partition by key

Ordering guarantees are expensive globally and cheap locally. Partition streams by entity key — order id, account id, aggregate id — so all events for one entity serialize through one partition. Cross-entity ordering still requires choreography or sagas. Within-entity ordering becomes tractable.

Version and sequence numbers

Every emitted event should carry a monotonic version for its aggregate. Consumers reject or buffer stale versions. This does not solve every race, but it stops silent regression where an older event overwrites newer truth.

Process managers and sagas

When a workflow spans services, something must remember where the workflow is. A process manager, saga orchestrator, or state machine — name it what you will — holds the cross-service sequence. Hoping consumers infer workflow state from unrelated topics is how organisations learn what a dead-letter queue is for.

“Decoupling services without decoupling accountability for outcomes is how distributed systems teach finance new reconciliation rituals.”Constrange

Duplicate events and the stories they tell

Duplicates arrive from retries, producer restarts, network timeouts, and consumer redelivery after partial failure. Each duplicate is a signal that the boundary between 'accepted' and 'done' was drawn in the wrong place. If publishing succeeds but processing fails, you will retry. If retry is not idempotent, you will incident.

Teams sometimes ban retries to stop duplicates. That trades duplicate work for lost work — usually worse. The fix is idempotent processing and explicit outbox patterns so publish and persist are one decision, not two hopes.

Replay: powerful and dangerous

Replay is the superpower of event logs. It is also how you send twelve months of price changes to a consumer that sends customer emails. Replay requires contracts: which topics, which offsets, which consumers, what side effects are safe, who approves production replay, how you prove idempotency held.

Development replays freely. Production replay without guardrails is a change event without a change ticket. Constrange recommends replay runbooks before you need them — written when calm, not during an outage when someone proposes 'just reset the offset'.

Dead-letter queues are symptoms, not solutions

A dead-letter queue is where messages go when a consumer gives up. It is not a strategy. It is a pile. DLQs accumulate poison messages, schema drift failures, and bugs masquerading as bad data. Without ownership, the DLQ becomes a shadow backlog someone checks after customers complain.

• Alert on DLQ depth and age — not only on consumer lag
• Classify failures: transient, bug, schema, poison payload
• Route poison messages to quarantine with context, not infinite retry
• Assign an owner who can re-drive or discard with audit trail
• Review DLQ trends weekly — spikes precede incidents

Debugging when causality is fragmented

Distributed workflows fail in ways monolith stack traces never showed. Traces must cross service boundaries. Logs must share correlation identifiers. Business identifiers must appear in every handler — order id, payment id, workflow id — not only request ids that change at each hop.

Support asks 'what happened to order 8842?' Engineering grep logs without a shared key and reconstructs fiction. The fix is not another dashboard. It is a correlation model agreed at design time and enforced in libraries, not left to each team's taste.

Questions that expose ordering debt

1. What can a user do while handlers are still catching up?
2. Which operations are safe to retry from the UI?
3. What happens if event B arrives before event A?
4. How do we prove a handler ran exactly once in effect?
5. What is the runbook for replaying last Tuesday?

Patterns that survive contact with operations

Outbox pattern for reliable publish. Inbox pattern for deduplication at consume. Explicit sagas for multi-step business processes. Read models that declare staleness. Compensating transactions where rollback is business-meaningful, not only database rollback.

Event-driven architecture works when teams treat messages as contracts: schema, ordering scope, idempotency, and failure semantics documented beside the topic name. It fails when topics are a dumping ground for 'something happened' strings and every consumer interprets differently.

A twelve-week hardening programme

1. Weeks 1–2: inventory topics and consumers — mark mutating handlers, list idempotency keys
2. Weeks 3–4: introduce correlation ids and trace propagation across all consumers
3. Weeks 5–6: chaos-test duplicate delivery and out-of-order delivery in staging
4. Weeks 7–8: DLQ ownership, alerts, classification runbook
5. Weeks 9–10: document replay procedure with approval gate
6. Weeks 11–12: measure reconciliation volume — did engineering debt drop?

Twelve weeks will not perfect the estate. It will show whether the organisation treats events as architecture or as a fire-and-forget integration trick.

Questions for your next architecture review

• Which user journeys require strict ordering — have we partitioned for them?
• What happens on duplicate delivery for each mutating consumer?
• Where is 'eventually' defined in time and what breaks if we miss it?
• Who owns the DLQ and what is the re-drive process?
• Can we reconstruct order 8842 from logs and events alone?

Event-driven systems scale teams and throughput. They do not scale ambiguity. When nobody knows the order anymore, the organisation pays in reconciliation, reversals, and trust — until someone designs causality on purpose.