Exactly-once is usually a lie: designing idempotent distributed workflows

Vendors promise exactly-once processing. Architects draw arrows labelled 'once.' Production sends the same webhook twice, the queue redelivers, the lambda times out after writing to the database but before acknowledging, and the customer gets two onboarding emails and one angry finance team. Exactly-once delivery across real systems is a marketing compression of a harder truth: networks duplicate, processes crash, and clocks lie.

What you can achieve is exactly-once effect — the business outcome happens once even when the work arrives many times. That requires idempotent design at every handoff, explicit deduplication keys, and honesty about which layer owns consistency.

Constrange maps workflows by counting failure points: webhook ingress, queue, consumer, outbound API, database commit, notification fan-out. Each point can retry, duplicate, or reorder. Exactly-once language hides the count; idempotent design surfaces it.

The pipeline nobody draws completely

A typical flow: partner webhook → API gateway → queue → worker → internal API → database → notification service → email provider. Seven hops. Each may retry independently. Drawing one arrow from 'event' to 'done' is how double billing ships.

Vocabulary that keeps teams honest

At-most-once: you might lose messages. At-least-once: you might duplicate. Exactly-once delivery end-to-end requires cooperation physics rarely gives you. Exactly-once effect: processing the same logical event twice does not change outcome — balance unchanged, row upserted, notification deduped.

Transactional outbox, idempotent consumers, and dedupe stores are implementation tactics for exactly-once effect — not magic words on a Kafka slide.

• Idempotency key carried from ingress to all side effects
• Natural keys where business defines uniqueness — invoice id, event id
• Outbox pattern for reliable publish after DB commit
• Inbox pattern for deduping consumer receives
• Status fields: pending, completed, failed — not blind insert

Designing idempotent side effects

Database writes

Prefer upserts and unique constraints aligned to business keys. 'Insert payment' fails on duplicate; 'apply payment id P' succeeds twice with same result. Avoid sequences that allocate new primary keys on every retry — duplicates become two rows forever.

Outbound API calls

Use provider idempotency headers where available. Where not, store 'already sent request X' before calling — or query provider state before retry. Timeout is the cruel case: unknown if call succeeded. Recovery requires reconcile loop or provider lookup — not blind retry.

Notifications

Customers forgive slow email once; not twice. Dedupe on (user, template, business event id). Separate 'should notify' from 'did notify' with persistent log.

The webhook trap

Partners retry until they see 200. Your handler must respond quickly and dedupe on event id before heavy work — or accept enqueue with dedupe at queue boundary. Processing synchronously in the webhook thread couples availability to partner retry storms.

Return 200 only when work is durably recorded — in inbox table or queue — not when processing finished. Finish asynchronously; dedupe at record boundary.

Sagas, compensation, and ordering

Multi-step workflows — reserve inventory, charge card, ship — need compensating actions or forward recovery. Idempotency at each step prevents double charge; saga state machine prevents double ship. Ordering matters when later steps assume earlier completion — use per-entity sequencing or versioning, not global queue order fantasy.

Partial completion is normal. Design explicit states: charged but not shipped triggers ops queue, not silent duplicate ship on retry.

“Retries are how distributed systems say 'maybe.' Idempotency is how you answer 'still no.'”Constrange

Observability for duplicates

Metric duplicate detections: same idempotency key seen twice, constraint violations caught, notification dedupe hits. Alert on rate spikes — partner config change, new deploy without keys, broken inbox cleanup.

Traces should carry idempotency key as baggage so support finds both attempts in one view. Without it, 'user clicked once' disputes devolve into log archaeology.

Testing what vendors omit

Chaos tests: replay webhooks, kill worker mid-transaction, delay API responses into timeout, redeliver queue messages deliberately. Unit tests with mock 'called twice' at each adapter. Staging that never duplicates is a demo environment.

Constrange often finds idempotency implemented only at the database — while notifications and CRM updates still double-fire because those adapters were 'later.' Later is production day one for duplicates.

When 'exactly-once' products fit

Some streams offer idempotent producers and transactional consume-process-produce within a broker boundary. Useful — bounded. The moment you call HTTP or send email, you left the boundary. Account for external hops explicitly.

Ledger and finance

Money requires exactly-once effect strongest. Use immutable ledger entries keyed by transaction id; reversals as new entries, not deletes. Never 'subtract twice' on retry — append compensating entry.

1. Assign stable idempotency key at system ingress
2. Persist dedupe before irreversible side effects
3. Align DB constraints to business natural keys
4. Instrument duplicate detection and unknown-timeout paths
5. Chaos-test redelivery at every hop before launch

Organizational symptoms

Support tickets: 'charged twice,' 'two accounts,' 'email spam.' Engineering blame partner retries without ingress dedupe. Finance escalates; engineering adds retry backoff — which does not stop duplicates, only delays them.

Product asks for 'real-time exactly-once' as one requirement. Split it: real-time visibility vs effect consistency — different designs, different costs.

Pragmatic rollout

Inventory the workflow hops. Mark which side effects are reversible vs irreversible. Implement idempotency on irreversible first — payments, provisioning, external comms. Reversible recompute can tolerate duplicate with waste; irreversible cannot.

Document idempotency key ownership: who generates, who stores, TTL, cleanup. Keys without TTL fill storage; keys without owner drift across services.

Poison messages and the idempotency boundary

A malformed payload that always throws will retry forever unless you move it aside. Idempotency does not fix bad data — it prevents good data from executing twice. Pair dedupe with failure classification: transient errors retry with backoff; permanent errors land in a dead-letter queue with the idempotency key attached so replay is a deliberate operator action, not an accidental loop.

Replay tooling must respect keys. Support 'run again' buttons that generate new keys are duplicate factories. Admin replay should reuse the original business event id or refuse when completed state already exists.

Cross-service contract

Publish idempotency key format and retention expectations to partner teams and vendors. Internal services that strip headers at the gateway undo your design. Treat the key like a correlation id — required on ingress, propagated on egress, logged in audit.

Accept the lie, ship the truth

Stop selling exactly-once across the pipeline. Ship at-least-once transport with exactly-once effect at business boundaries — named keys, constraints, dedupe stores, reconcile loops for unknown timeouts. Honest architecture reduces incidents more than optimistic labelling.

The contrarian win: teams that embrace duplication as default design faster — they test redelivery on day one. Teams that believe exactly-once discover duplication in production, under revenue recognition pressure.

Questions before the next launch

• Which side effects are irreversible if run twice?
• Where is the idempotency key generated — and who trusts it?
• What happens on timeout when success is unknown?
• Can support trace both attempts of the same logical event?
• Have we replayed webhooks and redelivered queue messages in staging?

Weak answers on any row are launch blockers for payment, provisioning, and external comms — not polish items for a later sprint.