Your Queue Is Hiding the Real Incident
Zero HTTP errors while depth and age climb. Lag is a first-class reliability metric with a recoverability horizon.
Queue-backed systems is discussed in architecture reviews and post-mortems, often as a reliability topic or a FinOps exercise. The conversation stops at policy slides. Constrange's Observe → Measure → Model series treats it as an engineering economics problem: what does the system actually do, what does that work cost, and what model explains the gap between intent and invoice.
Observe: monitor queue depth and message age alongside HTTP error rate. Incidents often start in lag, not 5xx.
Measure: recoverability horizon = backlog / processing rate. If horizon exceeds customer SLA, you are in incident.
Model: async systems decouple acceptance from completion. Green HTTP with growing lag is deferred failure.
HTTP green, system red
API returns 202 Accepted. Producer celebrates. Consumers fall behind. Depth climbs. Age crosses hours. Support tickets mention 'delayed notifications' while Grafana shows green. The queue absorbed the failure until it could not.
Lag as SLO
Define SLO on processing latency: 95% of messages processed within N minutes. Page on age p99, not only consumer restarts. Model drain time after producer spike — can you recover before business impact?
| Shows | Misses | |
|---|---|---|
| HTTP 5xx | Sync path failure | Backlog |
| Queue depth | Accumulated work | Per-message age distribution |
| Consumer CPU | Worker health | Poison messages |
| Age p99 | Oldest stuck work | Future arrival rate |
Incident playbooks
- Scale consumers with proven throughput, not hope
- Throttle producers when horizon exceeds SLA
- Quarantine poison messages to unblock drain
- Communicate customer delay honestly
“The incident you see in HTTP is often the incident your queue postponed.”Constrange
Questions for your next review
- What would we see if we measured queue lag for one week?
- Which dependency owns the majority of the cost or delay?
- What policy or architecture assumption does the data contradict?
- What is the smallest experiment that would change our model?
The organisations that improve queue lag are not those with the best slogans. They are those that observe honestly, measure without vanity metrics, and update the model when the bill or the trace disagrees with the diagram.
Frequently asked questions
Why zero errors during an outage?
Synchronous paths return 200 while async backlog grows — errors appear later or never on HTTP dashboards.
What metrics matter for queues?
Depth, age p99, consumer lag, and time-to-drain at current throughput.
What is recoverability horizon?
How long until backlog clears at steady processing rate — defines customer-visible delay.
When should queues page?
When age or depth breaches SLO, not only when consumers crash.