The Architecture of a 500ms Timeout
A client deadline is a budget across DNS, TLS, gateways, services, and databases — until one slow hop consumes the rest.
Client deadlines is discussed in architecture reviews and post-mortems, often as a reliability topic or a FinOps exercise. The conversation stops at policy slides. Constrange's Observe → Measure → Model series treats it as an engineering economics problem: what does the system actually do, what does that work cost, and what model explains the gap between intent and invoice.
Observe: trace each hop in a critical path with span timings. The client timeout is the envelope.
Measure: how much of 500ms DNS, TLS, gateway, service, cache, and database consume at p95 and p99.
Model: serial hops sum; the slowest hop near the end leaves no margin for recovery or retry.
A deadline is not a single knob
Mobile clients set 500ms. API gateways inherit it. Services copy it into HTTP clients. Nobody allocated budget per hop. DNS spikes. TLS handshake on cold connection. Gateway auth. Service logic. Cache miss to database. One slow hop does not take 20% of 500ms — it takes whatever is left.
| Typical p95 | Budget risk | |
|---|---|---|
| DNS | 5–30ms | Spikes on resolver issues |
| TLS | 20–80ms | Cold connections, cert chains |
| Gateway | 10–50ms | Auth, rate limits, WAF |
| Service | 50–200ms | Business logic, fan-out |
| Database | 10–300ms | Cache miss, lock, query |
Timeout policy as architecture
Per-hop timeouts should be shorter than the client deadline, with context passed downstream. Parent remaining budget beats static 500ms on every call. Without propagation, inner services think they have 500ms when the client has 50ms left.
Design for partial failure
- Return degraded response before deadline exhaustion
- Cache aggressively on the hot path
- Parallelise independent hops where possible
- Measure budget consumption per hop in traces
“A timeout without a budget map is a wish that the slowest hop will cooperate.”Constrange
Questions for your next review
- What would we see if we measured latency budgets for one week?
- Which dependency owns the majority of the cost or delay?
- What policy or architecture assumption does the data contradict?
- What is the smallest experiment that would change our model?
The organisations that improve latency budgets are not those with the best slogans. They are those that observe honestly, measure without vanity metrics, and update the model when the bill or the trace disagrees with the diagram.
Frequently asked questions
Is 500ms a good timeout?
Only if the budget is allocated across hops. A single global timeout hides which hop spent it.
How do I budget latency?
Sum p95 hop latency with margin; set per-hop timeouts less than client deadline.
Why do timeouts cascade?
Upstream waits for downstream; one slow database hop consumes gateway and TLS budget.
What should we model?
Deadline as a shared budget with per-hop caps and fail-fast at each layer.