Observability is not three dashboards

Observability initiatives often stop at tooling adoption. Logs go to a vendor. Metrics scrape everything. Traces sample at ten percent. Three dashboards appear in the ops channel. Incidents still begin with 'something is wrong' and end with 'we think it was a downstream timeout' — reconstructed from memory, not from evidence.

The three pillars — logs, metrics, traces — are necessary and insufficient. They are storage and transport. Observability is the ability to ask new questions about behaviour you did not predict and get trustworthy answers fast. That requires instrumentation design, correlation, ownership, and culture — not another panel showing CPU.

Constrange reviews platforms that bought observability and still cannot answer: what happened to customer 7721 across six services between 14:02 and 14:07? If causality cannot be reconstructed, observability is monitoring with a larger invoice.

Dashboards measure comfort, not understanding

Dashboards aggregate what someone thought would matter during setup. They go green. Incidents break assumptions. Novel failures do not have panels. On-call scrolls familiar graphs while users report a journey that crossed services nobody charted together.

Logs without correlation are noise

Terabytes of unstructured logs do not shorten incidents. They lengthen searches. Correlation identifiers — trace id, request id, business ids like order id and payment id — must propagate across every hop, every async handoff, every batch reprocessor. Without them, logs are parallel monologues.

Structured logging helps. Schema discipline helps more. If each team names fields differently, joins remain archaeology. Platform libraries should enforce correlation propagation; code review should treat missing ids as a defect, not style.

• Trace id on every log line
• Business ids at domain boundaries
• Consistent field names for user, tenant, workflow
• Log the decision: retry, reject, idempotent skip
• Avoid logging secrets and payloads at info in production

Metrics that mislead

Average latency hides tail pain. Success rate hides partial degradation. Service-level metrics hide journey failure. A checkout dependency can be 'healthy' while one region's payment path fails because routing rules send a slice of traffic to a bad pool.

Use metrics for SLOs users feel: journey success, queue age, error budget burn. Complement with exemplars linked to traces. Aggregate metrics tell you something hurts; exemplars and traces show where.

Traces that stop at the boundary

Distributed tracing fails when instrumentation stops at service edges teams do not own. A trace that dies at the API gateway, the message broker, or the vendor callback is a cliff, not a map. Async work needs trace context injected into message headers and restored in consumers.

Minimum viable trace continuity

1. Auto-instrument ingress and egress in shared libraries
2. Propagate context through HTTP headers and message metadata
3. Span per significant domain operation, not only per framework call
4. Record error attributes and retry attempts on spans
5. Sample intelligently — keep errors and high-latency paths

“A trace that ends at the queue is a story that ends at 'and then something happened elsewhere'.”Constrange

Reconstructing causality under pressure

Incident response asks causal questions: what triggered, what amplified, what user impact, what changed. Observability succeeds when any engineer can pivot from symptom to timeline without tribal knowledge. That requires runbooks tied to identifiers, not runbooks that say 'ask the payments team'.

High-cardinality is a design choice

Teams avoid high-cardinality labels because cost spikes. Then they cannot filter by customer or order during an incident. Balance matters: guard cardinality in steady state; allow scoped diagnostic modes during incidents; use trace backends for detail, metrics for aggregates.

Cost conversations should happen at design time — not after instrumentation is forbidden. Observability budget is part of platform economics, like compute.

Ownership: who makes signals true

Central ops cannot instrument product journeys. Teams that ship services must ship spans, logs, and SLO definitions with them. 'Observability ticket' backlogs mean the platform is visible only where ops guessed correctly.

• Definition of done includes instrumentation
• Service catalog links to dashboards and SLOs
• On-call runbooks reference trace queries by business id
• Blameless post-mortems check signal gaps
• Platform provides libraries — teams adopt them

Testing observability before production fire

Chaos and game days validate more than failover. Inject latency into a dependency and verify traces show the amplification. Publish duplicate messages and verify logs show idempotent skip. Can a new engineer follow one order id end-to-end in staging? If not, production will not magically be clearer.

The three-dashboard anti-pattern

Dashboard one: infrastructure. Dashboard two: API golden signals. Dashboard three: business KPI. None links an error spike to a named customer journey. Executives see green. Support drowns. Engineering toggles between tools manually. This is not observability. It is decoration.

What to build instead

1. Journey-centric SLOs with error budgets
2. Trace-first incident runbooks for top five user paths
3. Log schemas enforced by lint or CI
4. Correlation test in contract tests between services
5. Weekly 'can we answer this question?' drill with real ids

From signals to organisational learning

Post-mortems should update instrumentation, not only runbooks. Missing span? Add it. Could not filter by region? Fix labels. Observability matures when incidents make the next incident cheaper to diagnose.

Constrange treats observability as constraint reading for runtime: where does work actually go, where does knowledge stop, where do handoffs lose context? The answers belong in architecture — not only in a vendor contract.

A twelve-week observability programme

1. Weeks 1–2: pick three critical journeys — map services and current signal gaps
2. Weeks 3–4: deploy correlation libraries and schema standards
3. Weeks 5–6: define SLOs and burn-rate alerts per journey
4. Weeks 7–8: trace async paths through broker and workers
5. Weeks 9–10: game day — diagnose injected failure using traces only
6. Weeks 11–12: post-mortem retrospective on signal quality metrics

Twelve weeks will not perfect the estate. It will show whether the organisation can reconstruct causality — or only watch dashboards go green.

Vendor tools do not replace design conversations

Observability vendors sell consolidation. Consolidation helps only after teams agree what must be visible. Buying a platform before defining correlation standards produces expensive grep. The design conversation asks: which identifiers matter, which journeys earn SLOs, which tool calls are audit-relevant, which spans must never be sampled away.

Constrange recommends instrumenting three journeys deeply before instrumenting thirty shallowly. Depth teaches patterns — field names, escalation queries, evaluable SLOs — that platform teams can encode in libraries. Breadth-first instrumentation creates dashboards that look complete and incidents that still wander.

Questions for your next platform review

• Can we follow one order id across all services in one query?
• Do traces survive our async boundaries?
• Which incidents failed because signals were missing — not because fix was hard?
• Are SLOs defined on user journeys or only on CPU?
• Who owns instrumentation for services shipped last quarter?

Logs, metrics, and traces are ingredients. Observability is the meal: the ability to reconstruct what happened, why it mattered, and who felt it — while the system is still on fire.

Stop when the next incident answers a causal question in minutes — not when the dashboard count reaches three.

Support tickets that begin with 'what happened to my order?' are observability requirements written in customer language. If engineering cannot answer from signals alone, the next purchase of tooling will not help — design will.