Designing systems that know when they don't know

Autonomous systems are judged on what they do when they know the answer. Production judges them on what they do when they do not. Most failures are not wrong answers delivered confidently — they are actions taken past the boundary of valid input, stale context, or missing policy.

Systems that know when they do not know escalate, defer, or choose safe defaults instead of guessing. That capability is not a model feature alone. It is architecture: confidence gates at decision points, explicit unknown states, human paths that are faster than failure, and governance that rewards restraint.

Constrange designs autonomous and agentic systems with uncertainty as a first-class output — not an embarrassment to hide behind fluent prose.

Uncertainty is a signal, not a bug

Traditional software returns errors. ML and agent systems return completions. A completion looks like success even when it is fabrication. Designing for uncertainty means the system can output: I do not have enough context; this request is out of policy; confidence is below threshold; escalating to human. Those outputs must be as engineered as the happy path.

Confidence gates at decision points

A confidence gate is an enforced check before action: classify input, score match to known patterns, verify prerequisites, compare to policy rules. If any check fails or score falls below threshold, action does not proceed. The gate is code and policy — not a system prompt that says 'only act if sure'.

Gates belong at boundaries: before tool invocation, before production mutation, before sending external communication, before closing an incident. Each boundary has different thresholds. Reading documentation can tolerate lower confidence than deleting a namespace.

• Separate read and write thresholds — writes stricter
• Require corroboration from two signals for irreversible steps
• Log confidence scores and gate decisions for audit
• Test gates with adversarial and out-of-distribution inputs

Escalating unknowns without shame

Organisations punish escalation culturally while demanding autonomy technically. On-call learns to make the agent 'just handle it'. Developers disable gates to unblock demos. Escalation must be fast, cheap, and credited — not a failure ticket.

Design escalation routes with context bundles: what was attempted, what was uncertain, which policies conflicted, suggested next steps for the human. Humans should not re-research from scratch. Unknown is handoff, not reset.

Escalation SLAs are part of the system

If humans take four hours to respond, autonomous systems will absorb risk to avoid waiting. Escalation paths need owners, paging, and timeouts — same as any critical dependency. An agent blocked on human review is healthier than an agent guessing through the block.

Safe defaults: what to do when you cannot know

Safe defaults are pre-authorised behaviours for uncertainty: deny write, read-only probe, queue for batch review, apply least-privilege template, return partial answer with explicit gaps. Defaults must be boring and reversible. Creative defaults under uncertainty are incidents.

Defaults are policy decisions, not engineering afterthoughts. Security wants deny. Product wants defer. Operations wants alert. Align before deployment — not in post-mortem. Document which default applies per action class and who can override.

Agents and the illusion of understanding

Agents compound uncertainty because they chain steps. Step one misclassifies; step two acts on the misclassification with high tool confidence because the tool executed successfully. Success of execution is not success of intent. Gates must exist between steps, not only at the entrance.

Tool use should require explicit intent objects: what I believe I am doing, why, expected outcome, confidence per field. Downstream tools validate intent against capability. Mismatches escalate. This is tedious engineering. It is cheaper than undoing a production change nobody authorised.

“A system that cannot say 'I don't know' will eventually say 'I did' — about the wrong thing.”Constrange

Governance that rewards restraint

Metrics drive behaviour. If agents are scored only on tasks completed, they will complete tasks past certainty. Add metrics: escalation rate, gate blocks, human override frequency, incidents avoided by deferral. Review false negatives — escalations that wasted time — but do not optimise escalation to zero.

Policy must be machine-readable. Agents cannot know they do not know policy that lives in PDF. Encode rules, test them, version them. When policy changes, gates update together — not via retraining hope.

Testing for unknowns

Test suites full of happy paths prove nothing about uncertainty handling. Add chaos for context: missing fields, contradictory instructions, stale data, prompt injection, tools returning partial errors. Assert the system escalates or defaults — not that it produces fluent text.

1. Define confidence thresholds per action class in policy, not prompts
2. Implement gates in code between agent reasoning and tool execution
3. Build escalation bundles with context, not bare notifications
4. Choose safe defaults per class with named owners
5. Measure escalations and gate blocks as health signals

Human override without bypass culture

Overrides are necessary for edge cases. Overrides that skip gates without audit recreate shadow automation. Log who overrode, why, and what gate fired. Review overrides weekly. Patterns indicate bad gates, bad training data, or bad policy — not 'users who do not trust AI'.

Autonomous systems in regulated and operational contexts

Finance, health, and infrastructure contexts cannot tolerate confident wrong. Regulators ask for evidence of control — not model size. Uncertainty handling is control: provable gates, auditable escalations, defaults aligned to policy. Fluency is not compliance.

Constrange often finds the missing piece is not a better model but a defined unknown state in the workflow — a status incident tools understand, a queue humans monitor, a default that denies until Monday when certainty is impossible Friday night.

Mixed autonomy on the same platform

Most estates run several autonomy levels at once: fully manual runbooks, scripted remediation, workflow-gated changes, and experimental agents. Uncertainty handling must be consistent across levels — not strict for the agent and loose for the legacy webhook that still restarts production from chat.

When a human override bypasses a gate for the agent but not for the script, teams learn which path is easier, not which is safer. Align thresholds by action class, not by tool brand. A delete is a delete whether triggered by kubectl, Terraform, or an LLM with a tool adapter.

Observability for uncertainty

Instrument unknown states the way you instrument errors: rate of gate blocks, escalation depth, time waiting on human, override count, and actions taken after override. Dashboards that only show success rate hide the compensation behaviour that precedes failure. On-call should see 'stuck in uncertain' as clearly as 'stuck in retry'.

Questions for your autonomous system design

• What can the system do when confidence is below threshold — besides guess?
• How fast can a human receive context and decide?
• Which actions have deny-by-default under uncertainty?
• Where do chained steps re-validate intent?
• What metric rewards escalation instead of punishing it?

Systems that answer well build trust slowly. Systems that stop before harm build trust that survives the first incident.