MCP Servers in Production: Architecture, Security, and Reliability

AI agents show up in vendor decks and architecture reviews, often sold as a capability upgrade or a model selection problem. The conversation stops at demos. Constrange treats production MCP servers as an engineering discipline: observe what systems actually do in your environment, measure the cost and risk of that behaviour, and model the gap between the prompt and the platform that makes it safe to run.

Observe: map every MCP server — where it runs, who owns it, dependency chain to backends, and current SLO if any. Include IDE-local servers accidentally used for batch jobs.

Measure: tool p95 latency, error rate, availability, deploy frequency, and blast radius per server. Agents amplify downstream outages — track fan-out.

Model: MCP servers are microservices with agent-specific traffic patterns — bursty, verbose payloads, retry-heavy callers. Apply service mesh discipline.

From laptop to datacenter

MCP began as a developer convenience — local servers wrapping filesystem and git. Production agents need always-available tools backed by enterprise systems. That shift requires everything microservices needed a decade ago: deploy pipelines, secrets management, on-call, and capacity planning.

Constrange sees teams promote the same laptop config to production via a hurried Dockerfile. Missing: health checks, graceful shutdown, connection pooling, and backoff when backends throttle.

Treat MCP promotion as a service launch, not a config tweak.

Topology and scaling

Deploy MCP servers close to their backends — same VPC, low latency. Agent hosts may be remote; tool latency dominates task time. Scale horizontally for read-heavy tools. Queue mutating tools if backends cannot handle agent retry storms.

Separate servers by blast radius: read-only search vs write-capable admin tools. Policy routes agents to the right server class.

Measure concurrent sessions per server. MCP may hold connections open — load balancers need appropriate timeouts.

1. Health endpoints checking downstream connectivity
2. Circuit breakers on failing backends
3. Idempotent handlers for mutating tools
4. Graceful degradation — partial tool availability

Security and tenancy

Multi-tenant agents require tenant context propagated to MCP servers. Row-level security in tools, not only in apps. Test cross-tenant isolation explicitly.

Rotate credentials without redeploying agents — use secret broker integration. Break-glass access logged and time-bounded.

Rate limit per tenant to prevent one customer from exhausting shared tool capacity.

Reliability and SLOs

Define SLOs on tool success and latency — agents need error budgets too. Alert on error budget burn. Page owners who registered the server in the tool registry.

Chaos test: disable a backend and observe agent behaviour. Fail closed or degrade gracefully — never infinite retry into outage.

Version MCP manifests. Breaking schema changes should be dual-run with deprecation windows — same as REST APIs.

“An MCP server without an owner is a production incident with no runbook.”Constrange

Operating model

Register every server in the platform catalogue with owner, SLO, dependencies, and data classification. Onboarding a new tool is a ticket to the owning team — not a silent YAML edit.

Run game days for high-traffic agents pulling many tools. Fan-out amplifies small MCP outages into task failures.

FinOps: attribute MCP compute and downstream API cost per agent. Right-size instances after measuring real load — agents are spiky.

When not to run MCP

If only one agent needs one REST API, a thin wrapper in-process may beat operating a server. MCP earns its ops cost at reuse and standardisation scale.

Questions for your next review

• What would we see if we measured production MCP servers for one week in production?
• Which dependency or tool call owns the majority of failures or cost?
• What assumption in our agent design does the telemetry contradict?
• What is the smallest experiment that would change our operating model?

The teams that run production MCP servers reliably are not those with the best prompts. They are those that observe honestly, measure without vanity metrics, and update the control plane when traces disagree with the architecture diagram.

Load test MCP servers with agent-like retry patterns — not steady single-thread traffic.

Publish status pages for critical MCP dependencies when customer agents rely on them.

Automate manifest validation in CI — broken schemas should not reach production.

Keep canary deploys for MCP servers the same as for APIs.

Document maximum payload sizes — agents can generate verbose arguments.

Practice credential rotation quarterly without agent downtime.

Production agent systems fail in boring ways long before they fail in interesting ones. Timeouts, stale credentials, ambiguous tool schemas, and missing idempotency keys dominate incident logs. Measure those first. The model is rarely the first place to spend another sprint — the integration edge is.

Platform maturity shows up in how teams talk about agents in incident review. Immature teams debate prompts. Mature teams read traces, attribute cost, and ask which policy gate should have fired. Build the habits and tooling that make the second conversation inevitable.

Challenge vendor claims with your own telemetry. Demos optimise for completion; your customers optimise for correctness under load. Run the same task at ten times traffic with injected tool failures. If the architecture cannot explain behaviour post hoc, it is not ready for production traffic.

Agent roadmaps should include explicit kill criteria. If unit economics do not improve after guardrails, or incident rate exceeds threshold, retire the agent. Keeping a failed experiment running because sunk cost is familiar is how organisations accumulate operational debt.

Cross-functional ownership beats a single 'AI team' holding every agent. Domain teams own outcomes; platform teams own control planes; security owns policy patterns; FinOps owns attribution. Without divided ownership, agents become a bottleneck service nobody can scale.

Treat eval datasets like test fixtures in a payments system — versioned, reviewed, and protected from casual edits. A polluted golden set hides drift until customers report it. Measure eval coverage the same way you measure code coverage: necessary, imperfect, and better than nothing.

Documentation for agents must include failure behaviour, not only happy paths. What happens at budget cap? At policy denial? At tool timeout? Operators need runbooks, not README enthusiasm. Measure mean time to diagnose using only published docs — embarrassing results drive better writing.

Regulatory and privacy constraints belong in architecture, not in post-hoc legal review. Data residency, retention, and right-to-erasure apply to agent memory and logs. Measure what personal data enters prompts and tool payloads weekly. Surprises here are expensive in every sense.

Load test MCP servers with agent-like retry patterns — not steady single-thread traffic.

Publish status pages for critical MCP dependencies when customer agents rely on them.

Automate manifest validation in CI — broken schemas should not reach production.

Keep canary deploys for MCP servers the same as for APIs.

Document maximum payload sizes — agents can generate verbose arguments.

Practice credential rotation quarterly without agent downtime.

Production agent systems fail in boring ways long before they fail in interesting ones. Timeouts, stale credentials, ambiguous tool schemas, and missing idempotency keys dominate incident logs. Measure those first. The model is rarely the first place to spend another sprint — the integration edge is.

Platform maturity shows up in how teams talk about agents in incident review. Immature teams debate prompts. Mature teams read traces, attribute cost, and ask which policy gate should have fired. Build the habits and tooling that make the second conversation inevitable.

Challenge vendor claims with your own telemetry. Demos optimise for completion; your customers optimise for correctness under load. Run the same task at ten times traffic with injected tool failures. If the architecture cannot explain behaviour post hoc, it is not ready for production traffic.

Agent roadmaps should include explicit kill criteria. If unit economics do not improve after guardrails, or incident rate exceeds threshold, retire the agent. Keeping a failed experiment running because sunk cost is familiar is how organisations accumulate operational debt.

Cross-functional ownership beats a single 'AI team' holding every agent. Domain teams own outcomes; platform teams own control planes; security owns policy patterns; FinOps owns attribution. Without divided ownership, agents become a bottleneck service nobody can scale.

Treat eval datasets like test fixtures in a payments system — versioned, reviewed, and protected from casual edits. A polluted golden set hides drift until customers report it. Measure eval coverage the same way you measure code coverage: necessary, imperfect, and better than nothing.