How to Build a Multi-Agent AI System: Production Architecture Decisions
A working multi-agent AI demo takes an afternoon to build. Getting a multi-agent AI system architecture into production, one that handles real data, real latency, and real failure modes, is a different undertaking.
The decisions you make before writing the first agent, on how agents communicate, who verifies their output, and what happens when they loop, determine whether you deploy a system or a liability.
Table of Contents
What a Multi-agent AI System is and What it is Not
A multi-agent system is not several chatbot windows running in parallel. Each agent is a discrete reasoning unit: it holds a defined set of tools, it reads or writes state, and it can delegate subtasks to other agents.
The distinction from a single LLM with multiple tools is precise. In a tool-augmented single LLM, one model receives a task, selects from its available tools, and returns a result.
In a multi-agent system, different agents run different models, hold different permissions, and fail independently. An orchestrator can decompose a task and hand it to a worker without knowing how that worker will complete it. That separation is what enables genuinely complex workflows.
What it is not: calling three prompts sequentially in a Python function. That is a chained pipeline, and it is a sound approach for many use cases. A true agentic AI architecture involves agents that reason about whether to delegate, what to delegate, and when to stop.
The honest test for whether your use case needs this pattern: does the task require parallel reasoning, cross-validation of outputs, or dynamic routing based on intermediate results? If not, a single well-prompted LLM call is almost certainly faster, cheaper, and easier to debug.
Multi-agent architecture earns its complexity in three situations: tasks too long for a single context window, workflows that require independent verification of outputs, and processes with genuinely parallel subtasks that can run simultaneously.
Document processing at scale, research-and-draft workflows, and automated decision pipelines with compliance requirements are the cases we see most often. Content summarisation, classification, and single-turn Q&A are almost never the right fit.
Agent Roles in a Multi-agent AI System: Orchestrator, Worker, Verifier, and Memory
A production system needs four roles assigned explicitly. They do not each require a separate process, but each must be deliberately designed.
1. Orchestrator
Receives the top-level task and decomposes it into subtasks. Routes those subtasks to worker agents. Does not execute tools directly. Use a capable model here, GPT-4 class or Claude Sonnet or Opus tier. Errors in decomposition cascade through the entire system, so this is not where to cut model cost.
2. Worker agents
They execute scoped, specific tasks. A research worker calls external search APIs. A data worker runs database queries. A drafting worker produces text from structured inputs. Workers can run on smaller, less expensive models where the task is narrow and well-defined.
3. Verifier agent
Cross-checks the worker’s output against the original task specification or a defined rubric. In production, this is not optional. Without verification, hallucinations and task drift accumulate silently. The verifier typically needs to match the orchestrator in capability, because its job is to catch the mistakes the orchestrator could make.
4. Memory agent
Surfaces relevant prior context. In the agentic sales workflow we deployed for a 200-person B2B SaaS company in 2024, the memory agent retrieves summaries of the last three prospect interactions before the orchestrator decides which follow-up action to take. This is typically a retrieval-augmented generation (RAG) layer backed by a vector store.
#Important
Model cost profiles matter here. The orchestrator and verifier together rarely account for less than 60% of token spend, even when workers handle the majority of tool calls. Plan for this ratio when scoping an autonomous AI agent workflow against a budget.
How agents pass state. In most production deployments, agents do not share a live memory space. The orchestrator passes a structured context object to the worker at each delegation step. That object includes the original task, relevant prior outputs, and any constraints the worker needs.
The worker returns a structured result object. The verifier receives both. Keeping each agent stateless between calls makes horizontal scaling straightforward and eliminates race conditions when multiple tasks run concurrently.
Context window management. Every agent call consumes context tokens. In a multi-hop chain, the orchestrator may pass accumulated prior outputs to maintain coherence, and that context grows with each hop.
On long or complex tasks it can exceed the model’s context limit, causing silent truncation of earlier reasoning. Design agent interfaces with a maximum input size in mind. The memory agent exists partly to solve this: rather than forwarding raw history, it retrieves a compressed, relevant summary.
Without this discipline, long-running multi-agent workflows degrade quietly as the model loses access to its own earlier decisions.
Production Framework Choices for Multi-agent AI Architecture: CrewAI, AutoGen, and LangGraph
These three frameworks solve the same core problem in meaningfully different ways. The wrong choice costs weeks of re-engineering.
1. CrewAI
It is built around role-based pipelines. You define agents as crew members with roles, goals, and constraints. It works well when agent responsibilities are fixed and the workflow is sequential or lightly conditional.
The abstraction is high enough that non-engineers can read the configuration and understand what is happening. A straightforward CrewAI enterprise deployment can be production-ready in a matter of days for well-scoped workflows.
The trade-off are complex conditional routing between agents requires workarounds, and deep observability into each agent step needs additional logging setup.
2. AutoGen(from Microsoft Research)
Autogen is designed for conversational multi-agent patterns. Agents exchange messages. A human-in-the-loop pattern, where an agent pauses and requests approval before taking a consequential action, is native to the framework rather than bolted on.
It fits workflows that are inherently dialogue-based: a research agent that refines its approach based on feedback, or an approval workflow that routes risky actions to a human reviewer. Teams that hit edge cases in simpler pipeline tools often find AutoGen handles them more cleanly, at the cost of more initial configuration.
3. LangGraph
These models agent interactions as a directed graph. States are nodes; transitions are edges with defined conditions. If your workflow has deterministic branching logic, “if the verifier returns a confidence score below 0.7, re-route to the worker with the failure reason appended,” LangGraph makes that explicit, testable, and auditable.
It has the steepest learning curve of the three, but it is the most debuggable in production because every state transition is an observable event you can replay.
Rough guidance: CrewAI for fixed role-based pipelines, AutoGen for conversational or human-in-the-loop flows, LangGraph for anything that requires precise state control and deterministic failure paths.
Biggest Failure Mode in Production Multi-agent AI Systems: Agent Loops and Token Spend
The most expensive mistake in production multi-agent AI system architecture is an agent loop with no termination condition.
An orchestrator that cannot complete a task re-routes it to a worker. The worker returns an inadequate result. The orchestrator tries again. The cycle continues until a cost alert fires, if you built one.
Before go-live, engineer four controls:
- Maximum iteration limits. Every agent call chain must have a hard cap on the number of hops. Set this at the framework level, not in the agent prompt. Prompts are not reliable guardrails.
- Token budget per task. Define a total token budget for each task type and track spend as the run executes. When the budget is exceeded, terminate and return a partial result or an explicit error.
- Spending limits at the API level. Anthropic, OpenAI, and Google all support hard spending limits at the account or project level. Set these before you run a single production task.
- Circuit breakers on tool calls. If a tool call fails three consecutive times, the agent stops and surfaces the failure. Build this into the tool wrapper, not the agent prompt.
Our POV
Last year, our AI integration company deployed a fraud detection triage system for a financial services company. The system used a dual-path agent architecture: one path for real-time card transaction triage, one for batch review of flagged accounts.
During testing, a failure in the external data enrichment API caused the verifier agent to loop 47 times before the run timed out. That single test run cost more than a full day of projected production spend. The fix was a three-strike circuit breaker on all external API calls and a 20-hop hard limit on the orchestrator.
Over the following 90 days in production, the system processed 30,000 fraud triage decisions, reduced false positive escalations to human review by 84% compared to the previous rule-based system, and had zero runaway cost events.
The lesson here is that loops are not an edge case. They are the default failure mode of any system where an agent can re-queue its own task.
How to Connect a Production Multi-agent AI System to Internal Systems
Every tool an agent can invoke is a potential blast radius. The design question is not “what can the agent do?” but “what is the worst action it could take on a bad or malformed instruction?”
Structure tool access in layers:
1. Read-only by default.
An agent with read-only access to your CRM, database, and file system can retrieve and analyse information without risk of modification. Start every new agent capability at read-only. Add write access only when the workflow explicitly requires it.
2. Approval gates for consequential actions.
Sending an email to a customer, updating a payment record, triggering a webhook: these need an explicit approval step before execution. In AutoGen, this is the human-in-the-loop pattern. In LangGraph, it is a conditional node that routes to a human review queue before any write occurs. Do not rely on a prompt instruction like “always ask before sending.” Build the gate at the framework level.
3. Audit logging on every tool call.
Log the agent ID, tool name, input parameters, output, and timestamp for every tool invocation. Store these logs separately from application logs and retain them. When something goes wrong in production, the tool call audit log is typically the only way to reconstruct the sequence of events. For any system connected to financial or customer data, this log is also a compliance requirement.
4. Scope permissions to the agent role.
A drafting worker agent should not have access to the send-email tool. Use separate tool sets per agent type, enforced at the framework level, not through prompting.
Our AI development services team builds these permission boundaries as a standard part of every agent deployment.
Production Observability for Multi-agent AI Systems: what LangSmith and Arize Phoenix show you
Running a multi-agent system in production without observability means learning about problems when users notice them or when the cost dashboard spikes. Both are too late.
1. LangSmith
It provides trace visualisation for LLM call chains. In a multi-agent run, you see the full call tree: which agent called which tool, the inputs and outputs at each step, token consumption per hop, and latency by agent. The token cost per agent hop view is the most useful for cost optimisation.
A research agent that consistently consumes 40% of the token budget on tasks that return thin results is a candidate for model downgrade or scope reduction. The LangSmith documentation covers the trace schema and query API in detail.
2. ArizePhoenix
This one is better suited to evaluation workflows. It runs structured evaluations against defined rubrics, accuracy, groundedness, task completion, across a sample of production runs. Sampling 5 to 10% of runs through an evaluator is typically sufficient to detect quality drift before it becomes visible to users.
A simplified trace from the fraud detection deployment shows what instrumented output looks like in practice:
{
“run_id”: “run_8f2a1c”,
“task”: “triage_transaction_TXN-442219”,
“agents”: [
{
“role”: “orchestrator”,
“model”: “claude-opus”,
“tokens_in”: 412,
“tokens_out”: 87,
“latency_ms”: 1240,
“action”: “delegated_to_worker:data_enrichment”
},
{
“role”: “worker:data_enrichment”,
“model”: “claude-haiku”,
“tool_calls”: [
{ “tool”: “lookup_merchant”, “status”: “success”, “latency_ms”: 340 },
{ “tool”: “check_velocity”, “status”: “success”, “latency_ms”: 280 }
],
“tokens_in”: 1840,
“tokens_out”: 612,
“latency_ms”: 2100
},
{
“role”: “verifier”,
“model”: “claude-sonnet”,
“verdict”: “pass”,
“tokens_in”: 2240,
“tokens_out”: 134,
“latency_ms”: 980
}
],
“total_tokens”: 5325,
“total_latency_ms”: 4320,
“outcome”: “escalate_for_review”
}
The metrics that matter most in operations:
- End-to-end latency by task type. Track P95 and P99, not just the average. Tail latencies reveal which task types are unpredictably slow.
- Cost per successful task completion. Separate from raw token spend. A system that uses more tokens but completes tasks at a higher rate may be more cost-efficient overall.
- Failure attribution by agent role. Aggregating failures by role identifies where to invest in reliability improvements.
- Tool call success rate. A tool that fails 15% of the time shows up as agent-level failures in your top-level metrics. Instrument the tools directly, not just the agents.
If you are building the observability layer alongside an agent deployment, our generative AI integration services team scopes that work in the initial project phase rather than treating it as a separate engagement.
If you are moving a prototype toward production, the highest-value work in the next two weeks is auditing your termination conditions, your tool permission boundaries, and your observability setup. Those three determine whether the system is safe to run. Everything else can be iterated once it is live.
Frequently asked questions
How many agents is too many in a multi-agent system?
There is no universal rule, but complexity scales with the number of possible agent interactions. A system with 10 agents where each can communicate with any other has up to 90 communication paths to reason about and debug. In practice, we rarely deploy systems with more than six to eight active agent roles. Beyond that, the architecture typically needs to be decomposed into separate, loosely-coupled workflows rather than one large system.
What is the cost difference between a single LLM and a multi-agent system at scale?
A multi-agent system typically costs three to eight times more per task than a single LLM call, depending on the number of agent hops and the model tier at each role. The value case is completing tasks a single prompt cannot: parallel subtask execution, cross-verification, and dynamic routing based on intermediate results. If a task fits cleanly in a single well-prompted LLM call, use that instead. Multi-agent architecture earns its overhead when the workflow has genuine complexity that a single call cannot handle reliably.
Can multi-agent systems be deployed on-premise?
Yes, with one important constraint. The framework itself, LangGraph, AutoGen, or CrewAI, runs on-premise without issue. The constraint is the model. If you use cloud LLM APIs, data leaves your environment on every agent call. A genuinely on-premise deployment requires a self-hosted model: Llama 3, Mistral, or a fine-tuned variant on your own infrastructure. Orchestration quality typically drops relative to frontier models, and that gap has the largest impact on the orchestrator and verifier roles.
How do you test a multi-agent system before go-live?
Test each agent in isolation first: fixed inputs, defined expected outputs, consistent results across multiple runs. Then test agent-to-agent handoffs with known inputs. Then run end-to-end tests with adversarial inputs, what happens when a tool fails, when the task is ambiguous, when the task cannot be completed. Load test with at least 50 concurrent runs to confirm the system degrades gracefully rather than spiralling into loops under contention.
