Inside a Production Multi-Agent GenAI System

A multi-agent system is a distributed system with probabilistic components. All the old distributed systems failures still apply, and now you also have semantic failures to deal with. In this post we'll follow one user request from the gateway to the final answer, stopping at every component on the way. At each stop we'll look at what breaks in production, why it breaks, and the engineering fix. Then we'll finish with where candidates struggle when this comes up in an interview.

One user request can turn into twenty or more model calls across five agents, and any of them can return in 200 milliseconds with an answer that is well formed, on schema, and wrong. This post follows a single request through a production multi-agent system, component by component. The edge, the orchestrator, the agents, the tools, the shared state, and the merge. For each one we'll go through the failure modes that only show up under real traffic and the fix that closes them. We'll finish with the questions that catch people out in interviews.

A distributed system with probabilistic parts

Draw a multi-agent system on a whiteboard and it looks friendly. A box that plans, a few boxes that do the work, some tools on the right. Run the same design against real traffic and it behaves like every distributed system you have operated:

• Calls time out: a tool or a sub-agent hangs, and something upstream gives up before it answers.
• Two writers race each other: two agents update the same record, and one of them silently overwrites the other. These are lost updates.
• A retry duplicates a side effect: the first attempt actually succeeded, the response was just lost, so the refund goes out twice.
• Load multiplies: one user request becomes six model calls and roughly a dozen tool calls, so a service that was given enough capacity for one call per request now gets twelve.

One part of this system does not behave like the rest. A model is probabilistic, which means it gives you a likely answer, not a correct one and not an error. So a sub-agent can reply in 200 milliseconds with something that is well formed, passes its schema check, and is wrong. There is no status code for wrong. Your dashboards stay green, and the user reads a confident paragraph built on a number that was never in any source.

Every agent you add imports the entire distributed-systems problem set. Partial failure, retries, lost updates, backpressure, and cost fan-out all arrive together. The only thing that reliably pays for that bill is context isolation, and we'll come back to why.

So we'll follow one request all the way through, stopping at each component. The edge, the orchestrator, the agents, the tools, the shared state, the merge, and the observability around all of it. At each stop we'll name what breaks under real traffic, why it breaks, and the fix.

A few things are deliberately left out because they already have their own posts here. The general shape of an agent loop is in the technical architecture of agentic AI. Caching, reliability, and the surrounding infrastructure are in the production AI stack. Latency and availability targets are in non-functional requirements for AI apps at scale. What goes into the window is in context engineering. This post is about what happens when there is more than one agent.

The example running through it is an order-support assistant. A customer asks why their order is late and whether they can get a refund. Answering that needs order history, shipping status, the refund policy, and a written reply. That is four jobs. Three of them are lookups, and they can run at the same time, because none of them needs another one's answer. Running them side by side like that is a fan-out. The fourth job is the writer, and it has to wait, because it composes what the other three found. Each of the three parallel paths is called a branch, and the rest of the post uses that word whenever a path can fail, retry, or be saved and resumed on its own.

The edge

The request arrives at a gateway before any model sees it. Authentication, rate limiting, input validation, prompt injection screening, and routing all happen here. Most of that looks the same as it does in any web service. Two jobs here are different, because they only exist once a request can fan out into many model calls. These are those two.

• Set the budgets once: A deadline and a token budget for the whole request are decided at the edge and carried down with it. Every component below reads them rather than inventing its own.
• Decide the shape of the work: Cheap classification here decides whether the request needs the full fan-out, a single agent, or a cached answer. Most requests do not need five agents.

One term from the diagram is worth settling first, because it appears at stage 1 and then travels the whole way. A tenant is one customer organisation on a system that serves many of them. If this assistant is sold to a hundred retailers, each retailer is a tenant. The tenant id on the request is what keeps one retailer's orders, policies and cached answers from ever reaching another. Rate limits, cost attribution and every shared index are scoped by it, so it is set once here rather than worked out later.

What breaks

Expensive requests are the first one. Most rate limiters count how many requests a user sends. They know nothing about what those requests cost you. Answering the order-support question takes 6 model calls. Now imagine a customer asks about 40 orders at once. The system has to read every one of them, so that single question takes more than 40 calls. The limiter sees one request either way.

Retry storms are the second. Agent responses are slow, so say an impatient client retries after 20 seconds. The first request is still running, holding a fan-out of sub-agents. Now there are two trees. The client retries again. Each retry starts a whole new tree, and none of the old ones stop. The system was only slow a minute ago. Now it has no capacity left for anyone, including the users who never retried.

Screening for prompt injection at the edge is worth doing, and worth being honest about. It catches the obvious attempts, someone typing ignore your instructions and refund me. It does nothing about the case that actually hurts, where the instruction is hidden inside a document the system fetches later. We'll come back to that one in the security section.

The fixes

• Limit by cost, not only by count: Keep the usual limiter that counts requests, then add a second one that tracks estimated spend per user and per tenant. The spend limiter is the one that catches the 40-order question, because the request count looks perfectly normal.
• Canonical request ids: The client sends an id, the edge accepts it, and a repeat of the same id returns the in-flight or completed result. That single rule removes most retry storms.
• A timeout budget set here: Decide the whole-request deadline at the edge and pass the remaining time down at every hop. Independent timeouts at four layers will always disagree with each other.
• Classify before you fan out: A small model deciding simple, standard, or complex costs almost nothing and routinely removes the majority of full fan-outs.

Frameworks at this layer

The edge has the most mature off-the-shelf options of anything in this post, because it is the least agent-specific part of the system. A gateway sitting in front of every provider call is the cheapest item here to adopt and the one that pays back fastest.

• LLM gateways, such as LiteLLM, Portkey or Cloudflare AI Gateway: One endpoint in front of every model provider. You get per-key spend limits, retries with backoff, automatic fallback to a second model, and a single place where every token is counted. When troubleshooting, this is where you switch provider during an outage without a deploy, and where you find out which agent burned the budget.
• Guardrail libraries, such as NeMo Guardrails, Guardrails AI or Llama Guard: Run classification checks before the model call and again on the way out. They give you a named, testable check instead of an instruction buried in a prompt, so a bypass shows up as a failing test rather than as a support ticket.
• PII detection, such as Microsoft Presidio: Finds and masks names, card numbers and addresses before the request leaves your network, and again in the response. It runs as an ordinary library, so the same call works in the logging path as in the request path.
• Semantic caching, such as Redis vector search or GPTCache: When a new question is close enough to one already answered, the stored answer is served at stage 2 and the whole fan-out is skipped. It is also a trap worth knowing about. If the cached answer is out of date, users report bad answers, you go looking at the model, and the model was never involved.

Tools named in this post are ones in common production use as of mid-2026, and this is the fastest-moving part of the field. Learn the category each one belongs to, because the categories outlive the names. A team that understands why it needs a gateway can swap one gateway for another without redesigning anything.

Put the token budget and the deadline in the same object that already carries the trace id, so all three travel together. Keep them in separate variables and sooner or later someone writes a new code path that passes the trace id along and quietly drops the other two. That path will not be the one you test. It will be the one that runs when traffic is heaviest.

The orchestrator

The orchestrator turns the request into a plan, hands parts of it out, and decides when the work is done. Four topologies cover almost everything built in practice, and the differences between them are about who holds the context and who decides what happens next.

For the order-support assistant, the supervisor pattern fits. The three lookups are independent, they can run at once, and none of them needs to see another one's working. The writer runs after them, because it composes what they return. A sequential pipeline would be slower and would let a wrong shipping answer poison the refund decision. A hierarchy would add a layer for no gain at this size.

What breaks

The orchestrator context window is the bottleneck of the whole system, and this surprises people. Assume four agents each return two thousand tokens of careful work. The supervisor now holds eight thousand tokens of worker output on top of the plan, the tool schemas, and the conversation. Add a second round and the supervisor is holding more than it can use well. Its window is not full, but the part it reasons over reliably is. So the merged answer gets worse while every worker is still doing good work, and nothing in the system reports a problem.

Plan drift is next. The plan is written once and then updated by whatever comes back. Ten steps later the system is answering a question adjacent to the one that was asked, because each individual step was a reasonable response to the step before it. Nothing failed. The goal moved.

Infinite delegation is the failure that shows up on the bill. A supervisor delegates to a sub-agent that decides the task is large and delegates further, and in a hierarchy this can cycle. Without a hard cap it runs until the budget, a rate limit, or an operator stops it.

Lost updates land here too when the topology is a blackboard. Two agents read the shared plan, both amend it, and the second write erases the first. Neither agent gets an error, so the only evidence is a plan that has quietly lost a step.

The fixes

• Hard caps on depth and iterations: A maximum delegation depth and a maximum loop count per request, enforced in the runtime rather than requested in a prompt. A prompt is a suggestion, and the runtime is not.
• An explicit termination condition: Write down what finished means before you build the loop. All required fields populated, or the deadline reached, or the budget spent. A loop with no written stop rule always stops for the wrong reason.
• Plan checkpointing: Persist the plan and its progress after every step. A crash or a timeout then resumes from the last good step instead of rerunning the tree, which is also what makes a retry cheap. The branches in the diagram are the order, shipping, and policy lookups.
• Budget propagation: Pass the remaining deadline and the remaining token budget into every delegation. An agent that knows it has 6 seconds left will return its best partial answer. The same agent, assuming it has 30, will start another tool call and get cut off with nothing to show.
• Compress at the return: Workers return a summary against a schema, with the full output written to storage and referenced by id. The supervisor reads the summary and fetches the detail only when it needs it.

Frameworks that orchestrate

This is probably the most crowded category, and also the hardest choice to undo. Once you build on a framework, it shapes how state is stored, how steps connect, and how your workflow runs. Don't compare them by asking what kinds of agents they support, they can all build essentially the same agents. Compare them by asking what you're left with when a production run fails halfway through.

• LangGraph: Models the system as an explicit graph with shared state and a checkpointer. Because state is persisted per node, a failed run resumes from the last good step instead of restarting, and you can pause for a human approval mid-graph. That checkpoint is also the artefact you read when debugging, since it shows the exact state the failing step saw.
• OpenAI Agents SDK and the Claude Agent SDK: Lighter, with handoffs, sessions and guardrails as first-class ideas rather than things you assemble. Good when the topology really is a supervisor with a few workers, which covers most products.
• CrewAI and AutoGen, now AG2: Role-based and conversational styles. Fastest to a working demo, and the place teams most often hit the termination problem, because a conversation between agents has no natural end unless you impose one.
• LlamaIndex Workflows and Pydantic AI: Event-driven and type-first respectively. Pydantic AI is worth a look if the handoff contracts in the next section appeal to you, since it treats the typed result as the centre of the design.
• Temporal or Restate, underneath any of the above: Durable execution for the case where a run spans minutes, calls things that charge money, and must survive the process dying. This is the answer to multi-step side effects rather than anything in the agent frameworks themselves.

Pick the orchestration framework on its failure story, not its demo. Ask how a half-finished run is resumed, where the plan is stored, and what stops the loop. Every framework looks the same on the happy path.

A prompt that says stop when you have enough information is not a termination condition. It works in testing and fails on the request that is genuinely ambiguous, which is the request most likely to be expensive.

The agents themselves

Each agent gets one job, one set of tools, and one clean context. Counting them in the diagram, the order-support system has four sub-agents. Three run the lookups at stage 4, an order agent, a shipping agent and a policy agent, and the writer agent at stage 5 composes the reply. Above them sits the orchestrator, which is an agent as well, so the system runs five agents in total. It is easy to miss in the count, because it never appears in the fan-out row. It is also the one that costs the most, since it runs twice on the top-tier model at stages 3 and 6. The temptation is to give each one a personality and a long backstory. What matters is much smaller, the scope of the job, the tools it can call, and the exact shape of what it must return.

Why more than one agent at all

Answer this one honestly, because the three reasons people usually give are weaker than they sound.

• Speed: Running branches at once does help, but only a little. You still wait for the slowest branch, so three agents are not three times faster.
• Specialised prompts: Useful, but you do not need separate agents for it. One agent with clear instructions can usually do the same job.
• Modularity: This is about how your code is organised. You can split code into clean modules and still run it as one agent.

The reason that holds is context isolation. The policy agent works better when its window contains the refund policy and nothing else. The shipping agent works better without three thousand tokens of policy text it will never use. One window carrying all four jobs degrades on all four, and it degrades quietly, so you find out from user complaints rather than from an error.

So before you split, name the context that was overloaded and say which job made it worse. If you cannot, build one agent and give it more tools instead. That version is far simpler, and none of the failures described in the rest of this post can happen to it.

The policy agent works a little differently from the other two, and the full architecture diagram at the top of this post shows it. It never holds the whole refund policy in its prompt. The policy runs to tens of pages, and only a few paragraphs matter for any one question. So it fetches the relevant part per question, in four steps.

• Embed the question: Turn the customer question into a vector, the same way the policy text was turned into vectors when it was indexed.
• Vector search: Pull the nearest passages out of the vector store that holds the policy corpus. Fast, and approximate.
• Rerank: Score those candidates again with a slower, more accurate model, because nearest is not the same as most relevant.
• Top-k chunks: Keep the best few, the top-k, and put only those in the window. Everything else stays out.

That is retrieval, and it is a large subject with its own posts. What matters here is that the window stays small and every claim points back at a passage.

The half of retrieval that nobody draws

Those four steps are the read path. The corpus they read has a write path, and that is where the production failures live. Nothing in the four steps can tell you whether the passage it found is still true. For a refund assistant that is not an abstract risk. The policy changed last month, the index did not, and the system quotes the old rule. It cites a real passage while doing it, and refunds the wrong amount with a clean trace behind it.

• Seed it with a pipeline, not a script someone ran once: The job that chunks, embeds and upserts the policy documents is the same job that runs on every later update. A corpus built by hand in a notebook cannot be rebuilt when the embedding model changes, and nobody will remember which version of which document went in.
• Reindex from the source, on a change event: The system of record for the policy, whether that is a document store or a CMS, emits a change and the indexing job runs. A nightly full rebuild is an acceptable second best. What fails is any design where a human is expected to remember to reindex.
• Track index age as a metric, and alarm on it: Freshness is invisible in every dashboard by default, because a stale index is fast, healthy and green. Record when each document was last indexed, alert when the oldest crosses your threshold, and put the effective date into the chunk so the agent can see it and say it.
• Delete superseded versions, do not rely on the model to choose: The worst case is the old clause and the new clause both scoring well and both landing in the window. The model then picks, confidently, with no way to know which is current. Carry a version and an effective date as metadata, filter to the current one in the query, and tombstone what is retired.
• Make deletion cascade: Removing a source document has to remove its vectors and anything derived from them. Orphaned vectors keep answering questions about a document that no longer exists, and this is also what a deletion request from a customer legally requires you to be able to do.
• Rebuild the whole index when the embedding model changes: Vectors from two different embedding models cannot be compared, so a model upgrade is a full re-embed rather than an incremental one. Build the new index alongside the old, evaluate it, then switch the alias. A partial migration produces a silently worse retriever with no failing step.
• Treat chunking as a design decision, not a default: A refund clause split down the middle retrieves as two half-rules, and the answer built on it is wrong in a way that reads perfectly. Chunk on the document's own structure, clause or section, and keep enough surrounding context that a passage still means what it meant in place.

Two more things matter here.

Put the tenant id inside the search query. One index holds the policies of every retailer you sell to. When retailer A asks a question, the search itself must be told to look only at retailer A's documents. Do not search everything and drop the wrong rows afterwards. By then you have already pulled another retailer's text into your system, and that is a data breach whether or not it reaches the answer.

Test the search on its own. Write down about fifty real questions, and next to each one write which passage should come back. Run the search against that list and count how often the right passage appears. No agent, no answer, just the search.

Skip this and you only ever see the finished answer, which tells you very little. A wrong answer looks identical in both of these cases. The search found the wrong passage, or the search was fine and the writer misread it. You cannot tell which from the answer alone, so teams rewrite prompts for weeks to fix what was a chunking problem all along.

A stale corpus is the most dangerous failure in this post, because every signal says the system is healthy. The retrieval succeeded, the citation is real, the trace is green, and the answer is wrong. Index age is the metric that catches it, and almost nobody has it on a dashboard.

The handoff is the contract

Agents that hand each other free-form text, plain sentences instead of typed fields, are the single most common source of quiet failure in these systems. Free-form text has no schema, so nothing can reject it, and a confident sentence carries exactly the same weight as a checked fact.

Error propagation is what the diagram is really about. One agent invents a figure. The next agent has no way to know, so it reasons from it. The writer produces a fluent answer built on it. Every span in the trace is green, every step returned in good time, and the output is wrong. Add a third hop and small errors compound, because each agent adds its own uncertainty to something it has already accepted as true.

The fixes

• Validate at every boundary: Parse the output against a schema before it goes anywhere. Reject and retry once on a failure, then fall back. This is cheap, and it converts an invisible semantic problem into a normal error you can count.
• Carry provenance in the schema: Every claim comes with a source id and a confidence value. A claim with no source is a rejectable output rather than a sentence, which is the whole point of the contract.
• Compress on the way out: Summarise to the fields the next agent needs, and keep the full text addressable by id. This protects the receiving context and makes the handoff auditable at the same time.
• Add a verification stage where it pays: A second agent that scores the work against fixed criteria and sends it back, which is the evaluator-optimiser pattern. Worth the extra calls when the criteria can be written down, and wasted when they cannot.
• Version the schemas: Two agents shipped by two people drift apart the moment one adds a field. Version the contract and fail loudly on a mismatch, the same way you would with any service boundary.

Write the contract as a schema, then validate it at the boundary. Reject a bad handoff, retry once, and degrade on the second failure.

Claude Code's subagent model is a public example of the shape. A subagent runs with its own context and returns a report to the main thread, so the main context stays clean and the detail lives in the subagent. The reason it works is the reason above, and none of it depends on the agents being clever.

Frameworks for contracts and retrieval

The tools below do two different jobs. The first group checks what an agent hands over, so a bad handoff is rejected instead of passed along. The second group runs the search that finds the right policy paragraphs.

• Pydantic, with the provider structured-output modes: The contract in the code block above is a Pydantic model, and every major provider can be asked to emit output conforming to a schema. A rejection is then a normal validation error you can retry on, which is what turns a silent wrong answer into a visible failure.
• Instructor, Outlines or XGrammar: Constrained decoding, which restricts what the model may emit token by token so malformed output cannot be produced in the first place. Useful when a schema is complex enough that retrying on validation errors gets expensive.
• LlamaIndex or Haystack for the retrieval pipeline: The four steps drawn above, embedding through top-k, assembled for you including the chunking of the policy documents. Worth using rather than writing, because the boring parts are where retrieval quality is won.
• A vector store, such as pgvector, Qdrant, Weaviate or Pinecone: Where the indexed policy lives. If you already run Postgres, pgvector keeps the corpus in the database you already back up and already know how to operate, which matters more at this size than raw search speed.
• A reranker, such as Cohere Rerank or an open bge-reranker model: The third retrieval step. When the policy agent cites the wrong clause, this is almost always the component to look at before blaming the model, because the model can only reason over what it was handed.

Tools and the outside world

Tools are where an agent system stops being a text generator and starts changing things. The model picks a tool from its description, so the schema matters more than most teams expect. A vague argument description produces wrong calls far more often than a weaker model does. Standard interfaces such as the Model Context Protocol make the wiring easier and change nothing about the failure modes below.

The split that matters is between tools that read and tools that write. A search is safe to retry, safe to run in parallel, and safe to hand to an agent processing untrusted text. A refund is none of those things.

There is one boundary here that is easy to miss. Anything a tool returns gets put into the agent context, and that context is sent to the model provider on the next call. So if the order lookup returns a full customer record, the address and the card details have left your systems already, whether or not any of it shows up in the reply. Redact tool results on the way in, not only the finished answer on the way out.

What breaks

Silent tool failures come first. A tool catches its own exception and returns an empty string or an empty list. The model reads that as a finding, so the answer becomes there are no matching orders rather than the lookup failed. A missing result and an empty result look identical to a language model, and only one of them is true.

Non-idempotent retries come second. Say the refund tool takes 8 seconds, the client-side timeout is 5, and the framework retries automatically. The first call succeeded. The customer is refunded twice. Nothing in the trace is red, because both calls returned 200.

Slow tools are the third. Most calls to a service are fast, but a small share are far slower, and that slow share is what a fan-out picks up. Assume the shipping API answers in 400ms almost always, and takes 12 seconds for 1 request in 100. Your fan-out waits for every branch before it can merge, so 1 request in 100 takes 12 seconds however quick the other branches were. Run five branches and you have five chances to hit somebody's bad tail.

Multi-step side effects are the fourth. A plan books a courier, charges a fee, and updates the order. The third step fails. Two changes are now live in two systems, and there is no transaction across them to roll back.

The fixes

• Errors are data the model must acknowledge: Return a typed error object with a status and a message, never an empty result. The agent can then say the lookup failed, which is a true statement and a recoverable one.
• Idempotency keys on every write: The caller generates a key derived from the request id and the step. A repeated call returns the original result. This is the difference between a safe retry and a duplicate refund.
• Segregate read and write tools by role: Read-only tools go to the agents that gather information. Write-capable tools go to a small, separate agent whose input is already validated and structured. It is the same read and write split as a CQRS design, applied to permissions.
• A timeout per tool call, inside the request deadline: Each tool gets its own budget, derived from the time the request has left. A tool that cannot answer inside it returns a timeout error the agent can reason about.
• Sagas for multi-step effects: A saga is a sequence of changes across several systems where every step is paired with its own undo. You need one because no single database transaction spans those systems and rolls them all back for you. Book the courier, charge the fee, update the order, and write three matching undos. If step three fails, you run the undo for step two and then step one, in reverse. Build each undo in the same change as the step it undoes. Left for later, it never gets built, and you find that out during the first failure rather than before it.

In the running example the refund never happens inside the fan-out. All three lookups read, and the writer only drafts a reply. Issuing the money is a separate write that sits behind the human approval gate from the security section, and the idempotency key guards that write after approval.

Derive the key from the request id and the step, then route every write through one call path.

Frameworks for tools and side effects