Context Engineering is where AI Agents succeed or fail
When an AI Agent fails in production, the model is usually not the problem. The information it received at that moment is. Most of the work of building reliable agents comes down to context engineering: what reaches the model, when it reaches it, and how it is presented.
When an AI Agent makes a bad decision thirty steps into a workflow, the useful question is not whether the model is unreliable. It is what information was in the context window at that point, and why that information made the wrong answer more likely than the right one. That question is usually answerable, and usually fixable.
Why blaming the model is usually wrong
When an AI Agent fails in production, the first reaction is usually to blame the model. People say it hallucinated, was not smart enough, or that the next model release will fix everything. Then a larger model gets deployed, costs go up, and many of the same failures show up again.
In most cases, the model did not suddenly become worse. The real problem is the information it received at that moment.
An LLM is a function that maps input to output. It only sees the tokens inside its current context window. It has no awareness of yesterday's conversation, no understanding of your project beyond the text in front of it, and no hidden state carrying knowledge across requests.
When an AI Agent makes a bad decision thirty steps into a workflow, the useful question is not whether the model is unreliable. It is what information was in the context window at that point, and why that information made the wrong answer more likely than the right one. That question is usually answerable, and more importantly, usually fixable.
Most of the work involved in building reliable AI Agents comes down to context engineering. How good an AI Agent is depends mostly on what information reaches the model, when it reaches it, and how it is presented. It is also one of the least visible parts of the system until production traffic starts exposing the weaknesses.
This is now a mainstream discipline rather than a niche idea. Anthropic's engineering team formalised the term in late 2025 and now treats context engineering as close to the number one job of building a reliable agent. The core idea is to curate the smallest high-signal set of tokens the model needs at each step.
How context breaks
The first problem is excessive context
Large context windows created the impression that curation is no longer necessary. Teams started feeding entire conversations, documents, logs, and tool outputs into the model and expected it to figure out what mattered. More context does not automatically produce better results. Irrelevant history competes for attention with relevant information. Long conversations usually contain far more noise than signal, increasing latency, cost, and error rates.
For most decisions, only a small portion of previous information is actually relevant. Reliable systems continuously identify and retain that subset instead of carrying the entire history forward.
This did not go away as windows grew. By 2026 frontier models commonly ship context windows of a million tokens or more, yet curation still matters. In a study across 18 frontier models, researchers at Chroma found that every one grew less reliable as the input got longer, a pattern now widely called context rot. It sets in well before the window is full. A model with a 200,000 token window can show meaningful degradation by around 50,000 tokens. A bigger window raises the ceiling. It does not remove the need to decide what belongs in the prompt.
The second problem is missing context
Models can only reason about information that is present in the prompt. If a deadline, schema definition, business rule, or previous failure never reaches the context window, the model cannot account for it. The result usually looks like a hallucination, but the underlying issue is much simpler. The model never received the information required to produce the correct answer.
The third problem is unfiltered tool output
AI Agents frequently call APIs, query databases, read files, or process logs. Many systems take the raw output from these tools and place it directly into the next prompt. Large JSON payloads, verbose logs, and lengthy documents quickly consume context while hiding the few details that actually matter.
Tool outputs should almost always be processed before reaching the model. Extract relevant fields, summarise documents, trim logs to the important sections, and reduce large datasets into concise representations. What a tool returns and what the model sees should rarely be identical.
The fourth problem appears during long-running tasks
Research AI Agents, coding AI Agents, and multi-step workflows continuously accumulate context. Without a strategy for managing growth, the context window eventually fills with outdated or low-value information. Performance degrades long before the model reaches its technical token limit. This is context rot in its long-running form, and it is why long-horizon agents need an explicit plan for what to keep.
Managing context at scale
One common approach is compaction
As history grows, the system periodically replaces detailed interactions with summaries. The goal is to preserve decisions, objectives, constraints, and unresolved questions while discarding information that is no longer useful.
Another approach is external memory
Instead of storing everything inside the context window, important information is written to files, databases, scratchpads, or structured stores. The model only retrieves what it needs for the current step. This allows workflows to extend far beyond the limits of a single context window.
Sub-agents solve a similar problem through isolation
A parent agent delegates a focused task to a child agent operating in its own context. The child performs the work and returns only the final result. Intermediate reasoning remains isolated, preventing one task from polluting another.
Retrieval is usually more effective than preloading
Many systems attempt to place every potentially relevant document into the prompt upfront. A better approach is to provide retrieval tools and fetch information when required. This keeps each decision focused on relevant data instead of surrounding it with unnecessary context.
Cache the context that does not change
A large part of any prompt is stable from one step to the next, including system instructions, tool definitions, and fixed rules. Modern providers let you cache that stable prefix so it is not reprocessed on every call. A 2026 evaluation across OpenAI, Anthropic, and Google found that prompt caching cut API cost by 41 to 80 percent and improved time to first token, with cache reads on some providers up to 90 percent cheaper than fresh tokens. The catch ties straight back to tool output. The dependable pattern is to cache the stable context and keep dynamic tool results out of the cached prefix, so one changing payload does not invalidate everything behind it.
| Strategy | What it does | When to reach for it |
|---|---|---|
| Compaction | Replaces detailed history with summaries that keep decisions and constraints | Long conversations where old turns no longer matter but their conclusions do |
| External memory | Writes important state outside the window, retrieves per step | Workflows that must outlast a single context window |
| Sub-agents | Isolates a focused task in its own context, returns only the result | Multi-step work where intermediate reasoning would pollute the parent |
| Retrieval over preloading | Fetches information on demand instead of stuffing it upfront | Large corpora where only a small slice is relevant to each decision |
| Prompt caching | Reuses the stable prefix (system prompt, tools, rules) instead of reprocessing it each call | Long-horizon agents that resend the same context on every step |
A blunt check for whether context is really being engineered. If nobody is watching the token count of each request, nobody is managing context.
Close the feedback loop
A lot of agent failures come down to one thing. The agent never verifies its work. An agent that edits a file without re-reading it, writes a test without running it, or makes an API call without checking the response can confidently drift into the wrong state.
Reliability comes from closing the feedback loop. The agent acts, observes the result, verifies it against the goal, and corrects mistakes when needed. The most dependable agents do not assume success. They gather evidence that a task was completed correctly.
Verification, feedback, and self-correction usually matter more than a larger model when you are building systems people can actually trust.
Why demos hide the problem
Most Agentic AI demos are short, clean, and carefully controlled. Context never grows large enough to create problems. Tool outputs stay manageable. Information remains fresh and consistent. Everything works.
Production environments look very different. Tasks run longer, users behave unpredictably, tool outputs become messy, and context accumulates over time. Failures that appear to be model weaknesses are usually context management failures that only emerge under real workloads.
An AI Agent is not just a loop that calls a model repeatedly. Every step requires deciding what information should survive, what should be discarded, and what should be retrieved again later.
The context window is the agent's working memory. Every token competes for attention and incurs cost. Reliable agents treat context as a resource that must be actively managed, not a container that grows indefinitely. Most apparent model failures become far easier to explain once you inspect the information the model was actually given.
Where to go next
Everything above, context engineering, verification, and treating context as a resource, is the day to day work of the Forward Deployed Engineer, the fastest growing role in applied AI right now. It is the engineer who sits with a customer and ships agents that hold up in their environment. If that is the role you want to grow into, here is the path.
- FDE Roadmap: see where the Forward Deployed Engineer path sits across every track. Open the roadmap
- Dedicated FDE Track: the full Forward Deployed Engineer dedicated track, many modules from customer discovery through deployment and interview prep. Start learning
- FDE interview prep: one module from that path, a focused Q&A bank for Forward Deployed Engineer interview loops. Practice the questions