Production AI Engineering in 2026

A practical map of production AI engineering across thirteen layers. Retrieval, inference, evaluation, LLMOps, safety, cost optimisation, observability, governance, UX, agents, fine-tuning, and organisational mechanics. Plus how to prioritise when you cannot do everything at once.

Most "AI engineering" content stops at prompts and models. The actual discipline is the set of deterministic control layers built around probabilistic systems. This is the long-form map. 13 areas, every corner case that goes wrong in production, and where to start when you have to fix it under pressure.

What Production AI Engineering actually is

A model is probabilistic. Ask it the same thing twice and you can get two different answers, and neither one arrives with an error attached. Production AI engineering is the work of putting enough ordinary, predictable engineering around that component to make the system as a whole behave. The model is the easy part. The discipline is everything holding it up. The retrieval pipeline that feeds it. The evaluation suite that catches its regressions. The serving stack that holds its tail latency, the guardrails that absorb its failures, and the cost engineering that keeps it economic. Most production AI failures happen because one of those supporting systems is weak, not because the model is bad.

An LLM is non-deterministic. Everything around it must be deterministic enough that the system as a whole stays predictable for the people who depend on it.

Where AI engineering sits

What Quality Contract actually includes (full breakdown)

The diagram above shows compact summaries. The table below expands every contract dimension across the three architectures so the structural escalation is visible at a glance. Every row maps to a section later in the guide where the topic is covered in production-grade depth.

How to read this table. Each row escalates left-to-right. The multi-agent column does not replace the GenAI column, it adds on top of it. A multi-agent quality contract has to include every GenAI control plus the inter-agent layer (state, trust boundaries, termination, loop budgets) that single-LLM systems do not have. As a result, the reliability of a multi-agent system depends on its weakest agent-to-agent handoff, not its strongest individual agent.

AI engineering is a separate discipline that combines ideas from both software engineering and ML engineering, but operates differently from each. Traditional software engineering assumes deterministic behaviour where the same input produces the same output and correctness is verified through unit tests. AI systems are different because the model is probabilistic. Outputs can vary even for similar inputs. As a result, production AI systems rely more on evaluations, quality metrics, and behaviour distributions instead of strict pass/fail testing alone.

ML engineering mainly focuses on the upstream model lifecycle such as collecting data, training models, fine-tuning, and monitoring training drift. AI engineering usually starts after the model already exists. It focuses on building reliable production systems around the model, including retrieval, agents, prompting, evaluation, serving infrastructure, integrations, safety, cost control, observability, and user experience.

AI engineering is mostly a software engineering discipline with some applied ML and LLM knowledge. ML engineering is primarily model development, training, and the MLOps process around them.

AI systems become significantly more complex when they become agentic. Instead of only generating text, the LLM must make decisions, choose actions, call tools or other agents, evaluate results, and decide what to do next. This creates a continuous decision-making loop where the model plans, acts, observes, and iterates until the task is completed.

A multi-agent system goes further by coordinating multiple specialised agents, each with its own prompt, tools, memory, and decision logic. At that point, failures are no longer isolated to a single model response. Errors can propagate across agents and compound through the workflow.

Every additional agent increases system complexity. Agents have to stay in step with each other. Tool calls have to be idempotent, running the same call twice with no extra effect, so a retry cannot send the same email or issue the same refund again. And the output of one agent has to be checked before it becomes the input of another. A wrong answer from one agent spreads through the whole chain if the agents downstream of it trust it blindly. Systems also need safeguards that detect infinite loops and runaway execution costs.

In production multi-agent systems, the architecture matters more than the specific LLM being used. The orchestration, state management, safety boundaries, memory design, and recovery logic are what determine whether the system is reliable.

Worked example for the rest of this guide. Smart-claims-processor, a multi-agent insurance-claims pipeline. Source is the aiml-companion repo, blog

The stack at a glance.

• LangGraph orchestration: multi-agent pipeline modelled as an explicit state machine
• CrewAI fraud sub-crew: specialist sub-agents collaborating inside a single LangGraph node
• ChromaDB three-tier memory: short-term in LangGraph state, long-term and episodic in ChromaDB
• Per-claim guardrails: input PII masker, per-agent output classifier, hallucination scoring
• LLM-as-judge evaluation: automated scoring of every claim decision against a rubric
• Native interrupt/resume HITL: pipeline pauses below a confidence threshold; reviewers resume from the exact checkpoint
• React + Material UI frontend: claimant flow, status timeline, HITL queue, admin settings

Worked scenario. A claimant submits a $4,200 windshield-damage claim through the UI. Behind the visible status timeline and the final decision to approve or deny, at least seven distinct systems run.

• 1. Memory: short-term + long-term + episodic, with a memory router that decides what to recall
• 2. Multi-agent inference: LangGraph orchestrator routes through 7 specialist agents, with the CrewAI fraud sub-crew nested inside
• 3. Guardrails + PII: input PII masking, output classifier on every agent, hallucination scoring before commit
• 4. Evaluation + HITL: LLM-as-judge scores each step; confidence below threshold triggers interrupt() and routes to a human reviewer queue
• 5. Response + UX: streaming status timeline, structured approve/deny render with reasons, correctable outputs
• 6. Observability + cost: OpenTelemetry GenAI spans, per-claim token and dollar tracking, 7-year audit log
• 7. The harness: control loop, tool dispatcher, sandbox, state checkpoints, context manager, permissions, observability, coordinating everything above

The next section walks one claim through every layer with real file references. Every section after that is one of those layers in depth.

This guide is for architects, tech leads, and senior engineers making real production decisions. Every section names the actual alternatives, lays out the trade-offs that drove the choice, and tells you what teams shipping at scale are picking and why.

13 Areas that determine the success of an AI system

• 1. Retrieval and data layer: Chunking strategy, embedder selection, vector store choice, hybrid vs dense, reranking, freshness pipelines. The largest single lever in answer quality, and where most teams underinvest.
• 2. Inference infrastructure: vLLM vs TGI vs SGLang vs TensorRT-LLM. Quantisation tiers (FP16 vs FP8 vs INT8 vs INT4). Model cascading. The self-host vs API decision with the actual economic break-even.
• 3. Evaluation as a continuous discipline: Golden datasets, LLM-as-judge calibration, offline vs online, pairwise vs absolute scoring, the regression suite that gates every prompt and model change.
• 4. LLMOps lifecycle: Prompt registries, version-pinned model snapshots, shadow vs canary vs A/B, rollback paths, surviving silent vendor model updates.
• 5. Safety engineering: Direct, indirect, and tool-flow prompt injection. Jailbreaks. PII detection. Hallucination scoring. The defence-in-depth stack and where each layer can be bypassed alone.
• 6. Cost engineering: Token budgets per request, user, feature, and tenant. Prefix cache vs semantic cache (they are not the same). Prompt compression. Cost attribution observability.
• 7. LLM-specific observability: OpenTelemetry GenAI conventions, trace structure for retrieval and tool calls, drift detection, feedback signal capture, agent-loop traces.
• 8. Governance and compliance: Audit logs, model cards, data residency, EU AI Act risk tiers and high-risk obligations, NIST AI RMF, GDPR Article 22, multi-tenant isolation.
• 9. UX for probabilistic systems: Streaming, partial results, confidence signalling, citation surfaces, correction paths, graceful degradation. The UX that turns "the AI is wrong" into "the AI is correctable".
• 10. Agent-specific engineering: Planning, tool selection reliability, hard budgets, sandboxes, trust boundaries between sub-agents, idempotency on side-effecting tools, the harness that makes the difference between a demo and a system.
• 11. Fine-tuning and customisation: When to prompt vs retrieve vs SFT vs DPO vs LoRA vs distil. The actual decision tree, the catastrophic forgetting tax, the rewards of staying with prompting longer than feels comfortable.
• 12. Organisational mechanics: Feedback loops from users and domain experts back into evaluations. Blameless review of bad outputs. The cross-functional skill mix that real LLM systems require.
• 13. Where to start when you cannot do everything: The two highest-use layers, a 90-day sequence, and the anti-patterns to skip.

A full reference (~55 minutes, 13 core areas, 60+ glossary terms). Read it end-to-end to build a complete mental model of production AI systems, or jump directly to the section most relevant to the problem you're solving today.

The Mental Model

The label AI engineering gets attached to three different architectures, classical software engineering, GenAI applications, and multi-agent systems. This part grounds the distinction in the hardest of the three. The 13 layers that follow, retrieval through where-to-start, are the engineering surface area that absorbs the model's non-determinism. One customer claim is traced end-to-end through smart-claims-processor, so every layer the rest of the guide unpacks shows up first in a worked example.

By the end of this part you will be able to look at any AI product and recognise which architecture it is. You will know what the deterministic engineering around the model has to absorb, and why the just use a better model instinct is almost always the wrong fix.

AI Engineering in context, the Smart Claims Processor

This guide hangs on a real, production-style codebase (Smart-claims-processor). It is a multi-agent insurance-claims pipeline with LangGraph orchestration, a CrewAI fraud sub-crew, ChromaDB-backed three-tier memory, per-claim guardrails, LLM-as-judge evaluation, native LangGraph interrupt/resume HITL, and a React + Material UI frontend.

Walk one claim through the full stack. Every later section in this guide is one of these layers in depth, and every trade-off called out here gets unpacked further with named tools, version pins, and corner cases.

The user query

A claimant submits an auto-claim through the React UI, $4,200 of windshield damage, with photos. The frontend hits FastAPI (api/main.py), which kicks off a LangGraph pipeline. What the user sees is a status timeline and, eventually, a decision to approve or deny with concrete reasons. What runs underneath is at least seven distinct systems coordinated by a single state machine.

Step 1. Memory and retrieval (ChromaDB, three-tier)

Before any agent generates a token, the intake agent (src/agents/intake_agent.py) can consult memory via tool-calls. Search_similar_claims and search_fraud_episodes (src/tools/memory_tools.py). Memory is three tiers (src/memory/manager.py). Short-term lives in the LangGraph state object that travels with the run.

Long-term holds past claim outcomes indexed in ChromaDB with all-MiniLM-L6-v2 embeddings (src/memory/embeddings.py). Episodic stores human overrides, confirmed fraud cases, and quality-gate failures. The LLM, not the code, decides when to query memory.

Trade-off at Step 1, tool-call memory (LLM reasons about whether to search) versus always-inject (top-K stuffed into every prompt). This project picked tool-call, so context windows stay clean and tokens are only spent on memory when the agent asks. Trade. Occasional missed retrievals on edge cases. Storage considered. Pgvector for transactional colocation with the claims DB, Pinecone for managed scale, ChromaDB for embedded zero-ops. ChromaDB was right HERE because the corpus stays under 100K claims and the project doubles as a learning artefact where local-first matters.

Step 2. Multi-agent inference (LangGraph + CrewAI)

The LangGraph state machine in src/agents/graph.py orchestrates 7 specialist agents. Intake validation, fraud detection (a 3-agent CrewAI crew built in src/agents/fraud_crew.py. Pattern analyst, anomaly detector, social validator), damage assessment, policy compliance check, settlement calculation, LLM-as-judge evaluation, and claimant communication. Every agent emits confidence, reasoning, flags, and findings, all persisted. The model itself is pluggable (src/llm.py). Gemini 3.5 Flash (2.5 Flash is deprecating Oct 2026) or Groq llama-3.3-70b-versatile, switchable at runtime via .env or the /api/settings/llm endpoint.

Trade-off at Step 2, multi-agent specialist crew versus a single agent with a long system prompt. This project picked specialists because fraud benefits from separate prompts for separate reasoning modes (pattern recognition, anomaly detection, social validation), and the rest of the pipeline has cleanly separable decisions (intake, damage, policy). Trade. Coordination overhead, more LLM calls per claim, harder debugging. Mixed framework choice. Pure LangGraph for the main pipeline, CrewAI retained ONLY for the fraud sub-crew because role-playing crews suit multi-perspective fraud reasoning. Pure LangGraph everywhere would have been simpler to debug and one less framework version to pin. The team kept CrewAI where it actually pays off.

Step 3. Guardrails and PII

Every agent execution is wrapped by src/guardrails/manager.py. Pre-execution checks. Per-claim budget caps (max agent calls, max tokens, max dollar cost), loop detection, execution timeout.

Post-execution checks. Minimum confidence per agent type (intake 0.60, fraud 0.55, damage 0.65, policy 0.70, settlement 0.70), hallucination check (key facts must reference data the agent actually had), schema completeness. PII is masked before any LLM call by src/security/pii_masker.py. Every decision flows into the 7-year audit log at src/security/audit_log.py.

Trade-off at Step 3, input-side PII masking versus output-side. This project masks at input so the LLM never sees real names, policy numbers, or addresses. Trade. The agent loses personalisation context, but the regulatory surface shrinks dramatically because the LLM cannot leak what it never saw. Alternative considered. Managed PII service (AWS Comprehend, Google Cloud Sensitive Data Protection (formerly DLP), Private AI). The project chose custom because the entity set is small and insurance-specific, and owning the masking logic was worth more than the breadth a managed service buys.

Step 4. Evaluation and HITL (interrupt and resume)

After settlement is calculated, the LLM-as-judge in src/evaluation/evaluator.py scores the full pipeline. Was the decision well-supported, were the rules applied, is the customer communication concrete. In parallel, the HITL system (src/hitl/checkpoint.py, src/hitl/queue.py) listens for two triggers. Any per-agent confidence below its gate (defined in configs/base.yaml under confidence_gates) or any high-risk signal (high fraud score, high value, manual policy escalation).

When either fires, the pipeline calls LangGraph's interrupt(), persists state via SqliteSaver, and a human reviewer picks it up from the HITL queue. Their decision resumes the pipeline from the exact checkpoint.

Trade-off at Step 4, confidence-gated HITL versus blanket HITL. The project routes the majority of claims through fully automated processing, with the exact ratio set by the project's configurable confidence_gates (tuned operating point, not a fixed figure). Blanket HITL catches more errors but blocks scale. Full automation scales but misses high-value mistakes. The confidence gates are the dial that lets you pick the operating point per agent. Calibrating those thresholds is ongoing engineering work. The architecture just makes the dial visible and tunable without a redeploy.

Step 5. Response and UX (React Agent Trace)

The React + Vite + Zustand + MUI (Material UI) frontend (frontend/) renders an Agent Trace panel in the claim detail view. Every agent's confidence, reasoning, and flags are expandable. Denied claims show specific denial reasons rather than generic boilerplate (the communication agent is explicitly prompted to surface them).

The HITL queue is its own UI for reviewers. Their decisions resume the paused LangGraph state. A settings UI lets admins switch LLM provider, country profile (US versus India), and confidence thresholds at runtime without a redeploy.

Trade-off at Step 5, full transparency (every agent's reasoning visible to the reviewer) versus opaque outcome (just the final decision). The project picked transparency. Trade. More UI surface to build and maintain. Win. Trust. A reviewer who can see WHY the fraud crew flagged a claim validates in seconds. One who sees only a number takes minutes and is wrong more often. For an HITL-driven product, transparency is the feature.

Step 6. Observability and cost

Token counts and dollar cost are tracked per-pipeline-run via a LangChain callback handler and surfaced in the claim detail view as Total LLM Cost (USD). Analytics endpoints (api/routes_analytics.py) aggregate approval rate, HITL rate, cost breakdown, fraud trends, and evaluator pass rate. Audit log entries carry agent, claim, decision, reasoning, timestamp, and reviewer ID where applicable, retained per the project's configurable retention policy (default 7 years to cover common US-state insurance DOI minimums).

Trade-off at Step 6, per-claim live cost (what this project does) versus batch-aggregated cost. Per-claim live lets you spot a $0.42 expensive claim immediately and investigate. Batch is cheaper to compute but slower to surface runaway behaviour. Alternative considered. Full OpenTelemetry export to a managed observability tool. The project keeps tracing local because the LangChain callback handler is the more instructive primitive. A production deployment at scale would add OTel without disrupting the existing flow.

Step 7. The harness (LangGraph + FastAPI)

What holds all of this together is the harness. LangGraph's state machine plus FastAPI's routing layer. The LangGraph state object carries claim data, intermediate agent outputs, HITL decisions, retry counts, and budget consumption between every node. SqliteSaver persists checkpoints durably so an interrupt() can resume hours or days later after human review.

FastAPI (api/main.py and the routes_*.py modules) exposes claims, appeals, HITL queue, analytics, policies, and settings. JWT + bcrypt + roles (user, reviewer, admin) gate every route. Together LangGraph and FastAPI are the difference between a demo and a system.

A complete production AI stack for a single insurance claim. ChromaDB-based three-tier memory, LangGraph multi-agent orchestration, a CrewAI fraud-detection sub-crew, claim-specific guardrails, input-side PII masking, LLM-as-a-Judge evaluation, human-in-the-loop interrupt/resume workflows, a React-based agent trace UI, per-claim cost tracking, and a 7-year audit trail. Each section below explores one of these layers in depth. If a concept feels abstract, refer to the corresponding implementation in the Smart Claims Processor project to see how it works in practice.

Where Quality Lives or Dies

A majority of production AI failures (per Databricks and Anyscale RAG eval studies) trace back to retrieval, not the LLM itself. If the system retrieves the wrong, outdated, or incomplete information, even the best model will produce a poor answer. That is why retrieval is one of the most important parts of a production AI system. Key design decisions include how documents are chunked, which embedding model is used, and whether to combine keyword and vector search. Then how results are reranked, how new data is ingested, whether knowledge graphs are needed, how citations are enforced, and how often the knowledge base is updated (batch, near real-time, or streaming).

By the end of this part you will know why hybrid plus reranker is the cheapest quality lift available. Also why the metadata schema is more load-bearing than the cadence choice, and when RAG is the wrong answer entirely.

1. The Retrieval and Data Layer

Prompts are cheap to iterate on. Retrieval is not. Bad chunking, a wrong-for-the-domain embedding model, or a stale index all surface as the same user complaint, and all three take days to diagnose and weeks to fix once they ship.

The production ingestion pipeline. From source to vector index

Production retrieval is a continuous data pipeline, not a one-off load-embed-query script. Source documents change, new records arrive, events fire, and the retrieval index has to keep up.

This pipeline typically includes four core steps. Ingesting source data, chunking and cleaning the content, generating embeddings, and updating the vector database. It must also support different update frequencies, batch, near real-time, or streaming, and handle operational issues such as duplicate data, failed embeddings, and stale indexes.

Three ingestion modes (batch, near-real-time, streaming)

Pick the slowest mode that still meets the product's freshness requirement. Most teams over-engineer for streaming when batch or near-real-time would suffice. The trade-off is higher operational complexity, more retry handling, and increased embedding cost for each incoming event.

The four transform steps, with production best practices

• 1. Parse: Use a layout-aware parser (Unstructured, Docling, LlamaParse, Reducto etc) for any document heavier than a memo. Plain text extraction silently drops tables, headings, and figures, usually the highest-value content in financial filings and scientific papers. Best practice. Store the parser version per chunk so you can diff quality after a parser upgrade.
• 2. Chunk + metadata: Recursive chunking at ~300-500 tokens with ~50-100 token overlap is a reasonable starting default for prose, tune against your retrieval eval set. Layout-aware for structured docs, AST-aware for code. Every chunk carries metadata that the LLM never sees but the retrieval layer relies on. Source_id, doc_url, last_modified, owning_team, version, ACL tags, language. Without this, citations, freshness, and multi-tenant isolation are not possible.
• 3. Embed (batched): Almost every provider supports batched embedding requests (provider-specific. OpenAI up to 2048, Cohere Embed v4 ~96, Voyage 128). Use them. Pin the embedder model version explicitly in the chunk record. Plan ahead for re-embed windows when upgrading. Embedder upgrades invalidate the entire index. Best practice. Dual-index strategy. Build the new index alongside the old, evaluate against a labelled retrieval test set, alias-swap at the end.
• 4. Upsert (idempotent): Writes are keyed by a stable chunk ID (hash of source_id + position + chunk version). Same input produces the same ID, so retries are free. Source deletion triggers an explicit tombstone, NOT a silent skip. Source deletion is the single most missed step in production indexing pipelines and the leading cause of "the bot is citing a doc that does not exist any more" complaints.

Cloud-specific patterns (as of mid-2026)

All three major clouds now ship managed RAG primitives. The decision is per-component rather than a blanket managed-versus-roll-your-own call. Most production teams use a managed embedding API even when they self-host the vector store, or vice versa.

Pipeline failure modes the embedder docs do not warn you about

• Tombstone misses: A source document is deleted but its chunks remain in the index forever. The agent will happily cite a doc that does not exist. The fix. Full-set reconciliation job runs nightly (compare source set to index set, tombstone the diff). A recurring cause of production-RAG incidents we see in audits.
• Schema drift: A source schema changes (column rename, new field, format shift). Chunks were embedded against the old schema and now reference stale structure. The fix. Track schema_version in chunk metadata. On detection, trigger targeted re-embed for affected sources.
• Embedding-throughput bottleneck: Provider API rate limits cap how fast you can rebuild. A 10 million chunk re-embed at 1000 chunks per second takes ~2.8 hours. Plan windows accordingly. Reserve capacity for the cutover. Use exponential backoff and a retry budget.
• Cost explosion on streaming: Streaming embeds every event. At ~$0.00005 per chunk (text-embedding-3-large at $0.13/1M tokens × ~400 tokens) and 10 million events per day, that is roughly $500 daily, and scales linearly with chunk size. Most events never get retrieved. The fix. Aggregate or sample before embedding. Embed only events that will plausibly be retrieved. Defer the rest to a batch-tier index.
• Permissions desync: A user lost access to a document, but the chunk is still retrievable and ends up in their answer. The fix. Enforce ACL filtering at the index query level via metadata, never trust the LLM to honour a "do not show user X this content" instruction in the prompt. Stale ACLs are the most common multi-tenant data leak path.
• Dual-write inconsistency: Some chunks made it to the index, some did not, because an upsert worker crashed mid-batch. The fix. Idempotent writes keyed by stable chunk ID, plus periodic reconciliation that re-embeds missing chunks.
• Index size blow-up: Re-embed without alias swap doubles index storage temporarily. If not budgeted, the cutover OOMs or runs out of disk. The fix. Budget for 2x peak storage during cutover windows. Alert on free-space below the doubling threshold.