ML System Design Patterns

Modelling patterns are about the model. ML system design patterns go beyond the model itself. The challenge is that the data your system depends on is never fully under your control. It changes without warning, and other teams use it without telling you. In this post we'll go through the patterns that deal with that, one layer at a time. Features, training, serving, releases and monitoring. Then we'll look at the mistakes that quietly break real world systems, and how all of this changes in the age of language models.

Code dependencies are explicit and versioned. Data dependencies are implicit, silently changing, and shared by teams you have never met. Nearly every pattern in production machine learning exists to deal with that one difference. This post maps them layer by layer. Feature stores and point-in-time correctness, training pipelines, the four serving shapes, the release ladder, and the three kinds of drift. We'll finish with the mistakes that quietly break real world systems, and how all of this changes in the age of language models.

The data you do not fully control

Ask most data scientists about machine learning patterns and they will talk about the model. Which algorithm to pick. How to engineer the features. How to stop it overfitting. That work is real and worth learning. All of it is about the model itself. ML system design patterns are a different list. They are about everything around the model, because that is where a production system actually breaks.

The hard part is the data. Think about how your code handles a dependency. It is listed in a file, you pin it to a version, and a diff shows you exactly what changed. Data works nothing like that. A column can change meaning overnight, with no release and no warning. The same feature also gets written twice. A data scientist writes it in Python for training. A backend engineer rewrites it in Java or Go for the live service. The two versions slowly stop agreeing. On top of that, other teams read your data without ever telling you.

That is the whole problem. Nearly every pattern below makes a data dependency behave more like a code dependency, so you can see it, version it, and roll it back.

A model is only as good as data you do not own. Every pattern in this post does one of three things. It makes that data visible, or it makes a result repeatable, or it makes a change easy to undo. If a pattern does none of the three, you probably do not need it yet.

One more thing before the patterns. Most of these decisions come down to the same three-way trade-off, so it helps to name it now.

Ask these three questions at every layer. How fresh does this feature have to be. How fast does the answer have to arrive. How much is it worth spending per request. Most design arguments are two people pulling toward different corners without saying so.

The loop every ML system runs

Every production ML system runs the same loop, whatever the model is. Data comes in. Features are built from it. A model is trained, evaluated, and served. The results are monitored. Then it runs again.

Each stage fails in its own way, and the failures are specific enough to name. That is why the rest of this post is organised the same way. Every section takes one stage of this loop and lists the patterns that hold it up.

Two things about the loop matter more than the stages themselves.

• It is a loop, not a line: What you serve today shapes what you train on tomorrow. Most of the hardest failures in this post come from that return path rather than from any single stage.
• The stages belong to different people: Data sourcing, features, serving, and monitoring often sit with four different teams. Almost every skew and dependency problem below starts at one of those handovers.

This post assumes the loop and focuses on what goes wrong across it. If you want the stages themselves taught properly, with worked examples rather than a summary, two tracks in this app cover the full path. Start with ML Pipelines for data through training, then MLOps for deployment and monitoring.

One system, followed all the way through

Patterns are hard to judge in the abstract. So from here on we'll apply every one of them to a single system, and you can watch the same six numbers decide each choice.

The system is a card payment fraud detector. It sits inside checkout, and for every payment it decides whether to approve it, send it to a human reviewer, or block it.

Read those numbers before going on, because most of them close off options.

• 80ms at p99: A slow model is out. So is fetching a feature by running a query at request time.
• A chargeback arriving 5 to 60 days later: You cannot know today whether today's predictions were right. Anything that waits for labels is slow by two months.
• 0.3% fraudulent: A model that approves everything is 99.7% accurate. So accuracy is useless here, and the rare class needs deliberate handling.
• Three consumers: Checkout, the review queue, and a finance report all read the score. Any change you make lands on all three.

This is worth doing for your own system before you pick any pattern. Write down what it predicts, its traffic, its latency budget, where the label comes from and how late it is, the base rate, and who reads the output. Most design arguments end quickly once those six numbers are on the table.

Data and feature patterns

Everything else in this post depends on this layer being right. Get it wrong and your metrics will lie to you for months. For the fraud detector, this is the layer that decides whether the model sees the same numbers in training that it will see at checkout.

Feature store

You write the feature once. The store then keeps it in two places. The offline store holds the full history and is used to build training sets. The online store holds only the latest value for each user or item, and the live service reads it in a few milliseconds. Both come from the same definition, so they cannot drift apart.

You do not need to buy a product to get the benefit. You need one place where the feature is defined, with both the training code and the serving code generated from it. In the fraud detector, cards_seen_from_this_device_24h has to mean exactly the same thing in the training set and at checkout, or the model is scoring something it was never trained on. The moment a data scientist writes the transform in Python and a backend engineer rewrites it in Go, you have a bug that will not show up for months.

Point-in-time correctness

Features have to be computed as of the prediction timestamp, not as of now. This sounds obvious and it gets broken constantly. The easy way to build a training set is to take the feature values you have today and attach them to old labels. That is exactly the mistake.

This one bites the fraud detector hard, because a chargeback arrives up to 60 days after the payment. Build a training row today and it is very easy to include what you learnt in those 60 days. The reason this deserves its own section is that it fails silently, and it makes your numbers look better. Offline accuracy goes up. Everyone is pleased. Live accuracy is worse and nobody connects the two, because the leakage is in the join, not the model.

If your offline score is much better than your live score and you cannot explain the gap, check the point-in-time join before you touch the model. This is one of the most common causes, and it is almost never the first thing teams look at.

The rest of the data layer

Four more that are less famous and still earn their place.

• Transform-in-graph: Bundle your preprocessing with the saved model so the exact same logic runs during training and serving. If two different code paths are parsing dates, you're inviting inconsistencies.
• Repeatable splitting: Split data using a stable hash instead of a random seed. Otherwise, records drift between the training and test sets every time the pipeline runs, gradually contaminating your evaluation.
• Windowed aggregation: Precompute rolling features, like a 30-day event count, in your data pipeline or streaming layer. At inference time, the model simply reads the value instead of computing it under a tight latency budget.
• Bridged schema: When a table's schema changes over time, backfill older records to match the new structure. That lets you train on the full history instead of discarding valuable data.

Data validation at ingest

Start every pipeline with schema and distribution checks. If the data is invalid, fail fast. They're cheap to implement and prevent most real production failures. A column that silently became null last Tuesday won't be caught later in the stack.

Training patterns

The pipeline, not the notebook

This pattern is called a workflow pipeline. It is the whole path from raw data to a registered model, broken into separate steps that run in order. Each step can be retried on its own, each one caches its output, and the entire run can be rebuilt from a commit.

The value shows up on the day someone asks which data produced the model currently serving traffic. In a notebook world that question has no answer. In a pipeline world it is a lookup.

Continued training with explicit triggers

For the fraud detector, a schedule alone is a bad fit, because fraud patterns change when an attacker decides they should, not on the first of the month. Retrain on a schedule, on a data volume threshold, or on a drift or performance alarm. Pick one deliberately and write it down. Retraining whenever someone remembers to is not a trigger, and that is what most teams are actually running on.

Pair it with a warm start policy. Resuming from the previous checkpoint is much faster, so use it for the routine cadence. Then retrain from scratch periodically, because warm starts accumulate drift that no single run makes visible.

Shaping the problem before you shape the model

These three change the question rather than the model. All three apply to the fraud detector, where 0.3% of payments are fraudulent and plain accuracy tells you nothing.

• Rebalancing: For skewed classes, downsample with weighting, reframe the problem, or use a cascade. Which one you pick depends on how rare the rare class actually is, so measure that before choosing.
• Reframing: Turn a regression into classification over buckets, or the reverse, because it fits the product decision better and hands you a distribution instead of a single number.
• Cascade: Break a conditional problem into a sequence of models. Powerful and dangerous. The whole chain has to be trained and evaluated as one pipeline, or errors compound silently between the stages.

Useful overfitting is a real pattern, not a mistake. Sometimes memorising is exactly the goal. Training a small model to copy a big one. Replacing a slow simulation with a model that has learnt its answers. Overfitting a single batch just to check your code runs.

Serving patterns

These four patterns answer one question. When do you compute the prediction, and who is waiting while you do it. For the fraud detector the answer is forced. A payment that has not happened yet cannot be precomputed, and a customer is waiting at checkout, so it has to be online and it has to fit in 80ms.

All four sit on top of the stateless serving function. The server keeps nothing about a user in its own memory, so any machine can answer any request. Once that is true, you handle more traffic by adding machines rather than rewriting the app.

Retrieval and ranking

The load-bearing pattern of every search, feed, and recommendation system. Retrieve cheaply over millions of candidates, then score expensively over the few hundred that survive.

This is the one pattern the fraud detector does not need, because there are no candidates to narrow down, just one payment to score. It is here because the same shape runs every search, feed and recommendation system, and it shows up again in LLM stacks under different names.

The four that keep serving honest

• Keyed predictions: The client passes an identifier that comes back with the prediction, so asynchronous and batched results can be rejoined to their inputs without guessing at ordering.
• Dynamic batching: Accumulate requests for a few milliseconds to exploit hardware parallelism. It trades a little tail latency for a lot of throughput, which is usually the right trade on a GPU.
• Graceful degradation: An explicit fallback for when the model is down or slow, whether that is the last known prediction, a heuristic, or a global average. Design it before you need it.
• Prediction logging: Log the inputs, the features, the output, and the model version. That one record is your debugging tool, your monitoring input, and your next training set.

If you build one thing from this section first, build prediction logging. Every monitoring pattern later in this post reads from it, and no amount of clever work recovers a request you failed to record.

Release patterns

A model release is riskier than a code release, because nothing crashes when a model gets worse. It just quietly starts giving worse answers. The ladder below exists so each step retires one specific risk before the next.

Shadow mode matters more than usual for the fraud detector, because a bad model here does not return an error, it declines a real customer at checkout. A canary deployment is the rung most teams reach for first, but two of these get confused often enough to be worth separating. Shadow mode proves the model runs correctly on the real distribution, with zero user impact. An A/B test proves the model is better for the business, which is a completely different claim.

You need both, because offline metrics and online metrics disagree constantly. A model with a better validation loss routinely loses on engagement, revenue, or completion rate, and no amount of offline work tells you that in advance.

• Champion and challenger: A standing incumbent that every candidate must beat on a fixed evaluation before promotion. It turns shipping from a judgement call into a rule.
• Multi-armed bandit: Traffic shifts automatically toward whichever version is winning. Right when you want to lose as little as possible while you learn, wrong when you need a clean measurement.
• Model registry: Immutable versions carrying their lineage, which is the data snapshot, the code commit, and the hyperparameters. Promotion moves a pointer, so rollback is moving it back.

The registry is what makes rollback boring, and boring rollback is what makes teams willing to ship. If reverting a model means a rebuild, people stop shipping and start arguing instead.

Monitoring patterns

The dangerous failure in an ML system is the slow decline that no alarm catches, not the crash. So this layer is about measuring quality rather than uptime.

Drift, and why one word is three problems

The three are covariate shift, label shift, and concept drift. Teams that treat them as one thing end up retraining on a schedule and wondering why it does not help. Retraining fixes covariate shift. It does not fix concept drift. The labels you would retrain on were created under the old rule, so they teach the model something that is no longer true.

Skew detection

Compare live feature distributions against the training distribution. This is the fastest alarm you can build, because it does not wait for labels. For the fraud detector that gap is the whole point, since waiting for labels means waiting up to 60 days. A broken upstream pipeline shows up here in minutes and in your accuracy metrics in weeks.

The rest of the monitoring layer

• Continuous evaluation: Score predictions against labels as they arrive, while accepting that labels are delayed and biased toward the cases someone bothered to review.
• Slice-based evaluation: Aggregate metrics hide per-segment failures. Define the slices up front, by device, region, tenant, or customer size, and alert on each one.
• Human-in-the-loop escalation: Route low-confidence cases to people, and treat their answers as labelled data. The review queue becomes a training set, which is a rare case of monitoring paying for itself.

If you want this layer in much more depth, including how a real incident gets investigated end to end, the observability post picks it up from here.

The anti-patterns

Most of these come from the paper Hidden Technical Debt in Machine Learning Systems, which is a decade old now and still describes the failure modes better than anything since. None of them are modelling mistakes. They are all structure, and they all get worse quietly.

CACE is the one worth internalising first. No machine learning input is truly independent, so removing a feature shifts every other weight to compensate. That is why you have to test one change at a time, and why adding one more signal is never as small as it sounds.

• Correction cascades: A model patching another model output, then patched again. Each layer buys a week and makes the stack harder to unwind. The fix is usually to retrain the base model, which is exactly the work each patch was avoiding.
• Pipeline jungles and glue code: The codebase becomes scrapers and adapters with a model somewhere in the middle. It grows a little at a time, so the only real defence is deleting paths rather than adding them.
• Unstable data dependencies: Consuming a feature that another team retrains or redefines on their own schedule. Version it and snapshot it, or accept that your model changes when theirs does.
• Configuration debt: Untested, unreviewed config files carrying as much logic as the code. In ML systems the config often contains the actual decisions, which makes this worse than it sounds.

The fraud detector has a textbook example. Block a payment and it can never produce a chargeback, so a blocked fraud attempt and a blocked innocent customer look identical in the data forever. Degenerate feedback loops are the hardest of the four to see, because the system looks like it is improving. The model shapes the data it is next trained on, through position bias, filter bubbles, or self-fulfilling risk scores. More data makes it worse. Breaking it means deliberately showing some things the model would not have picked, so the next training set is not just a record of what you already recommended.

The LLM-era additions

The LLM stack did not replace these patterns. It renamed most of them and added a few that are genuinely new, and seeing the correspondence makes the new stack much less mysterious.

Three of these carry a warning worth stating plainly.

• RAG quality is retrieval quality: Chunking and retrieval dominate the result. The generator is rarely the bottleneck, so tuning prompts when retrieval is broken wastes weeks.
• A judge has to be calibrated: Score the judge against human labels on a sample first. An uncalibrated judge produces a confident number that measures nothing.
• The tool boundary is untrusted: In an agent loop, treat every tool call as an external system with timeouts, retries, and a hard step limit. The loop is the new failure mode.

If this is the layer you are working in, the technical architecture of agentic AI and how to evaluate AI systems with real code go a long way further into it.

How to choose

The list above is a catalogue, not a plan. Adopting all of it before you have a model in production is the most expensive mistake in this space. It is also a common one, because the catalogue is much easier to buy than the judgement is to build.

Start with the heuristic benchmark. Ship the rule-based version, instrument it, and let its numbers define the bar. Two useful things happen. You get a baseline every model has to beat, and often you discover no model is needed at all.

Then add patterns on evidence. A gap between offline and live scores justifies a feature store. An unreproducible result justifies a pipeline. A silent decline justifies drift monitoring. A release that scares you justifies shadow mode. Each pattern costs real maintenance, so it should be paying for a problem you have already met.

The uncomfortable summary is that most production ML problems are data dependency problems, not modelling problems. That is why the fix is almost never a better architecture.

If you want the modelling side of this instead, the tracks in this app cover it directly.

• Feature Engineering: How features are built, and where the definitions that cause skew come from. Open the module.
• Model Versioning: Immutable versions, lineage, and promotion, which is what makes rollback boring. Open the module.
• Monitoring in Production: Drift, skew, and the alarms that catch a silent decline. Open the module.
• Eval-First Engineering: Building the evaluation before the feature, as a working habit. Open the module.

Sources

The primary references behind the patterns and anti-patterns above.

• Sculley, Holt, Golovin et al., Google: Hidden Technical Debt in Machine Learning Systems, NIPS 2015. The origin of CACE, correction cascades, undeclared consumers, glue code, pipeline jungles, and configuration debt. Link
• Lakshmanan, Robinson and Munn: Machine Learning Design Patterns, O'Reilly 2020. The 30-pattern catalogue behind much of the data, training, and serving sections here, including transform, bridged schema, cascade, reframing, keyed predictions, useful overfitting, and heuristic benchmark. Link
• Breck, Cai, Nielsen, Salib and Sculley: The ML Test Score, a rubric for ML production readiness and technical debt reduction, IEEE Big Data 2017. The basis for treating data validation and skew detection as testable requirements. Link
• Breck, Polyzotis, Roy, Whang and Zinkevich: Data Validation for Machine Learning, SysML 2019. How schema and distribution checks are run at ingest at Google scale, as part of TFX. Link
• Chip Huyen: Designing Machine Learning Systems, O'Reilly 2022. Reference for the drift taxonomy, continuous evaluation, and degenerate feedback loops. Link
• Google Cloud Architecture Centre: MLOps, continuous delivery and automation pipelines in machine learning. The maturity levels behind the pipeline pattern, and the retrain triggers of schedule, new data, performance degradation, and drift. Link