MLOps Architecture Diagram: From Training Data to a Monitored Model

A model in a notebook is not a system. The famous figure in Hidden Technical Debt in Machine Learning Systems (Sculley et al., NeurIPS 2015) makes the point with boxes: only a small fraction of a real-world ML system is the ML code. The rest is data collection, feature extraction, serving, monitoring and the plumbing between them, and that is where production problems come from.

An MLOps architecture diagram draws the rest. This post builds one for a fraud-scoring model, explains the two loops every production model has, and lists what to label so the diagram answers the questions that come up when a model misbehaves.

Two loops and a trigger

Every production model has a training loop and a serving loop:

• Training loop. Data is collected, turned into features, used to train a candidate, evaluated against the current model and registered.
• Serving loop. A request arrives, features are looked up, the model scores it, and the prediction is logged.
• The trigger connects them. Monitoring of the serving loop, a schedule or new data starts the training loop again.

Google's MLOps guide describes three maturity levels: level 0 is a manual process, level 1 automates the training pipeline for continuous training, and level 2 adds CI/CD for the pipeline itself. Draw the level you have, not the one you plan, and mark the manual steps.

Training-serving skew

The bug an MLOps diagram prevents most often is training-serving skew: a feature computed one way in the training SQL and another way in the serving code. The model looks good offline and worse in production, and nobody can say why. The same Google guide recommends a feature store as the single source for experimentation, continuous training and online serving, so a feature has one definition.

The second classic bug is leakage through time. A training row for a payment must use feature values as they were at the time of that payment, not as they are today. Label this point-in-time requirement on the edge from the feature store to the training pipeline.

The example: fraud scoring for card payments

A payments API asks a model for a fraud score before it authorizes a card payment. The design:

1. Events. The payments API publishes every payment to Kafka. A stream job keeps rolling features, such as payments per card in the last hour, in an online store. The same events land in the warehouse.
2. Labels. Whether a payment was fraud is learned from chargebacks, which arrive weeks or months later. They are loaded into the warehouse and joined to the original payments.
3. Features. The feature store defines each feature once, computes the offline values from the warehouse and materializes the latest values to the online store.
4. Training. An orchestrated pipeline builds a point-in-time training set, validates the data, trains a candidate, compares it with the live model on the same evaluation set, and registers it with its metrics.
5. Release. An approved version is deployed to the scoring endpoint, first in shadow mode, scoring copies of live traffic without affecting decisions, then for a small share of traffic, then for all of it.
6. Monitoring. Every score is logged with its features. Monitoring compares input and score distributions with the training data and, as labels arrive, computes precision and recall. A drift or quality alert starts retraining.

How to read it:

• Top row, training. Chargeback labels and event history meet in the warehouse. The feature store builds training sets for the pipeline, which registers candidates in the model registry.
• Bottom row, serving. The payments API publishes events to Kafka and calls the scoring endpoint synchronously. The endpoint reads features from the online store and writes every score to the prediction log.
• The registry edge is the only way a model reaches production: an approved version, never a file copied from a notebook.
• Right, the loop back. Monitoring reads predictions and late labels. Drift or a drop in quality raises an alert that starts the training pipeline.

What to label

• Point-in-time rules on the training set edge.
• The model version and stage on the endpoint and the registry.
• Latency budgets for feature lookups and scoring, because the payment waits for them.
• Label delay. If labels take weeks, say so: it decides how fast you can measure quality.
• Retraining triggers: schedule, drift threshold, data volume.
• Owners. Data, features, models and the endpoint often belong to different teams.

Common mistakes

• One box called "ML model". It hides the feature store, the registry and the monitoring loop, which is where incidents start.
• No feedback path. If labels never come back, nobody knows when the model got worse.
• Notebooks in the production path. A model should reach serving only through the pipeline and the registry.
• Two implementations of one feature. Training SQL and serving code that compute "the same" value differently cause skew.
• No rollback. Keep the previous version in the registry and draw how the endpoint switches back.
• Infrastructure-only monitoring. Latency and error rates can be perfect while the model is wrong; monitor inputs, scores and outcomes too.

The serving side is ordinary distributed systems work. The event-driven architecture diagram covers the Kafka side, and for language models with retrieval, the RAG architecture diagram covers indexing and serving.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. From the ML pack, add a Feature store, a Model for the registry and an Endpoint for scoring.
3. From the Data pack, add Kafka, a Data Warehouse, Redis for the online store and a Topic for the prediction log.
4. From the DevOps pack, add a Pipeline for training, a Metrics node for model monitoring and an Alert for the retraining trigger.
5. Add a C4 Model Container for the payments API and an External System for the card network's chargebacks.
6. Connect them, label each edge with the data it carries, and make the late labels and the retraining trigger dashed. Add a Core Markdown block with the release steps.

The example uses 13 diagram nodes, within the Free plan. With an AI agent connected over MCP, you can describe the pipeline and let it place the nodes, then adjust the labels.

FAQ

Do I need a feature store? Not on day one. You need one definition per feature used by both training and serving. A feature store is the usual way to get that once several models share features or you need fresh features online.

What is the difference between a model registry and an artifact store? An artifact store keeps files. A registry adds versions, metrics, lineage and an approval state, so the deployment step can ask for the approved version instead of a path.

How often should a model be retrained? As often as the data changes in ways that matter. Start with a schedule, add drift and quality triggers once monitoring is in place, and record why each retraining ran.

How do I monitor a model when labels arrive late? Watch what you can see now, such as input distributions, score distributions and the share of payments flagged, and compute precision and recall when labels arrive. Draw both paths.

Should training and serving share infrastructure? They can share the feature definitions and the registry, but they have different load profiles. Draw them as separate paths that meet in the feature store and the registry.

Start free — one seat, no credit card.