Reference: The MLOps Lifecycle
A one-page cross-link map of the standard machine-learning operations lifecycle — the loop a model travels from first training run through production and back again. Each stage links to the corpus article that owns the deep dive on it.
The lifecycle as one picture
The lifecycle is a directed cycle with one main path and one recovery path:
TRAIN → VERSION → DEPLOY → SHADOW → CANARY → A/B TEST → MONITOR → DRIFT-DETECT → RETRAIN
│
◄────────────────────────────────────────────────────────────────────────────────────────┘
loop closes: retrain feeds a new TRAIN cycle
ROLLBACK is a side-exit, not a stage: from CANARY, A/B TEST, or MONITOR
you can drop back to the previous good VERSION without waiting for a retrain.
Read it left-to-right for a brand-new model going through the chain for the first time. A model that has been in production for six months instead re-enters the diagram at monitor, having looped back through the return arrow after its last retrain — it lives in the right third of the diagram. The left third — train, version, deploy — is the initial build. The middle third — shadow, canary, A/B — is the gauntlet a candidate model runs before it is allowed to serve real traffic. The right third — monitor, drift-detect, retrain — is where a model spends the rest of its life: it is being monitored, its inputs are being checked for drift, and when drift crosses a threshold it is retrained, which starts the cycle over at train.
Roster
| Stage | One-line definition | When it runs | Corpus deep dive |
|---|---|---|---|
| Train | Fit model weights on a dataset, with the code, data, and hyperparams all pinned so the run is reproducible. | First model and every retrain cycle. | Building an ML pipeline that avoids train/serve skew), Experiment tracking done right), What is a feature store?) |
| Version | Assign a semantic version (or content hash) to the model artifact and its metadata so it can be retrieved, compared, or restored later. | After every successful train. | Model versioning and rollbacks) |
| Deploy | Package the model + its serving code into an artifact (container, wheel, serverless function) that runs identically in prod as in dev. | When a version is promoted. | Containerizing a model with Docker and FastAPI, CI/CD for ML) |
| Shadow | Send a copy of live traffic to the new model, serve its predictions to nobody, log them for offline comparison. | First production exposure of a candidate. | A/B testing deployed models) |
| Canary | Route a small fraction of live traffic to the new model and serve its predictions for real; watch for errors. | After shadow looks healthy. | A/B testing deployed models) |
| A/B test | Split traffic between the current model and the candidate, measure a business metric, and decide promotion on a statistical bound. | Before full rollout. | A/B testing deployed models), Is your model actually better?) |
| Monitor | Track serving latency, error rate, and a proxy for prediction quality (labels arrive late) in production. | Continuously, once deployed. | Monitoring model performance) |
| Drift-detect | Compare the distribution of live input features to the distribution the model was trained on; alert when they diverge. | Continuously, on a schedule. | How to detect and handle data drift) |
| Retrain | Pull fresh data, re-run the training pipeline, produce a new candidate version. | On a schedule or when drift/alerts fire. | When should you retrain?) |
| Rollback | Swap the serving endpoint back to the previous good version. Not a stage — a recovery action available from canary, A/B, or monitor. | On alarm, any time after deploy. | Model versioning and rollbacks) |
Which stage catches which failure
The lifecycle exists because every stage in it is the answer to a production failure that happened before that stage existed. Pairing stages to the failures they catch is the fastest way to decide what your MLOps stack actually needs versus what you can defer.
| Stage | Dominant failure mode it exists to catch | Symptom you see when you skip it |
|---|---|---|
| Train | “We can’t reproduce this model.” | A great model in the notebook; nobody can say which data or code produced it. |
| Version | “Which model is in production right now?” | Two teams debugging the same bug against different model artifacts. |
| Deploy | “Works on my laptop.” | The model passes review and 500s in prod because a system library differs. |
| Shadow | “The new model is fine on the test set.” | Offline metrics look great; live inputs are a different distribution. |
| Canary | “Full-fleet outage.” | A bad model is promoted to 100% of traffic and breaks the product. |
| A/B test | “More accurate, but worse for the business.” | Accuracy went up; revenue or retention went down and nobody can explain why. |
| Monitor | “Silent quality decay.” | The model serves happily for weeks while its outputs quietly degrade. |
| Drift-detect | “The world changed, the model didn’t.” | Inputs drifted, labels are slow to arrive, and you find out from a customer. |
| Retrain | “The model is stale.” | Performance has been declining for months; nobody scheduled a refresh. |
| Rollback | “No way back.” | A bad deploy is live and the only fix is to rebuild and redeploy from scratch. |
Decision tree: where does my symptom point?
- “I can’t reproduce a model a teammate built.” → the gap is in Train (experiment tracking) and Version (artifact storage). You need pinned code + data + hyperparams and a versioned model registry.
- “The model works in dev and 500s in prod.” → the gap is in Deploy (containerization) and possibly Version (you may be deploying a different artifact than the one that was tested).
- “Offline metrics say the new model is better, but I’m afraid to ship it.” → you are missing Shadow and Canary. Run shadow first, then a small canary, before any traffic.
- “I shipped the new model and revenue dropped.” → you skipped the A/B test (or ran it without a statistical decision rule). Roll back and run a proper test.
- “The model has been in prod for months and I have no idea if it’s still good.” → the gap is Monitor. You need a serving metric and, where possible, a quality proxy.
- “Users say the model is worse, but my accuracy dashboard says it’s fine.” → the gap is Drift-detect. Your inputs have shifted; the model is confidently wrong on a new slice.
- “I know the model is stale, but I don’t know when to retrain.” → the gap is Retrain policy — a schedule or a drift-triggered retrain, not a human noticing.
- “A bad model is live and I can’t undo it.” → the gap is Rollback. You need the previous version in a registry and a one-command swap.
The lifecycle as a tiny state machine in Python — not production code, just a sketch of how the stages connect and where rollback plugs in:
# Sketch of the MLOps lifecycle as a state machine.
# Illustrative — not production code.
from enum import Enum
class Stage(Enum):
TRAIN = "train"
VERSION = "version"
DEPLOY = "deploy"
SHADOW = "shadow"
CANARY = "canary"
AB_TEST = "ab_test"
MONITOR = "monitor"
DRIFT_DETECT = "drift_detect"
RETRAIN = "retrain"
# The forward path, in order.
FORWARD = [
Stage.TRAIN, Stage.VERSION, Stage.DEPLOY,
Stage.SHADOW, Stage.CANARY, Stage.AB_TEST,
Stage.MONITOR, Stage.DRIFT_DETECT, Stage.RETRAIN,
]
def advance(current: Stage, health_ok: bool, drift_ok: bool) -> Stage:
"""Decide the next stage given the current one and its health signals.
health_ok : did the candidate pass the gate at this stage?
drift_ok : is the production model still within drift tolerance?
"""
# Rollback is a side-exit, not a stage: from canary, A/B, or monitor
# you can drop back to the previous good version without retraining.
if current in (Stage.CANARY, Stage.AB_TEST, Stage.MONITOR) and not health_ok:
# In real code: swap the serving endpoint to the previous version.
print("ROLLBACK: reverting to previous good version")
return Stage.MONITOR # back to watching the old model
if current == Stage.MONITOR:
# Monitor always hands off to drift-detect; drift-detect is the
# only stage that branches on drift_ok. (A previous version of
# this function branched here too, which made drift-detect
# unreachable — monitor either oscillated with drift-detect
# forever or jumped straight past it to retrain.)
return Stage.DRIFT_DETECT
if current == Stage.DRIFT_DETECT:
return Stage.RETRAIN if not drift_ok else Stage.MONITOR
if current == Stage.RETRAIN:
return Stage.TRAIN # loop closes: retrain feeds a new train cycle
# Default: move to the next stage in the forward path.
idx = FORWARD.index(current)
return FORWARD[(idx + 1) % len(FORWARD)]
# Example walk-through of one promotion + one drift-driven retrain.
state = Stage.TRAIN
for step in range(12):
print(f"step {step}: {state.value}")
if state == Stage.CANARY:
state = advance(state, health_ok=False, drift_ok=True) # bad canary → rollback
elif state == Stage.MONITOR:
state = advance(state, health_ok=True, drift_ok=False) # drift detected
elif state == Stage.DRIFT_DETECT:
state = advance(state, health_ok=True, drift_ok=False) # still drifting
else:
state = advance(state, health_ok=True, drift_ok=True)
Verified output of the walk-through above (run as written, Python 3.x, no randomness involved):
step 0: train
step 1: version
step 2: deploy
step 3: shadow
step 4: canary
ROLLBACK: reverting to previous good version
step 5: monitor
step 6: drift_detect
step 7: retrain
step 8: train
step 9: version
step 10: deploy
step 11: shadow
The advance function is the whole lifecycle in miniature: a forward path, a drift-triggered loop on the right, and a rollback exit available from three points. Monitor always hands off to drift-detect unconditionally; drift-detect is the sole place that checks drift_ok and decides between looping back to monitor or falling through to retrain. In a real system each branch is a pipeline stage with its own failure mode and its own corpus article — which is what the roster table above maps.
Cross-references
- Building an ML pipeline that avoids train/serve skew)
- Experiment tracking done right: MLflow vs Weights & Biases)
- What is a feature store — and do you actually need one?)
- Model versioning and rollbacks: treating models like code)
- Containerizing a model with Docker and FastAPI
- CI/CD for machine learning: what should you actually automate?)
- A/B testing deployed models: shadow, canary, and full rollout)
- Is your model actually better? A plain-English guide to model comparison)
- Monitoring model performance in production without labels)
- How to detect and handle data drift in production)
- When should you retrain? Building a simple retraining policy)
Further reading
- Google Cloud / Google Research, Practitioners Guide to MLOps: A framework for efficient MLOps (whitepaper, 2023) — the reference taxonomy most MLOps stacks implicitly follow; pairs cleanly with the stage list above.
- Databricks, Introducing MLOps with MLflow — the MLflow model-registry and experiment-tracking view of the same lifecycle.
- Library docs: MLflow, Weights & Biases, Kubeflow, Evidently (drift), Grafana (monitoring).
Related articles
- MLOps Under review
Building an ML Pipeline That Avoids Train-Serving Skew
Learn why your ML model works in training but fails in production, and how to build a skew-resistant pipeline with shared features, validation, and monitoring.
- MLOps Under review
When Should You Retrain? Building a Simple Retraining Trigger
Learn to build a retraining trigger that uses drift detection and performance monitoring to retrain ML models only when they need it, not on a fixed schedule.
- MLOps Under review
Monitoring Model Performance in Production (Without the Fancy Tools)
Models fail silently in production. Learn to detect data drift and prediction collapse with a lightweight Python dashboard before revenue drops.
- MLOps Under review
CI/CD for Machine Learning: What Should You Actually Automate?
Stop shipping models by hand — learn which ML pipeline steps to automate first, from pytest unit tests to GitHub Actions training and Docker deployment.
Looking for something else?
Search every article by title, summary or topic.