CI/CD for Machine Learning: What Should You Actually Automate?

1. The ‘Oops’ That Costs a Weekend

Last time, Dev packaged his product-recommendation model into a Docker container — writing a Dockerfile, building the image, and serving predictions through a FastAPI /predict endpoint. The model that once crashed with ModuleNotFoundError on the engineering team’s server now ran identically inside a sealed container. But deploying it still meant SSHing into the server, copying files, and running docker build by hand.

Friday, 4:00 PM. Dev just finished retraining the recommender — the model that suggests products to ten million e-commerce users — and it hits 98% accuracy on the validation set. He feels like a hero. He copies the .pkl file to the production server, updates the code, and heads home.

Saturday morning, his phone starts blowing up. The production environment is crashing. Why? His local machine had scikit-learn 1.2.2 while the server was running 1.0.1. Or maybe he had /Users/dev/projects/ hardcoded in a preprocessing script.

This is the ‘it works on my machine’ curse. Regular software only worries about the code. Machine Learning adds a trinity of breakage: the code, the data, and the environment. Shift any one slightly and the whole system collapses.

Here’s a typical ‘manual’ script that causes these headaches:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
import joblib

# PROBLEM: This path only exists on YOUR laptop
data = pd.read_csv('/Users/dev/projects/my_ml_project/data/raw_data.csv')

def train():
    # Imagine this data has a hidden dependency on a local cleanup script
    X = data.drop('target', axis=1)
    y = data['target']
    
    model = RandomForestClassifier()
    model.fit(X, y)
    
    # PROBLEM: Saving locally without versioning
    joblib.dump(model, 'model_final_v2_REALLY_FINAL.pkl')
    print("Model saved!")

if __name__ == "__main__":
    train()

This is the kind of script Dev was running every time he retrained the recommender by hand — the script that produced the .pkl file he shipped on Friday afternoon.

• import pandas as pd — the standard data-manipulation library. Nothing wrong here, but the version of pandas on Dev’s laptop may differ from the version on the server, which can change how DataFrames handle edge cases (empty columns, NaN handling, dtype inference).
• from sklearn.ensemble import RandomForestClassifier — the model class. This is where the version mismatch bites: RandomForestClassifier in scikit-learn 1.2.2 may serialize differently than in 1.0.1, so a model saved on Dev’s laptop can fail to load (or silently produce different predictions) on the server.
• data = pd.read_csv('/Users/dev/projects/my_ml_project/data/raw_data.csv') — reads training data from a hardcoded absolute path that only exists on Dev’s laptop. On the production server, this path doesn’t exist and the script crashes with FileNotFoundError. This is the same class of bug that containerization solved for the model artifact, but it’s still alive in the training script.
• X = data.drop('target', axis=1) / y = data['target'] — separates features from the target column. axis=1 means “drop a column,” not a row. If the CSV doesn’t have a column named exactly target, this silently drops the wrong column or crashes.
• model = RandomForestClassifier() — creates a model with all default hyperparameters. No explicit configuration means no reproducibility — if Dev runs this again after a sklearn upgrade changes the defaults, he gets a different model with no record of what changed.
• model.fit(X, y) — trains the model on the full dataset. No train/test split, no cross-validation, no held-out evaluation — just a single fit with no check that the model generalizes.
• joblib.dump(model, 'model_final_v2_REALLY_FINAL.pkl') — serializes the model to disk with an ad hoc filename. There’s no version number tied to the data version, the code version, or the hyperparameters. The next model might be model_final_v3_ACTUALLY_FINAL.pkl, and nobody will know which one is in production.
• if __name__ == "__main__": — standard Python idiom that runs train() only when the script is executed directly (not when imported). This is fine, but it means the training only runs when Dev remembers to run it by hand.

Your model becomes a ‘black box’ that only you can reproduce. Lose your laptop or catch the flu, and the company loses the model. The goal of automation is to make experiments repeatable without you being there.

2. Think of CI/CD as a Robot Assistant

DevOps terms like CI and CD sound like corporate jargon. They’re not. Think of a kitchen prep cook checking ingredients before service.

• CI (Continuous Integration): Checking your work every time you save. Push code to GitHub, and a robot runs your tests to catch math or data-loading errors.
• CD (Continuous Delivery): Getting the model ready for the world. Instead of dragging files to a server by hand, the robot packages the model and ships it where it needs to go.
• CT (Continuous Training): The piece specific to ML. If your data shifts — new customer behavior, say — the robot retrains the model automatically.

Here’s what that flow looks like:

# A conceptual view of our robot assistant's logic
def ml_pipeline(new_code, new_data):
    # 1. CI Phase
    if not run_unit_tests(new_code):
        return "Stop! The code is broken."
    
    # 2. CT Phase
    model = train_model(new_data, new_code)
    metrics = evaluate(model)
    
    if metrics['accuracy'] < 0.90:
        return "Stop! The model performance dropped."
        
    # 3. CD Phase
    register_model(model)
    deploy_to_staging(model)
    print("Model is ready for review!")

This is pseudo-code — the functions run_unit_tests, train_model, evaluate, register_model, and deploy_to_staging are placeholders for real implementations. The point is the flow: test first, train second, deploy last, with hard stops at each gate.

• if not run_unit_tests(new_code): — the CI gate. If unit tests fail, the pipeline stops before any training happens. No wasted GPU time on broken code.
• model = train_model(new_data, new_code) — the CT phase. Training happens with the new data and new code together, so the model is always consistent with the latest version of both.
• metrics = evaluate(model) — runs the model against a held-out evaluation set and collects metrics. The specific metrics (accuracy, F1, RMSE) depend on the model.
• if metrics['accuracy'] < 0.90: — a quality gate. Even if the code passes and training succeeds, if the model’s accuracy drops below 90%, the pipeline stops. This prevents a regression — a model that’s technically running but performing worse than its predecessor — from reaching production.
• register_model(model) — the CD phase begins. The model is registered in a model registry (like MLflow), which assigns it a version number and stores its metadata.
• deploy_to_staging(model) — moves the model to a staging environment for final review before it goes live. In a fully automated pipeline, a final check (or human approval) gates the promotion from staging to production.

3. Step 1: Automating the ‘Sanity Checks’ (CI)

Automate the boring stuff first, not the complex AI. You want a robot catching your typos so you don’t have to. The most common error in ML is a shape mismatch—your data has 10 columns but your model expects 11.

We use Unit Tests for this. Think of a unit test as a tiny guard dog for a specific function.

import pandas as pd
import pytest

def preprocess_data(df):
    # Let's say we always expect a 'price' column
    df['price_normalized'] = df['price'] / df['price'].max()
    return df

def test_preprocess_data():
    # We create a tiny 'fake' dataset to test our logic
    fake_data = pd.DataFrame({'price': [10, 20, 30]})
    processed = preprocess_data(fake_data)
    
    # Check if the new column exists
    assert 'price_normalized' in processed.columns
    # Check if the math is right (30/30 = 1.0)
    assert processed['price_normalized'].max() == 1.0

# When you run 'pytest', it checks these rules automatically.

This is the kind of unit test Dev would write for the recommender’s preprocessing pipeline — a tiny, fast test that catches broken logic before it reaches training.

• import pytest — the testing framework. pytest discovers any function whose name starts with test_ and runs it automatically when you type pytest in the terminal.
• def preprocess_data(df): — the function being tested. It normalizes the price column by dividing every value by the column’s maximum, so the largest price becomes 1.0. In Dev’s recommender, an equivalent function might normalize product prices or user session durations before feeding them to the model.
• df['price'] / df['price'].max() — divides every value in the price column by the column’s maximum value. This is a simple max-scaling: [10, 20, 30] becomes [0.33, 0.67, 1.0].
• pd.DataFrame({'price': [10, 20, 30]}) — creates a tiny 3-row DataFrame as test input. The point of a unit test is to use the smallest possible input that exercises the logic — if the function works on 3 rows, it’ll work on 3 million.
• assert 'price_normalized' in processed.columns — checks that the function actually created the new column. If someone renames price to cost in the function, this assertion fails immediately and pytest reports the failure.
• assert processed['price_normalized'].max() == 1.0 — checks the math: 30/30 = 1.0, so the max of the normalized column should be exactly 1.0. If the normalization logic changes (e.g., dividing by min instead of max), this catches it.

Rename price to cost by accident and this test fails immediately. The robot catches the small stuff so you can focus on the math.

4. Step 2: The Hardest Part—Automating the Training (CT)

The hardest part of MLOps is managing the data. Don’t just run python train.py on a server. The server might not have the same data you have.

Tools like GitHub Actions can trigger training automatically. The problem: training is expensive. Spending $50 on cloud GPUs every time you fix a typo in your README makes no sense. Train only when the data or training logic changes.

Here’s a simplified YAML config that tells GitHub to run training only when files in the data/ or models/ folder change:

# (Simplified logic represented in Python comments for a YAML structure)
# name: Train Model
# on:
#   push:
#     paths:
#       - 'data/**'
#       - 'src/train.py'
# jobs:
#   train:
#     runs-on: ubuntu-latest
#     steps:
#       - run: python src/train.py
#       - run: python src/evaluate.py --output report.md

This is a GitHub Actions workflow file (YAML), shown as Python comments for readability. In a real project, this would be a file at .github/workflows/train.yml that GitHub reads automatically.

• name: Train Model — the workflow name, shown in the GitHub Actions UI and in status checks on pull requests.
• on: push: paths: — the trigger. This workflow runs only when code is pushed to the repository AND the changed files match the specified paths. This is the key to not wasting money: a typo in README.md won’t trigger a $50 GPU run, but a new CSV in data/ will.
• 'data/**' — a glob pattern matching any file inside the data/ directory (recursively). ** means “any depth,” so data/raw/2024-01.csv and data/processed/features.parquet both match.
• 'src/train.py' — if the training script itself changes, retrain. Someone might fix a preprocessing bug in train.py, and you want the model retrained with the fix.
• runs-on: ubuntu-latest — GitHub provides a virtual machine (a “runner”) with Ubuntu Linux pre-installed. This is the clean, disposable environment where training runs — not Dev’s laptop, not the production server.
• python src/train.py — runs the training script. In Dev’s case, this would retrain the recommender on the latest data.
• python src/evaluate.py --output report.md — runs evaluation and writes a report. In a real setup, this report would be posted as a comment on the GitHub Pull Request so reviewers can see the metrics before approving the merge.

In a real-world setup, you’d use DVC (Data Version Control). Think of DVC as Git for huge data files. It guarantees that when the robot trains the model, it uses the exact same version of the data you used on your laptop.

5. Step 3: Shipping the Model Without Breaking Things (CD)

Once the model is trained, you ship it. Docker is the shipping container for your code. It bundles everything needed to run, so the destination doesn’t matter — Linux server or Windows cloud, same container.

Before real customers touch the model, run a Smoke Test. It’s a simple check: can the model predict ‘1’ before it sees real traffic?

import joblib
import numpy as np

def verify_model_health(model_path):
    try:
        model = joblib.load(model_path)
        # Create a dummy input based on expected features
        dummy_input = np.array([[0] * 10]) 
        prediction = model.predict(dummy_input)
        
        print(f"Health Check Passed: Model predicted {prediction}")
        return True
    except Exception as e:
        print(f"Health Check Failed: {e}")
        return False

# If this returns True, the robot is allowed to 'push' the model to production.

This is a smoke test — the simplest possible check that the model is alive and can produce a prediction. Dev would run this after training but before deploying the recommender to production.

• joblib.load(model_path) — deserializes the model from disk. If the file is missing, corrupted, or incompatible with the installed scikit-learn version (the exact problem that caused the Saturday morning crash), this throws an exception.
• np.array([[0] * 10]) — creates a 1×10 array of zeros. [0] * 10 repeats the value 0 ten times, producing [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]. The double brackets [[...]] make it 2-dimensional (one row, ten columns), because sklearn’s predict expects a 2-D feature matrix. The number 10 must match the number of features the model was trained on.
• model.predict(dummy_input) — runs inference on the dummy input. If the model expects a different number of features (e.g., 8 instead of 10), this throws a shape mismatch error. If it succeeds, the model is “alive” — it loaded and can predict.
• try/except Exception as e: — catches any error (file not found, shape mismatch, deserialization error) and returns False instead of crashing the deployment pipeline. The error message is printed for debugging.
• return True / return False — the boolean return is the signal the CD pipeline uses. True means “safe to deploy”; False means “stop the pipeline and alert Dev.” This is the gate between training and production — if the smoke test fails, the model never reaches the ten million users.

If the test passes, the robot moves the model to a ‘Model Registry’ (like MLflow). It’s a library where every model carries a version number, a date, and a ‘pass’ certificate from your tests.

6. So, What Should You Automate First?

You don’t need a spaceship when a bicycle will do. Automation works best as iteration, not a one-shot project.

CI vs. CD vs. CT — which should Dev automate first, and which can wait?

Dev’s team is small — just him and one platform engineer. They can’t build all three pipelines at once, so which matters most?

Stage	What to do	Tool Example
Day 1	Automated Linting & Unit Tests	Pytest, Flake8
Month 1	Automated Training (CT)	GitHub Actions, DVC
Month 3	Automated Deployment (CD)	Docker, MLflow, FastAPI

What this actually means for you:

1. Start with Pytest. Write three tests for your data cleaning function today.
2. Use GitHub Actions. Set up a simple script that runs your tests every time you push code.
3. Stop manual naming. Never name a file model_final_v2.pkl again. Use a versioning tool.

The tradeoff — which gate pays off first for a small team:

Automation type	What it prevents	Effort to set up	Risk of NOT having it	Worth it for Dev right now?
CI (unit tests + linting)	Typos, shape mismatches, hardcoded paths, broken imports	Low: write a few pytest functions, add a GitHub Actions YAML that runs them on push	You ship bugs that crash on the first request; debugging happens in production	Yes — start here. Cheapest to set up, catches the highest-frequency failures, and gives immediate confidence
CD (automated deployment)	Manual deploy mistakes, environment mismatches, no rollback path	Medium-high: container registry, deployment pipeline, smoke tests, rollback mechanism	Saturday-morning crashes from manual deploys; no way back if a bad model ships	After CI, before CT. Dev’s Friday deploy disaster is exactly what CD prevents — but it depends on CI being in place first (you can’t safely automate deploying something you haven’t tested)
CT (automated retraining)	Stale models, manual retraining errors, inconsistent training environments	Medium: data versioning (DVC), training runner, trigger on data changes	Model goes stale; someone forgets a preprocessing step; data drift goes unnoticed	Last. Only worth the investment once the model is live in production and drift is a real concern

When to break this order: If Dev’s team is already manually retraining and deploying every week under time pressure, and the manual deploy is the riskiest step (it was — the Saturday crash came from a manual deploy), then automating CD first might be the right call. The “unit tests first” advice assumes you have time to build foundation. If you’re already bleeding from manual deploys, stop the bleeding first.

You don’t need to be a ‘DevOps expert.’ Automation is about confidence. When your tests run automatically, Friday night stops being a gamble on what Saturday morning brings.

Ready to try it? Write one unit test for your current project and see how many hidden bugs you find.

Dev’s deploys were automated now — the robot caught his typos and retrained on schedule. The recommender shipped through a smoke-tested pipeline. But when his manager asked which model version was actually live and what hyperparameters it was trained with, Dev stared at a folder of files named model_final_v2_REALLY_FINAL.pkl with no answer. He’d need experiment tracking for that.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What do CI, CD, and CT stand for in the article’s “kitchen prep cook” framing, and what does each one automate?

Understand In your own words, explain why the article says you shouldn’t retrain the model every time you push a code change—what does the “paths” filter in the GitHub Actions example prevent?

Apply Using the article’s verify_model_health smoke test pattern, would this test catch a bug where the model was trained correctly but the training script accidentally saved a model expecting 8 features instead of 10, given the dummy_input in the example?

Analyze The article opens with a Friday-4pm deploy that broke because of a scikit-learn version mismatch between the developer’s laptop and the production server. Walk through which of the article’s three automation stages (CI, CT, or CD) would have caught this specific failure, and why the other two wouldn’t have.

Evaluate The article’s “Day 1 / Month 1 / Month 3” rollout table recommends unit tests first, training automation second, deployment automation last. Critique this ordering for a team that’s already manually retraining and deploying models weekly under real time pressure: is “unit tests first” still the right priority, or would automating the riskiest manual step first make more sense for them specifically?

Create Design a CI check (not full code, just the logic) that would have caught the article’s opening disaster — a hardcoded local file path (/Users/dev/projects/...) that only exists on the developer’s machine. What would the check look for, and at what point in the pipeline would you run it?

---

Related articles

• Containerizing a Model with Docker and FastAPI for Production — Dev packages the recommender into a Docker container with FastAPI, the foundation that this article’s CD pipeline deploys.
• Experiment Tracking Done Right: MLflow vs. Weights & Biases) — Dev adopts experiment tracking after realizing his CI/CD pipeline has no organized record of which model version, trained with which hyperparameters, is actually live in production.

References & Further reading

• Google Cloud. (2021). Practitioner’s Guide to MLOps Whitepaper. — defines the CI/CD/CT maturity model for ML systems and the automation stages this article walks through.
• GitHub Actions documentation — https://docs.github.com/en/actions — reference for workflow YAML syntax, path filters (on.push.paths), and self-hosted vs. GitHub-hosted runners.
• GitLab CI/CD documentation — https://docs.gitlab.com/ee/ci/ — alternative CI/CD platform with built-in container registry and ML pipeline templates; useful for teams not on GitHub.

---