Model Versioning and Rollbacks: Treating Models Like Code You Can Revert

This is the metadata record Dev wishes he’d had when the snow-boots-in-July incident hit — a single JSON file that answers “which code, which data, which version?” without guessing.

• import json / import datetime / import hashlib — three standard-library modules. json for serializing the metadata file, datetime for timestamping, hashlib for fingerprinting data (used later in the article).
• def create_model_metadata(model_name, version, metrics, data_path, git_hash): — a function that bundles everything needed to reproduce a model into one dictionary. For Dev’s recommender, model_name might be "recommender", version might be "1.0.2", and metrics might be {"ctr": 0.034, "ndcg": 0.72}.
• "model_name": model_name — the human-readable name of the model. This is what Dev searches for when he needs to find the old version.
• "version": version — a semantic version string like "1.0.2". This is the identifier that lets Dev say “go back to 1.0.1” instead of “go back to the one before the bad one.”
• "timestamp": datetime.datetime.now().isoformat() — an ISO 8601 timestamp (e.g., 2024-07-16T14:32:05.123456). This answers “when was this trained?” — critical for debugging seasonal issues like the July snow-boots incident.
• "metrics": metrics — the evaluation results from the training run. For Dev’s recommender, this might include click-through rate, NDCG, or recall@K. This is the number that tells Dev whether this version was actually good before the deploy.
• "data_source": data_path — a pointer to the training data. In Dev’s case, this might be s3://recommender-data/2024-07-interactions.parquet. Without this, Dev can’t retrain even if he has the exact code.
• "git_commit": git_hash — the Git commit hash of the code that trained the model. This is what lets Dev git checkout a1b2c3d and reproduce the training run exactly — the thing he couldn’t do when he’d made ten commits since the last good run.
• "framework": "scikit-learn-1.2.2" — the ML library and version. A model trained with scikit-learn 1.2.2 may not load correctly in 1.3.0. This field catches that class of silent breakage.
• with open(f"{model_name}_v{version}_metadata.json", "w") as f: / json.dump(metadata, f, indent=4) — writes the metadata dictionary to a JSON file with the model name and version in the filename. indent=4 makes it human-readable. The filename pattern (recommender_v1.0.2_metadata.json) means Dev can find the metadata for any version by name, not by guessing.
• return metadata — returns the dictionary so the calling code can use it (e.g., to log it to MLflow or print it).
• meta = create_model_metadata("demand_forecaster", "1.0.2", {"rmse": 45.2}, "s3://data/v1/train.csv", "a1b2c3d") — an example call creating metadata for version 1.0.2 of a demand-forecasting model with RMSE 45.2, trained on data from an S3 path, at Git commit a1b2c3d.

This is the bare-minimum versioning workflow Dev should have run before every deploy — train, save with a versioned filename, and write a metadata sidecar file. Without these three steps, “rollback” means “I hope I can find the old file somewhere.”

• import json — this block is meant to be run on its own, and it calls json.dump at the end, so the import belongs here too rather than relying on Section 3 having run first.
• import joblib — joblib is scikit-learn’s preferred serialization library. It handles NumPy arrays efficiently and is the standard way to save sklearn models. Unlike pickle, it compresses large numerical arrays.
• from sklearn.ensemble import RandomForestClassifier / from sklearn.datasets import load_iris — imports the model class and the Iris dataset. In Dev’s recommender, this would be his collaborative-filtering or gradient-boosted model and his user-interaction dataset.
• from sklearn.model_selection import train_test_split / from sklearn.metrics import accuracy_score — needed to hold out a test set and score against it, so the accuracy written into the metadata is a measured number, not a guess.
• data = load_iris() / X, y = data.data, data.target — loads the Iris dataset (150 samples, 4 features, 3 classes) and unpacks features (X) and labels (y). This is a toy example; Dev’s real training data would be millions of user-item interactions.
• X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y) — holds out 20% of the data for evaluation. stratify=y keeps the three Iris classes proportionally represented in both splits.
• model = RandomForestClassifier(n_estimators=10, random_state=42) — creates a Random Forest with 10 trees. n_estimators is the number of decision trees in the forest. More trees generally mean better accuracy but slower training and inference.
• model.fit(X_train, y_train) / accuracy = accuracy_score(y_test, model.predict(X_test)) — trains only on the training split, then scores on the held-out test split. This run prints 0.9667 — that’s the number that goes into the metadata below, not a placeholder.
• model_name = "iris_model" / version = "1.0.0" — model_name is new here so the metadata filename can carry it, matching Section 3’s convention (see below). version is a semantic version string: MAJOR.MINOR.PATCH.
• model_dir = "model_registry/" / os.makedirs(model_dir, exist_ok=True) — creates a local directory called model_registry/ to hold versioned models and their metadata. exist_ok=True means it won’t crash if the directory already exists.
• model_filename = f"{model_name}_v{version}.joblib" — constructs a versioned filename: iris_model_v1.0.0.joblib. The v{version} in the filename is what prevents the new model from overwriting the old one — the exact mistake that left Dev with no way back. Note that the version here is the full semantic string, 1.0.0 — Section 8’s rollback loader has to agree with that, not reconstruct a different filename from a bare major-version number.
• joblib.dump(model, os.path.join(model_dir, model_filename)) — serializes the trained model to disk inside the model_registry/ directory. os.path.join handles path separators correctly across operating systems.
• metadata = {"version": version, "accuracy": accuracy, "filename": model_filename} — a small metadata dictionary. In a real run, Dev would include the Git commit, data hash, and framework version (as in Section 3). Here it’s minimal to show the pattern, but accuracy is now the measured held-out score, and filename is the literal name the model was just saved under — Section 8 reads this field instead of re-deriving the path.
• metadata_filename = f"{model_name}_v{version}_metadata.json" — uses the same model-name-carrying convention as Section 3 ({model_name}_v{version}_metadata.json), rather than a different pattern with no model name. One convention, everywhere metadata gets written.
• with open(os.path.join(model_dir, metadata_filename), "w") as f: json.dump(metadata, f) — writes the metadata as a JSON sidecar file alongside the model.
• print(f"Model and metadata saved to {model_dir}") — confirms the save. After this, Dev has two files in model_registry/: the .joblib model and the .json metadata — a versioned pair he can roll back to.

Lightweight versioning (JSON metadata + local files) — the approach from Sections 3–4.

When to use lightweight versioning (JSON metadata):

• Solo projects or prototyping
• Air-gapped environments where you can’t run an external server
• Teams too small to justify registry maintenance
• Dev’s situation before the snow-boots incident — one person, one model, one deploy

When to graduate to a full model registry:

• More than one person deploys models — you need a shared, central source of truth
• Your deployment server is different from your training server — the deploy box needs to pull “production” by name, not by SSH-ing into your laptop
• You need stage transitions (Staging → Production → Archived) with audit trails and approval workflows
• You’re managing more than 2–3 model versions and can’t keep track of which JSON file is which

Rule of thumb for Dev’s team: Start with JSON metadata files (Sections 3–4). The day someone asks “which model is live in production right now?” and you can’t answer without SSH-ing into a server, graduate to MLflow Model Registry.

This is the graduation step — instead of saving a .joblib file to a local folder, Dev logs the model to MLflow’s Model Registry, which gives it a name, a version number, and a central database that any server can query.

• import mlflow / import mlflow.sklearn — the core MLflow library plus the sklearn integration. MLflow’s sklearn module knows how to serialize scikit-learn models and their environment metadata (library versions, Python version) into the registry.
• with mlflow.start_run(): — opens an MLflow run context. Everything logged inside this block (params, metrics, the model itself) is attached to one run record. This is the same call Dev used for experiment tracking in the previous article — now he’s adding the model artifact to the registry.
• model = RandomForestClassifier().fit(X, y) — trains a Random Forest on the Iris data (X and y were defined in Section 4). All default hyperparameters are used. In Dev’s recommender, this would be his full training pipeline.
• mlflow.sklearn.log_model(sk_model=model, artifact_path="iris-model", registered_model_name="IrisClassifier") — the key call that does three things at once: (1) serializes the model as an artifact inside the run, (2) stores it under the path iris-model within that run, and (3) registers it as a new version of the named model IrisClassifier in the Model Registry. If IrisClassifier doesn’t exist yet, MLflow creates it and assigns Version 1. If it does exist, MLflow auto-increments to the next version number.
• registered_model_name="IrisClassifier" — the name under which the model is registered. This is the stable identifier Dev’s deployment server will use to load the model: “give me the production version of IrisClassifier.” The version number is managed by MLflow, not by Dev.
• print("Model registered in MLflow!") — confirms registration. Dev can now open the MLflow UI, navigate to the Models tab, and see IrisClassifier with its version history, stage tags, and metadata — all in one place.

This is the rollback mechanism Dev needed during the snow-boots incident — instead of hunting for the right .pkl file, he changes one number in a config file and the next request loads the old model.

• import json / import joblib / import os — this block runs on its own, so it needs its own imports rather than relying on earlier sections having already run.
• def set_active_version(version, config_path="deployment_config.json"): — writes deployment_config.json. Earlier drafts of this pattern only ever showed this file as a comment and never wrote it — which meant load_production_model below would raise FileNotFoundError on a clean directory. This function is what actually creates it, both at first deploy and on every rollback.
• def load_production_model(model_name, model_dir=..., config_path=...): — loads whichever model version is currently tagged as active. model_name (e.g., "iris_model") is used to find the right metadata file. In a real MLflow-based app, this whole function collapses to something like mlflow.sklearn.load_model(f"models:/{model_name}/Production").
• with open(config_path, "r") as f: config = json.load(f) — reads the deployment config file. This file is the “single source of truth” for which model is live. It’s the one file Dev needs to edit to roll back.
• version = config["active_model_version"] — extracts the version string (e.g., "1.0.0") from the config — the same full semantic-version string Section 4 used when it built the filename, not a shortened one.
• metadata_path = os.path.join(model_dir, f"{model_name}_v{version}_metadata.json") — builds the path to the metadata sidecar using the same {model_name}_v{version}_metadata.json convention Section 3 and Section 4 both write. Opening this file, not the model file, is the fix for the mismatch: the metadata’s "filename" field says exactly what the model file is called, so there’s nothing left to guess or reconstruct.
• model_path = os.path.join(model_dir, metadata["filename"]) — the actual model path, read verbatim from the metadata rather than rebuilt from the version string. If Section 4’s naming convention ever changes, this line doesn’t need to.
• return joblib.load(model_path) — deserializes the model from disk and returns it. The calling code (e.g., a FastAPI endpoint) uses this model object to make predictions.
• set_active_version("1.0.0") / load_production_model("iris_model") — deploys version 1.0.0 (the version Section 4 actually saved) and then loads it back. Prints Loaded: RandomForestClassifier. To roll back to an earlier version, call set_active_version again with that version’s string — no retraining, no redeploy.

This is the data fingerprint Dev should store in every model’s metadata — a hash that changes the instant any row in the training data changes, so he can detect silent data swaps without inspecting the dataset by hand.

• import pandas as pd — imports pandas, the standard data-manipulation library. pd.util.hash_pandas_object is a pandas utility that computes a hash for every element in a DataFrame/Series.
• def get_data_hash(df): — a function that takes a pandas DataFrame and returns a single hash string representing its entire contents. If even one cell changes, the hash changes.
• pd.util.hash_pandas_object(df).values — hash_pandas_object returns a Series of per-element hashes (one hash per row). .values extracts the underlying NumPy array of those individual hashes.
• hashlib.sha256(...).hexdigest() — takes the array of per-element hashes and computes a single SHA-256 hash over all of them. .hexdigest() returns the hash as a 64-character hexadecimal string (e.g., a3f5b2c1d4e6...). This is the compact fingerprint Dev stores in the model metadata.
• df = pd.DataFrame(X) — wraps the NumPy array X (the Iris features from Section 4) in a pandas DataFrame, because hash_pandas_object requires a pandas object.
• data_hash = get_data_hash(df) — computes the hash. For the same data, this always returns the same string. For different data (even one row changed), it returns a different string.
• print(f"Data Hash: {data_hash}") — prints the hash so Dev can compare it against the hash stored in the model’s metadata. If they match, the data hasn’t changed. If they don’t match, the training data has been modified since the model was trained.
• # Store this hash in your model metadata! — the comment reminds Dev to save this hash in the metadata JSON (from Section 3) or in the MLflow run. Without storing it, computing the hash is useless — you have a fingerprint but nothing to compare it against.

This is the environment snapshot Dev needs alongside every model — a pip freeze output that pins every library version, so the model that worked on his laptop loads correctly on the deploy server instead of crashing on a scikit-learn version mismatch.

• import subprocess — Python’s standard library module for running external commands. subprocess lets you shell out to system commands (like pip) from within Python and capture their output.
• def capture_environment(): — a function that snapshots the current Python environment and writes it to a file. Dev calls this right after training, so the environment record is tied to the exact moment the model was created.
• subprocess.check_output(["pip", "freeze"]) — runs the command pip freeze in a subprocess and captures its stdout. pip freeze outputs every installed package and its version in name==version format (e.g., scikit-learn==1.2.2, pandas==2.0.1). check_output raises an error if the command fails (non-zero exit code).
• .decode("utf-8") — check_output returns bytes; .decode("utf-8") converts them to a Python string. This is necessary in Python 3 where strings and bytes are distinct types.
• with open("env_requirements.txt", "w") as f: f.write(installed_packages) — writes the full pip freeze output to a file called env_requirements.txt. This file is the exact equivalent of a requirements.txt — someone can pip install -r env_requirements.txt to recreate the environment. Dev stores this file alongside the model metadata so he can reproduce the environment if the model needs retraining months later.
• capture_environment() — calls the function, executing the snapshot. In a real workflow, Dev would call this inside the training script right after model.fit(), so the environment is captured at training time — not later, when it might have changed.