Experiment Tracking Done Right: MLflow vs. Weights & Biases

1. Why Your Spreadsheet of Model Runs Is Costing You Money

Last time, Dev automated the recommender’s deploys with CI/CD. The robot caught his typos, retrained on schedule, and shipped the model through a smoke-tested pipeline. But when his manager asked which model version was live and what hyperparameters it used, Dev stared at a folder of files named model_final_v2_REALLY_FINAL.pkl with no answer.

Have you ever spent three days training a model, finally hit 92% accuracy, and then… forgotten which learning rate you used?

Dev has. He spent a long weekend retraining the product-recommendation model — the one that serves ten million e-commerce users — and found a configuration that lifted click-through rate by three points. He celebrated, wrote recommender_good_run.pkl to a shared drive, and went home. Monday, his manager asked: “Great results — what learning rate did you use? And which version of the training data?” Dev opened his notebook, scrolled through forty cells of experiments, and realized he had no idea. The run that worked was indistinguishable from the fifty that didn’t.

It starts simply enough. A Jupyter Notebook. Then model_v1.ipynb, model_v1_final.ipynb, and eventually model_v1_final_FINAL_use_this_one.ipynb. Maybe even a spreadsheet where you type in your results by hand.

This is the hardest part of early-stage ML development. It feels productive. You’re building a house of cards. If a teammate asks how you got that result, or you need to reproduce that model three months from now for production, you’re stuck guessing. The hidden cost isn’t just lost time — it’s the inability to learn from your

2. What Experiment Tracking Actually Does

Think of an experiment tracker as a “Black Box Flight Recorder” for your machine learning models. Not magic. Just a very organized, automated diary.

An experiment tracker logs four main things:

1. Hyperparameters: The settings you chose (like learning rate or tree depth).
2. Metrics: The results (like Loss, Accuracy, or F1-score).
3. Code/Environment: Exactly what version of the code ran and which libraries were installed.
4. Artifacts: The file containing the trained model weights.

So you can eventually ask: “Show me every run from last Tuesday where the dropout was higher than 0.2,” and the tool hands back a ranked list.

3. MLflow: The Open-Source Workhorse

MLflow is the “DIY” favorite. It’s open-source, free, and runs entirely on your local machine if you want it to. That’s useful if you have strict data privacy rules or just don’t want to sign up for another cloud service.

MLflow is built on three pillars: Tracking (logging the data), Projects (packaging the code), and Registry (handling model versions).

Here’s what happens when we log a simple run locally:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# 1. Start an experiment
mlflow.set_experiment("My_First_MLflow_Project")

with mlflow.start_run():
    # Define parameters
    n_estimators = 100
    max_depth = 5
    
    # Train a dummy model
    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)
    model.fit([[0, 1], [1, 0]], [0, 1])
    
    # Log parameters and metrics
    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)
    mlflow.log_metric("accuracy", 0.95)
    
    # Save the model artifact
    mlflow.sklearn.log_model(model, "random_forest_model")

print("Run finished! Type 'mlflow ui' in your terminal to see the results.")

This is the kind of tracking call Dev would add to his recommender’s training script — four lines that turn an anonymous .pkl file into a queryable record.

• import mlflow / import mlflow.sklearn — the core MLflow library plus the sklearn integration, which knows how to serialize scikit-learn models as MLflow artifacts.
• mlflow.set_experiment("My_First_MLflow_Project") — creates (or opens) a named experiment. All runs inside this block are grouped together in the MLflow UI, so Dev can see all his recommender experiments in one table instead of scattered across notebooks.
• with mlflow.start_run(): — opens a “run” context manager. Everything logged inside this with block — params, metrics, artifacts — is attached to a single run record. When the block exits, the run is automatically closed.
• n_estimators = 100 / max_depth = 5 — hyperparameters defined as plain Python variables. These are the values Dev would lose track of without tracking — the exact numbers that produced the good run.
• model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth) — creates a RandomForest with the specified hyperparameters. In Dev’s recommender, this might be a collaborative-filtering model or a gradient-boosted tree instead, but the tracking pattern is the same.
• model.fit([[0, 1], [1, 0]], [0, 1]) — trains on a tiny 2-row dummy dataset. In production, Dev would pass his full training matrix here. The dummy data keeps the example runnable without downloading a dataset.
• mlflow.log_param("n_estimators", n_estimators) — logs the hyperparameter value to MLflow. This is the key call: it writes “n_estimators = 100” into the run’s metadata so Dev can query it later. Without this, the value lives only in Dev’s memory.
• mlflow.log_metric("accuracy", 0.95) — logs a metric. Unlike params (which are set once), metrics can be logged multiple times (e.g., once per epoch) and MLflow tracks the history. Here it’s a single final value.
• mlflow.sklearn.log_model(model, "random_forest_model") — serializes the trained model as an artifact inside the run’s directory. This is what lets Dev load the exact model later — the weights, the sklearn version, and the model type are all captured.
• print("Run finished! ...") — the run is complete. Typing mlflow ui in the terminal opens a local web dashboard where Dev can see every run, its params, its metrics, and its artifacts in a sortable table.

Interpretation: When you run this, MLflow creates a local folder called mlruns. It stores the number 0.95 and the parameter 100 in a simple file. It’s lightweight. Everything stays on your hard drive.

4. Weights & Biases: The Collaborative Cloud Platform

Weights & Biases (W&B) is like Instagram for your model metrics. It’s a hosted service, so your data goes to their servers and you get an interactive web dashboard back.

MLflow is functional. W&B is polished. It’s built for teams who want to leave comments on each other’s graphs and track real-time hardware usage—like how hot your GPU is getting.

import wandb

# 1. Initialize W&B (requires a free account and API key)
wandb.init(project="my-awesome-project", config={
    "learning_rate": 0.01,
    "architecture": "CNN",
    "dataset": "CIFAR-10"
})

# 2. Log metrics during training
for epoch in range(5):
    loss = 0.5 / (epoch + 1)
    wandb.log({"loss": loss, "epoch": epoch})

# 3. Finish the run
wandb.finish()

This is the W&B equivalent of the MLflow logging above — same data captured, different philosophy: everything goes to the cloud and renders as interactive charts.

• import wandb — the Weights & Biases client library. Unlike MLflow, which stores locally by default, W&B sends data to their hosted servers. You need a free account and API key (set via wandb login or the WANDB_API_KEY environment variable).
• wandb.init(project="my-awesome-project", config={...}) — starts a run and registers it under a named project. The config dictionary is W&B’s equivalent of MLflow’s log_param calls — it records the hyperparameters and settings for this run. In Dev’s recommender, this might include {"model_type": "als", "factors": 64, "regularization": 0.01}.
• "learning_rate": 0.01 / "architecture": "CNN" / "dataset": "CIFAR-10" — config values logged as key-value pairs. These appear as columns in the W&B dashboard, so Dev can sort and filter runs by any of them.
• for epoch in range(5): — a training loop simulating 5 epochs. In a real recommender training run, this might be 50 or 100 epochs over millions of user-item interactions.
• loss = 0.5 / (epoch + 1) — a fake loss that decreases each epoch. Epoch 0: 0.5, epoch 1: 0.25, epoch 2: 0.167, etc. This simulates a converging loss curve for the demo.
• wandb.log({"loss": loss, "epoch": epoch}) — logs a metric step. Unlike MLflow’s log_metric (which can also be called per epoch), W&B renders this as a live-updating line chart in the dashboard. Dev can watch the loss drop in real time from his browser.
• wandb.finish() — closes the run and uploads any remaining buffered data to W&B’s servers. Without this call, the run may show as “running” in the dashboard indefinitely.

Interpretation: Here’s the catch—you need an internet connection. In exchange, you get a URL you can send your boss, one that shows a live-updating graph of your loss decreasing. The config dictionary is your single source of truth for settings.

5. Head-to-Head: MLflow vs. W&B

Feature	MLflow	Weights & Biases
Setup	5 mins (Local)	2 mins (Cloud Sign-up)
Cost	Free (Open Source)	Free for individuals; Paid for teams
UI	Basic, functional tables	Beautiful, interactive charts
Data Privacy	You own the server	Data lives in their cloud
Collaboration	Requires manual server setup	Built-in team dashboards

6. A Concrete Example: Training a Scikit-Learn Model

Let’s compare them by running a real classifier. We want to see which tool makes it easier to find the “best” model.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
import mlflow
import wandb

data = load_iris()
X_train, X_test, y_train, y_test = train_test_split(data.data, data.target)

def train_model(C_value):
    # Log to MLflow
    with mlflow.start_run(run_name=f"SVM_C_{C_value}"):
        model = SVC(C=C_value)
        model.fit(X_train, y_train)
        acc = model.score(X_test, y_test)
        
        mlflow.log_param("C", C_value)
        mlflow.log_metric("accuracy", acc)
        
    # Log to W&B
    run = wandb.init(project="iris-comparison", name=f"SVM_C_{C_value}", reinit=True)
    wandb.log({"accuracy": acc, "C": C_value})
    run.finish()

# Try two different settings
for c in [0.1, 1.0]:
    train_model(c)

This is a side-by-side comparison — the same model, same data, same hyperparameter sweep, logged to both tools at once so Dev can see how each dashboard presents the results.

• from sklearn.datasets import load_iris — loads the Iris dataset, a classic 3-class classification benchmark (150 flowers, 4 features). In Dev’s recommender, the equivalent would be loading user-interaction logs, but Iris keeps the example lightweight.
• train_test_split(data.data, data.target) — splits the data into training and test sets. By default, 75% train / 25% test, with shuffling. X_train and y_train are used for fitting; X_test and y_test for evaluation.
• def train_model(C_value): — a function that trains an SVM with a given C value and logs the result to both MLflow and W&B. C is the regularization parameter: smaller C = stronger regularization (more generalization, possibly underfitting); larger C = weaker regularization (more fitting, possibly overfitting).
• with mlflow.start_run(run_name=f"SVM_C_{C_value}"): — opens an MLflow run with a descriptive name like SVM_C_0.1. The f"..." is an f-string that interpolates the C_value into the run name — so Dev can tell at a glance which run used which setting.
• model = SVC(C=C_value) — creates a Support Vector Classifier with the specified C. The other hyperparameters (kernel, gamma, etc.) use sklearn defaults.
• model.fit(X_train, y_train) — trains the SVM on the training split.
• acc = model.score(X_test, y_test) — computes accuracy on the held-out test set. This is the number that tells Dev whether C=0.1 or C=1.0 performed better.
• mlflow.log_param("C", C_value) / mlflow.log_metric("accuracy", acc) — logs the hyperparameter and the resulting accuracy to MLflow. After both calls, the MLflow UI shows two rows (one per C value) with accuracy as a sortable column.
• wandb.init(project="iris-comparison", name=f"SVM_C_{C_value}", reinit=True) — starts a W&B run in the same project. reinit=True allows W&B to start a new run in the same process (otherwise it would complain about re-initializing). The name matches the MLflow run name for easy cross-referencing.
• wandb.log({"accuracy": acc, "C": C_value}) — logs both the metric and the hyperparameter to W&B in a single call. W&B renders this as a point on a scatter plot of accuracy vs. C.
• run.finish() — closes the W&B run. Each iteration of the loop creates and finishes one W&B run and one MLflow run.
• for c in [0.1, 1.0]: — tries two C values. After the loop, Dev has two runs in each tool and can compare which C gave higher accuracy.

In practice: Open the MLflow UI and you’ll see two rows in a table. Open W&B and you’ll see two points on a scatter plot. Both answer the same question: which C value performed better. The choice comes down to how you prefer to view that data.

7. When to Choose MLflow

Choose MLflow if:

• Privacy is #1: You are working with sensitive medical or financial data that cannot leave your network.
• You love Open Source: You want to be able to modify the tool or avoid paying a monthly subscription as your team grows.
• Simple Needs: You just need a local log of what you did yesterday.

8. When to Choose Weights & Biases

Choose W&B if:

• You work in a team: You want to see what your colleague in a different time zone is working on without asking them for a CSV.
• Deep Learning focus: W&B has incredible integrations for PyTorch and TensorFlow that automatically log system metrics like GPU memory usage.
• Visuals matter: You need to present your findings to stakeholders who respond better to clean charts than raw tables.

9. Hybrid Approach: Using Both (Or Neither)

But wait—you don’t strictly have to choose. Some teams use MLflow for the heavy lifting (storing massive model files on their own servers) and W&B just for the high-level metric dashboards.

However, for most people, picking one and sticking to it is better. The worst choice is “neither.” Building a custom logging system using logging.info() and text files is a recipe for technical debt.

10. Getting Started: Your First Experiment

Here’s a 60-second action plan:

1. Run pip install mlflow.
2. Add mlflow.autolog() to the top of your script.
3. Run your training code.
4. Type mlflow ui in your terminal.

Interpretation: autolog() automatically captures parameters and metrics for most popular libraries. The fastest way to stop tracking experiments by hand.

11. Common Pitfalls and How to Avoid Them

• Logging too much: Don’t log the loss every millisecond. The UI bogs down and the charts get unreadable. Log once per epoch, or every 100 batches.
• Vague names: Skip run names like “test1” and “test2”. Tag them with what they actually are — “baseline”, “experiment_lr_search”, “final_candidate”.
• Forgetting the data: A model is Code + Data. Change your dataset without logging a version number or a hash of the data, and the experiment isn’t truly reproducible.

12. What’s Next: From Tracking to Reproducibility

Experiment tracking is where MLOps begins. Once you can track what happened, the real challenge is making sure you can repeat it. In the next part of this series, we’ll turn to Model Registries—the process of taking a “winning” experiment and stamping it as “Ready for Production.”

13. Recap and Decision Framework

• MLflow is your private, free, local toolbox.
• W&B is your collaborative, cloud-based visual dashboard.
• The Goal: Stop using spreadsheets. Start logging every parameter and metric automatically.

Decision Tree: Solo on a private server? Go MLflow. On a team where the UI matters, go Weights & Biases.

---

With experiment tracking in place, Dev could finally answer his manager’s question. He logged every recommender run — hyperparameters, metrics, data version, model artifacts — and the dashboard made it trivial to see which configuration won. He promoted the best run to production. Within hours, the click-through rate dropped. The new model was worse than the old one, and Dev needed to roll back — fast. He reached for the previous model, only to find he’d never built a real way to revert. He had tracking, but he didn’t have versioning.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What are the four main things an experiment tracker logs, according to the article?

Understand In your own words, explain why the article calls a folder of notebooks named model_v1_final_FINAL_use_this_one.ipynb “a house of cards” rather than just messy but harmless.

Apply Using the article’s decision tree (private server vs. team + best UI), which tool would you recommend for a solo researcher at a hospital working with patient data that legally cannot leave the internal network?

Analyze The article’s Pitfall section warns against “Forgetting the Data” — logging code and metrics but not a version or hash of the dataset used. Walk through why a model with perfectly logged hyperparameters and metrics could still be unreproducible three months later if the underlying dataset changed silently in that time.

Evaluate The article recommends mlflow.autolog() as “the fastest way to get out of spreadsheet hell.” Critique blindly relying on autologging as your only tracking strategy: what kind of experiment-specific context (the “why” behind a run) does autologging not capture that a human still needs to record manually?

Create Design a tagging and naming convention (following the article’s advice against “test1”, “test2”) for a team running a systematic learning-rate sweep across three model architectures. What tags or run-name pattern would let a teammate filter to “all runs for architecture B with LR > 0.01” six months from now?

---

Related articles

• CI/CD for Machine Learning: What Should You Actually Automate? — Dev automated the recommender’s deploys with CI/CD, but had no organized record of which model version was live — the gap that experiment tracking fills.
• Model Versioning and Rollbacks: Treating Models Like Code — Dev promotes a tracked model to production, discovers it’s worse, and needs to roll back fast — building the versioning and rollback system he never had.

References & Further reading

• MLflow documentation — https://mlflow.org/docs/latest/ — official reference for the Tracking API, Model Registry, autolog() integrations, and the MLflow UI.
• Weights & Biases documentation — https://docs.wandb.ai/ — official reference for wandb.init, wandb.log, team dashboards, and framework integrations (PyTorch, TensorFlow, scikit-learn).

---