Building a Baseline in 10 Minutes: A Practical AutoML Workflow

The 10-Minute Challenge

You just got handed a new dataset. Your boss wants results by end of day. You could spend hours exploring the data, testing algorithms, and tuning hyperparameters — but honestly, you’ve got three other meetings this afternoon.

What’s the fastest way to get a decent model that tells you if your data even has signal?

That’s exactly what a baseline model does. A baseline is a simple, fast model that sets a floor for performance. Think of it as checking if the oven works before you start baking a complex soufflé.

Here’s the dirty secret experienced data scientists know: Google’s Rule #4 for machine learning tells us to “keep the first model simple and get the infrastructure right” (source). You don’t need a production-ready masterpiece in 10 minutes. You need a sanity check.

By the end of this tutorial, you’ll have a working baseline in under 10 minutes using AutoML tools. And you’ll know whether your dataset is worth investing more time in — or if you should go back to the drawing board.

But wait — why bother with a baseline at all? Let’s dig into that first.

Why Bother with a Baseline? (Intuition Before Formalism)

Imagine you’re a chef building a complex dish like a beef Wellington. Before you spend two hours assembling the perfect layers of puff pastry and mushroom duxelles, what do you do first?

You taste the individual ingredients. You check whether the beef is fresh, whether the salt is still good, whether the oven actually heats evenly.

A baseline is that taste test for your model.

Here’s the plain-English definition: A baseline is the simplest model you can build. For classification, that often means just predicting the most common class every time. For regression, it means predicting the average value. Period.

Without a baseline, you can’t tell if your fancy model is actually learning or just memorizing noise. A model that gets 85% accuracy sounds impressive — until you find out that 85% of your data belongs to one class, and your “smart” model just guessed that class every time.

This is the hardest part for beginners to accept: You need to measure how well a dumb model performs first, because that gives you the true floor. Any real model must beat this floor to be useful.

scikit-learn gives you two tools for this: DummyClassifier and DummyRegressor. The docs call them “simple baselines” that ignore the input features entirely (source). They’re not fancy. They’re not smart. But they’re essential.

Let’s see what that looks like in code.

Your First Baseline: The Dummy Model (Code + Interpretation)

Let’s build a real baseline on a familiar dataset: the Titanic survival data from seaborn. You’ve seen this dataset a hundred times. Now let’s see how a truly dumb model performs on it.

# Self-contained: imports, data loading, dummy model, evaluation
import pandas as pd
import seaborn as sns
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load the Titanic dataset
# This is a real dataset available in seaborn
# We'll use only numeric columns to keep it simple
data = sns.load_dataset('titanic')
data = data[['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']].dropna()

X = data.drop('survived', axis=1)
y = data['survived']

# Split the data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a DummyClassifier that always predicts the most common class
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
y_pred_dummy = dummy.predict(X_test)

# Evaluate
dummy_accuracy = accuracy_score(y_test, y_pred_dummy)
print(f"Dummy (most frequent) accuracy: {dummy_accuracy:.4f}")
print(f"Most common class found: {dummy.classes_[dummy.predict_proba(X_test)[0].argmax()]}")
print()
print("In plain English: If we always guess 'died' (the most common outcome),")
print(f"we get {dummy_accuracy*100:.1f}% accuracy. Any real model must beat this.")

What the numbers mean: The DummyClassifier got about 58-62% accuracy on the Titanic dataset (it varies depending on the random split). That’s the floor. If your fancy model can’t beat 58%, you have a problem — either your features aren’t predictive, or something is wrong with your data.

According to the Deepchecks article on baselines, “setting a baseline performance using a simple model offers a ground-level measurement” (source). That ground-level measurement is your starting point.

Level Up: A Simple Model (Logistic Regression) as a Stronger Baseline

The DummyClassifier is the true floor, but it’s not very useful as a point of comparison. A logistic regression — a simple linear model — is a much more practical baseline. It’s still simple, but at least it uses your features.

Let’s build one and compare it to the dummy.

# Self-contained: load data, train both models, compare
import pandas as pd
import seaborn as sns
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load data (same as before)
data = sns.load_dataset('titanic')
data = data[['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']].dropna()

X = data.drop('survived', axis=1)
y = data['survived']

# Split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Dummy baseline
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
y_pred_dummy = dummy.predict(X_test)
dummy_accuracy = accuracy_score(y_test, y_pred_dummy)

# Logistic regression baseline
logreg = LogisticRegression(max_iter=1000, random_state=42)
logreg.fit(X_train, y_train)
y_pred_logreg = logreg.predict(X_test)
logreg_accuracy = accuracy_score(y_test, y_pred_logreg)

# Print comparison
print(f"Dummy baseline accuracy:      {dummy_accuracy:.4f}")
print(f"Logistic regression accuracy: {logreg_accuracy:.4f}")
print()
print("If logistic regression barely beats the dummy, your features might not be predictive.")
print(f"Here, logistic regression {'beats' if logreg_accuracy > dummy_accuracy else 'ties or loses to'} the dummy.")

What the numbers mean: Odds are, the logistic regression beats the dummy by a decent margin — maybe 78-80% vs. 58-62%. That tells us our features (passenger class, age, fare, etc.) actually have some predictive power. If the gap were only 1-2%, we’d know something was wrong.

As the Towards Data Science article on baselines puts it: “If trained models can’t beat the baseline, it could be a sign that the data set lacks predictive power” (source).

Now let’s see how AutoML can do this comparison for us — automatically trying dozens of models in seconds.

Enter AutoML: PyCaret in 5 Lines

Now for the fun part. PyCaret is an open-source AutoML library that can compare multiple models and pick the best one in just a few lines of code. Let’s see it in action.

# Self-contained: install-free, uses PyCaret functional API
# Note: Requires pycaret installed (pip install pycaret)
import pandas as pd
import seaborn as sns
from pycaret.classification import *

# Load the same Titanic data
data = sns.load_dataset('titanic')
data = data[['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']].dropna()

# PyCaret setup handles everything: missing values, encoding, scaling, split
# The 'target' parameter tells it which column is our target
s = setup(data=data, target='survived', session_id=42, verbose=False)

# Compare models — trains 15+ models and returns a leaderboard
best_model = compare_models(verbose=False)

# We'll print the leaderboard manually from the results
# The compare_models() function prints a table; we can also access the results
# For interpretation, let's evaluate the top model on the test set
print("PyCaret's top model:", type(best_model).__name__)
predictions = predict_model(best_model, data=data)
print("\nLook at the Accuracy column in the output above to see how it compares to our baselines.")
print("PyCaret tried 15+ models and picked the best one.")

What’s actually going on here? The setup() function automatically handles missing value imputation, categorical encoding, feature scaling, and train/test splitting. The compare_models() function then trains a suite of algorithms — decision trees, random forests, gradient boosting machines, SVMs, and more — and ranks them by your chosen metric (default is accuracy for classification).

The Analytics Vidhya tutorial on PyCaret calls this workflow “building a machine learning model in seconds” (source), and they’re not exaggerating. The LearnDataSci tutorial similarly shows how compare_models() gives you a quick leaderboard without writing loops (source).

But wait — PyCaret can be slow for large datasets. It’s trying 15+ models, some of which are computationally expensive. For the small Titanic dataset, it’s fine. But if you have 100,000+ rows, you might be waiting a while. Let’s look at a faster alternative.

But Wait — PyCaret Can Be Slow: Enter FLAML

FLAML (Fast Lightweight AutoML) is a Microsoft-developed AutoML library that emphasizes speed and resource efficiency. The docs call it “an economical and fast AutoML engine as a scikit-learn style estimator” (source).

Here’s the workflow in just three lines.

# Self-contained: FLAML model fit and evaluation
# Requires: pip install flaml
import pandas as pd
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from flaml import AutoML

# Load data
data = sns.load_dataset('titanic')
data = data[['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']].dropna()
X = data.drop('survived', axis=1)
y = data['survived']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# FLAML automl object
# The time_budget parameter tells FLAML how many seconds to search
# Here we give it 60 seconds to find the best model
automl = AutoML()
automl.fit(X_train, y_train, task='classification', time_budget=60, verbose=False)

# Evaluate
y_pred_flaml = automl.predict(X_test)
flaml_accuracy = accuracy_score(y_test, y_pred_flaml)

print(f"FLAML best estimator: {automl.best_estimator}")
print(f"FLAML accuracy: {flaml_accuracy:.4f}")
print()
print("In 60 seconds, FLAML searched multiple models and found a good one.")
print("The best estimator varies — it could be XGBoost, LightGBM, or a Random Forest.")

What the numbers mean: In 60 seconds, FLAML found a model — likely an XGBoost or LightGBM variant — that matches or beats what PyCaret found. The key difference: FLAML is budget-aware. It starts with simple models and allocates more time to promising ones, rather than trying everything exhaustively.

The MLJAR blog comparison of AutoML frameworks notes that PyCaret is “often slower compared to lightweight AutoML frameworks,” while FLAML “employs budget-aware optimization strategies” (source). That’s exactly what we’re seeing here.

What About Heavier Tools? (A Quick Mention of auto-sklearn)

Before we wrap up, let’s briefly mention the heavier option. auto-sklearn is described as a “drop-in replacement for a scikit-learn estimator” that “frees the user from algorithm selection and hyperparameter tuning” (source).

It’s powerful — it uses Bayesian optimization and builds ensembles of the best models. But it’s slower. Much slower. For a 10-minute baseline, stick with PyCaret or FLAML. Save auto-sklearn for when you have an hour or more to dedicate to finding the absolute best model.

This is the hardest part of the AutoML toolkit: knowing when to use which tool. For speed and sanity, PyCaret and FLAML are your go-tos. For competition-grade performance, auto-sklearn is waiting in the wings.

Putting It All Together: Your 10-Minute Workflow

Here’s the recipe you can follow with any new dataset:

Step 1: Build a dummy baseline (30 seconds)

• Train a DummyClassifier or DummyRegressor
• Get the absolute floor performance

Step 2: Build a simple model baseline (1 minute)

• Train a logistic regression (classification) or linear regression (regression)
• Compare to the dummy — if it’s close, your data might be weak

Step 3: Run PyCaret compare_models() (5 minutes)

• Let AutoML try 15+ models
• Pick the top model as your stronger baseline

Step 4: Run FLAML with a time budget (3 minutes)

• Give FLAML 60-300 seconds
• Compare the result to PyCaret’s top model

Step 5: Compare all baselines in a table

• Which model won? Is it meaningfully better than the dummy?
• If all models are close to the dummy baseline, your data may need more work

Let’s tie it all together in one code block.

# Self-contained: complete baseline workflow comparison
# Requires: pip install pycaret flaml seaborn scikit-learn
import pandas as pd
import seaborn as sns
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from flaml import AutoML

# Load and prepare data
data = sns.load_dataset('titanic')
data = data[['survived', 'pclass', 'age', 'sibsp', 'parch', 'fare']].dropna()
X = data.drop('survived', axis=1)
y = data['survived']

# Split once, use everywhere
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 1. Dummy baseline
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
y_pred_dummy = dummy.predict(X_test)
dummy_acc = accuracy_score(y_test, y_pred_dummy)

# 2. Logistic regression baseline
logreg = LogisticRegression(max_iter=1000, random_state=42)
logreg.fit(X_train, y_train)
y_pred_logreg = logreg.predict(X_test)
logreg_acc = accuracy_score(y_test, y_pred_logreg)

# 3. FLAML baseline (60 seconds)
automl = AutoML()
automl.fit(X_train, y_train, task='classification', time_budget=60, verbose=False)
y_pred_flaml = automl.predict(X_test)
flaml_acc = accuracy_score(y_test, y_pred_flaml)

# Print comparison
print("=== 10-Minute Baseline Comparison ===\n")
print(f"1. Dummy (most frequent): {dummy_acc:.4f}")
print(f"2. Logistic regression:    {logreg_acc:.4f}")
print(f"3. FLAML (60s search):     {flaml_acc:.4f} (best estimator: {automl.best_estimator})")
print()
print("Interpretation:")
print(f"- The dummy sets the floor at {dummy_acc*100:.1f}%")
print(f"- Logistic regression {'handily beats' if logreg_acc > dummy_acc + 0.1 else 'is similar to'} the dummy")
print(f"- FLAML {'improved on' if flaml_acc > logreg_acc else 'matched'} the logistic regression")
print()
print("Next steps: If FLAML's accuracy is much higher, explore these models further.")
print("If all models are similar to the dummy, your features may need more work.")

Recap: What You Learned

Here’s what you walked away with:

• A baseline is the simplest model that sets a performance floor — you need it to know if your models are actually learning.
• DummyClassifier and DummyRegressor are the true floor — models that ignore all features.
• PyCaret gives you a multi-model baseline in 5 lines — it compares 15+ models automatically.
• FLAML gives you a fast, resource-efficient baseline with a time budget.
• Always compare your fancy models to these baselines before celebrating a score.

Check Your Understanding

Remember: What is a baseline model in machine learning?

Understand: Why is it important to build a baseline before training complex models?

Apply: Given a new dataset (say, a customer churn prediction problem), write the pseudocode to build a dummy baseline and a PyCaret baseline.

Analyze: If your PyCaret baseline is only 2% better than the dummy baseline, what might that indicate about your data?

Evaluate: Compare PyCaret and FLAML. Under what circumstances would you choose one over the other?

Create: Design a 10-minute baseline workflow for a regression problem instead of classification. What would your steps be?

Related Articles

• Part 1: What AutoML Actually Automates (and What It Still Can’t) — Introduces the core concepts of automated machine learning that power the baselines you just built.
• Part 3: Auto-Sklearn and H2O AutoML: Open-Source AutoML You Can Run Locally — Explores the heavier AutoML tools you might use when you have more time than 10 minutes.
• Part 5: When AutoML Beats a Hand-Tuned Model — and When It Quietly Fails — Shows why those 0.96 accuracy scores sometimes fail in production, and how your baselines can help catch that.
• Part 7 (next): From Baseline to Production: Deploying Your AutoML Model — Shows how to take the baseline you just built and turn it into a production-ready model.