Auto Sklearn And H2O Automl Open Source Automl You

The Problem: You’ve Tuned a Model Before. It Wasn’t Fun.

You know the feeling. You’ve spent hours tweaking hyperparameters — adjusting the learning rate, changing the number of trees, trying different kernels. You run a grid search, wait forever, and get back a model that’s barely better than the default. Then you wonder: did I miss a better combination? Should I have tried a different algorithm entirely?

It’s frustrating. Grid search and random search are brute force — they don’t learn from past runs. They try combinations blindly, wasting time on bad ideas and missing good ones.

But what if the machine could do that searching for you — and do it smarter? What if it could learn from each failed attempt, adjust its strategy, and combine the best models into something even better?

That’s exactly what AutoML promises. It automates the model selection + hyperparameter tuning + ensemble pipeline. But here’s the catch: many AutoML solutions are cloud-only (like Vertex AI AutoML or SageMaker Autopilot from the last article) or cost money. They lock you into a vendor’s ecosystem.

This tutorial covers two free, open-source AutoML tools you can run on your own laptop: auto-sklearn and H2O AutoML. By the end, you’ll have run both on a real dataset and know which one fits your workflow.

What AutoML Actually Does Under the Hood (Intuition First)

Before we run any code, let’s build a mental model of what happens when you call automl.fit(). AutoML is not magic — it’s a structured search over a space of ML pipelines.

Think of it this way: You’re a chef trying to find the best recipe for a dish. AutoML is like having a sous-chef who systematically tries different ingredient combinations, learns from each batch, and then blends the top recipes together.

AutoML has three core components:

1. A search space — which algorithms + which hyperparameters to consider. This is like the list of possible ingredients and cooking techniques.
2. A search strategy — how to explore that space efficiently. This is like the method the sous-chef uses to decide which recipe to try next.
3. A way to combine the best models — ensembling. This is like blending the top recipes into a single dish that’s better than any one recipe.

Now here’s the interesting part: auto-sklearn and H2O AutoML approach these components differently.

auto-sklearn uses Bayesian optimization — a ‘smart’ search that learns from past evaluations. It builds a probability model of which hyperparameters are likely to work well, then focuses on those. It also uses meta-learning — it warms up by looking at similar datasets it’s seen before. This is like a sous-chef who remembers what worked for similar dishes and starts from there.

H2O AutoML uses a different approach. It trains a fixed set of algorithms (GBMs, GLMs, deep nets, etc.) using random grid search, then builds a stacked ensemble — a meta-model that learns how to combine the predictions of all the base models. This is like a sous-chef who cooks every possible variation, then has a master chef taste all of them and decide how to blend them.

The key difference: auto-sklearn optimizes the entire pipeline (preprocessing + model), while H2O AutoML focuses on the model and uses its own in-memory distributed engine.

Meet the Contenders: auto-sklearn vs. H2O AutoML at a Glance

Here’s a side-by-side comparison to help you decide which one to try first:

Feature	auto-sklearn	H2O AutoML
Search strategy	Bayesian optimization + meta-learning	Random grid search + stacked ensembles
Pipeline scope	Full pipeline (preprocessing + model)	Model-level (preprocessing handled separately)
Parallelization	Limited (single machine)	Distributed in-memory engine
Output format	Ensemble of scikit-learn models	Leaderboard with best single model + stacked ensemble
Learning curve	Low (drop-in sklearn replacement)	Medium (Java backend, H2OFrame API)
Best for	Small to medium datasets, sklearn workflows	Large datasets, interpretability
License	BSD 3-clause	Apache 2.0

auto-sklearn is a wrapper around scikit-learn. It’s a drop-in replacement for any sklearn estimator — you can use it in your existing pipelines with minimal changes. It outputs an ensemble of scikit-learn models.

H2O AutoML is a standalone Java-based engine with a Python API. It’s better for large datasets (it can distribute work across multiple cores or machines) and for interpretability (it gives you a leaderboard with variable importance).

Both are free and open-source. Let’s get them installed.

Setting Up Your Environment: Installing Both Tools

Installing these tools can be the hardest part. Let’s walk through it step by step.

Installing auto-sklearn

auto-sklearn has stricter system dependencies. It’s officially supported on Linux and macOS — Windows users will need WSL or Docker.

On Linux (Ubuntu/Debian), you’ll need build tools and SWIG:

sudo apt-get install build-essential swig python3-dev
pip install auto-sklearn

On macOS, use Homebrew for SWIG:

brew install swig
pip install auto-sklearn

Let’s verify the installation:

# This code block is fully self-contained
import autosklearn
print(f"auto-sklearn version: {autosklearn.__version__}")

Installing H2O AutoML

H2O AutoML requires Java (JRE or JDK) to be installed and on your PATH. Check if you have Java:

java -version

If you don’t have Java, download it from java.com.

Then install H2O:

pip install h2o

Let’s verify:

# This code block is fully self-contained
import h2o
print(f"H2O version: {h2o.__version__}")

Fallback plan: If installation fails, use Google Colab (it has H2O pre-installed) or a Docker image. Both tools can be installed in a conda environment to avoid conflicts.

Your First auto-sklearn Run: 10 Lines of Code

Let’s run auto-sklearn on the digits dataset from scikit-learn. This is a small dataset (1,797 images of handwritten digits) that trains in a few minutes.

# This code block is fully self-contained
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from autosklearn.classification import AutoSklearnClassifier

# Load the digits dataset
X, y = load_digits(return_X_y=True)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create the AutoML classifier
# time_left_for_this_task: total budget in seconds (2 minutes)
# per_run_time_limit: max time per model in seconds (30 seconds)
automl = AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    seed=42
)

# Fit the model (this will take about 2 minutes)
automl.fit(X_train, y_train)

# Make predictions
y_pred = automl.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Test accuracy: {accuracy:.4f}")

What’s happening here?

• time_left_for_this_task=120 tells auto-sklearn it has 2 minutes total to find the best model
• per_run_time_limit=30 limits each individual model to 30 seconds — this prevents one slow model from eating the whole budget
• auto-sklearn handles preprocessing, algorithm selection, and hyperparameter tuning automatically
• The output is an ensemble of models, not just a single model

What Just Happened? Interpreting auto-sklearn’s Output

This is the hardest part of using auto-sklearn — understanding what it actually did. Let’s peek under the hood.

# This code block is fully self-contained
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from autosklearn.classification import AutoSklearnClassifier

# Load and split data
X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train auto-sklearn (same setup as before)
automl = AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    seed=42
)
automl.fit(X_train, y_train)

# Print the leaderboard — all models evaluated, ranked by validation score
print("=== Leaderboard ===")
leaderboard = automl.leaderboard()
print(leaderboard)

# Print the ensemble — which models were selected and their weights
print("\n=== Ensemble Models ===")
print(automl.show_models())

# Print performance over time — how the ensemble improved
print("\n=== Performance Over Time ===")
print(automl.performance_over_time_.head())

What does this mean?

• The leaderboard shows all 40+ models auto-sklearn evaluated, ranked by validation score. You can see which algorithms it tried (gradient boosting, random forest, SVM, etc.) and their hyperparameters.
• The ensemble is built using greedy ensemble selection (Caruana et al. 2004). auto-sklearn doesn’t just pick the best model — it selects a set of models that work well together. The output shows which models were selected and their weights in the ensemble.
• Performance over time shows how the ensemble’s accuracy improved as more models were evaluated. If the curve plateaus early, you’re done. If it’s still climbing, you need more time.

In plain English: auto-sklearn tried 40+ different pipelines, picked the 5 that worked best together, and combined them into a single ensemble that outperforms any individual model.

Your First H2O AutoML Run: The Same Dataset, Different Approach

Now let’s run H2O AutoML on the same digits dataset. The API is different — H2O is a full platform, not just a sklearn wrapper.

# This code block is fully self-contained
import h2o
from h2o.automl import H2OAutoML
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import pandas as pd

# Initialize H2O cluster (starts Java backend)
h2o.init()

# Load digits dataset
X, y = load_digits(return_X_y=True)

# Convert to pandas DataFrame and add target column
df = pd.DataFrame(X)
df['target'] = y

# Split into train and test
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)

# Convert pandas DataFrames to H2OFrames
train_h2o = h2o.H2OFrame(train_df)
test_h2o = h2o.H2OFrame(test_df)

# Define target and feature columns
target = 'target'
features = [col for col in train_h2o.columns if col != target]

# Convert target to categorical (for classification)
train_h2o[target] = train_h2o[target].asfactor()
test_h2o[target] = test_h2o[target].asfactor()

# Create and train the AutoML model
aml = H2OAutoML(max_runtime_secs=120, seed=42)

# Note: the API uses column names, not arrays
aml.train(x=features, y=target, training_frame=train_h2o)

# Get the leaderboard
print("=== Leaderboard ===")
lb = aml.leaderboard
print(lb)

# Get the best model
print("\n=== Best Model ===")
print(aml.leader)

# Make predictions on test set
predictions = aml.leader.predict(test_h2o)

# Shut down H2O cluster
h2o.cluster().shutdown()

What’s happening here?

• h2o.init() starts a Java backend in the background — this is the H2O cluster
• Data must be converted to H2OFrame — H2O’s own data format
• train() uses column names (x and y) instead of numpy arrays
• The leaderboard shows all models trained, ranked by default metric (AUC for classification, RMSE for regression)
• aml.leader is the top-ranked model — often a Stacked Ensemble

What Just Happened? Interpreting H2O AutoML’s Output

The Stacked Ensemble is powerful, but it’s also a black box. Here’s how to peek inside.

# This code block is fully self-contained
import h2o
from h2o.automl import H2OAutoML
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import pandas as pd

# Initialize H2O
h2o.init()

# Load and prepare data
X, y = load_digits(return_X_y=True)
df = pd.DataFrame(X)
df['target'] = y
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
train_h2o = h2o.H2OFrame(train_df)
test_h2o = h2o.H2OFrame(test_df)
target = 'target'
features = [col for col in train_h2o.columns if col != target]
train_h2o[target] = train_h2o[target].asfactor()
test_h2o[target] = test_h2o[target].asfactor()

# Train AutoML
aml = H2OAutoML(max_runtime_secs=120, seed=42)
aml.train(x=features, y=target, training_frame=train_h2o)

# Inspect the leaderboard — see all models and their metrics
print("=== Full Leaderboard ===")
print(aml.leaderboard)

# If the leader is a Stacked Ensemble, inspect the metalearner
# The metalearner shows which base models contribute most to the ensemble
if aml.leader.algo == "stackedensemble":
    print("\n=== Metalearner Coefficients ===")
    metalearner = aml.leader.metalearner()
    print(metalearner.coef_norm())

# Get variable importance for the best single model
# (not the ensemble, but the top individual model)
print("\n=== Variable Importance (Best Single Model) ===")
best_single = aml.leaderboard[1, 0]  # Second row (first is often ensemble)
model = h2o.get_model(best_single[0])
varimp = model.varimp(use_pandas=True)
print(varimp.head())

# Shut down H2O
h2o.cluster().shutdown()

What does this mean?

• The leaderboard shows all models trained, with columns: model_id, auc, logloss, etc. It’s sorted by the default metric (AUC for classification).
• The Stacked Ensemble (usually the leader) combines the predictions of all base models using a metalearner (default: GLM with non-negative weights). You can inspect the metalearner to see which base models contribute most.
• Variable importance shows which features matter most for the best single model. This is useful for interpretability.

In plain English: H2O trained 8+ models, then blended them into a super-model that’s better than any single one. The XGBoost model might be the best individual, but the ensemble improves AUC by a small but meaningful amount.

Head-to-Head: Which One Should You Use?

Let’s compare the two tools on practical dimensions:

Ease of use: auto-sklearn wins if you’re already using scikit-learn. It’s a drop-in replacement — you can use it in your existing pipelines with minimal changes. H2O AutoML has a steeper learning curve due to the Java backend and H2OFrame API.

Speed: H2O AutoML is generally faster on larger datasets due to its distributed in-memory engine. auto-sklearn can be slow on datasets with many features or rows (it’s limited to a single machine).

Interpretability: H2O AutoML wins with its leaderboard, variable importance, and metalearner inspection. auto-sklearn’s ensemble is harder to interpret — you can see which models were selected, but understanding why is more difficult.

Dataset size: H2O AutoML scales better to large datasets (100k+ rows). auto-sklearn is best for small to medium datasets (up to ~50k rows).

Integration: auto-sklearn integrates seamlessly with sklearn pipelines and model persistence (pickle). H2O AutoML requires saving models in H2O’s format (MOJO/POJO) for production.

Benchmark reality check: On the MLJAR benchmark (30 datasets, 1-hour budget), neither tool is uniformly better. H2O AutoML often wins on logloss, but auto-sklearn can be competitive on smaller datasets. The best tool depends on your specific data and constraints.

Going Further: Tuning the AutoML Itself

Both tools have knobs you can turn. Let’s see how changing parameters affects the output.

auto-sklearn: Increasing the Time Budget

# This code block is fully self-contained
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from autosklearn.classification import AutoSklearnClassifier

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Run with 5 minutes instead of 2
automl = AutoSklearnClassifier(
    time_left_for_this_task=300,  # 5 minutes
    per_run_time_limit=30,
    ensemble_size=10,  # Allow more models in the ensemble
    seed=42
)
automl.fit(X_train, y_train)

y_pred = automl.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Test accuracy (5 min): {accuracy:.4f}")

# Compare with previous run (2 min) — accuracy should improve slightly
print(f"Number of models evaluated: {len(automl.leaderboard())}")
print(f"Ensemble size: {automl.ensemble_size}")

H2O AutoML: Using max_models Instead of Time

# This code block is fully self-contained
import h2o
from h2o.automl import H2OAutoML
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import pandas as pd

h2o.init()

X, y = load_digits(return_X_y=True)
df = pd.DataFrame(X)
df['target'] = y
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42)
train_h2o = h2o.H2OFrame(train_df)
test_h2o = h2o.H2OFrame(test_df)
target = 'target'
features = [col for col in train_h2o.columns if col != target]
train_h2o[target] = train_h2o[target].asfactor()
test_h2o[target] = test_h2o[target].asfactor()

# Use max_models instead of max_runtime_secs for predictable runtime
aml = H2OAutoML(max_models=20, seed=42)

# Exclude deep learning (slow on wide data)
# aml = H2OAutoML(max_models=20, exclude_algos=["DeepLearning"], seed=42)

aml.train(x=features, y=target, training_frame=train_h2o)

print("=== Leaderboard (max_models=20) ===")
print(aml.leaderboard)

h2o.cluster().shutdown()

Key parameters to remember:

• auto-sklearn: time_left_for_this_task, per_run_time_limit, ensemble_size, initial_configurations_via_meta_learning
• H2O AutoML: max_runtime_secs, max_models, sort_metric, exclude_algos, seed

The Catch: When AutoML Fails (and What to Do About It)

AutoML is not a silver bullet. Here are common failure modes and how to diagnose them.

Overfitting

AutoML can overfit if the time budget is too large relative to dataset size. The ensemble selection in auto-sklearn acts as regularization, but it’s not foolproof.

# This code block is fully self-contained
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from autosklearn.classification import AutoSklearnClassifier

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train auto-sklearn
automl = AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    seed=42
)
automl.fit(X_train, y_train)

# Compare train vs. test accuracy to check for overfitting
y_train_pred = automl.predict(X_train)
y_test_pred = automl.predict(X_test)
train_acc = accuracy_score(y_train, y_train_pred)
test_acc = accuracy_score(y_test, y_test_pred)
print(f"Train accuracy: {train_acc:.4f}")
print(f"Test accuracy: {test_acc:.4f}")
print(f"Gap: {train_acc - test_acc:.4f}")

# A gap > 0.05 suggests overfitting
if train_acc - test_acc > 0.05:
    print("Warning: Possible overfitting detected. Consider reducing time budget.")
else:
    print("No significant overfitting detected.")

Extracting a Single Model for Deployment

If the ensemble is too complex to deploy, you can extract individual models:

# This code block is fully self-contained
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from autosklearn.classification import AutoSklearnClassifier

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

automl = AutoSklearnClassifier(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    seed=42
)
automl.fit(X_train, y_train)

# Access individual models from the ensemble
# The ensemble is stored in automl.ensemble_
# Each model is a scikit-learn estimator
print("=== Individual Models in Ensemble ===")
for i, model in enumerate(automl.get_models_with_weights()):
    print(f"Model {i+1}: {model[1]} (weight: {model[0]:.3f})")

Common Gotchas

• auto-sklearn on Windows: Use WSL or Docker. Native support is not guaranteed.
• H2O memory: The Java backend can consume significant memory. Use h2o.init(max_mem_size="2G") to limit it.
• High-dimensional data: Pre-filter features before running AutoML. H2O’s exclude_algos can remove slow algorithms like deep learning.
• Performance plateau: Check the performance over time. If it plateaus early, you’re done. If it’s still climbing, you need more time.

What We Learned and Where to Go Next

Let’s recap what we covered:

• auto-sklearn is a drop-in sklearn replacement that optimizes the full pipeline using Bayesian optimization + meta-learning. It’s best for small to medium datasets and sklearn workflows.
• H2O AutoML is a standalone platform that trains a diverse set of models and builds a Stacked Ensemble. It’s better for large datasets and when interpretability matters.
• Both are free, open-source, and run locally. The choice depends on your workflow: sklearn integration (auto-sklearn) vs. scalability and interpretability (H2O).

Next in the series: ‘AutoML in the Cloud: Google Vertex AI vs. SageMaker Autopilot’ — we’ll compare cloud-based AutoML services and when to use them instead of local tools. Or: ‘Deep Dive into H2O AutoML: Customizing the Stacked Ensemble’ — for readers who want to go deeper into H2O’s ensemble mechanics.

Check Your Understanding

Remember: What are the three core components of AutoML? (Search space, search strategy, ensembling)

Understand: Explain in your own words how auto-sklearn’s Bayesian optimization is different from H2O’s random grid search.

Apply: Given a dataset with 10,000 rows and 50 features, which tool would you recommend and why?

Analyze: Look at the leaderboard from your auto-sklearn run. Which algorithm performed best? Was it the one you expected?

Evaluate: Compare the accuracy of your auto-sklearn model with the H2O model on the digits dataset. Which one performed better? What tradeoffs did each make?

Create: Design a simple experiment to compare auto-sklearn and H2O AutoML on a dataset of your choice. What metrics would you track? How would you ensure a fair comparison?

Related articles

• Part 1: What AutoML Actually Automates (and What It Still Can’t) — Builds the foundational understanding of AutoML’s capabilities and limitations.
• Part 2: Google Cloud AutoML vs. Azure AutoML vs. AWS SageMaker Autopilot: Choosing Your AutoML Platform — Compares cloud-based AutoML services, which is useful context for deciding when to use local tools vs. cloud.