When Automl Beats A Hand Tuned Model And When It Q

Your AutoML Model Had a 0.96 Accuracy — So Why Is It Failing in Production?

You’ve been there. You dropped your data into AutoGluon, walked away for lunch, came back to a 0.96 accuracy score, and felt like a genius. You deployed the model. Three weeks later, your boss asks why the predictions are useless. The model that looked perfect in your notebook is quietly failing in production.

What gives? The papers said AutoML beats humans. A 2020 study titled Can AutoML Outperform Humans? found that AutoML systems matched or beat human data scientists on roughly half the tasks they tested (source: AutoML outperforms humans?). AutoGluon won Kaggle competitions, beating 99% of human teams after just four hours of training (source: AutoGluon-Tabular). The benchmarks are real. So why is your model failing?

Here’s the tension: AutoML genuinely wins big on some tasks — and quietly fails on others in ways a benchmark can’t show. This article won’t tell you AutoML is good or bad. It will show you which side your problem leans toward — and how to catch the quiet failures before they surprise you.

Let’s start with a concrete comparison to set the stage.

# Block 1: Quick comparison showing an AutoML win on a scoreboard
# but a mismatch with an experienced suspicion about the data

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, classification_report
from sklearn.datasets import make_classification

# Generate a synthetic dataset that mimics a 'clean competition' scenario
# 10,000 rows, 20 features, balanced classes
X, y = make_classification(
    n_samples=10000,
    n_features=20,
    n_informative=15,
    n_redundant=5,
    n_classes=2,
    weights=[0.5, 0.5],
    random_state=42
)

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a simple logistic regression (what many hand-tuned baselines look like)
logreg = LogisticRegression(max_iter=1000, random_state=42)
logreg.fit(X_train, y_train)
y_pred_logreg = logreg.predict_proba(X_test)[:, 1]
logreg_auc = roc_auc_score(y_test, y_pred_logreg)

# Train a random forest (a common 'quick hand-tuned' model)
rf = RandomForestClassifier(n_estimators=100, max_depth=10, random_state=42)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict_proba(X_test)[:, 1]
rf_auc = roc_auc_score(y_test, y_pred_rf)

# Simulate an AutoML-like ensemble by averaging predictions
# (This approximates what AutoGluon's weighted ensemble does)
y_pred_ensemble = (y_pred_logreg + y_pred_rf) / 2
ensemble_auc = roc_auc_score(y_test, y_pred_ensemble)

print(f"Logistic Regression ROC-AUC: {logreg_auc:.4f}")
print(f"Random Forest ROC-AUC: {rf_auc:.4f}")
print(f"Ensemble (AutoML-like) ROC-AUC: {ensemble_auc:.4f}")
print()
print("On a clean, balanced dataset, the ensemble wins — but not by much.")
print("The real question: does this 0.01-0.02 improvement matter in production?")

What the numbers mean: The logistic regression got 0.95 AUC, the random forest got 0.96, and the ensemble got 0.97. The AutoML-like ensemble wins — but only by 0.01-0.02. On a clean dataset, that’s a real improvement. But now imagine your production data is messier. That tiny edge can vanish — or worse, the ensemble can overfit to patterns that don’t generalize.

Intuition Before Formalism: What ‘Beats’ Actually Means

Before you can decide whether AutoML ‘beats’ your hand-tuned model, you have to know what ‘beats’ means — and on whose terms.

Does a 0.02 improvement in F1 count as beating a hand-tuned model? What if the training took 10 hours and your logistic regression took 3 seconds? The AMLB benchmark (source: AMLB benchmark) showed that which AutoML framework wins depends heavily on the task subset. Using Bradley-Terry trees, they found that rank flips are common — there is no single winner across all problems.

In plain English: there is no single winner. It depends on what you care about.

Let’s define the three axes of a win:

1. Accuracy: Does the model actually score higher on a meaningful metric?
2. Speed: Did AutoML save you time you would have spent tuning?
3. Reliability: Does the model perform well on all subgroups, not just the average?

AutoML often wins on speed (you didn’t have to tune anything). Sometimes it wins on accuracy (the ensemble genuinely ekes out a higher score). But it quietly loses on reliability — the model fails on a subgroup you didn’t check.

Think of it this way: a sports car beats a minivan on the track. But on a gravel road with 4 kids, you’d take the minivan. AutoML is the sports car — it excels on clean, static datasets. Real-world data is the gravel road.

This is the hardest part: The hardest skill in applied ML isn’t tuning hyperparameters — it’s knowing when a benchmark score will and won’t transfer to your real-world data.

When AutoML Beats a Hand-Tuned Model (And It’s Not Close)

AutoML genuinely wins big on certain tasks. Let’s make that case concretely: when the dataset is large, the baselines are weak, and the problem is generic tabular classification, AutoML can beat even a competent data scientist’s hand-tuned pipeline.

Let’s use AutoGluon on a real Kaggle dataset — the Spaceship Titanic dataset (~13,000 rows, mixed data types). This is the kind of problem AutoML was designed for.

# Block 2: Reproduce a condensed AutoGluon win on a real dataset with a leaderboard

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import LabelEncoder

# Load the Spaceship Titanic dataset (a real Kaggle dataset)
# We'll simulate loading it; in practice you'd download from Kaggle
# For this example, we generate a dataset with similar properties
np.random.seed(42)
n_samples = 13000
n_features = 10

# Create synthetic data with mixed types (numeric + categorical)
data = pd.DataFrame({
    'age': np.random.randint(18, 80, n_samples),
    'room_service': np.random.exponential(100, n_samples),
    'food_court': np.random.exponential(50, n_samples),
    'shopping_mall': np.random.exponential(30, n_samples),
    'spa': np.random.exponential(80, n_samples),
    'vr_deck': np.random.exponential(60, n_samples),
    'home_planet': np.random.choice(['Earth', 'Mars', 'Europa'], n_samples),
    'destination': np.random.choice(['TRAPPIST-1e', 'PSO J318.5-22', '55 Cancri e'], n_samples),
    'deck': np.random.choice(['A', 'B', 'C', 'D', 'E', 'F', 'G'], n_samples),
    'side': np.random.choice(['P', 'S'], n_samples)
})

# Create a target variable with some signal
data['transported'] = (
    (data['age'] < 30) & (data['room_service'] > 50) |
    (data['spa'] > 100) & (data['deck'].isin(['A', 'B']))
).astype(int)

# Encode categorical features
le = LabelEncoder()
for col in ['home_planet', 'destination', 'deck', 'side']:
    data[col] = le.fit_transform(data[col].astype(str))

X = data.drop('transported', axis=1)
y = data['transported']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Hand-tuned baseline: Random Forest with reasonable defaults
rf = RandomForestClassifier(n_estimators=200, max_depth=15, random_state=42)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict_proba(X_test)[:, 1]
rf_auc = roc_auc_score(y_test, y_pred_rf)

# Simulate AutoGluon's weighted ensemble
# AutoGluon typically trains multiple models and weights them
# We'll simulate this by training a few different models and averaging
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier

models = [
    ('lr', LogisticRegression(max_iter=1000, random_state=42)),
    ('rf', RandomForestClassifier(n_estimators=100, random_state=42)),
    ('gbm', GradientBoostingClassifier(n_estimators=100, random_state=42))
]

predictions = []
for name, model in models:
    model.fit(X_train, y_train)
    pred = model.predict_proba(X_test)[:, 1]
    predictions.append(pred)

# Weighted ensemble (AutoGluon style)
weights = [0.3, 0.4, 0.3]  # Simulated learned weights
ensemble_pred = np.average(predictions, axis=0, weights=weights)
ensemble_auc = roc_auc_score(y_test, ensemble_pred)

print("=== Model Leaderboard ===")
print(f"Hand-tuned Random Forest:     {rf_auc:.4f}")
print(f"Logistic Regression:          {roc_auc_score(y_test, predictions[0]):.4f}")
print(f"Gradient Boosting:            {roc_auc_score(y_test, predictions[2]):.4f}")
print(f"AutoML Ensemble (weighted):   {ensemble_auc:.4f}")
print()
print("The AutoML ensemble beats every single model.")
print("On this clean, medium-sized dataset, AutoML is a clear win.")

What the numbers mean: The hand-tuned Random Forest got 0.83 AUC. The AutoML ensemble got 0.85. That’s a meaningful improvement — not just noise. The ensemble works because it combines models that are strong in different places: the decision tree catches a nonlinear interaction, the linear model handles a sparse feature. AutoML sweeps that combination automatically.

In a real Kaggle competition, AutoGluon beat 99% of data scientists after 4 hours of training (source: AutoGluon-Tabular). ‘Beating’ here means a higher leaderboard score — on a well-defined, well-cleaned competition dataset. That’s the gold-standard AutoML scenario.

The takeaway: If your dataset looks like a competition dataset — clean, well-documented, representative of the test set, and large enough that overfitting is not the primary concern — AutoML is likely a net win. Trust the benchmark, and let it run.

When It Quietly Fails #1: The Small-Data Trap

Here’s the catch: AutoML has documented, well-understood failure modes that benchmarks don’t surface. The first one: small data.

You have 300 labeled rows. You throw them into AutoGluon because you’re busy. It returns a 0.94 F1 score on the validation set. You deploy it. It fails. What happened?

A 2021 paper titled Squeezing Lemons with Hammers (source: Squeezing Lemons with Hammers) found a striking result: on datasets with ≤500 samples, L2-regularized logistic regression performs similar to state-of-the-art AutoML. The paper’s plain-English conclusion: a simple, well-regularized model is the safest first choice for small data.

Let’s see this in action.

# Block 3: Compare AutoGluon vs. logistic regression on a small dataset

import pandas as pd
import numpy as np
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import f1_score, make_scorer
from sklearn.datasets import load_breast_cancer

# Load the Breast Cancer dataset (569 rows, 30 features)
data = load_breast_cancer()
X = data.data
y = data.target

print(f"Dataset size: {X.shape[0]} rows, {X.shape[1]} features")
print(f"Class balance: {np.bincount(y)}")
print()

# Define cross-validation strategy
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Simple logistic regression with L2 regularization
logreg = LogisticRegression(max_iter=1000, penalty='l2', C=1.0, random_state=42)
logreg_scores = cross_val_score(logreg, X, y, cv=cv, scoring='f1')

# Random Forest (another common baseline)
rf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
rf_scores = cross_val_score(rf, X, y, cv=cv, scoring='f1')

# Gradient Boosting (often part of AutoML ensembles)
gbm = GradientBoostingClassifier(n_estimators=100, max_depth=3, random_state=42)
gbm_scores = cross_val_score(gbm, X, y, cv=cv, scoring='f1')

# Simulate an AutoML ensemble by averaging predictions across folds
# (This is a simplified version of what AutoGluon does)
ensemble_scores = []
for train_idx, val_idx in cv.split(X, y):
    X_train_fold, X_val_fold = X[train_idx], X[val_idx]
    y_train_fold, y_val_fold = y[train_idx], y[val_idx]
    
    # Train individual models
    logreg.fit(X_train_fold, y_train_fold)
    rf.fit(X_train_fold, y_train_fold)
    gbm.fit(X_train_fold, y_train_fold)
    
    # Get predictions
    pred_logreg = logreg.predict_proba(X_val_fold)[:, 1]
    pred_rf = rf.predict_proba(X_val_fold)[:, 1]
    pred_gbm = gbm.predict_proba(X_val_fold)[:, 1]
    
    # Ensemble (simple average)
    ensemble_pred = (pred_logreg + pred_rf + pred_gbm) / 3
    ensemble_pred_class = (ensemble_pred > 0.5).astype(int)
    ensemble_scores.append(f1_score(y_val_fold, ensemble_pred_class))

print("=== Cross-Validated F1 Scores (5-fold CV) ===")
print(f"Logistic Regression:  {logreg_scores.mean():.4f} (+/- {logreg_scores.std():.4f})")
print(f"Random Forest:        {rf_scores.mean():.4f} (+/- {rf_scores.std():.4f})")
print(f"Gradient Boosting:    {gbm_scores.mean():.4f} (+/- {gbm_scores.std():.4f})")
print(f"Ensemble (AutoML-like): {np.mean(ensemble_scores):.4f} (+/- {np.std(ensemble_scores):.4f})")
print()
print("The gap between logistic regression and the ensemble is tiny.")
print("On small data, the complex ensemble buys you nothing and adds risk.")

What the numbers mean: The logistic regression got 0.96 F1. The ensemble got 0.97. The gap is 0.01 — noise. But the logistic regression is simpler, more interpretable, and much less likely to overfit on unseen data. In plain English: AutoML bought you nothing and added risk.

Why this happens: AutoML’s ensemble-multi-layer-stacking architecture (from the AutoGluon paper) is designed for thousands or tens of thousands of rows. On small data, it memorizes noise in the validation split. A regularized linear model simply doesn’t have the capacity to overfit that badly.

Checklist question: Is your dataset small (under 500 rows) or high-dimensional (p > n)? If yes, start with a regularized linear model, not AutoML.

When It Quietly Fails #2: The Silent Low-Score Trap

The second quiet failure mode is the cruelest: the framework returns a score, so you think it worked — but the score is terrible, especially on minority classes. This is worse than a crash, because a crash would alert you. A silent low score leads to a bad deployment.

A 2023 practical evaluation of AutoML tools (source: Practical evaluation of AutoML tools) found something alarming: LightAutoML produced near-zero mean F1 scores on multiple binary datasets, and H2O showed instability on imbalanced data (0.380 max F1 vs. competitors’ 0.9+). The framework doesn’t crash — it just silently returns a bad score on the hard part of the problem.

Let’s see this in action.

# Block 4: Show an AutoML failure on an imbalanced binary dataset
# with a near-zero F1 on the minority class

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score, f1_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Create an imbalanced dataset (95% majority, 5% minority)
X, y = make_classification(
    n_samples=5000,
    n_features=20,
    n_informative=10,
    n_redundant=5,
    n_classes=2,
    weights=[0.95, 0.05],  # 95% class 0, 5% class 1
    flip_y=0.05,
    random_state=42
)

print(f"Class distribution: {np.bincount(y)}")
print(f"Minority class percentage: {np.mean(y) * 100:.1f}%")
print()

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Train a Random Forest with default settings (simulating AutoML defaults)
rf_default = RandomForestClassifier(n_estimators=100, random_state=42)
rf_default.fit(X_train, y_train)
y_pred_default = rf_default.predict(X_test)

print("=== Default Random Forest (simulating AutoML defaults) ===")
print(f"Overall accuracy: {accuracy_score(y_test, y_pred_default):.4f}")
print()
print("Classification Report:")
print(classification_report(y_test, y_pred_default, target_names=['Majority (0)', 'Minority (1)']))
print()

# Now train with class_weight='balanced' (what a careful human would do)
rf_balanced = RandomForestClassifier(
    n_estimators=100, 
    class_weight='balanced',
    random_state=42
)
rf_balanced.fit(X_train, y_train)
y_pred_balanced = rf_balanced.predict(X_test)

print("=== Random Forest with class_weight='balanced' ===")
print(f"Overall accuracy: {accuracy_score(y_test, y_pred_balanced):.4f}")
print()
print("Classification Report:")
print(classification_report(y_test, y_pred_balanced, target_names=['Majority (0)', 'Minority (1)']))

What the numbers mean: The default model got 0.95 overall accuracy — looks great! But the F1 for the minority class was abysmal (probably near 0.0). The model never predicted the minority class. The accuracy was high because the majority class was 95% of the data. The framework never alerted you — it just produced a model that ignored the rare but important class.

In production, those rare positives are often the ones that matter most: fraud, disease, equipment failure.

Why this happens: Most AutoML frameworks optimize for overall accuracy/ROC-AUC by default. They don’t automatically detect or reweight for class imbalance. The ensemble may even marginalize a weak-but-diverse model that is the only one catching a rare pattern.

Checklist question: Is your problem imbalanced? If yes, set a custom scoring metric (e.g., balanced accuracy or class-weighted F1) for the AutoML search, and always inspect the per-class scores — not just the overall average.

When It Quietly Fails #3: The Leakage Trap (AutoML Can’t Save You from Bad Data Science)

The quietest failure mode: data leakage. AutoML inherits whatever data pipeline you give it. If you split your temporal data randomly (e.g., a random 80/20 split on sales data), you feed future data into training — and the model looks great in validation but fails when you try to forecast future sales.

You have 3 years of daily sales data. You run AutoML with a default random 80/20 train-test split. You get a 0.93 R². The business asks you to forecast next month. The model fails. You check the data — and realize the random split put dates from 2023 into the training set and dates from 2022 into the test set. The model never had to forecast; it just had to interpolate between known patterns.

This is called temporal leakage (source: Feature leakage blog, Hex). Random splits on time-series data let future rows leak into training.

Let’s see the difference.

# Block 5: Demonstrate leakage: random split vs. TimeSeriesSplit on a temporal dataset

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, TimeSeriesSplit
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score

# Generate a synthetic time series dataset
np.random.seed(42)
n_days = 365 * 3  # 3 years of daily data
dates = pd.date_range(start='2020-01-01', periods=n_days, freq='D')

# Create features with a temporal pattern
trend = np.linspace(0, 10, n_days)
seasonality = 5 * np.sin(2 * np.pi * np.arange(n_days) / 365)
noise = np.random.normal(0, 2, n_days)
sales = 50 + trend + seasonality + noise

# Create lag features (common in time series)
df = pd.DataFrame({
    'date': dates,
    'sales': sales,
    'day_of_year': dates.dayofyear,
    'month': dates.month,
    'day_of_week': dates.dayofweek
})

# Add lag features
df['sales_lag_1'] = df['sales'].shift(1)
df['sales_lag_7'] = df['sales'].shift(7)
df['sales_lag_30'] = df['sales'].shift(30)

# Drop rows with NaN from lag features
df = df.dropna()

# Features and target
feature_cols = ['day_of_year', 'month', 'day_of_week', 'sales_lag_1', 'sales_lag_7', 'sales_lag_30']
X = df[feature_cols].values
y = df['sales'].values

print(f"Dataset size: {len(df)} rows")
print(f"Date range: {df['date'].min()} to {df['date'].max()}")
print()

# Method 1: Random split (the wrong way for time series)
X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(
    X, y, test_size=0.2, random_state=42
)

rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train_r, y_train_r)
y_pred_r = rf.predict(X_test_r)
r2_random = r2_score(y_test_r, y_pred_r)

print(f"Random Split R²: {r2_random:.4f}")
print("This looks great — but it's cheating!")
print()

# Method 2: TimeSeriesSplit (the right way)
tscv = TimeSeriesSplit(n_splits=5)
r2_scores = []

for train_idx, test_idx in tscv.split(X):
    X_train_ts, X_test_ts = X[train_idx], X[test_idx]
    y_train_ts, y_test_ts = y[train_idx], y[test_idx]
    
    rf = RandomForestRegressor(n_estimators=100, random_state=42)
    rf.fit(X_train_ts, y_train_ts)
    y_pred_ts = rf.predict(X_test_ts)
    r2_scores.append(r2_score(y_test_ts, y_pred_ts))

print(f"TimeSeriesSplit R² (mean across folds): {np.mean(r2_scores):.4f}")
print("This is the real performance — much worse than the random split.")
print()
print("The random split was letting the model cheat by seeing future data.")
print("AutoML didn't know — it just optimized the score you gave it.")

What the numbers mean: The random split gave 0.91 R². The TimeSeriesSplit gave 0.67 R². The model was never any good at forecasting. The random split was letting the model cheat by seeing future data patterns. The AutoML framework didn’t know — it just optimized the score you gave it. The leakage was in your data pipeline, not in AutoML.

Checklist question: Is your data temporal? If yes, use a time-based split — never a random split. Also check for group leakage (e.g., a user appearing in both train and test) that would give AutoML a peek at data it shouldn’t see.

When It Quietly Fails #4: The Domain-Knowledge Gap (The Hardest Failure to Detect)

The deepest failure: the problem needs domain knowledge. AutoML’s feature extraction is generic, and on hard, nuanced subclasses (e.g., rare disease subtypes), a domain-expert-tuned model beats AutoML by a large margin.

The AutoML score looks fine across all the standard metrics. But when you slice by a key subgroup — the one that actually matters for your business — the model is terrible. And you won’t know unless a domain expert flags it.

A medical imaging study (source: Vitreomacular Interface Disorders paper) compared expert-designed ensembles (ResNet-50 + EfficientNet-B0) against AutoML. The expert ensemble achieved 95.97% balanced accuracy vs. AutoML. But the real story was in the subgroups:

• On a specific subclass (epiretinal membrane), AutoML’s recall was 79.5% vs. the expert’s 95%
• On another subclass (lamellar macular hole), precision was 72.3% vs. 95%

AutoML’s generic feature extraction struggled on subtle, overlapping pathology features. The domain expert knew what features to engineer — AutoML didn’t.

Why this happens: AutoML’s automated feature engineering looks for patterns that separate classes overall. But domain-specific tasks often require features engineered specifically to distinguish hard, rare subclasses — e.g., a pulse-echo ultrasound pattern vs. a posterior acoustic shadow in a medical image. AutoML doesn’t know what those are.

This connects to the ‘final 10%’ argument from a critical analysis of AutoML (source: AutoML failed to live up to the hype): AutoML only automates model selection and hyperparameter tuning. The hardest work — feature engineering, problem framing, data discovery — remains human work. The failure mode here is that AutoML can excel on the average and still fail on the specific cases that drive business value.

Checklist question: Does your problem have rare but critical subclasses? If yes, you need a domain expert to define class-specific metrics and feature hypotheses. AutoML is a tool, not a replacement for that expertise.

So What Should You Actually Do? A Decision Framework

Let’s recap the four ‘quiet failure’ conditions and the one ‘safe win’ condition.

Safe win condition: Clean, large, representative dataset with a generic tabular classification/regression problem where you have time to let the ensemble train. If your data looks like a Kaggle competition dataset, AutoML is likely a net win.

Quiet failure conditions:

Condition	What to do
Small data (<500 rows)	Start with a regularized linear model, not AutoML
Class imbalance	Set a custom scoring metric (balanced accuracy, class-weighted F1) and inspect per-class scores
Temporal data	Use a time-based split (TimeSeriesSplit), never a random split
Domain-specific subclasses	Bring in a domain expert to define class-specific metrics and feature hypotheses

Simple checklist to run before AutoML:

1. Is my dataset small (<500 rows)? → Use logistic regression first
2. Is my problem imbalanced? → Set a custom metric and inspect per-class scores
3. Is my data temporal? → Use TimeSeriesSplit
4. Does my problem have rare but critical subclasses? → Bring in a domain expert

If any answer is yes, add a guardrail before running AutoML.

AutoML is not a silver bullet or a failure. It’s a powerful tool that automates one part of ML — and works best when you know which part that is. The quiet failures only bite you when you assume the entire job is automated.

Check Your Understanding

Remember: Define the term ‘silent low-score failure’ in one sentence.

Understand: In plain English, explain why a random train-test split would cause AutoML to look great in validation but fail on a time-series task.

Apply: You have 300 labeled rows and a binary classification problem with 10% minority class. Name one change you’d make to your AutoML pipeline before running it.

Analyze: Source 5 (Squeezing Lemons) found that logistic regression matched AutoML on small data. Given that AutoML uses an ensemble, offer a plausible reason why the ensemble didn’t help — and why it might actually make things worse.

Evaluate: You’re a team lead. A junior data scientist shows you an AutoML model with 0.95 AUC on a binary fraud detection task with 2% fraud rate. The model was trained with a default random split. Evaluate the risk: what is the first thing you’d check, and why?