Grid Search Vs Random Search Vs Bayesian Optimizat

1. Why Hyperparameter Tuning Matters (and Why You Shouldn’t Do It by Hand)

You’ve just trained a Random Forest on a new dataset, and the accuracy is… okay. Not great, not terrible. You know the model has knobs you can twist—things like max_depth, min_samples_split, and n_estimators—but you have no idea which settings will unlock better performance. So you start guessing. You try a few values, retrain, check the score, tweak again. An hour later you’ve run maybe a dozen experiments, and you’re still not sure if you’ve found the best combination.

Sound familiar? Manual hyperparameter tuning is slow, biased (you tend to stick with what you know), and doesn’t scale. When you have more than two or three knobs, the number of possible combinations explodes, and your gut feeling becomes useless.

Hyperparameters control model complexity. For a Random Forest, max_depth limits how deep each tree can grow—too shallow, and the model underfits; too deep, and it overfits. min_samples_split sets the minimum number of samples required to split an internal node—small values let the tree grow more, large values keep it simple. Get these right, and your model generalizes better. Get them wrong, and you leave performance on the table.

You don’t know the right values ahead of time. You need a systematic way to search for them. And not all search methods are created equal.

In this article, we’ll compare three popular approaches: grid search, random search, and Bayesian optimization. We’ll use a concrete example throughout: tuning a Random Forest on the breast cancer dataset (a small, fast-to-train dataset from scikit-learn). Let’s start by establishing a baseline.

# Load data and train a default Random Forest
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load the breast cancer dataset
X, y = load_breast_cancer(return_X_y=True)

# Split into train and test sets (we'll use the test set only at the end)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train a Random Forest with default hyperparameters
rf_default = RandomForestClassifier(random_state=42)
rf_default.fit(X_train, y_train)

# Predict and evaluate
y_pred_default = rf_default.predict(X_test)
baseline_accuracy = accuracy_score(y_test, y_pred_default)
print(f"Baseline accuracy (default params): {baseline_accuracy:.4f}")

Output:

Baseline accuracy (default params): 0.9649

This output shows that the Random Forest achieves approximately 96.5% accuracy using the library’s default settings. Not bad—but can we do better? Let’s see how systematic tuning can squeeze out extra performance.

2. Grid Search: The Brute-Force Approach

Grid search is the most intuitive method: you define a grid of candidate values for each hyperparameter, then try every possible combination. It’s exhaustive—if the best combination is inside your grid, grid search will find it.

Let’s try a small grid for max_depth and min_samples_split. We’ll use GridSearchCV from scikit-learn, which automatically performs cross-validation.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import time

# Load data again (self-contained block)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Define a small grid
param_grid = {
    'max_depth': [3, 5, 10],
    'min_samples_split': [2, 5, 10]
}

# Create GridSearchCV object
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(
    estimator=rf,
    param_grid=param_grid,
    cv=5,          # 5-fold cross-validation
    scoring='accuracy',
    n_jobs=-1      # use all CPU cores
)

# Time the search
start = time.time()
grid_search.fit(X_train, y_train)
grid_time = time.time() - start

print(f"Grid search best params: {grid_search.best_params_}")
print(f"Grid search best CV accuracy: {grid_search.best_score_:.4f}")
print(f"Grid search total time: {grid_time:.2f} seconds")

Output:

Grid search best params: {'max_depth': 10, 'min_samples_split': 2}
Grid search best CV accuracy: 0.9692
Grid search total time: 1.23 seconds

Grid search found a combination that gives 96.92% accuracy on cross-validation, which is a small improvement over the baseline. However, notice that it took 1.23 seconds for just 9 combinations. While that seems fast, the number of combinations grows exponentially with each new hyperparameter added. This is called the curse of dimensionality, which means that as you add more variables to the search, the volume of the search space grows so quickly that the data becomes sparse and the computational cost becomes unmanageable.

In practice, as Bergstra and Bengio (2012) showed, only a few hyperparameters actually matter for performance. Grid search wastes most of its budget on irrelevant dimensions because it samples each dimension uniformly. For anything larger than a very small number of hyperparameters, we need a smarter approach.

3. Random Search: Smarter Sampling, Same Budget

Random search takes a different approach: instead of trying every combination, it samples random combinations from the parameter space. With the same number of trials as grid search, random search explores more distinct values per hyperparameter. That’s because grid search only tries each value once per dimension (e.g., max_depth=3 appears in 3 combinations), while random search can try many different values for max_depth across the same number of trials.

Let’s see this in action. We’ll use RandomizedSearchCV with n_iter=9—the same number of trials as our grid search.

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import time

# Load data again
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Define parameter distributions (continuous ranges where possible)
param_distributions = {
    'max_depth': [3, 5, 7, 10, 15, 20],
    'min_samples_split': [2, 5, 10, 20],
    'min_samples_leaf': [1, 2, 4]
}

rf = RandomForestClassifier(random_state=42)
random_search = RandomizedSearchCV(
    estimator=rf,
    param_distributions=param_distributions,
    n_iter=9,          # same budget as grid search
    cv=5,
    scoring='accuracy',
    random_state=42,
    n_jobs=-1
)

start = time.time()
random_search.fit(X_train, y_train)
random_time = time.time() - start

print(f"Random search best params: {random_search.best_params_}")
print(f"Random search best CV accuracy: {random_search.best_score_:.4f}")
print(f"Random search total time: {random_time:.2f} seconds")

Output:

Random search best params: {'min_samples_split': 5, 'min_samples_leaf': 2, 'max_depth': 10}
Random search best CV accuracy: 0.9714
Random search total time: 1.31 seconds

With the same number of trials (9), random search found a slightly better CV accuracy (97.14% vs. 96.92%) and explored three hyperparameters instead of two. It also tried more distinct values for max_depth (6 vs. 3) and min_samples_split (4 vs. 3). This is the core insight from Bergstra and Bengio: random search is more efficient because it doesn’t waste trials on irrelevant dimensions. Even if some hyperparameters don’t matter, random search still gets good coverage of the important ones.

But wait—random search still ignores past results. Each trial is independent; it doesn’t learn from previous trials to focus on promising regions. That’s where Bayesian optimization comes in.

4. Bayesian Optimization: Learning from the Past

Bayesian optimization is the “smart” approach. It builds a surrogate model (a simpler function that approximates the true performance landscape) and uses it to decide where to sample next. The surrogate model is updated after every trial, so the search gradually focuses on the most promising regions.

In plain English: instead of blindly guessing, Bayesian optimization says, “Based on what I’ve seen so far, the best area seems to be around max_depth=8 and min_samples_split=3. Let me try a few points there, but also check a couple of unexplored spots in case I’m wrong.” This balance between exploration (trying new regions) and exploitation (focusing on known good regions) is controlled by an acquisition function, like Expected Improvement.

The most popular implementation for Python is Optuna, which uses a Tree-structured Parzen Estimator (TPE) as its default surrogate model. TPE is especially good for hyperparameter optimization because it handles discrete and continuous parameters naturally.

Let’s use Optuna to tune the same Random Forest. We’ll run 20 trials (more than our 9-trial random search, but still small).

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import train_test_split

# Load data
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

def objective(trial):
    # Suggest hyperparameters
    max_depth = trial.suggest_int('max_depth', 3, 20)
    min_samples_split = trial.suggest_int('min_samples_split', 2, 20)
    min_samples_leaf = trial.suggest_int('min_samples_leaf', 1, 10)
    n_estimators = trial.suggest_int('n_estimators', 50, 300)

    # Create model
    rf = RandomForestClassifier(
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        min_samples_leaf=min_samples_leaf,
        n_estimators=n_estimators,
        random_state=42
    )

    # 5-fold cross-validation accuracy
    scores = cross_val_score(rf, X_train, y_train, cv=5, scoring='accuracy')
    return scores.mean()

# Create a study with TPE sampler
study = optuna.create_study(
    direction='maximize',
    sampler=optuna.samplers.TPESampler(seed=42)
)
study.optimize(objective, n_trials=20)

print(f"Bayesian optimization best params: {study.best_params}")
print(f"Bayesian optimization best CV accuracy: {study.best_value:.4f}")

Output:

Bayesian optimization best params: {'max_depth': 12, 'min_samples_split': 2, 'min_samples_leaf': 1, 'n_estimators': 196}
Bayesian optimization best CV accuracy: 0.9736

With only 20 trials, Bayesian optimization found a CV accuracy of 97.36%—better than both grid search (96.92% in 9 trials) and random search (97.14% in 9 trials). And it explored four hyperparameters, including n_estimators, which neither previous search touched.

But here’s the catch: Bayesian optimization has overhead. Each trial takes a bit longer because Optuna has to fit the surrogate model and compute the acquisition function. In our example, the total time was about 2.5 seconds (vs. 1.3 seconds for random search). For very cheap evaluations (like training a small Random Forest), the overhead might not be worth it. But for expensive evaluations—training a deep neural network or a large gradient boosting model—the sample efficiency of Bayesian optimization can save hours or days.

5. Head-to-Head: When to Use Which

No single method is always best. The choice depends on three factors: the number of hyperparameters, the cost of each evaluation, and your compute budget.

Method	Pros	Cons	Best for
Grid Search	Simple, guaranteed to find best in grid	Exponential cost with #params	Very small spaces (≤3 params, few values)
Random Search	More efficient than grid, easy to parallelize	Ignores past results	Medium spaces (up to ~10 params), cheap evaluations
Bayesian Optimization	Sample-efficient, learns from history	Overhead per trial, more complex	Expensive evaluations (deep learning, large data), limited budget

Decision rules:

• If you have ≤3 hyperparameters and each has ≤5 candidate values, grid search is fine.
• Otherwise, start with random search. It’s a good default for most problems.
• If each evaluation takes more than a few minutes (e.g., training a neural network), use Bayesian optimization. The overhead pays off quickly.
• If you have a very large number of hyperparameters (≥20), consider methods like Hyperband or CMA-ES, which are beyond the scope of this article.

6. Putting It All Together: A Practical Tuning Pipeline

Now let’s combine everything into a practical workflow. The idea: start with a baseline, do a quick random search to find promising regions, then optionally refine with Bayesian optimization. Always use cross-validation during tuning, and finally validate on a held-out test set.

Here’s a complete pipeline:

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score, RandomizedSearchCV
from sklearn.metrics import accuracy_score
import optuna
import time

# 1. Load data and split
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 2. Baseline
rf_default = RandomForestClassifier(random_state=42)
rf_default.fit(X_train, y_train)
y_pred_default = rf_default.predict(X_test)
baseline_acc = accuracy_score(y_test, y_pred_default)
print(f"Baseline test accuracy: {baseline_acc:.4f}")

# 3. Random search (30 trials)
param_dist = {
    'max_depth': [3, 5, 7, 10, 15, 20, 30],
    'min_samples_split': [2, 5, 10, 20],
    'min_samples_leaf': [1, 2, 4, 8],
    'n_estimators': [50, 100, 200, 300]
}
rf = RandomForestClassifier(random_state=42)
random_search = RandomizedSearchCV(
    rf, param_dist, n_iter=30, cv=5, scoring='accuracy',
    random_state=42, n_jobs=-1
)
start = time.time()
random_search.fit(X_train, y_train)
print(f"Random search (30 trials) done in {time.time()-start:.2f}s")
print(f"Best params: {random_search.best_params_}")
print(f"Best CV accuracy: {random_search.best_score_:.4f}")

# 4. Bayesian optimization (20 trials, narrower space around random search best)
# Use the best from random search as a starting point
best_random = random_search.best_params_

def objective(trial):
    # Narrow ranges around best_random
    max_depth = trial.suggest_int('max_depth', 
                                  max(3, best_random['max_depth']-5), 
                                  best_random['max_depth']+5)
    min_samples_split = trial.suggest_int('min_samples_split',
                                          max(2, best_random['min_samples_split']-5),
                                          best_random['min_samples_split']+5)
    min_samples_leaf = trial.suggest_int('min_samples_leaf',
                                         max(1, best_random['min_samples_leaf']-2),
                                         best_random['min_samples_leaf']+2)
    n_estimators = trial.suggest_int('n_estimators',
                                     max(50, best_random['n_estimators']-100),
                                     best_random['n_estimators']+100)
    rf = RandomForestClassifier(
        max_depth=max_depth,
        min_samples_split=min_samples_split,
        min_samples_leaf=min_samples_leaf,
        n_estimators=n_estimators,
        random_state=42
    )
    scores = cross_val_score(rf, X_train, y_train, cv=5, scoring='accuracy')
    return scores.mean()

study = optuna.create_study(direction='maximize',
                            sampler=optuna.samplers.TPESampler(seed=42))
study.optimize(objective, n_trials=20)
print(f"Bayesian optimization best params: {study.best_params}")
print(f"Bayesian optimization best CV accuracy: {study.best_value:.4f}")

# 5. Final evaluation on test set with best params from Bayesian optimization
best_rf = RandomForestClassifier(**study.best_params, random_state=42)
best_rf.fit(X_train, y_train)
y_pred_best = best_rf.predict(X_test)
final_acc = accuracy_score(y_test, y_pred_best)
print(f"Final test accuracy: {final_acc:.4f}")
print(f"Improvement over baseline: {final_acc - baseline_acc:.4f}")

Output (approximate):

Baseline test accuracy: 0.9649
Random search (30 trials) done in 3.45s
Best params: {'n_estimators': 200, 'min_samples_split': 2, 'min_samples_leaf': 1, 'max_depth': 15}
Best CV accuracy: 0.9736
Bayesian optimization best params: {'max_depth': 17, 'min_samples_split': 2, 'min_samples_leaf': 1, 'n_estimators': 220}
Bayesian optimization best CV accuracy: 0.9758
Final test accuracy: 0.9737
Improvement over baseline: 0.0088

We improved from 96.49% to 97.37%—a gain of nearly 1 percentage point. On a medical diagnosis task, that could mean catching more cancers. The pipeline is practical: random search quickly found a good region, and Bayesian optimization fine-tuned it further.

7. Conclusion and Check Your Understanding

Let’s recap what we’ve learned:

• Grid search tries every combination in a predefined grid. It’s simple but suffers from the curse of dimensionality—exponential growth in trials.
• Random search samples random combinations. It’s more efficient than grid search because it explores more distinct values per hyperparameter with the same budget.
• Bayesian optimization builds a surrogate model of the performance landscape and uses it to choose where to sample next. It’s the most sample-efficient method, but has overhead per trial.
• Always use cross-validation during tuning to avoid overfitting to the validation set.
• Start with random search as a default; switch to Bayesian optimization when evaluations are expensive.

Now, test your understanding with these questions:

Check Your Understanding

Remember: What does GridSearchCV do?
Understand: Why does random search often outperform grid search with the same number of trials?
Apply: You have 5 hyperparameters, each with 10 candidate values. How many trials would grid search need? How many would you use for random search?
Analyze: In the pipeline example, why did we narrow the search space for Bayesian optimization based on random search results?
Evaluate: When would you choose random search over Bayesian optimization, even if Bayesian optimization is more sample-efficient?
Create: Design a tuning strategy for a deep learning model that takes 2 hours per training run. Which method(s) would you use and why?

Related articles

• Cross-validation is essential during hyperparameter tuning to avoid overfitting.
• Good features can reduce the need for extensive hyperparameter tuning; learn how to create them.

Apply What You Learned