XGBoost vs. LightGBM vs. CatBoost: How to Actually Choose Without a Math Degree

Last time, Sam got regularization working to tame overfitting on the linear models. Now the HomeMatch team wants to try gradient-boosted trees for the real sellability model. Sam’s not ready for that yet. He wants to compare the three major libraries on a smaller practice problem first, before betting on the real listings.

Sam felt like he was staring at a restaurant menu that ran on too long. He just wanted to build a model that predicts something—used car prices, say—but he was staring down three libraries that all claimed to do the same job. He’d heard of XGBoost, LightGBM, and CatBoost. All “Gradient Boosting” models, which just means they learn by making a mistake, fixing it, and repeating the process hundreds of times.

But if they all do the same thing, why are there three of them?

1. The ‘Better’ Trap: Why Three Libraries Exist for the Same Job

If one were strictly better, the other two would have disappeared years ago. “Better” depends on your data’s personality. Think of them as workshop tools—XGBoost is the reliable scalpel, LightGBM is the high-powered chainsaw, CatBoost is the versatile multi-tool.

Sam decided to practice on a smaller benchmark first—a used-car dataset—before applying the winner to HomeMatch’s real listings. The data has numbers (mileage), categories (brand, fuel type), and some missing values.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# Let's create a dummy dataset that mimics a used car market
data = {
    'brand': ['Toyota', 'Ford', 'BMW', 'Toyota', 'Ford', 'BMW'] * 100,
    'mileage': [50000, 80000, 30000, 45000, 120000, 15000] * 100,
    'fuel_type': ['Gas', 'Diesel', 'Gas', 'Hybrid', 'Diesel', 'Electric'] * 100,
    'price': [15000, 12000, 25000, 18000, 9000, 45000] * 100
}
df = pd.DataFrame(data)

# Introducing some missing values to make it realistic
df.iloc[0, 1] = np.nan 

X = df.drop('price', axis=1)
y = df['price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(f"Dataset loaded with {len(df)} rows and {len(df.columns)} columns.")

• import pandas as pd and import numpy as np bring in the two core data-science libraries: pandas for tabular data manipulation, numpy for numerical operations.
• from sklearn.model_selection import train_test_split imports the function that splits data into training and test sets so we can measure generalization.
• The data dictionary creates three repeating lists of 600 entries each (6 base values × 100) to simulate a used-car market — the * 100 repeats the pattern to give each model enough rows to learn from.
• pd.DataFrame(data) converts the dictionary into a pandas DataFrame with columns brand, mileage, fuel_type, and price.
• df.iloc[0, 1] = np.nan deliberately injects a missing value into the mileage column of the first row — real data is messy, and this tests how each library handles NaNs.
• X = df.drop('price', axis=1) separates features from the target; y = df['price'] isolates the price column as the thing being predicted.
• train_test_split(X, y, test_size=0.2, random_state=42) splits 80% of rows for training and 20% for testing, with a fixed random seed so the split is reproducible.

The dataset is small but has the “hard stuff”: words like ‘Toyota’ and ‘Hybrid’ that computers can’t naturally read. How each library handles those words is the first big difference we’ll see.

2. The Intuition: How They Grow Their Trees

Before the code, it helps to understand how these models “think.” They all build decision trees — they just grow them differently.

XGBoost is the meticulous architect. One floor at a time. It won’t start the second floor until every room on the first is finished. In technical terms, this is “Level-wise” growth. The tree stays balanced and stable.

LightGBM is the speed-runner. It ignores floors. It finds the room that yields the most information and builds there — even if that room is on the 10th floor while the 2nd sits empty. This is “Leaf-wise” growth. It’s much faster, but there’s a catch: it can fixate on one specific “room” and lose the big picture, which leads to overfitting on small datasets.

CatBoost is the master of categories. It builds “Symmetric Trees” — every split at the same level uses the exact same rule. That sounds restrictive, but it pays off. Predictions get fast once the model is trained, and the structure helps prevent memorizing the training data.

3. XGBoost: The Reliable Workhorse

For years, XGBoost dominated Kaggle competitions. The algorithm is famously robust, but it remains a bit old-school about data handling. Historically, it rejected strings outright. You had to convert “Toyota” into a number, like 1, yourself.

This means extra work before training begins. Applying “One-Hot Encoding” — turning one column into five columns of 0s and 1s — makes your data much “wider.” That extra width consumes more memory.

from xgboost import XGBRegressor
from sklearn.preprocessing import LabelEncoder

# XGBoost traditionally needs numbers, so we encode categories manually
X_xgb = X_train.copy()
encoder = LabelEncoder()
for col in ['brand', 'fuel_type']:
    X_xgb[col] = encoder.fit_transform(X_xgb[col])

xgb_model = XGBRegressor(n_estimators=100, learning_rate=0.1)
xgb_model.fit(X_xgb, y_train)
print("XGBoost training complete.")

• from xgboost import XGBRegressor imports XGBoost’s regression model (as opposed to XGBClassifier for classification tasks).
• from sklearn.preprocessing import LabelEncoder brings in a tool that converts string labels into integers — “Toyota” becomes 0, “Ford” becomes 1, etc.
• X_xgb = X_train.copy() makes a copy of the training features so the original X_train is preserved for the other libraries that don’t need encoding.
• The for col in ['brand', 'fuel_type'] loop applies LabelEncoder to each categorical column in place — this manual step is the “extra work” XGBoost traditionally requires before training can begin.
• XGBRegressor(n_estimators=100, learning_rate=0.1) creates a model with 100 boosted trees and a learning rate of 0.1 — each tree corrects the residual errors left by the previous ones.

Newer XGBoost versions handle categories natively, but most practitioners still encode manually to avoid surprises. For a medium-sized dataset where you need a model that won’t throw unexpected errors, sticking to the manual approach remains the standard safe choice.

4. LightGBM: The Speed Demon for Big Data

If you have a million rows, XGBoost might take an hour to train. LightGBM will likely do it in minutes.

It uses a trick called “binning.” Instead of looking at every single mileage value (like 50,001 and 50,002), it groups them into “bins” (like 50,000 to 51,000). Think of it like rounding numbers to make them easier to add in your head. It saves a lot of RAM.

import lightgbm as lgb

# LightGBM needs categories to be the 'category' data type
X_lgb = X_train.copy()
for col in ['brand', 'fuel_type']:
    X_lgb[col] = X_lgb[col].astype('category')

lgb_model = lgb.LGBMRegressor(n_estimators=100, learning_rate=0.1, verbose=-1)
lgb_model.fit(X_lgb, y_train)
print("LightGBM training complete.")

• import lightgbm as lgb imports the LightGBM library, which uses histogram-based binning for speed and memory efficiency.
• X_lgb = X_train.copy() makes a fresh copy of the training features so the original is preserved for CatBoost (which needs the raw strings).
• X_lgb[col] = X_lgb[col].astype('category') converts string columns to pandas’ categorical dtype — LightGBM can natively handle this type without manual integer encoding, but it still needs the dtype conversion.
• lgb.LGBMRegressor(n_estimators=100, learning_rate=0.1, verbose=-1) silences LightGBM’s training output with verbose=-1 while using the same 100-tree / 0.1-learning-rate configuration as XGBoost for a fair comparison.

Notice how much faster that felt? Well, if your dataset is tiny (like 100 rows), LightGBM’s aggressive “leaf-wise” growth might find patterns that are just noise. It’s a chainsaw: great for clearing a forest, but maybe too much for a bonsai tree.

5. CatBoost: The ‘Set It and Forget It’ Miracle

Here’s what throws most people off. CatBoost actually prefers you don’t encode your data. You hand it the raw strings, tell it which columns are categories, and it handles the rest.

It uses “Ordered Boosting” to prevent “data leakage.” Most models “cheat” a little by looking at the average price of all Toyotas across the whole dataset. CatBoost only looks at the Toyotas it has seen before the current row, which mirrors how things work in the real world.

from catboost import CatBoostRegressor

# No encoding needed! Just tell it which columns are categories.
cat_features = ['brand', 'fuel_type']
cat_model = CatBoostRegressor(iterations=100, learning_rate=0.1, verbose=0)
cat_model.fit(X_train, y_train, cat_features=cat_features)
print("CatBoost training complete.")

• from catboost import CatBoostRegressor imports CatBoost’s regression model — no separate encoder import needed, unlike the XGBoost example.
• cat_features = ['brand', 'fuel_type'] tells CatBoost which columns are categorical — no manual encoding required; CatBoost handles the raw strings internally with its own target-based encoding.
• CatBoostRegressor(iterations=100, learning_rate=0.1, verbose=0) uses iterations (CatBoost’s term for n_estimators) and silences output with verbose=0 — the parameter naming differs from XGBoost/LightGBM but serves the same purpose.
• cat_model.fit(X_train, y_train, cat_features=cat_features) trains directly on the original X_train with no preprocessing — the key “set it and forget it” advantage that saves the most lines of code.

CatBoost is often the most accurate “out of the box” because its default settings are smart. It’s the multi-tool that just works.

6. The Verdict: Which One Should You Use?

So how do you choose? Here’s how the data stacks up against what we need:

• Use XGBoost if: You have a smaller, clean dataset (under 50,000 rows) and you want a tried-and-true model that is very hard to break.
• Use LightGBM if: You have massive data (millions of rows) or very limited RAM. It is the king of efficiency.
• Use CatBoost if: Your data is full of strings and categories. It will save you hours of preprocessing and usually gives the best results with zero tuning.

Feature	XGBoost	LightGBM	CatBoost
Training Speed	Medium	Fastest	Slowest (usually)
Ease of Use	Medium	Medium	Easiest
Categorical Support	Manual/Basic	Good	Best
Best for…	Stability	Big Data	Categorical Data

In our car example, the “Time to Train” for LightGBM was roughly 3x faster than XGBoost, but CatBoost required the fewest lines of code to handle the brand names.

What’s next? Sam picked his library for the HomeMatch sellability model—but before trusting an ensemble of hundreds of trees, he wanted to understand what’s actually happening inside a single one. In the next part of the series, we’ll step back and look at Decision Trees from scratch—the building block that XGBoost, LightGBM, and CatBoost all grow hundreds of, so you can see exactly what’s happening inside each “room” of the tree. Then start building.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What do “Level-wise,” “Leaf-wise,” and “Symmetric” tree growth mean, and which library uses each?

Understand In your own words, explain why LightGBM’s leaf-wise growth is described as risking overfitting on small datasets, using the article’s “chainsaw vs. bonsai tree” analogy.

Apply Using the article’s decision table (Training Speed, Ease of Use, Categorical Support), if you had a 5-million-row dataset with heavy categorical features and limited RAM, which single library would best satisfy both constraints, and which column of the table would you have to compromise on?

Analyze The article explains CatBoost’s “Ordered Boosting” as only looking at Toyotas “seen before the current row” instead of averaging over the whole dataset. Walk through why looking at the whole dataset’s average price for “Toyota” (including rows that come later) counts as a form of data leakage, even though it seems like it would just make the model more accurate.

Evaluate The article calls CatBoost “the most accurate ‘out of the box’” and says it “usually gives the best results with zero tuning.” Critique this framing: given the article’s own decision table shows CatBoost as “Slowest (usually),” what’s the tradeoff a team is implicitly accepting if they default to CatBoost purely because it needs less tuning?

Create Design a quick benchmarking plan (not full code) to actually test which of the three libraries is best for a specific new dataset: 200,000 rows of e-commerce transactions with 15 categorical features (product category, region, payment method) and a moderate compute budget. What would you measure, and what result would tip your decision toward each library?

---

Related articles

• P03: Regularization Explained — Why Your Models Overfit and How to Stop It — where Sam tamed overfitting with Ridge, Lasso, and Elastic Net on the linear models before graduating to gradient-boosted trees.
• P05: Decision Trees from Scratch — How Splits Actually Get Made — the next step for Sam: opening up a single tree to understand the building block that all three boosting libraries grow hundreds of.

References & Further reading

• Chen, T., & Guestrin, C. (2016). “XGBoost: A Scalable Tree Boosting System.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. — the foundational paper that introduced XGBoost, its level-wise tree growth, and the system design choices that made it a Kaggle legend.
• Ke, G., Meng, D., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). “LightGBM: A Highly Efficient Gradient Boosting Decision Tree.” Advances in Neural Information Processing Systems, 30. — the paper that introduced LightGBM’s leaf-wise growth and histogram-based binning for dramatic speed and memory gains.
• Kaggle: House Prices — Advanced Regression Techniques — a competition where XGBoost, LightGBM, and CatBoost dominate the leaderboard for tabular housing prediction.
• XGBoost documentation: xgboost.readthedocs.io — official API reference, parameter guide, and tutorials.
• LightGBM documentation: lightgbm.readthedocs.io — official parameter reference including the leaf-wise vs. level-wise growth configuration.
• CatBoost documentation: catboost.ai — official docs covering Ordered Boosting, symmetric trees, and categorical handling.

---