Encoding Categorical Variables One Hot Label Targe

You’ve cleaned the data, handled the missing values, and built a beautiful feature set. Then you hit the wall: your best column is full of words like “blue,” “green,” and “red,” and the machine learning model you’re about to train only eats numbers. What do you do?

Most of us reach for the obvious fix—just assign a number to each color. Red becomes 1, blue becomes 2, green becomes 3. It runs without an error, so it must be fine, right?

Not quite. That tiny decision can quietly break your model in ways that are hard to spot until you’re wondering why your predictions are terrible in production. This article is about making that decision intentionally. We’ll walk through four encoding strategies—one-hot, label, target, and embedding encoding—and, more importantly, we’ll build the intuition for when each one is the right tool and when it’s a trap.

1. The Problem: Why Can’t We Just Feed ‘Blue’ to a Model?

Let’s make this concrete. Imagine you’re building a model to predict house prices. You’ve got a column called Neighborhood with values like “CollgCr,” “Veenker,” and “Edwards.” You load the data, split it, and try to fit a linear regression. The moment you pass that column to the model, it throws an error—something about strings and floats.

Your first instinct is probably something like this:

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import LabelEncoder

# Load a small, relatable housing dataset
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

# We'll use a few columns to keep things clear
features = ["Lot Area", "Neighborhood"]
target = "SalePrice"

# The Neighborhood column is full of strings
print("First few neighborhoods:")
print(df["Neighborhood"].head())
print(f"\nUnique neighborhoods: {df['Neighborhood'].nunique()}")

# The naive approach: map each neighborhood to an integer
X = df[features].copy()
y = df[target].copy()

le = LabelEncoder()
X["Neighborhood_encoded"] = le.fit_transform(X["Neighborhood"])

print("\nMapping from neighborhood to integer:")
for i, neighborhood in enumerate(le.classes_[:5]):
    print(f"  {neighborhood} -> {i}")

# Now fit a linear model
X_model = X[["Lot Area", "Neighborhood_encoded"]]
X_train, X_test, y_train, y_test = train_test_split(X_model, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

print(f"\nCoefficients:")
print(f"  Lot Area: {model.coef_[0]:.2f}")
print(f"  Neighborhood_encoded: {model.coef_[1]:.2f}")

The model runs. It even gives you a coefficient for the neighborhood: a single number that says, “For every one-unit increase in the neighborhood code, the predicted price changes by this many dollars.”

But what does a “one-unit increase in neighborhood” actually mean? The encoder assigned “CollgCr” to 0, “Edwards” to 1, “Gilbert” to 2, and so on. A linear model takes that number literally. It thinks “Gilbert” (2) is twice “Edwards” (1) in some meaningful sense. It thinks the distance between “CollgCr” and “Edwards” is the same as the distance between “Edwards” and “Gilbert.” None of that is true—these are just names. We’ve lied to the model about the structure of the data, and it has obediently learned that lie.

This is the central tension of categorical encoding: how do we represent categories as numbers without lying to the model about their relationships? The answer depends on three things: how many categories you have, what kind of model you’re using, and whether the target variable can safely inform the encoding. Let’s work through the four main answers, starting with the one everybody learns first.

2. One-Hot Encoding: The Safe Default (and When It Breaks)

If the integer-encoding lie bothers you, the natural fix is to stop pretending there’s an order at all. Instead of one column with numbers 0, 1, 2, you create one column per category and put a 1 where that category is present and a 0 everywhere else. This is one-hot encoding, and it’s the first real solution most data scientists reach for.

Think of it this way: you’re not telling the model “this row is a 1.” You’re telling it “this row is red—yes, blue—no, green—no.” Every category gets its own on/off switch, and the model can learn a separate weight for each switch. No false ordering, no imaginary distances.

Here’s what that looks like in code:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

# Load the same housing dataset
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

# We'll focus on a lower-cardinality column to see the pattern clearly
# MS Zoning has only a handful of categories
print("MS Zoning categories:")
print(df["MS Zoning"].value_counts())

# Before encoding
print(f"\nShape before one-hot encoding: {df[['MS Zoning']].shape}")

# Apply OneHotEncoder
ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
# We'll fit on the whole column for demonstration
encoded_array = ohe.fit_transform(df[["MS Zoning"]])
encoded_df = pd.DataFrame(encoded_array, columns=ohe.get_feature_names_out(["MS Zoning"]))

print(f"Shape after one-hot encoding: {encoded_df.shape}")
print("\nFirst 3 rows of the one-hot encoded data:")
print(encoded_df.head(3))

A column that had 7 unique values became 7 columns. Each row has exactly one 1 and six 0s. The model now has seven honest switches to work with instead of one lying number.

Now here’s the interesting part. What happens when a column has 1,000 categories instead of 7? You get 1,000 new columns. If you have a million rows, that’s a billion cells, most of them zeros. This is the dimensionality explosion, and it’s the reason one-hot encoding isn’t always the right answer. It can blow up your memory, slow training to a crawl, and—for tree-based models especially—create sparse splits that hurt performance because the model struggles to find useful signal across a thousand mostly-empty columns.

There’s one more subtle gotcha worth knowing. Many tutorials tell you to drop one of the one-hot columns to avoid multicollinearity—the “dummy variable trap.” And for some statistical models, that’s good advice. But scikit-learn’s own documentation warns that using the drop parameter “breaks the symmetry of the original representation and can therefore induce a bias in downstream models.” If you’re using a regularized linear model (like Ridge or Lasso), the regularization already handles the multicollinearity, and dropping a column can actually make things worse. Unless you have a specific reason to drop, it’s usually safer to keep all the columns.

One last feature that matters in production: handle_unknown='ignore'. Let’s see why it’s critical:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

# Recreate the encoder on the same data
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
ohe.fit(df[["MS Zoning"]])

# Imagine a new house shows up with a zoning category we've never seen
new_data = pd.DataFrame({"MS Zoning": ["Unknown_Zone"]})
encoded_new = ohe.transform(new_data)

print("Encoded row for an unseen category:")
print(encoded_new)
print("\nAll zeros—the encoder silently handles the unknown category.")
print("Without handle_unknown='ignore', this would raise an error and crash your pipeline.")

Rule of thumb: one-hot encode when cardinality is low (fewer than 10–15 categories) and you’re using a linear model, logistic regression, or a neural network. Avoid it for tree-based models with moderate-to-high cardinality—we’ll see a better option for those next.

3. Label / Ordinal Encoding: When Order Actually Makes Sense

Remember the integer encoding we called dangerous in the introduction? It’s time to revisit it—because it’s not always wrong. In fact, there are two very specific situations where it’s exactly the right call.

First, the obvious one: when your categories have a real, meaningful order. “Small,” “medium,” and “large” aren’t just arbitrary labels—large really is bigger than medium, and medium really is bigger than small. “Education level” (high school < bachelor’s < master’s < PhD) has a genuine progression. For these ordinal variables, mapping them to 0, 1, 2, 3 isn’t a lie—it’s encoding the truth. The model should understand that PhD > master’s, and an integer encoding tells it exactly that.

But here’s the second, less obvious use case—and it’s the one that surprises a lot of people. Tree-based models don’t care about the false order. A decision tree doesn’t multiply a coefficient by the encoded value the way a linear model does. Instead, it asks questions like “is the encoded neighborhood <= 3?” That question is really just a way of grouping categories: neighborhoods 0, 1, 2, and 3 go left; neighborhoods 4, 5, and 6 go right. The tree doesn’t know or care that 4 is “twice” 2—it’s just using the integer as a bucket label. The false order we worried about with linear models is invisible to trees.

Let’s check that against the data:

import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import OrdinalEncoder
import numpy as np

# Load the housing data
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

# We'll use Neighborhood again—it's nominal, not ordinal
features = ["Lot Area", "Neighborhood"]
target = "SalePrice"

X = df[features].copy()
y = df[target].copy()

# Ordinal (label) encoding
ord_enc = OrdinalEncoder()
X_ord = X.copy()
X_ord["Neighborhood"] = ord_enc.fit_transform(X[["Neighborhood"]])

# Split the data
X_train, X_test, y_train, y_test = train_test_split(X_ord, y, test_size=0.2, random_state=42)

# Linear model with ordinal encoding (this should be bad for nominal data)
lr = LinearRegression()
lr_scores = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_mean_squared_error')
lr_rmse = np.sqrt(-lr_scores.mean())

# Random forest with the same encoding (this should be fine)
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf_scores = cross_val_score(rf, X_train, y_train, cv=5, scoring='neg_mean_squared_error')
rf_rmse = np.sqrt(-rf_scores.mean())

print(f"Linear regression RMSE with ordinal encoding: ${lr_rmse:,.0f}")
print(f"Random forest RMSE with ordinal encoding: ${rf_rmse:,.0f}")
print("\nThe linear model struggles because it's been fed a false order.")
print("The random forest doesn't care—it just splits on thresholds.")

The random forest handles the nominal data just fine with a simple integer encoding. The linear model, on the other hand, is trying to learn a slope across neighborhoods that don’t have a meaningful order—and its performance suffers for it.

Scikit-learn’s own documentation for OrdinalEncoder actually calls this out: it positions ordinal encoding as the natural fit for tree-based models while directing you to OneHotEncoder for linear models with nominal categories. The library itself is telling you the heuristic.

The hard part is knowing whether your categories are truly ordinal. “Education level” is ordinal. “City” is not. “Star rating” is ordinal. “Product category” is not. Getting this wrong bakes a false assumption into your model, and for linear models especially, that assumption will quietly degrade your predictions.

Rule of thumb: if you’re using a tree-based model (decision trees, random forests, XGBoost, LightGBM), try label encoding first—even for nominal data. It’s memory-efficient and often performs just as well as one-hot encoding without the dimensionality explosion. If you’re using a linear model and your data is ordinal, use ordinal encoding and preserve the true order. If you’re using a linear model and your data is nominal, go back to one-hot encoding.

4. Target Encoding: Using the Target to Encode Categories (Without Cheating)

So far you’ve got two tools: one-hot encoding (safe but explodes with high cardinality) and label encoding (compact but lies to linear models). What do you do when you’ve got a column with 50, 100, or 1,000 categories and you’re using a model that can’t handle the dimensionality explosion?

Enter target encoding. Instead of creating a thousand binary columns or assigning arbitrary integers, you replace each category with a single number: the average target value for that category. If you’re predicting whether a house sells above the median price, and 73% of houses in “CollgCr” do, then “CollgCr” becomes 0.73. You’ve compressed the entire category into one meaningful number that captures its relationship with the thing you’re trying to predict.

This is incredibly powerful. But it’s also dangerous—and this is the hardest part of categorical encoding.

If you compute the target mean using the same rows you’re using to train the model, you’ve leaked the target into your features. Your model will look brilliant in training because you’ve essentially handed it a cheat sheet with some of the answers already filled in. Then you deploy it, and the performance collapses because the cheat sheet doesn’t exist in the real world.

Scikit-learn’s TargetEncoder prevents this with an internal cross-fitting scheme. It splits the data into folds, computes the target mean for each category using only the other folds, and assigns those “out-of-fold” estimates. This means fit_transform(X, y) deliberately does not equal fit(X, y).transform(X). The first uses cross-fitting; the second doesn’t, and it’s leaky.

Let’s see the difference:

import pandas as pd
from sklearn.preprocessing import TargetEncoder

# Load the housing data and create a binary target
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

# Create a binary target: above or below median price
median_price = df["SalePrice"].median()
df["AboveMedian"] = (df["SalePrice"] > median_price).astype(int)

# We'll use Neighborhood—it has 25+ categories, high enough to matter
X = df[["Neighborhood"]]
y = df["AboveMedian"]

# Fit the encoder
te = TargetEncoder(target_type="binary")

# The RIGHT way: fit_transform uses cross-fitting internally
te_safe = TargetEncoder(target_type="binary")
encoded_safe = te_safe.fit_transform(X, y)

# The WRONG way: fit, then transform on the same data—no cross-fitting
te_leaky = TargetEncoder(target_type="binary")
te_leaky.fit(X, y)
encoded_leaky = te_leaky.transform(X)

# Compare the two
comparison = pd.DataFrame({
    "Neighborhood": X["Neighborhood"],
    "Safe (cross-fit)": encoded_safe.flatten(),
    "Leaky (no cross-fit)": encoded_leaky.flatten()
})

print("First 8 rows—notice the safe and leaky values differ:")
print(comparison.head(8))
print("\nThey're different because the safe version used out-of-fold estimates.")
print("The leaky version peeked at the target for the same row it's encoding.")

The numbers are different—sometimes subtly, sometimes dramatically. The safe version is a honest estimate of what the category means for the target, computed without peeking at the row it’s encoding. The leaky version has already seen the answer.

Target encoding also has a clever protection against overfitting for rare categories. If a neighborhood appears only three times in the data, its target mean could be extreme just by chance—maybe all three happened to be above median. The encoder shrinks that estimate toward the global mean, so a rare category with an extreme value gets pulled back toward the average. This is called shrinkage, and it’s built into scikit-learn’s implementation.

What do the encoded values actually mean? Let’s interpret them:

import pandas as pd
from sklearn.preprocessing import TargetEncoder

# Reload and re-encode to get a clean mapping
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)
median_price = df["SalePrice"].median()
df["AboveMedian"] = (df["SalePrice"] > median_price).astype(int)

X = df[["Neighborhood"]]
y = df["AboveMedian"]

te = TargetEncoder(target_type="binary")
encoded = te.fit_transform(X, y)

# Show the average target-encoded value per neighborhood
result = pd.DataFrame({
    "Neighborhood": X["Neighborhood"],
    "Encoded_Value": encoded.flatten()
})
neighborhood_means = result.groupby("Neighborhood")["Encoded_Value"].mean().sort_values(ascending=False)

print("Neighborhoods ranked by target-encoded value (higher = more likely above median):")
print(neighborhood_means.head(10))
print("\nIn plain English: houses in the top neighborhood have roughly")
print("an 80%+ chance of being above the median price, while houses")
print("in the bottom neighborhoods are much less likely.")

A value of 0.82 doesn’t mean “this house is 82% likely to be above median.” It means “houses in this neighborhood, on average, have an 82% rate of being above median, and this single number captures that neighborhood’s overall relationship with the target.” The model can then use that signal alongside other features.

Rule of thumb: use target encoding for high-cardinality categoricals (50+ categories) when you’re using a model that benefits from a single numeric feature per category—linear models, neural networks, and even gradient-boosted trees when you want to give them a strong starting signal. Always use fit_transform (or an explicit cross-fitting pipeline) to avoid leakage. Never compute target means on the full dataset and then train on those same rows.

5. Embedding Encoding: Letting a Neural Network Learn the Representation

All the encoders we’ve seen so far are hand-engineered: we decide the rules for turning categories into numbers. One-hot encoding says “make a binary column for each category.” Target encoding says “use the average target value.” But what if we could let the model invent its own representation?

That’s what embedding encoding does. Instead of assigning each category a single number, it assigns each category a dense vector of numbers—say, 32 floats. And here’s the key: those numbers aren’t chosen by us. They’re learned by the neural network during training, adjusted to minimize the loss just like the network’s weights.

Think of it like word embeddings in natural language processing. In a word2vec model, the words “king” and “queen” end up with similar vectors because they appear in similar contexts. The model discovers that relationship on its own. Embedding encoding does the same thing for your categorical variables: if two neighborhoods behave similarly with respect to house prices, their embedding vectors will drift close together in the learned space. The network invents its own features.

This is fundamentally different from target encoding. Target encoding gives you one number per category based on a simple statistic. Embedding encoding gives you a rich, multi-dimensional representation that can capture complex, non-linear relationships between categories. A neighborhood might be similar to another in some dimensions (price appreciation) but different in others (lot size), and the embedding space can represent that nuance.

The process has two steps. First, you map each category to an integer using a vocabulary lookup—just like label encoding. Second, you feed that integer into an Embedding layer that learns a dense vector for each integer index. The embedding layer is just another layer in your neural network, trained end-to-end with everything else.

Let’s build a minimal example:

import numpy as np
import pandas as pd
import tensorflow as tf
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load the housing data
url = "https://raw.githubusercontent.com/plotly/datasets/master/ames-housing.csv"
df = pd.read_csv(url)

# We'll use Neighborhood as our high-cardinality categorical
# and a couple of numeric features
cat_col = "Neighborhood"
num_cols = ["Lot Area", "Year Built"]
target_col = "SalePrice"

# Build the vocabulary: map each neighborhood to a unique integer
vocab = {name: idx for idx, name in enumerate(df[cat_col].unique())}
vocab_size = len(vocab)
print(f"Vocabulary size (unique neighborhoods): {vocab_size}")

# Map the categorical column to integers
df["Neighborhood_idx"] = df[cat_col].map(vocab)

# Prepare features and target
X_cat = df[["Neighborhood_idx"]].values
X_num = df[num_cols].values
y = df[target_col].values.reshape(-1, 1)

# Scale numeric features and the target
num_scaler = StandardScaler()
X_num_scaled = num_scaler.fit_transform(X_num)

y_scaler = StandardScaler()
y_scaled = y_scaler.fit_transform(y)

# Split
X_cat_train, X_cat_test, X_num_train, X_num_test, y_train, y_test = train_test_split(
    X_cat, X_num_scaled, y_scaled, test_size=0.2, random_state=42
)

# Build a model with an embedding layer for the categorical feature
embedding_dim = 8  # Each neighborhood gets an 8-dimensional vector

cat_input = tf.keras.layers.Input(shape=(1,), name="neighborhood_input")
embedding = tf.keras.layers.Embedding(
    input_dim=vocab_size,
    output_dim=embedding_dim,
    name="neighborhood_embedding"
)(cat_input)
embedding_flat = tf.keras.layers.Flatten()(embedding)

num_input = tf.keras.layers.Input(shape=(len(num_cols),), name="numeric_input")

combined = tf.keras.layers.Concatenate()([embedding_flat, num_input])
dense = tf.keras.layers.Dense(32, activation="relu")(combined)
output = tf.keras.layers.Dense(1)(dense)

model = tf.keras.Model(inputs=[cat_input, num_input], outputs=output)
model.compile(optimizer="adam", loss="mse")

print("\nTraining the model—the embedding layer learns alongside the dense layers...")
history = model.fit(
    [X_cat_train, X_num_train], y_train,
    validation_split=0.2,
    epochs=30,
    batch_size=32,
    verbose=0
)

print(f"Final training loss: {history.history['loss'][-1]:.4f}")
print(f"Final validation loss: {history.history['val_loss'][-1]:.4f}")

# Extract the learned embeddings
embedding_weights = model.get_layer("neighborhood_embedding").get_weights()[0]
print(f"\nLearned embedding shape: {embedding_weights.shape}")
print(f"(That's one 8-dimensional vector for each of the {vocab_size} neighborhoods)")

# Show a few learned vectors
print("\nFirst 3 neighborhoods and their learned embedding vectors:")
for i, (name, idx) in enumerate(list(vocab.items())[:3]):
    print(f"  {name}: {embedding_weights[idx][:4]}... (first 4 of 8 dimensions)")

The network learned an 8-dimensional vector for each neighborhood, and those vectors were optimized to help predict sale prices. Two neighborhoods that affect prices similarly will have vectors that are close together in this 8-dimensional space. The model discovered that structure on its own—we didn’t have to tell it which neighborhoods are similar.

This approach comes from a 2016 paper by Guo and Berkhahn called “Entity Embeddings of Categorical Variables.” They showed that learned embeddings consistently outperform one-hot encoding on high-cardinality categoricals, especially when the categories have meaningful relationships the network can discover. In one experiment on a taxi dataset with hundreds of pickup-location IDs, the embedding approach beat one-hot encoding by a significant margin.

The cost is real, though. You need a neural network, which adds complexity, training time, and hyperparameter decisions (how many dimensions in the embedding? what architecture?). This isn’t the first tool you reach for. It’s the tool for when you already have a neural network in your pipeline, you’ve got high-cardinality features, and you want to squeeze out extra performance by letting the model learn its own categorical representations.

One important note: the category_encoders library—which is excellent and worth knowing about—does not implement embedding encoding. Its high-cardinality tools are hashing and statistical encoders. For embedding encoding, you use a deep learning framework’s Embedding layer directly, as we just did.

6. Decision Framework: Which Encoder Should You Actually Use?

We’ve covered four encoders, each with its own strengths and failure modes. The question now is: when you’re staring at a new dataset, which one do you pick?

The answer depends on three questions:

1. How many unique categories? (cardinality)
2. What kind of model are you using? (linear, tree-based, or neural network)
3. Do the categories have a natural order? (nominal vs. ordinal)

Here’s the decision framework, distilled into a few rules:

• Low cardinality (<15) + linear model or neural net → One-hot encoding. It’s the safe default. No false ordering, no leakage risk, and the dimensionality is manageable.

• Any cardinality + tree-based model → Label/ordinal encoding. Trees split on thresholds, so the false integer order doesn’t hurt them. This is memory-efficient and often performs just as well as one-hot encoding for tree-based models.

• High cardinality (>50) + linear model or neural net → Target encoding (with cross-fitting) or embedding encoding (if using a neural network). Target encoding compresses the category into a single meaningful number; embedding encoding learns a rich multi-dimensional representation.

• Ordinal data (any cardinality) → Ordinal encoding, preserving the true order. If “small < medium < large” is real, tell the model that truth.

Let’s make that executable:

def recommend_encoder(cardinality, model_type, is_ordinal=False):
    """
    Return a recommended encoding strategy based on the decision framework.
    
    Parameters:
        cardinality: int, number of unique categories
        model_type: str, one of 'linear', 'tree', 'neural_network'
        is_ordinal: bool, whether the categories have a natural order
    """
    if is_ordinal:
        return "OrdinalEncoder (preserve the true order)"
    
    if model_type == 'tree':
        return "OrdinalEncoder / LabelEncoder (trees handle splits, not magnitudes)"
    
    if model_type in ('linear', 'neural_network'):
        if cardinality < 15:
            return "OneHotEncoder (safe default, manageable dimensionality)"
        else:
            return "TargetEncoder with cross-fitting (or Embedding layer for neural networks)"
    
    return "Unknown model type—please specify 'linear', 'tree', or 'neural_network'"

# Test the function
print("Neighborhood (28 categories, linear model, nominal):")
print(f"  -> {recommend_encoder(28, 'linear')}")

print("\nNeighborhood (28 categories, random forest, nominal):")
print(f"  -> {recommend_encoder(28, 'tree')}")

print("\nMS Zoning (7 categories, linear model, nominal):")
print(f"  -> {recommend_encoder(7, 'linear')}")

print("\nOverall Quality (10 categories, linear model, ordinal):")
print(f"  -> {recommend_encoder(10, 'linear', is_ordinal=True)}")

Before we wrap up, let’s name the two mistakes I see most often in the wild:

The most common mistake: using label encoding with a linear model on nominal data. The model learns a false magnitude relationship, and you get coefficients that make no sense. If you’re using a linear model and your categories don’t have an order, one-hot encode them.

The most dangerous mistake: computing target encoding on the full dataset without cross-fitting. Your validation metrics will look fantastic, and your production performance will be a disaster. Always use fit_transform (which cross-fits internally) or build an explicit cross-fitting pipeline. Never compute target means on the same rows you’re training on.

If you need to go beyond scikit-learn’s TargetEncoder, the category_encoders library offers additional target-style encoders—CatBoost encoding, James-Stein encoding, Leave-One-Out encoding, and Weight-of-Evidence encoding—each with slightly different shrinkage and smoothing strategies. None of them are embedding encoders, but they give you more options when the standard target encoder isn’t quite right for your data.

---

Recap

• The core problem: models eat numbers, not strings. The naive integer mapping lies to linear models by inventing an order that doesn’t exist.
• One-hot encoding removes the false order by creating binary columns, but explodes with high cardinality. Use it for low-cardinality features with linear models and neural networks.
• Label/ordinal encoding is correct for ordinal data and, surprisingly, works fine for nominal data with tree-based models because trees split on thresholds, not magnitudes.
• Target encoding compresses high-cardinality features into a single meaningful number, but cross-fitting is non-negotiable—without it, you’re leaking the target.
• Embedding encoding lets a neural network learn its own categorical representations, capturing complex inter-category relationships that hand-engineered encodings miss.
• The decision framework: ask about cardinality, model type, and whether the categories are ordinal—then pick the encoder that fits.

Check Your Understanding

Remember: What is the key difference between nominal and ordinal categorical variables? Why does that difference matter for encoding?

Understand: In your own words, explain why fit_transform(X, y) and fit(X, y).transform(X) give different results for scikit-learn’s TargetEncoder. What is the encoder doing differently in each case?

Apply: You have a dataset with a “City” column containing 200 unique city names, and you’re building a linear regression model. Which encoding strategy should you use, and why?

Analyze: A colleague tells you they used LabelEncoder on a “Product Category” column (50 categories, no natural order) and trained a logistic regression model that achieved 95% accuracy. What questions would you ask about their approach, and what potential issues would you investigate?

Evaluate: Compare one-hot encoding and target encoding for a dataset with a 100-category feature and a binary classification target. What are the trade-offs in terms of dimensionality, leakage risk, and the kind of signal each encoding captures?

Create: Design an encoding strategy for a dataset with three categorical columns: “Country” (3 categories, nominal), “Education Level” (5 categories, ordinal), and “Product ID” (10,000 categories, nominal). You’re building a gradient-boosted tree model. Justify each choice.