Reference: Feature Engineering

A standalone reference covering categorical encoding, numeric scaling, derived features (binning / interactions / polynomial terms), time-series feature construction, and distance metrics. Use it as a lookup table when you’re about to fit a model and need to decide what to do with the columns you’ve got.

---

Roster

Encoders

Encoder	One-line definition	Output range	When to use	Corpus article
One-hot	Dummy column per category level	0/1	Low-cardinality categoricals (< ~15 levels); any model	XGBoost vs LightGBM vs CatBoost)
Label	Integer per category level	0 .. k-1	Only when the integers carry no order (rare; mostly legacy / CatBoost internal)	XGBoost vs LightGBM vs CatBoost)
Ordinal	Integer per level, mapped by an ordered mapping	0 .. k-1	Genuine ordered categories ("low" < "med" < "high")	Feature engineering for tabular data)
Target (mean)	Replace category with mean of y over training rows in that category	continuous	High-cardinality categoricals; only with CV folds or smoothing	Feature engineering for tabular data)
Hashing	Apply a hash function to category string, mod n_buckets	0/1 or count	Very high cardinality, online / streaming, no stable vocab	Feature engineering for tabular data)

Scalers

Scaler	Formula	Output range	When to use	Corpus article
Standard (z-score)	(x - μ) / σ	~ centered at 0, std 1 (not bounded)	Gaussian-ish features; linear models, SVMs, NNs	Feature scaling)
Robust	(x - median) / IQR	centered at 0 (IQR-scaled)	Skewed / outlier-heavy features; same model families	Feature scaling)
Min-max	(x - min) / (max - min)	0 .. 1	When a bounded range is required (image pixels, NN inputs); distance models that need same scale	Feature scaling)
Unit-vector (L2 norm)	x / ‖x‖₂	row norm = 1	Text / embedding vectors; cosine turns into dot product	Vector databases compared), RAG pipeline)

Derived features

Transform	What it adds	Cost	When to use
Binning (k-bins)	Converts a continuous variable into k ordinal bins	Loses information, requires bin-edge selection	Non-linear effect of a single numeric on y; “age bands”, “price buckets”
Interaction terms	x1 * x2, x1 & x2 (logical)	Combinatorial blow-up; only for linear models that can’t see interactions	When the effect of one feature depends on another (e.g., “price * region”)
Polynomial features	x, x², x³, x1·x2, ...	Exponentially growing column count with degree	Linear regression with non-linear surface; regularization essential

Time-series features

Feature	Definition	When to use	Corpus article
Lag	y[t-k] as a feature for predicting y[t]	When past values predict the present (autoregressive structure)	Time-series feature engineering)
Rolling window	Aggregation (mean/std/min/max) over a trailing window	When recent volatility / level matters	Time-series feature engineering)
Expanding window	Aggregation from series start to current point	When all history is relevant (cumulative mean, running min)	Time-series feature engineering)
Fourier / seasonal	sin(2πkt/T), cos(2πkt/T) for seasonal periods T	Strong periodicity with known period (daily/weekly/yearly)	Time-series feature engineering)

Distance metrics

Metric	Formula	When to use	Corpus article
Euclidean (L2)	√Σ(xᵢ - yᵢ)²	Dense numeric vectors of the same scale; default “as the crow flies” distance	kNN in high dimensions)
Manhattan (L1)	`Σ	xᵢ - yᵢ	`
Minkowski	`(Σ	xᵢ - yᵢ	^p)^(1/p)`
Cosine	1 - (x·y)/(‖x‖·‖y‖)	Direction matters more than magnitude; embeddings, text, sparse high-dim	Vector databases compared)
Jaccard	`1 -	A∩B	/
Mahalanobis	√((x-y)ᵀ Σ⁻¹ (x-y))	Correlated features; accounts for covariance so correlated columns don’t double-count	kNN in high dimensions)
Hamming	Σ 1[xᵢ ≠ yᵢ] / n	Categorical / string alignment; number of mismatched positions	Vector databases compared)

---

Decision tree: which encoder + which scaler + which distance for a given model family

The single most useful heuristic on this page is the “tree models don’t care, distance/linear models do” rule. Tree-based models (random forest, XGBoost, LightGBM, CatBoost) split on thresholds and are invariant under monotonic feature transforms — scaling changes nothing, ordinal encoding of categoricals is fine because the tree only checks x < threshold. Linear models, SVMs, kNN, k-means, and neural networks all do care about scale and about which distance metric you pick.

Pick the encoder:

• Is the column categorical?
  • Low cardinality (< 15 levels)?
    • One-hot, unless you’re training a tree with native categorical handling (CatBoost, LightGBM categorical_feature=).
  • Genuine order ("bad" < "ok" < "good")?
    • Ordinal with an explicit mapping.
  • High cardinality (e.g., ZIP, user ID, product SKU)?
    • Target encoding with CV folds and smoothing — never fit it on the full training y and use it on the same rows (leakage).
    • If you can’t afford CV scaffolding: hashing with a bucket count ≥ 2× cardinality.

Pick the scaler:

• Model is a tree (RF / XGBoost / LightGBM / CatBoost)?
  • Don’t scale. (Won’t hurt, won’t help.)
• Model is linear / SVM / NN / kNN / k-means?
  • Features roughly Gaussian, few outliers?
    • StandardScaler.
  • Heavy outliers, long tails?
    • RobustScaler.
  • Need a bounded range (NN inputs, image pixels)?
    • MinMaxScaler to [0, 1].
  • Working with text / embedding vectors and a cosine objective?
    • L2-normalize rows so cosine = dot product.

Pick the distance metric:

• Vectors are dense numeric, same scale, low-medium dim?
  • Euclidean (Minkowski p=2).
• Sparse / high-dim, or L1-robustness wanted?
  • Manhattan (Minkowski p=1).
• Direction matters more than magnitude (embeddings, text)?
  • Cosine.
• Data is presence/absence sets (tags, tokens)?
  • Jaccard.
• Features are correlated (e.g., height_cm and height_in) and you don’t want them to double-count?
  • Mahalanobis, with covariance estimated from training data.
• Data is categorical / strings of equal length?
  • Hamming.
• Very high dimensional (d ≫ 1000) and sparse?
  • Prefer cosine; see kNN in high dimensions).

---

Worked Python examples

Encoding

import numpy as np
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, TargetEncoder

# Toy dataset: a single categorical column with 3 levels + a binary target
df = pd.DataFrame({
    "city":      ["LA", "SF", "NY", "LA", "NY", "SF", "LA", "NY", "SF", "LA"],
    "purchase": [1, 0, 1, 1, 0, 1, 0, 1, 0, 1],
})
y = df["purchase"].to_numpy()

# --- One-hot (low cardinality) ---
ohe = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
X_ohe = ohe.fit_transform(df[["city"]])
# X_ohe.shape == (10, 3)   # 3 cities (LA, NY, SF); no `drop=` set, so every level gets its own column
# columns: LA, NY, SF (one-hot, handle_unknown=ignore)

# --- Ordinal (only when there is a real order) ---
ord_ = OrdinalEncoder(categories=[["bad", "ok", "good"]])
X_ord = ord_.fit_transform(pd.DataFrame({"rating": ["good","ok","bad","good"]}))
# array([[2.], [1.], [0.], [2.]])   # bad=0, ok=1, good=2

# --- Target encoding WITH leakage-safe CV ---
te = TargetEncoder(target_type="binary", cv=4, shuffle=True, random_state=0)
X_te = te.fit_transform(df[["city"]], y)
# X_te is a (10, 1) array; each row's city is replaced by the mean purchase rate
# computed from the OTHER 3 folds -- never from its own row.
print(X_te.round(3).ravel())
# actual output:  [0.657 0.543 0.524 0.645 1.    0.    1.    1.    0.    1.   ]

• OneHotEncoder(sparse_output=False) returns a dense array — fine for small data; keep sparse_output=True for high cardinality to save memory.
• handle_unknown="ignore" is critical: at inference time you’ll see new cities, and you want them to map to an all-zero row instead of crashing.
• OrdinalEncoder takes an explicit categories= list ordered from smallest to largest. Use this only when the order is real — otherwise you’re feeding the model fake signal.
• TargetEncoder(cv=...) is the leakage-safe form, but cv must be an integer fold count — passing a KFold(...) splitter instance raises sklearn.utils._param_validation.InvalidParameterError (“The ‘cv’ parameter of TargetEncoder must be an int in the range [2, inf)”), because TargetEncoder builds its own internal K-fold split rather than accepting an external splitter object. Control that internal split with shuffle= and random_state= instead. For each row, the target mean is computed on the other folds. The plain-vanilla “replace category with y.mean() over the full training set” leaks the label into the feature and inflates CV scores dramatically. Always use the cv= argument.
• target_type="binary" tells the encoder to use the class mean; "continuous" uses the regression mean.

Scaling

import numpy as np
from sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler, normalize

rng = np.random.default_rng(0)
X = np.column_stack([
    rng.normal(50, 10, size=(1000, 1)),     # roughly Gaussian
    rng.lognormal(mean=3, sigma=1, size=(1000, 1)),  # heavy right tail
])

# Standard
xs = StandardScaler().fit_transform(X)
print("Standard   mean:", xs.mean(axis=0).round(3), "std:", xs.std(axis=0).round(3))
# mean: [0. 0.]  std: [1. 1.]

# Robust
xr = RobustScaler().fit_transform(X)
# Centers on the MEDIAN, scales by the interquartile range (75th - 25th percentile).
print("Robust     median:", np.median(xr, axis=0).round(3))
# median: [0. 0.]

# Min-max
xm = MinMaxScaler().fit_transform(X)
print("MinMax     min:", xm.min(axis=0), "max:", xm.max(axis=0))
# min: [0. 0.]  max: [1. 1.]

# L2 row normalization (used for embeddings before cosine)
emb = rng.standard_normal(size=(5, 128))
emb_l2 = normalize(emb, norm="l2")
print("L2 row norms:", np.linalg.norm(emb_l2, axis=1).round(3))
# [1. 1. 1. 1. 1.]   # each row now has unit norm; cosine == dot product

• StandardScaler uses mean and std. It is not robust to outliers — a single bad row can inflate std so most rows compress into a tiny range.
• RobustScaler swaps mean → median and std → IQR. The IQR is the range of the middle 50% of the data, so it ignores extreme tails. Use this for income, prices, click counts.
• MinMaxScaler guarantees the output is in [0, 1]. That’s a feature, not a side effect — it’s the right scaler for NN inputs and image pixels, and it’s required if you want to interpret distances as “fraction of the possible range.”
• normalize(X, norm="l2") normalizes rows, not columns. Every row ends up with ‖x‖₂ = 1. After this, cosine_similarity(a, b) == a @ b. That’s why vector databases L2-normalize embeddings once at index time and then use plain dot-product for retrieval — it’s the same as cosine but cheaper.
• Fit scalers on training data only. fit_transform(X_train), then transform(X_val), transform(X_test). Fitting on the full dataset leaks val/test statistics into the train pipeline.

Binning, interactions, polynomial features

import numpy as np
from sklearn.preprocessing import KBinsDiscretizer, PolynomialFeatures
from sklearn.linear_model import Ridge

rng = np.random.default_rng(0)
n = 500
age    = rng.uniform(18, 90, n).reshape(-1, 1)
income = rng.uniform(20_000, 200_000, n).reshape(-1, 1)
y      = 0.02 * age.ravel() + 5e-6 * income.ravel()**2 + rng.normal(0, 2, n)

# Binning: convert age into 5 ordinal bins
kbd = KBinsDiscretizer(n_bins=5, encode="ordinal", strategy="quantile")
age_binned = kbd.fit_transform(age)
print("bin edges:", kbd.bin_edges_[0].round(1))
# actual: [18.  34.3 50.2 63.5 77.2 89.8]

# Polynomial features (degree 2): x1, x2, x1^2, x1*x2, x2^2
X = np.hstack([age, income])
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
print("poly feature names:", poly.get_feature_names_out(["age", "income"]))
# ['age', 'income', 'age^2', 'age income', 'income^2']

# A Ridge regression on the polynomial terms picks up the non-linear surface
# that a plain linear regression would have missed.
ridge = Ridge(alpha=1.0).fit(X_poly, y)
print("R² on training set:", round(ridge.score(X_poly, y), 3))
# actual: 1.0 -- the income^2 term (up to ~2e5) dwarfs the injected noise
# (std=2), so the polynomial fit is essentially exact on this toy data.

The polynomial-feature mapping for a 2-dimensional input x = (x1, x2) at degree d=2 is the explicit lifting

$$\phi(x) = (x_1,\; x_2,\; x_1^2,\; x_1 x_2,\; x_2^2).$$

The linear regression on the lifted space is

$$\hat{y} = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_1 x_2 + w_5 x_2^2,$$

which is a degree-2 polynomial surface in the original coordinates. The interaction term x1 * x2 is what lets the model express “the effect of income depends on age.”

Plain English	Statistical symbol	Python equivalent
Input vector	x = (x1, x2)	np.array([x1, x2])
Degree-2 lift	φ(x)	PolynomialFeatures(degree=2).fit_transform(X)
Linear predictor	ŷ = wᵀφ(x)	Ridge().fit(X_poly, y).predict(X_poly)
Number of features for d-dim degree-p	C(d+p, p)	len(PolynomialFeatures(degree=p).fit_transform(X).T)

Time-series features

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
idx = pd.date_range("2022-01-01", periods=200, freq="D")
y   = pd.Series(np.sin(np.arange(200) * 2 * np.pi / 7) + rng.normal(0, 0.2, 200), index=idx)

df = pd.DataFrame({"y": y})
# Lag features: y[t-1], y[t-7]
df["y_lag1"]  = df["y"].shift(1)
df["y_lag7"]  = df["y"].shift(7)
# Rolling window: 7-day mean and std
df["y_roll7_mean"] = df["y"].rolling(7).mean()
df["y_roll7_std"]  = df["y"].rolling(7).std()
# Expanding window: cumulative mean from series start
df["y_expand_mean"] = df["y"].expanding().mean()
# Fourier seasonal term for weekly period (T=7)
t = np.arange(len(df))
df["sin_dow"] = np.sin(2 * np.pi * t / 7)
df["cos_dow"] = np.cos(2 * np.pi * t / 7)
print(df.head(10).round(3))
#                y  y_lag1  y_lag7  y_roll7_mean  y_roll7_std  y_expand_mean  sin_dow  cos_dow
# 2022-01-01  0.025     NaN     NaN           NaN          NaN          0.025    0.000    1.000
# 2022-01-02  0.755   0.025     NaN           NaN          NaN          0.390    0.782    0.623
# ...

For a deeper dive see Feature engineering for time series: lags, rolling windows).

Distance metrics

import numpy as np
from sklearn.metrics import pairwise_distances
from scipy.spatial.distance import cdist

rng = np.random.default_rng(0)
X = rng.standard_normal(size=(4, 3))
S = rng.integers(0, 2, size=(3, 5))   # 3 binary sets of length 5

print("Euclidean:")
print(pairwise_distances(X, metric="euclidean").round(2))
# [[0.   0.49 2.09 1.59]
#  ...

print("Manhattan:")
print(pairwise_distances(X, metric="manhattan").round(2))

print("Cosine (1 - cos):")
print(pairwise_distances(X, metric="cosine").round(2))

print("Jaccard on binary sets (1 - J):")
print(pairwise_distances(S.astype(bool).view(np.uint8),
                         metric="jaccard").round(2))

print("Mahalanobis (cov from X):")
Sigma = np.cov(X.T)
VI = np.linalg.pinv(Sigma)
print(cdist(X, X, metric="mahalanobis", VI=VI).round(2))

print("Hamming on 5-char strings:")
a = np.array(["A", "A", "C", "G", "T"])
b = np.array(["A", "T", "C", "G", "G"])
print((a != b).sum() / len(a))   # 0.4  -> 2 of 5 positions differ

The Minkowski family generalizes the integer-distance cases:

$$d_p(x, y) = \left( \sum_i |x_i - y_i|^p \right)^{1/p}.$$

• p = 1 → Manhattan: Σ|xᵢ - yᵢ|.
• p = 2 → Euclidean: √Σ(xᵢ - yᵢ)².
• p → ∞ → Chebyshev: maxᵢ |xᵢ - yᵢ|.

Cosine distance is not a Minkowski metric — it’s a shape distance:

$$d_{\cos}(x, y) = 1 - \frac{x \cdot y}{\|x\|_2 \, \|y\|_2}.$$

After L2-normalizing rows so ‖x‖₂ = ‖y‖₂ = 1, the denominator collapses and d_cos = 1 - x·y. That’s why vector databases L2-normalize at index time.

Mahalanobis “whitens” the difference using the inverse covariance:

$$d_{\text{Mah}}(x, y) = \sqrt{(x - y)^\top \Sigma^{-1} (x - y)}.$$

If Σ = I this reduces to Euclidean. If the features are correlated, Mahalanobis accounts for that correlation so the same information doesn’t get double-counted by x₁ and x₂ being near-duplicates.

Plain English	Statistical symbol	Python equivalent
L2 norm	‖x‖₂ = √Σxᵢ²	np.linalg.norm(x)
Dot product	x·y	x @ y
Cosine similarity	x·y / (‖x‖ ‖y‖)	1 - pairwise_distances(X, metric="cosine")
Covariance matrix	Σ = Cov(X)	np.cov(X.T)
Mahalanobis distance	√((x-y)ᵀ Σ⁻¹ (x-y))	cdist(X, X, metric="mahalanobis", VI=pinv(Σ))

---

Edge cases + common mistakes

Encoders

1. Target encoding without CV folds. The classic mistake: replace each category with the mean of y over the entire training set, then train and cross-validate on the same rows. The leakage is severe — a category with one row gets a feature equal to that row’s label, and the model looks perfect in CV. Fix: use sklearn.preprocessing.TargetEncoder with an integer cv= (a fold count, not a KFold splitter object), or implement out-of-fold encoding by hand. Always smooth toward the global mean when a category has few samples.

2. One-hotting high-cardinality columns. A user_id column with 100k unique values produces a 100k-column sparse matrix that no linear model handles gracefully and that swells tree training memory. Use target encoding (with CV) or hashing.

3. Hashing collisions destroying signal. With n_buckets too small, two unrelated categories collide into the same bucket. Rule of thumb: n_buckets ≥ 2 × cardinality, and don’t expect interpretability — you can’t recover a category from its hash.

Scalers

4. Fitting scalers on the full dataset, then splitting train/val/test. The val/test statistics leak into the training mean and std. The right order: split first, then fit the scaler on X_train only and transform the rest. Pipeline + ColumnTransformer makes this automatic.

5. Scaling tree features. It does nothing for tree performance and costs you interpretability of split thresholds. Skip it. (It doesn’t hurt, but it doesn’t help.)

Derived features

6. Polynomial features without regularization. PolynomialFeatures(degree=3) on 20 columns produces C(20+3, 3) = 1771 columns. A plain LinearRegression will overfit catastrophically. Use Ridge or Lasso, and cross-validate alpha.

7. Interactions for trees. Trees already find x1 & x2 interactions automatically via successive splits. Adding x1 * x2 columns to a tree model wastes space; add them only for linear models.

8. Binning then dropping the raw feature. Binning throws away information; in a tabular model that can use both, keep the raw numeric and the binned version, or use target-driven bin edges.

Time-series features

9. Lag features without respecting time order. df.shift(1) is correct only if the index is sorted by time. If you shuffled rows for any reason, the lag is meaningless. Re-sort by timestamp first, and never use a future value (shift(-1)) as a feature for predicting the present.

10. Rolling windows with leakage in CV. A 30-day rolling mean computed across the full series will have used future information at training time. Recompute rolling features inside each CV fold’s training window, or use only trailing statistics.

Distance metrics

11. Cosine on non-negative count vectors without normalization. If raw counts vary in magnitude, cosine will be dominated by the rows with the largest norms. L2-normalize first.

12. Mahalanobis with a singular covariance. If d > n or features are linearly dependent, Σ is singular. Use np.linalg.pinv (pseudo-inverse), or regularize with Σ + λI.

13. Jaccard on non-binary data. Jaccard is a set metric. Applying it to continuous vectors gives nonsense. Convert to presence/absence first, or use cosine.

14. Distance metrics in high dimensions (the curse of dimensionality). In d ≫ 100 dimensions, Euclidean distances between random points converge — every pair looks about equally far. Use cosine or reduce dimensionality first. See kNN in high dimensions).

---

Cross-references

This reference draws on and is used by the following corpus articles:

• Feature scaling — why your model might be ignoring half your data) — the scaler-specific article; pairs with the scaling section here.
• Feature engineering for tabular data: the technique) — the encoders and derived-features article.
• XGBoost vs LightGBM vs CatBoost: how to actually choose) — native categorical handling in tree libraries; the “don’t scale trees” rule in context.
• Feature engineering for time series: lags, rolling windows) — full time-series addendum; this reference only summarizes.
• Why your kNN model fails in high dimensions) — the distance-metrics article in context of the curse of dimensionality.
• Vector databases compared: when you actually need one) — cosine vs L2 in vector retrieval.
• Building your first RAG pipeline: chunking, embedding, retrieval) — L2 normalization of embeddings, cosine retrieval.

---

Further reading

Foundational books

• Kuhn, M. & Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models. Chapman & Hall/CRC. The canonical tabular feature-engineering text; chapters 5–8 cover encoding and rescaling in depth.
• Zheng, A. & Casari, A. (2018). Feature Engineering for Machine Learning: Principles and Techniques for Data Scientists. O’Reilly. Practical, code-first treatment of encoding, binning, and feature selection.

Library docs

• scikit-learn preprocessing user guide — encoders, scalers, polynomial features.
• scikit-learn target encoding — the leakage-safe TargetEncoder introduced in sklearn 1.3.
• scipy.spatial.distance — full distance-metric reference.

Kaggle competition

• House Prices: Advanced Regression Techniques — the canonical “encode the categoricals, scale the numerics, engineer the interactions” playground; every technique on this page shows up in the top public notebooks.