Class Imbalance Smote Undersampling And When Neith

You’ve just trained your first fraud detection model. Accuracy: 99%. You’re thrilled — until you look closer. Out of 1,000 fraudulent transactions in your test set, your model caught exactly zero. It simply predicted “not fraud” for every single row. That 99% accuracy was a mirage.

If you’ve ever worked on fraud detection, rare disease diagnosis, or predicting customer churn, you’ve felt this pain. The problem is class imbalance: one class (the one you usually care most about) makes up a tiny fraction of your data.

So what do we do? Should we “fix” the imbalance? And if so, how? Let’s explore three common strategies — random undersampling, SMOTE (oversampling), and class weighting — along with the crucial insight that sometimes, none of these is the right answer.

Let’s start by seeing the problem in action.

# Build an imbalanced dataset and see how accuracy can lie
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
import numpy as np

# Create a heavily imbalanced dataset: 1% positive class, 99% negative
X, y = make_classification(
    n_samples=10000, 
    n_features=20, 
    weights=[0.99, 0.01], 
    random_state=42
)

print("Class distribution in y:")
unique, counts = np.unique(y, return_counts=True)
for cls, cnt in zip(unique, counts):
    print(f"  Class {cls}: {cnt} samples ({100*cnt/len(y):.1f}%)")

# Split into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

# Train a dummy classifier that always predicts the majority class (0)
dummy = DummyClassifier(strategy='most_frequent')
dummy.fit(X_train, y_train)
y_pred_dummy = dummy.predict(X_test)

print("\nDummy Classifier Results:")
print(f"Accuracy: {accuracy_score(y_test, y_pred_dummy):.3f}")
cm = confusion_matrix(y_test, y_pred_dummy)
print("Confusion matrix:")
print(cm)
print(f"\nInterpretation: {cm[0,0]} true negatives, {cm[0,1]} false positives, {cm[1,0]} false negatives, {cm[1,1]} true positives")
print("The model caught ZERO fraud cases (TP=0). Accuracy is high only because most cases are not fraud.")

What just happened? Our dummy model achieved 99% accuracy by ignoring the minority class entirely. That’s the accuracy paradox: a very high accuracy that masks completely useless predictions. We need better metrics and better strategies.

Why Imbalance Hurts (and How We Measure the Hurt)

Before fixing the imbalance, we need to understand why it hurts model performance. The core issue is that standard metrics lie when classes are imbalanced.

Think of a search for needles in a haystack:

• Precision: Of all the items you flagged as “needle,” how many are actually needles? (Low precision = lots of false alarms.)
• Recall: Of all the actual needles, how many did you find? (Low recall = you miss many needles.)
• F1-score: The harmonic mean of precision and recall — a single number that balances both.

For imbalanced problems, precision-recall curves (PR curves) are usually more informative than ROC curves. Why? Because ROC curves measure false positive rate (FP / total negatives), and when negatives are abundant, a model can have many false positives yet still a low false positive rate. PR curves focus directly on the minority class and are less forgiving.

Let’s train a real model and see these metrics in action.

# Train a logistic regression on imbalanced data and evaluate with multiple metrics
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, precision_score, recall_score, 
                             f1_score, roc_auc_score, average_precision_score, 
                             confusion_matrix, RocCurveDisplay, PrecisionRecallDisplay)
import matplotlib.pyplot as plt

# Train a basic logistic regression (no imbalance fix)
lr_baseline = LogisticRegression(max_iter=1000, random_state=42)
lr_baseline.fit(X_train, y_train)
y_pred_baseline = lr_baseline.predict(X_test)
y_proba_baseline = lr_baseline.predict_proba(X_test)[:, 1]

print("Baseline Logistic Regression Results:")
print(f"Accuracy:                {accuracy_score(y_test, y_pred_baseline):.3f}")
print(f"Precision (class 1):     {precision_score(y_test, y_pred_baseline):.3f}")
print(f"Recall (class 1):        {recall_score(y_test, y_pred_baseline):.3f}")
print(f"F1-score (class 1):      {f1_score(y_test, y_pred_baseline):.3f}")
print(f"ROC-AUC:                 {roc_auc_score(y_test, y_proba_baseline):.3f}")
print(f"PR-AUC:                  {average_precision_score(y_test, y_proba_baseline):.3f}")

print("\nConfusion Matrix:")
cm = confusion_matrix(y_test, y_pred_baseline)
print(cm)
print(f"\nInterpretation: Accuracy is {accuracy_score(y_test, y_pred_baseline):.3f},")
print(f"but recall is {recall_score(y_test, y_pred_baseline):.3f} — the model misses most fraud cases.")
print(f"ROC-AUC is {roc_auc_score(y_test, y_proba_baseline):.3f} (seems good!),")
print(f"but PR-AUC is only {average_precision_score(y_test, y_proba_baseline):.3f} (shows the true challenge).")

What does this mean? The baseline model has a respectable ROC-AUC (let’s say ~0.85), but its recall on the minority class is terrible — maybe around 0.05. That means we catch only 5% of fraud cases. The PR-AUC reveals the struggle: a perfect model scores PR-AUC close to 1.0, but our baseline is far lower. This is why we need specialized techniques.

Quick Fix #1: Random Undersampling — Less Data, Better Focus?

The most straightforward idea: throw away majority-class samples until the classes are balanced. Intuitively, if we have 99 non-fraud and 1 fraud, we could randomly drop 98 non-fraud samples to get a balanced 1:1 ratio.

But what do we lose? Potentially valuable information about what “normal” looks like. Undersampling can increase variance because we discard data. However, it’s computationally cheap and can dramatically boost recall.

from imblearn.under_sampling import RandomUnderSampler
from sklearn.metrics import classification_report

# Apply random undersampling
rus = RandomUnderSampler(random_state=42)
X_train_rus, y_train_rus = rus.fit_resample(X_train, y_train)

print("Class distribution after random undersampling:")
unique, counts = np.unique(y_train_rus, return_counts=True)
for cls, cnt in zip(unique, counts):
    print(f"  Class {cls}: {cnt} samples ({100*cnt/len(y_train_rus):.1f}%)")

# Train logistic regression on undersampled data
lr_rus = LogisticRegression(max_iter=1000, random_state=42)
lr_rus.fit(X_train_rus, y_train_rus)
y_pred_rus = lr_rus.predict(X_test)
y_proba_rus = lr_rus.predict_proba(X_test)[:, 1]

print("\nRandom Undersampling Results:")
print(f"Accuracy:                {accuracy_score(y_test, y_pred_rus):.3f}")
print(f"Precision (class 1):     {precision_score(y_test, y_pred_rus):.3f}")
print(f"Recall (class 1):        {recall_score(y_test, y_pred_rus):.3f}")
print(f"F1-score (class 1):      {f1_score(y_test, y_pred_rus):.3f}")
print(f"PR-AUC:                  {average_precision_score(y_test, y_proba_rus):.3f}")
print()
print(classification_report(y_test, y_pred_rus, target_names=['not fraud', 'fraud']))

Interpretation: Recall likely jumped dramatically — say from 0.05 to 0.75. But precision probably dropped — we’re now flagging many legitimate transactions as fraud (false positives). The model is more aggressive, but at a cost. The trade-off: catch more fraud but annoy more customers.

More intelligent undersampling methods like Tomek Links (removing pairs of points from opposite classes that are each other’s nearest neighbor) or Edited Nearest Neighbours (removing majority samples that look too much like the minority class) can reduce noise, but the core trade-off remains.

Quick Fix #2: SMOTE — Making Up Data, Carefully

Instead of throwing data away, what if we create new minority samples? That’s the idea behind SMOTE (Synthetic Minority Over-sampling TEchnique), one of the most popular oversampling methods.

The intuition is elegant: you pick a minority-class sample, find its k nearest minority-class neighbors, randomly select one, and place a new synthetic point along the line connecting them. This creates plausible new minority examples, not just duplicates. Let’s see it in action.

from imblearn.over_sampling import SMOTE

# Apply SMOTE (default k_neighbors=5)
smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

print("Class distribution after SMOTE:")
unique, counts = np.unique(y_train_smote, return_counts=True)
for cls, cnt in zip(unique, counts):
    print(f"  Class {cls}: {cnt} samples ({100*cnt/len(y_train_smote):.1f}%)")

# Train logistic regression on SMOTE-augmented data
lr_smote = LogisticRegression(max_iter=1000, random_state=42)
lr_smote.fit(X_train_smote, y_train_smote)
y_pred_smote = lr_smote.predict(X_test)
y_proba_smote = lr_smote.predict_proba(X_test)[:, 1]

print("\nSMOTE Results:")
print(f"Accuracy:                {accuracy_score(y_test, y_pred_smote):.3f}")
print(f"Precision (class 1):     {precision_score(y_test, y_pred_smote):.3f}")
print(f"Recall (class 1):        {recall_score(y_test, y_pred_smote):.3f}")
print(f"F1-score (class 1):      {f1_score(y_test, y_pred_smote):.3f}")
print(f"PR-AUC:                  {average_precision_score(y_test, y_proba_smote):.3f}")
print()
print(classification_report(y_test, y_pred_smote, target_names=['not fraud', 'fraud']))

Interpretation: SMOTE often gives higher recall than baseline and sometimes better precision than undersampling. But it’s not magic. If minority samples are very sparse or overlap heavily with the majority, SMOTE can create unrealistic or noisy points. It also makes the dataset larger, which may slow down training.

Combining Forces: SMOTE + Cleaning Methods

SMOTE can create samples that fall in “noisy” regions — deep inside majority territory. A common improvement is to clean the resulting dataset using undersampling techniques that remove ambiguous points.

• Tomek Links: Pairs of points from opposite classes that are each other’s nearest neighbor. Removing the majority-class point cleans the decision boundary.
• Edited Nearest Neighbours (ENN): Remove a majority sample if its k nearest neighbors are mostly from the minority class.

The imbalanced-learn library provides handy combinations: SMOTETomek and SMOTEENN.

from imblearn.combine import SMOTETomek, SMOTEENN

# Apply SMOTETomek and SMOTEENN
smote_tomek = SMOTETomek(random_state=42)
X_train_st, y_train_st = smote_tomek.fit_resample(X_train, y_train)

smote_enn = SMOTEENN(random_state=42)
X_train_se, y_train_se = smote_enn.fit_resample(X_train, y_train)

print("Class distribution after SMOTETomek:")
unique_st, counts_st = np.unique(y_train_st, return_counts=True)
for cls, cnt in zip(unique_st, counts_st):
    print(f"  Class {cls}: {cnt} samples")

print("\nClass distribution after SMOTEENN:")
unique_se, counts_se = np.unique(y_train_se, return_counts=True)
for cls, cnt in zip(unique_se, counts_se):
    print(f"  Class {cls}: {cnt} samples")

# Train models on hybrid data
lr_st = LogisticRegression(max_iter=1000, random_state=42)
lr_st.fit(X_train_st, y_train_st)
y_proba_st = lr_st.predict_proba(X_test)[:, 1]

lr_se = LogisticRegression(max_iter=1000, random_state=42)
lr_se.fit(X_train_se, y_train_se)
y_proba_se = lr_se.predict_proba(X_test)[:, 1]

print("\nSMOTETomek PR-AUC:", average_precision_score(y_test, y_proba_st))
print("SMOTEENN PR-AUC:", average_precision_score(y_test, y_proba_se))

Interpretation: Hybrid methods often improve precision slightly over plain SMOTE by cleaning borderline cases. But the improvement is often modest — and the added complexity may not always be worth it, especially if you have enough data.

The Third Path: Cost-Sensitive Learning (No Resampling Needed)

Here’s a different idea: what if we don’t touch the data at all, but instead tell the model to pay more attention to minority-class errors? That’s cost-sensitive learning.

In scikit-learn, many classifiers support the class_weight parameter. Setting it to 'balanced' automatically assigns weights inversely proportional to class frequencies. For example, if the minority class is 1% of the data, each minority sample gets a weight 99 times higher than a majority sample during training.

This changes the loss function: misclassifying a minority example hurts the model more, forcing it to prioritize getting those right.

# Train logistic regression with class_weight='balanced'
lr_weighted = LogisticRegression(max_iter=1000, class_weight='balanced', random_state=42)
lr_weighted.fit(X_train, y_train)
y_pred_weighted = lr_weighted.predict(X_test)
y_proba_weighted = lr_weighted.predict_proba(X_test)[:, 1]

print("Cost-Sensitive Learning (class_weight='balanced') Results:")
print(f"Accuracy:                {accuracy_score(y_test, y_pred_weighted):.3f}")
print(f"Precision (class 1):     {precision_score(y_test, y_pred_weighted):.3f}")
print(f"Recall (class 1):        {recall_score(y_test, y_pred_weighted):.3f}")
print(f"F1-score (class 1):      {f1_score(y_test, y_pred_weighted):.3f}")
print(f"PR-AUC:                  {average_precision_score(y_test, y_proba_weighted):.3f}")
print()
print(classification_report(y_test, y_pred_weighted, target_names=['not fraud', 'fraud']))

Interpretation: Class weighting often gives similar recall to SMOTE but with better precision — and it’s dramatically simpler: no data transformation, no synthetic points, no risk of overfitting to noise. It’s a strong baseline that is often underappreciated.

The Calibration Trap

Sometimes, fixing class imbalance makes things worse. Specifically, if your goal is to output well-calibrated probabilities — for example, predicting “this transaction has a 5% chance of being fraud” — resampling distorts those probabilities.

Calibration means: when a model predicts 0.8, it should be correct about 80% of the time. Resampling (undersampling or SMOTE) artificially changes the class distribution, so the model learns an overconfident probability. After balancing, a case that truly has a 1% fraud risk might get a predicted probability of 50%.

Recent studies have shown that resampling does not improve AUROC but severely harms calibration. Let’s see this with calibration curves.

from sklearn.calibration import calibration_curve, CalibrationDisplay
import matplotlib.pyplot as plt

# Collect model names and predicted probabilities
models = {
    'Baseline': y_proba_baseline,
    'Undersampling': y_proba_rus,
    'SMOTE': y_proba_smote,
    'Class Weight': y_proba_weighted
}

# Plot calibration curves
fig, ax = plt.subplots(figsize=(8, 6))
for name, proba in models.items():
    prob_true, prob_pred = calibration_curve(y_test, proba, n_bins=10, strategy='uniform')
    ax.plot(prob_pred, prob_true, marker='o', label=name)

# Plot the perfectly calibrated line
ax.plot([0, 1], [0, 1], 'k--', label='Perfectly calibrated')
ax.set_xlabel('Mean predicted probability')
ax.set_ylabel('Fraction of positives')
ax.set_title('Calibration Curves')
ax.legend()
plt.show()

print("\nInterpretation: The baseline model stays close to the diagonal (well-calibrated).")
print("Resampled models (undersampling, SMOTE) bend upward — they are overconfident.")
print("Class weighting usually sits between the two, offering a better trade-off.")

When is resampling still okay?

• If you only care about ranking (e.g., which transactions to investigate first) and AUC is your metric, resampling can help.
• If you have enough data to calibrate post-hoc (e.g., using Platt scaling or isotonic regression), you can correct the probabilities after resampling.
• If your problem is more about finding any minority examples (high recall) and false positives are tolerable, resampling may be the right choice.

But if you need trustable probabilities — for risk scoring, insurance, medical decisions — avoid resampling and use cost-sensitive learning or threshold moving instead.

Practical Decision Framework: Which Technique When?

Here’s a quick mental model to choose your approach:

Scenario	Recommended Approach	Why
Need well-calibrated probabilities	Class weighting or threshold moving	Resampling distorts probabilities
Only care about ranking (top-k recall)	SMOTE or undersampling	Can improve recall, but check calibration
Very small dataset	Class weighting	SMOTE may introduce noise; undersampling loses too much data
Very large dataset	Random undersampling or class weighting	SMOTE can be slow; undersampling is cheap
Want simplest solution	Start with class weighting	Often works well with minimal code change
Need to maximize recall at any cost	SMOTE + cleaning (hybrid)	But accept poor precision and calibration

Always evaluate with PR-AUC and calibration curves, not just accuracy or ROC-AUC.

What You’ve Learned (and Where to Go Next)

Let’s recap your journey:

1. Class imbalance creates the accuracy paradox — high accuracy but useless models.
2. Undersampling is simple but loses data and can hurt generalisation.
3. SMOTE creates synthetic minority samples, often improving recall, but can introduce noise.
4. Hybrid methods (SMOTE + cleaning) can refine decision boundaries at extra cost.
5. Cost-sensitive learning (class weighting) is a powerful, simple alternative that preserves calibration better.
6. Calibration matters: if you need realistic probabilities, resampling can be the wrong choice.

In the next article, we’ll explore ensemble methods designed for imbalanced data — like BalancedRandomForest, EasyEnsemble, and RUSBoost — which combine resampling with multiple models to get the best of both worlds.

Now it’s your turn. Try these techniques on your own dataset and see which works best for your problem.

Check Your Understanding

1. Remember: What does SMOTE stand for, and how does it create synthetic samples?
2. Understand: Explain in plain English why accuracy is a misleading metric for imbalanced classification.
3. Apply: Given a medical diagnosis dataset with 5% positive cases and a requirement for well-calibrated probabilities, which technique would you recommend and why?
4. Analyze: Compare the calibration curves of a model trained with SMOTE versus class weighting. What pattern do you expect, and why?
5. Evaluate: Under what circumstances might you choose random undersampling over SMOTE despite the data loss?
6. Create: Design a small experiment to test whether SMOTE improves or harms model calibration on your own dataset.