Calibration Curves: When Your Model's Probabilities Are Lying to You

1. The Probability Trap: Why Your Model’s Confidence Is Fake

Imagine you are building a model to predict whether a loan applicant will default. Your model looks at a new application and says there is a 90% probability of successful repayment. You feel great. You approve the loan.

But here is the catch: what if, among all the people your model gave a “90% probability” to, only 60% actually paid back? Then your model isn’t just wrong. It is overconfident — claiming certainty it can’t back up.

Most of us focus on Accuracy (how often is the model right?) or AUC (how well does it rank people?). We often forget Calibration. Calibration is about honesty. If a weather forecaster says there is a 70% chance of rain every day for a month, it better rain on exactly 21 of those 30 days. If it only rains on 10, the forecaster is miscalibrated.

Let’s check whether a standard model is honest with us.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

# Create a fake dataset
X, y = make_classification(n_samples=10000, n_features=20, n_informative=2, n_redundant=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a Random Forest
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# Get probabilities for the positive class
probs = model.predict_proba(X_test)[:, 1]

# Look at cases where the model is very confident ( > 90%)
high_conf_mask = probs > 0.9
actual_rate = y_test[high_conf_mask].mean()
mean_predicted = probs[high_conf_mask].mean()
print(f"When the model says >90% probability, the actual success rate is: {actual_rate:.2%}")
print(f"Average predicted probability in that group: {mean_predicted:.2%}")
# Output: When the model says >90% probability, the actual success rate is: 96.97%
# Output: Average predicted probability in that group: 97.73%

Run it yourself and the story isn’t the dramatic one you might expect. The actual success rate among these confident predictions is 96.97%, barely three-quarters of a point below the 97.73% average probability the model assigned — well within noise for a group this size (1,088 predictions). This particular Random Forest came out close to honest on this data. That’s not a guarantee: Random Forests don’t have to be well calibrated (Section 4 explains why), and it’s easy to build one that skews a lot more confident than this. The rest of this article shows you how to check for yourself, rather than assume either way.

2. Calibration in One Sentence: What We’re Actually Measuring

Calibration is a reality check for your model. To measure it, we group predictions into buckets.

Everything the model predicted between 0% and 10% goes in Bucket A. Then 10% to 20% in Bucket B, and so on. For each bucket, we calculate the actual percentage of positive outcomes.

If the model is perfectly calibrated, the average predicted probability in the bucket should equal the actual fraction of positives. So let’s work through it manually.

# Create bins
bins = np.linspace(0, 1, 11)
df = pd.DataFrame({'pred': probs, 'actual': y_test})
df['bin'] = pd.cut(df['pred'], bins)

# Calculate the average prediction vs actual outcome per bin
calibration_data = df.groupby('bin', observed=True).agg({'pred': 'mean', 'actual': 'mean'})
print(calibration_data)

The output has a pred column (what the model thought) and an actual column (what really happened). If pred is 0.75 and actual is 0.60, the model is over-promising.

3. The Calibration Curve: Reading the Graph

Rather than scan raw tables, we plot a Calibration Curve (also called a Reliability Diagram).

• The Diagonal Line: The “Perfectly Calibrated” line. A model sitting here is 100% honest.
• Below the Diagonal: The model is overconfident. It says 80%, but the truth is 60%.
• Above the Diagonal: The model is underconfident. It says 20%, but the truth is 40%.

import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve

prob_true, prob_pred = calibration_curve(y_test, probs, n_bins=10)

plt.plot(prob_pred, prob_true, marker='o', label='Random Forest')
plt.plot([0, 1], [0, 1], linestyle='--', label='Perfectly Calibrated')
plt.xlabel('Mean Predicted Probability')
plt.ylabel('Fraction of Positives')
plt.legend()
plt.show()

The Random Forest curve usually takes an “S” shape. Too cautious near the middle, too extreme at the edges.

4. Why Do Models Lie? Common Causes of Miscalibration

Why doesn’t every model just tell the truth?

1. Algorithm Design: Logistic Regression maximizes the likelihood of the data, which naturally produces well-calibrated probabilities. Random Forests take a different route — they average votes. If 90 out of 100 trees pick “Class A,” you get 0.9. But those trees are correlated, so that “90%” isn’t a true statistical probability.
2. Overfitting: When a model memorizes noise in the training data, it grows overly “certain” about its decisions, pushing probabilities toward 0 or 1.
3. Imbalanced Data: If only 1% of your data is the positive class, models struggle to find the right “baseline” for their confidence.

5. Fixing It: Calibration Methods That Actually Work

The good news: you don’t have to retrain your whole model. A post-processing step does the job.

Two main approaches fix a lying model:

1. Platt Scaling: Train a tiny Logistic Regression on top of your model’s outputs. It learns to “stretch” or “squish” the probabilities back toward reality.
2. Isotonic Regression: More flexible — it fits a non-decreasing line to the data. Works well with plenty of examples, but it can overfit on small samples.

from sklearn.calibration import CalibratedClassifierCV
from sklearn.frozen import FrozenEstimator

# Platt Scaling (method='sigmoid') — wrap the already-fitted model in
# FrozenEstimator so CalibratedClassifierCV only fits the calibration step
calibrated_platt = CalibratedClassifierCV(FrozenEstimator(model), method='sigmoid')
calibrated_platt.fit(X_test, y_test) # Usually you use a separate validation set here!

# Isotonic Regression
calibrated_iso = CalibratedClassifierCV(FrozenEstimator(model), method='isotonic')
calibrated_iso.fit(X_test, y_test)

# Let's see the improvement
probs_platt = calibrated_platt.predict_proba(X_test)[:, 1]
prob_true_platt, prob_pred_platt = calibration_curve(y_test, probs_platt, n_bins=10)

plt.plot(prob_pred_platt, prob_true_platt, marker='s', label='Platt Scaled')
plt.plot([0, 1], [0, 1], linestyle='--')
plt.legend()
plt.show()

6. When to Calibrate (and When Not To)

Do you always need to do this? No.

Calibration matters if:

• You’re making decisions based on a threshold (e.g., “Only call customers with a >15% churn risk”).
• You’re calculating expected value (e.g., “Probability of Sale * Price”).
• You’re combining multiple models.

Calibration doesn’t matter if:

• You only care about ranking. If you just need to know who is more likely to buy than someone else, the exact probability won’t change that order.

7. Putting It Together: A Real Workflow

In a real project, you’ll want three data splits: Train for building the model, Calibration for fixing the probabilities, and Test to see how you actually did.

# 1. Split data
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2)
X_train, X_calib, y_train, y_calib = train_test_split(X_temp, y_temp, test_size=0.2)

# 2. Train model
rf = RandomForestClassifier().fit(X_train, y_train)

# 3. Calibrate on the calibration set
calibrator = CalibratedClassifierCV(FrozenEstimator(rf), method='isotonic')
calibrator.fit(X_calib, y_calib)

# 4. Evaluate on the final test set
final_probs = calibrator.predict_proba(X_test)[:, 1]

8. The Catch: Calibration Isn’t Magic

Here’s the hard truth: calibration does not make your model smarter.

A low-AUC model that can’t tell good loans from bad ones stays that way after calibration. What you get is an “honestly bad” model. Instead of confidently giving the wrong answer, it tells you, “I’m not sure, it’s about a 50/50 chance.”

Worth noting: calibrating on a tiny dataset can mean fitting the noise. Always check your calibration on fresh data.

9. Next Steps: Calibration in Production

Real-world data shifts. That’s Concept Drift. A model perfectly calibrated in January can turn overconfident by June as customer behavior changes.

To handle this:

1. Monitor ECE: Track the Expected Calibration Error — the average gap between the curve and the diagonal.
2. Recalibrate often: You don’t always need to retrain the heavy Random Forest. Sometimes updating just the Platt Scaling (the Logistic Regression on top) is enough to fix the drift.

Summary:

• Accuracy tells you whether the model is right; calibration tells you if it knows when it’s right.
• Use calibration_curve to visualize how honest your model is.
• Use CalibratedClassifierCV to fix overconfident or underconfident models.
• Calibration matters for risk management and financial decisions.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What does it mean for a model to be “overconfident” versus “underconfident” on a calibration curve, relative to the diagonal line?

Understand In your own words, explain why Random Forests tend to be less well-calibrated than Logistic Regression, using the article’s explanation of how each algorithm produces its probability.

Apply Using the article’s bucket logic (average predicted probability vs. actual positive rate), if a bucket of predictions has a mean predicted probability of 0.75 but the actual positive rate in that bucket is 0.55, is the model overconfident or underconfident for that bucket, and by how many percentage points?

Analyze The article says calibration “does not make your model smarter” — a low-AUC model just becomes “honestly bad” instead of confidently wrong. Walk through why fixing calibration (a post-processing step on probabilities) can’t improve a model’s ability to rank good loans above bad ones, even though it changes the actual probability numbers.

Evaluate The article’s Section 7 recommends a three-way split: Train, Calibration, and Test. Critique the code in Section 5, which instead calibrates and evaluates CalibratedClassifierCV on the same X_test/y_test. What specifically goes wrong with a calibration curve that’s plotted using the same data the calibrator was fit on?

Create Design a monitoring plan for calibration drift in production for a credit-risk model: what would you track (referencing the article’s “Expected Calibration Error” idea), how often would you check it, and what would trigger a recalibration versus a full model retrain?

---