Why Did the Model Say No? Explaining Black-Box Decisions to Your Boss Without the Math

The ‘Black Box’ problem: Why ‘Trust me’ isn’t enough

Over the last three articles, the data science team built up an explainability toolkit for loan-denial predictions. They used SHAP values to decompose a prediction into individual feature contributions. They compared SHAP against LIME for speed versus consistency. And they used PDP and ICE curves to see how income and other features shape approval odds across all applicants — not just one. Now it all comes together. Mr. Owens needs to take that technical work and turn it into the actual explanation given back to an applicant — one that satisfies a regulator too.

Imagine you are a loan officer at a local bank. Maria, a long-time customer, walks in, applies for a small business loan, and five minutes later your screen flashes a bright red “REJECTED”.

Maria looks you in the eye and asks, “Why? My credit is decent, and my revenue is up.” Mr. Owens is standing behind you, waiting for the same answer. So is the regulator who asked for documentation on every automated denial this quarter.

If your only answer is “I don’t know, the computer just said so,” you’ve lost more than a customer. You’ve lost their trust. And Mr. Owens has nothing to show the regulator. In the business world, “trust me” is a dangerous phrase. A model that makes decisions in the dark is a liability — in banking, healthcare, retail, or anywhere else.

A “black box” isn’t magic. It is usually just a model with so many moving parts — thousands of decision branches or millions of weights — that a human brain can’t track them all at once. If you can’t explain it, your stakeholders won’t use it, no matter how accurate it is.

So let’s build a “messy” model that is impossible to explain just by looking at it. We’ll use a Random Forest, which is essentially a forest of hundreds of tiny decision trees all shouting their opinions at once.

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestClassifier

# Let's create a fake dataset of loan applicants
# Features: Credit Score, Annual Income (in $k), and Debt-to-Income Ratio
data = {
    'credit_score': [780, 560, 650, 720, 590, 700, 620, 810, 540, 760, 610, 690],
    'income_k':     [40,   95, 110,  55,  30, 120,  85,  50, 100,  65,  45,  75],
    'debt_ratio':   [0.10, 0.20, 0.15, 0.45, 0.55, 0.50, 0.35, 0.45, 0.15, 0.25, 0.60, 0.10],
    'approved':     [1, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1]
}
df = pd.DataFrame(data)

# Train a Random Forest with 100 trees
# Even with 3 features, 100 trees make it a 'black box'
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(df.drop('approved', axis=1), df['approved'])

# Predict for a new customer: decent income, weaker credit, moderate debt
new_customer = np.array([[600, 115, 0.30]])
prediction = model.predict(new_customer)
probability = model.predict_proba(new_customer)[0][1]

print(f"Prediction: {'Approved' if prediction[0] == 1 else 'Rejected'}")
print(f"Probability of Approval: {probability:.2f}")

• data = {...} — A Python dictionary where each key is a column name and each value is a list of that column’s values. This is the simplest way to define tabular data by hand. The three features are deliberately spread out so no two of them move in lockstep — high credit scores show up with both low and high incomes, and low debt ratios show up at both ends of the credit-score range. That matters later: if credit score, income, and debt ratio all rose and fell together, no explanation method could tell you which one the model actually used.
• pd.DataFrame(data) — Converts the dictionary into a pandas DataFrame — a table where each key becomes a column and each list becomes that column’s rows.
• RandomForestClassifier(n_estimators=100, random_state=42) — Creates (but does not yet train) a random forest with 100 decision trees. random_state=42 seeds the internal bootstrapping so the forest is identical on every run.
• model.fit(df.drop('approved', axis=1), df['approved']) — Trains the forest on all features except approved (the features) against approved (the label). axis=1 means “drop a column.”
• new_customer = np.array([[600, 115, 0.30]]) — Creates a 2-D NumPy array with one row and three columns: credit score 600, income $115k, debt ratio 0.30 — a candidate with a below-average credit score but above-average income, so credit score and income pull in opposite directions on the prediction. In the narrative, these are Maria’s actual values.
• model.predict(new_customer) — Returns an array of predicted class labels (0 = Rejected, 1 = Approved) for each input row.
• model.predict_proba(new_customer)[0][1] — predict_proba returns an array of shape (n_rows, n_classes). [0] selects the first (and only) row; [1] selects the probability of class 1 (Approved).

The output tells us Maria was rejected, with a 35% chance of approval — below the 50/50 line, but not a landslide. Looking at the 100 trees inside that model variable won’t tell you whether it was the debt ratio or her credit score that mattered more. This is the hardest part of modern machine learning: the trade-off between accuracy and understanding.

SHAP: Giving every feature a ‘receipt’

To solve this, we use SHAP (SHapley Additive exPlanations). The easiest way to think about it is a group project.

Imagine four friends building a house. At the end, the house is 20 feet tall. How much did each person contribute? You can’t just look at the final height. You have to check how tall the house would have been if Friend A didn’t show up, or if Friend B worked alone.

SHAP treats your model features (like Credit Score or Income) like teammates. It calculates a “payout” for each feature by comparing the prediction when that feature is present versus when it’s missing. What this actually means: we break the final prediction into a sum of its parts. Every feature gets a “receipt” for its contribution.

import shap

# Initialize the explainer
explainer = shap.TreeExplainer(model)

# Calculate SHAP values for our new customer
shap_values = explainer.shap_values(new_customer)

# Let's look at the contributions toward the 'Approved' probability (class 1)
feature_names = ['credit_score', 'income_k', 'debt_ratio']
contributions = pd.DataFrame({
    'feature': feature_names,
    'contribution': shap_values[0, :, 1]
})

print(contributions)

# Verify additivity instead of just eyeballing the numbers
base_value = explainer.expected_value[1]
approval_proba = model.predict_proba(new_customer)[0][1]
assert abs(base_value + shap_values[0, :, 1].sum() - approval_proba) < 1e-6
print(f"base ({base_value:.4f}) + sum(phi) ({shap_values[0, :, 1].sum():.4f}) = {base_value + shap_values[0, :, 1].sum():.4f}, model says {approval_proba:.4f}")

• shap.TreeExplainer(model) — Creates a SHAP explainer specialised for tree-based models (random forests, gradient boosting, XGBoost, etc.). It uses the exact TreeSHAP algorithm, which is polynomial-time and much faster than the general-purpose KernelExplainer.
• explainer.shap_values(new_customer) — Computes SHAP values for every feature in the single row new_customer. On a current shap install this returns a single NumPy array shaped (n_rows, n_features, n_classes) — here, (1, 3, 2). Older versions returned a list of two arrays, one per class, which is why you may see code elsewhere written as shap_values[1][0]; on the current shape that indexing silently grabs the wrong thing ([1] becomes “test row 1,” which doesn’t exist for a single-row input, so it errors immediately rather than returning a plausible-looking wrong number).
• shap_values[0, :, 1] — Selects row 0 (our single customer), every feature, and class 1 (Approved) — the correct way to reach the per-feature contributions on the current array shape.
• pd.DataFrame({'feature': feature_names, 'contribution': shap_values[0, :, 1]}) — Constructs a two-column DataFrame pairing each feature name with its SHAP contribution, so the printed output is human-readable instead of a bare NumPy array.
• explainer.expected_value[1] — The model’s average predicted probability of approval across the training data. Index [1] selects the positive class.
• assert abs(base_value + shap_values[0, :, 1].sum() - approval_proba) < 1e-6 — Turns the additivity guarantee into something the code actually checks, rather than a number printed and hoped to match.

When you run this, you’ll see numbers like -0.2659 for credit_score and +0.1605 for income_k. A negative number means that feature dragged the approval score down (making rejection more likely), while a positive number pushed it up. No more guessing — we have the receipt for this applicant’s denial, and the additivity check confirms it: base value 0.4883 plus the sum of the three contributions (-0.1383) lands exactly on the model’s own 0.35 approval probability.

This article doesn’t introduce new mathematics — it applies the tools from the first three articles to a real stakeholder conversation. Here is a recap of the key terms used across the cluster so you can connect the business explanation back to its technical roots.

Plain English	Statistical symbol	Python equivalent	Tool
SHAP value (credit assigned to feature $i$ for one prediction)	$\phi_i$	shap_values[0, i, 1]	SHAP (Article 1)
Model’s baseline / average prediction	$\mathbb{E}[f(x)]$	explainer.expected_value[1]	SHAP (Article 1)
Additive guarantee: base + all SHAP values = prediction	$f(x) = \mathbb{E}[f(x)] + \sum_i \phi_i$	base_value + shap_values[0, :, 1].sum()	SHAP (Article 1)
LIME local linear weight for feature $i$	$w_i$	weight in exp.as_list()	LIME (Article 2)
Partial dependence (average prediction as feature $x_S$ varies)	$\bar{f}_S(x_S)$	y-value on the PDP curve	PDP/ICE (Article 3)
One row’s prediction as $x_S$ varies (an ICE curve)	$f(x_S, x_C^{(i)})$ for fixed $i$	One line in the kind='both' plot	PDP/ICE (Article 3)

How they connect in this article: The Force Plot Mr. Owens shows the applicant is a visual rendering of $\phi_i$ for each feature — the same SHAP values from Article 1. The Summary Plot he shows the compliance team aggregates $|\phi_i|$ across all applicants, the same aggregation introduced in Article 1 and extended in Article 3. The PDP curves from Article 3 provide the “shape of the effect” backdrop that gives the individual SHAP receipt its broader context.

Reading the ‘Force Plot’: The tug-of-war for a decision

Now, how do you show this to Mr. Owens — and, eventually, to Maria? You don’t show them a spreadsheet of decimals. You show them a Force Plot.

Think of it as a visual tug-of-war.

1. The Base Value: The starting point — the average approval rate across all customers in our history.
2. Red Arrows: Features pushing the score higher, toward approval.
3. Blue Arrows: Features pulling the score lower, toward rejection.

The final prediction is where the rope settled after the pull.

# Note: In a real notebook, this displays an interactive HTML graphic
shap.initjs()
shap.force_plot(
    explainer.expected_value[1], 
    shap_values[0, :, 1], 
    new_customer, 
    feature_names=feature_names
)

• shap.initjs() — Initialises the JavaScript plotting backend required for SHAP’s interactive (non-Matplotlib) force plots. Required once per notebook session before calling force_plot without matplotlib=True.
• shap.force_plot(...) — Draws the classic SHAP force plot: a horizontal bar starting at the base value, with red arrows pushing right (toward higher approval probability) and blue arrows pushing left (toward lower probability).
• explainer.expected_value[1] — The model’s average predicted probability of approval across the background dataset. This anchors the left edge of the plot at the baseline. Index [1] selects the positive class (Approved).
• shap_values[0, :, 1] — This applicant’s SHAP values for the Approved class: the lengths and directions of each arrow.
• new_customer — This applicant’s feature values, displayed as labels on each arrow so the reader can see what value of each feature drove the contribution.
• feature_names=feature_names — Passes the column labels so the plot annotates arrows with human-readable names instead of integer indices.

Here it’s a genuine tug-of-war, not just a one-sided pull: you can tell Mr. Owens, “The average customer has just under a 49% chance of approval. This applicant dropped to 35%. Her Credit Score — the big blue arrow, at -0.27 — pulled the score down the hardest. Her Income (the red arrow, +0.16) genuinely pushed back the other way, but not far enough. Her Debt Ratio added one more small blue arrow (-0.03) on top.” That’s the answer he can give Maria, and the one he can put in the regulator’s file — and this time, both arrow colors actually appear on the plot.

The ‘Global’ view: What does the model care about most?

Explaining one decision is great for Maria, but Mr. Owens wants to know if the model is biased or broken before it goes live — and the regulator will ask the same question. They want to see the model’s “personality.”

A Summary Plot (often called a Beeswarm plot) does this. It moves from one specific decision to the whole model. The plot shows whether the model is “obsessed” with a particular variable.

# We'll calculate SHAP values for the whole training set to see the big picture
shap_values_all = explainer.shap_values(df.drop('approved', axis=1))

# Generate the summary plot
shap.summary_plot(shap_values_all[:, :, 1], df.drop('approved', axis=1))

• explainer.shap_values(df.drop('approved', axis=1)) — Computes SHAP values for every row in the training set (not just one applicant), producing an (n_rows, n_features, n_classes) array — here (12, 3, 2). This is the same call used in Article 1 for the summary bar plot.
• shap.summary_plot(shap_values_all[:, :, 1], df.drop('approved', axis=1)) — Draws the beeswarm plot: each dot is one applicant’s SHAP value for that feature, coloured by the feature’s actual value (red = high, blue = low). Features are ranked top-to-bottom by mean absolute SHAP value — the model’s overall priority list.

Averaged across all 12 applicants, credit_score has the highest mean |SHAP value| (0.23), well ahead of income_k (0.08) and debt_ratio (0.07) — so on this dataset, credit score really is the model’s primary driver, both for this one applicant and across the whole book of business.

What if a feature like “Zip Code” showed up at the top instead? That’s a red flag. The model could be using location as a proxy for race or income level, which could lead to legal trouble. You can check the model’s priorities against a human expert’s intuition. If it cares more about the day of the week the loan was applied for than the person’s debt, the model is “dumb” and needs fixing.

Which Explanation Do You Show Whom?

The team has now built three different explainability tools across this series — SHAP values, LIME, and PDP/ICE curves. But Mr. Owens faces a practical question: which one does he actually show to each audience? The answer depends on who’s sitting across the table.

Audience	What they need	Best tool	Why
The applicant (denied)	A clear, human reason for her denial — no jargon	Force Plot (SHAP)	One visual tug-of-war; she can see which features pushed her toward “No” without any math. The additive guarantee means the explanation is provably correct, not an approximation.
The regulator / compliance team	Proof the model isn’t using forbidden or proxy variables; documentation per denial	Summary Plot (global SHAP) + Force Plot (per-denial)	The summary plot proves the model’s global priorities are legitimate; the force plot documents each individual denial. SHAP’s additivity gives the regulator something to check line by line, which is exactly what a documentation requirement wants.
Mr. Owens (internal management)	A fast gut-check during operations; understanding the model’s overall behaviour	PDP/ICE curves for shape + LIME for speed	PDPs show “how does income affect approval across all applicants?” in one line; LIME gives a quick approximate explanation when a customer is on the phone and there’s no time to wait for SHAP.
The data science team (debugging)	Full decomposition; interaction detection; bias auditing	SHAP dependence plots + ICE curves	SHAP dependence plots automatically surface interactions (colour by the most correlated second feature); ICE curves reveal heterogeneity the PDP hides.

When to simplify

• ✅ Maria doesn’t need the math. She needs to hear: “Your credit score was the biggest factor working against you, and your income wasn’t quite enough to offset it.” The force plot is the visual behind that sentence — she never sees a Shapley value.
• ✅ The regulator needs the proof. They need to see that the force plot is mathematically guaranteed to decompose the prediction (SHAP’s additivity), and that the summary plot shows no proxy variables at the top.
• ✅ Mr. Owens needs the pattern. The PDP from Article 3 (“how does income affect approval overall?”) plus the force plot for this specific denial gives him both the forest and the tree in one meeting.

The short version

Show Maria the force plot. Show the regulator the force plot plus the summary plot. Show Mr. Owens the PDP curve plus the force plot. Reach for LIME specifically when speed matters more than an additivity guarantee — a live call where a fast local approximation beats making the customer wait for SHAP. Reach for SHAP when the explanation itself needs to be reproducible and provably decompose the prediction, such as anything that ends up in a compliance file. The tool that’s “right” depends on who’s asking, what they’ll do with the answer, and how much they need the additivity guarantee versus the speed.

The ‘So What?’—Turning charts into business actions

Mr. Owens doesn’t care about Shapley values or game theory. He cares about risk and results — and right now, he wants to give Maria an honest answer.

Here’s how to put these tools to work:

• Don’t show the code; show the tug-of-war. Use the Force Plot to walk stakeholders through individual decisions. It turns a “no” into a “no, because…”
• Use the Summary Plot as a safeguard. Before deploying, show the model’s top features to your legal or compliance team. It proves you aren’t using forbidden data.
• Debug the “dumb” behavior. If SHAP shows the model ignoring Income, you may have a data quality issue that slipped past training.

Explainability isn’t a “nice to have.” It’s what lets a business trust the math you’ve built.

What we learned:

1. Black boxes are just complex models we can’t track mentally.
2. SHAP gives every feature a receipt for its contribution.
3. Force Plots show the tug-of-war behind a single decision.
4. Summary Plots reveal the model’s overall personality and potential biases.

---

Maria sits across the desk from Mr. Owens. He doesn’t show her a beeswarm plot or a list of Shapley values. He shows her one picture — the force plot — and says: “Your credit score pulled your approval odds down the hardest. Your income pushed back the other way — it wasn’t nothing — but it wasn’t enough to overcome the gap, and your debt ratio added one more small pull down.” Maria nods. She may not love the answer, but she has one — specific to her, not “the computer said so.” Mr. Owens files the same force plot in the regulator’s documentation packet, alongside the summary plot that confirms the model’s global priorities are sound — credit score, consistently, is what this model leans on hardest. Next time a loan is denied, he knows which tool to reach for and which story to tell. The black box isn’t magic. It’s just math — and now everyone in the room can see it.

Check Your Understanding

Questions below move from simple recall up to open-ended design, roughly following Bloom’s Taxonomy.

Remember What do red arrows and blue arrows represent on a SHAP Force Plot?

Understand In your own words, explain the article’s “four friends building a house” analogy — how does it capture what a SHAP value actually measures for one feature?

Apply Using the article’s contribution table format (feature, contribution), if credit_score has a contribution of -0.27 and income_k has +0.16, which feature is pushing harder toward rejection, and by how much more?

Analyze The article says a Summary Plot showing “Zip Code” as the top feature would be “a red flag” because it might be “a proxy for race or income level.” Walk through why a feature can create legal/fairness risk even when it was never labeled as a sensitive attribute and the model was never told about race directly — what mechanism lets a proxy variable smuggle that information in?

Evaluate The article’s example dataset has only 12 rows, yet it’s used to demonstrate a “black box” Random Forest with 100 trees. Critique this pedagogical choice: with so few training examples, is the model’s SHAP explanation actually revealing genuine learned patterns, or could it just as easily be reflecting overfit noise from a dataset too small to generalize from?

Create Design a stakeholder-facing explanation package (following the article’s Force Plot + Summary Plot pattern) for a different model: an insurance claims model that just denied a claim. Describe what you’d show the claimant (individual explanation) versus what you’d show your compliance team (global/aggregate explanation), and why those two audiences need different views of the same model.

---

Related articles

• SHAP Values: Why Your Model Predicted That
• LIME vs SHAP: Choosing the Right Translator for You)
• Partial Dependence Plots and ICE Curves: See How Your Model Really Thinks)

---

References & Further reading

• Lundberg, S. M., & Lee, S.-I. (2017). “A Unified Approach to Interpreting Model Predictions.” Advances in Neural Information Processing Systems 30 (NeurIPS 2017). https://arxiv.org/abs/1705.07874 — The foundational paper introducing SHAP values and the TreeSHAP algorithm; the mathematical backbone behind every force plot and summary plot in this article.
• Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ‘16). https://arxiv.org/abs/1602.04938 — The foundational paper introducing LIME; relevant here for the tradeoff discussion of when a fast local approximation is acceptable versus when SHAP’s consistency is required.
• Friedman, J. H. (2001). “Greedy Function Approximation: A Gradient Boosting Machine.” Annals of Statistics, 29(5), 1189–1232. https://projecteuclid.org/euclid.aos/1013203451 — The foundational paper introducing Partial Dependence Plots; the “shape of the effect” backdrop that gives Mr. Owens the global pattern behind an individual denial.
• Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. https://christophm.github.io/interpretable-ml-book/ — A comprehensive, free online textbook covering SHAP, LIME, PDP, ICE, and the broader explainability landscape; an essential reference for anyone building a repeatable explainability workflow like the one this article describes.
• shap library documentation. https://shap.readthedocs.io — Full API reference for TreeExplainer, force_plot, summary_plot, and all other SHAP utilities used throughout this series.
• Kaggle: “Home Credit Default Risk” competition. https://www.kaggle.com/c/home-credit-default-risk — A real-world loan-default dataset where SHAP explanations are widely demonstrated in top-scoring notebooks; an excellent sandbox for practising the end-to-end workflow from force plot to summary plot to stakeholder conversation.

---