Machine learning models are everywhere, making decisions about loans, medical diagnoses, and what you see online. But often, they are black boxes. You get an output—a prediction, a score—without a clear reason why. I started asking a simple question: How can we trust a system we don’t understand? This search led me to model interpretability, and specifically to SHAP. If you want to move from just using models to truly understanding them, follow along. I’ll show you how.

Think of a complex model like a team making a decision. Who contributed what to the final call? SHAP, or SHapley Additive exPlanations, answers this. It’s based on a concept from game theory that fairly distributes credit. For any single prediction, SHAP tells you how much each feature, like ‘income’ or ‘age’, pushed the model’s output higher or lower. This makes the model’s reasoning transparent, feature by feature.

Let’s look at it in action. First, you’ll need the library.

import shap
import xgboost as xgb
from sklearn.datasets import load_breast_cancer

# Load data and train a model
data = load_breast_cancer()
X, y = data.data, data.target
model = xgb.XGBClassifier().fit(X, y)

# Create a SHAP explainer
explainer = shap.Explainer(model)
shap_values = explainer(X)

With just a few lines, you have a powerful explainer object. But how does it actually calculate these values?

The core idea is to ask “what if?” What if we didn’t know this person’s age? What if their income was the average value? SHAP runs many of these scenarios by creating combinations of features. It measures the model’s output with and without a specific feature to see its isolated effect. The final SHAP value is a weighted average of all these contributions. It’s a rigorous way to measure importance. Doesn’t that make more sense than just looking at feature weights in a linear model?

The real power is in the explanations. For a single prediction, you can see exactly what drove it.

# Explain the first prediction in the dataset
shap.plots.waterfall(shap_values[0])

This creates a waterfall chart. It starts with the model’s average output across all data. Then, each bar shows how a feature changed that base value for this specific person. A long red bar for ‘worst radius’ pushing the prediction up, a blue bar for ‘smoothness error’ pulling it down. You get a clear, visual story for that one decision.

What about the model as a whole? You can move from local to global insights. A summary plot shows patterns across your entire dataset.

shap.summary_plot(shap_values, X, feature_names=data.feature_names)

Each dot is a person. The position shows the feature’s value (low or high), and the color shows its SHAP value (positive or negative impact). You instantly see, for example, that a high ‘worst area’ generally increases the risk score across many patients. It shows you the model’s general logic.

Why not use simpler methods? Feature importance from a random forest tells you what matters overall but not for a single case. Coefficients in linear models can be misleading with correlated features. SHAP provides a consistent, local accuracy that other tools often lack. It works for almost any model, from a logistic regression to the most complex neural network, which is why it has become so widely trusted.

There are advanced techniques, too. You can use SHAP to find interactions between features or to validate model behavior on specific subgroups. This is crucial for catching bias. If a model consistently gives high SHAP values to ‘zip code’ in a loan approval system, you might have an ethical and legal problem on your hands.

So, where do you start? Begin with a simple model and a dataset you know well. Use the TreeExplainer for tree-based models like XGBoost; it’s fast and exact. For neural networks, the GradientExplainer is a good choice. Remember, interpretability is not a one-time check. Build it into your workflow. Explain your model during development, not just at the end for a report.

I began this journey skeptical of black boxes. SHAP turned the light on inside them. It transformed how I build, validate, and present models. It builds trust—with me, my team, and the people affected by the predictions. Give it a try on your next project. What surprising logic might your model reveal?

If this guide helped you see your models more clearly, please share it with a colleague. Have you used SHAP to discover something unexpected in a model? Let me know in the comments—I read every one.