Interpreting Models with Permutation Importance and SHAP
Explore how to interpret machine learning models by measuring feature impact using permutation importance and explaining individual predictions with SHAP values. Understand the difference between global and local explanations, learn to evaluate feature contributions without implying causation, and communicate model insights effectively.
A loan-default model can achieve a strong receiver operating characteristic area under the curve (ROC AUC) and still be unusable for deciding which applications need manual review. A reviewer needs two different answers. Which inputs drive the model’s predictions overall? Why did the model assign a high default-risk score to this application?
Interpretability is the ability to explain which model inputs influenced a prediction and in which direction. A global explanation describes model behavior across many applications, such as whether credit_utilization usually matters more than recent_delinquencies. A local explanation describes one prediction, such as why an application with income=52000, debt_to_income=0.41, and recent_delinquencies=2 received a high risk score.
A linear model may expose coefficients, and a single decision tree may expose rules. Those outputs do not always explain a complete prediction pipeline well. Transformations can change inputs before the estimator sees them. Tree-based importance values also measure split behavior inside one model, not necessarily the practical loss in predictive performance when an input becomes unavailable.
Measure global reliance by shuffling one feature
Permutation importance answers the first review question by measuring what happens when we disrupt one input on a held-out set. Permutation importance is the average decrease in a chosen evaluation score after randomly shuffling one feature’s values while leaving all other columns unchanged.
Shuffling recent_delinquencies breaks its observed relationship with defaulted. If the model used that relationship, ROC AUC falls. Shuffling a feature the model does not use produces little or no score change. The method is model-agnostic because it works by evaluating predictions, not by inspecting coefficients, tree splits, or internal estimator details.
The held-out rows must represent the applications the model will score later. If the evaluation set differs from the real decision setting, a score drop describes the wrong population.
The following analysis fits a preprocessing-and-classification ...