Learn about the SHAP explainability algorithm, which connects game theory with local explanations.

SHapely Additive exPlanations

SHapley Additive exPlanations (SHAP) is a popular explainability algorithm that connects game theory with local explanations. SHAP aims to explain the prediction for any input (e.g., an image) as a sum of contributions from its feature values (e.g., image pixels).

SHAP assumes that the individual features (e.g., image pixels) in the input (e.g., an image) participate in a cooperative game whose payout is the model prediction. The algorithm uses game theory to distribute the payout among these features fairly. The payout is known as the Shapely value of a feature.

What are Shapely values?

Let’s assume that an image XX can be represented as a set of pixels X={Xij:0i<H,0j<W}X = \{X_{ij}: 0 \leq i < H, 0 \leq j < W \}, where HH and WW are the height and width of the image. Now, given a neural network f(.)f(.), let fS(X)f_S(X) represent the prediction on the image XX when only pixels in the set SS are considered while making the prediction (in other words, pixels not present in set SS are set to zero).

Now, let Pij={S:SX AND XijS}P_{ij} = \{ S: S \subset X \text{ AND } X_{ij} \in S \} denote all such subsets of XX that contain pixel XijX_{ij}.

Let’s consider one such subset SS. We define MCS(i,j)\text{MC}_S(i,j), the “Marginal contribution” of (i,j)th(i,j)^{th} pixel corresponding to set SPijS \in P_{ij} as the change in prediction score when the pixel XijX_{ij} is removed from set SPijS \in P_{ij}. In other words,

Get hands-on with 1200+ tech skills courses.