Gradient Masking: False Security and Warning Signs
Explore how gradient masking can create a false sense of security in machine learning defenses by obscuring gradients that attacks rely on. Learn to differentiate between true model robustness and artifacts of evaluation failures. This lesson helps you critically assess defense claims by recognizing warning signs such as non-adaptive attackers, sensitivity to small changes, and lack of diverse evaluation methods. You'll gain skills to scrutinize robustness reports and improve your understanding of trustworthy defenses against adversarial attacks.
A new defense for our image-classification model reports very high robust accuracy against a common attack, and the reported results initially appear convincing. When the evaluation is rerun under a modified attack configuration, the reported robustness drops sharply even though the model weights and data are unchanged. That discrepancy is an early indication that the evaluation may be reflecting a limitation of the attack rather than genuine model robustness.
Gradient masking, also called obfuscated gradients, is one common way this happens. The defense or evaluation pipeline distorts or obscures the gradient signal, so a gradient-based attack cannot identify a reliable optimization direction even when adversarial examples still exist within the allowed perturbation region.
Use the table below to trace how a defense change can inflate reported robustness by distorting the gradient signal, and contrast that with a defense that genuinely changes the decision boundary: