Threat Modeling a Training-Time Data or Pipeline Attacker
Explore how to build structured threat models for training-time attackers targeting machine learning pipelines. Understand assets, attacker goals, access points, and constraints affecting dataset integrity and model behavior. Learn to identify trust boundaries and predict observable signs of data or pipeline tampering to enhance ML security.
A new batch of images arrives for the same image-classification system; the training job runs overnight, and the model gets promoted because overall validation accuracy and loss look consistent with the last release.
Two days later, downstream monitoring shows something odd. A narrow slice of inputs that used to be stable now fails in a systematic way, and the failures cluster around one class boundary rather than looking like random noise or drift.
Notice what’s different from the last lesson’s attacker: this one never sends a request to POST /Predict at all. Everything happens before the model is promoted; the interaction channel is the pipeline, not the API. That pattern forces a different question than the usual deployed endpoint story. If the model artifact passed normal checks, then the relevant ...