From Life Cycle to Attack Surfaces: Shift, Error, and Adversaries
Explore how to inventory attack surfaces throughout an ML system's lifecycle, linking interfaces to assets and security goals. Understand how to differentiate ordinary errors, natural data shifts, and deliberate adversarial actions by examining observable patterns and plausible access points, preparing you to create actionable security models for ML pipelines.
Accuracy dropped for a subset of images in the running classification API, and two explanations can both fit the same symptom. A natural distribution shift can change the kinds of inputs arriving at the API, so a model that worked last month fails today. An adversary can also cause a performance drop, but only if they have a real interaction channel and a plausible objective.
The goal here is not to decide what happened from a single metric. The goal is to turn your lifecycle trace into an explicit attack surface inventory, so you can point to concrete places where someone could influence confidentiality, integrity, availability, or privacy. That inventory is the chapter’s output artifact, and it becomes the input to the next chapter’s threat model.
What an attack surface means in ML
An attack surface in an ML system is the set of interfaces, inputs, and dependencies through which an adversary could influence a security objective. In the running classification API, that includes obvious entry points like the HTTP request body, plus less obvious ones like dataset refresh jobs, labeling tools, model artifact storage, and monitoring dashboards that drive automated responses.
To ...