What Makes ML Security Different From Traditional Security?

An image-classification model behind an HTTP prediction API can change its mind without anything being malformed. The same JPEG that used to come back as cat can later come back as dog after a retrain, a change in preprocessing, or a slow shift in what users upload, even when the request body is valid and the endpoint returns 200.

That visible behavior breaks a common security instinct that says if inputs are well-formed and code paths are tested, the system should behave the same way tomorrow. In this chapter, the goal is not to jump straight to attacks. The goal is to build an ML-system view, so security claims are anchored to clear objectives and a declared system boundary before anyone argues about whether a defense works.

ML security protects objectives, not just availability

ML security means protecting an ML system’s objectives under adversarial interaction, not merely preventing crashes, preventing unauthorized access, or keeping the server up. For a classification API, an objective might be that correct labels are returned often enough to support a downstream business decision, or that the system does not become reliably wrong for a targeted set of inputs.

Traditional software security often leans on deterministic validation. A parser either accepts JSON or rejects it, an authorization check either grants access or denies it, and unit tests try to lock in a stable input-output contract. ML systems have deterministic parts, but the core behavior you care about is statistical and data-dependent, so the evidence you use to validate behavior changes as well.

In the comparison below, focus on what signal counts as a failure. In deterministic code, a single bad output can be enough. In an ML model, some errors are expected, so you look for shifts in rates, patterns, and worst cases.

A useful split for the prediction pipeline is deterministic versus statistical.

• Deterministic parts include request parsing, schema checks, authentication, rate limiting, and converting bytes into an image tensor.

• Statistical parts include the model’s decision boundary and its calibrated confidence, which depend on training data, labels, and optimization.

This matters because a validator that proves the API accepts only valid images does not prove the model will behave acceptably on those images, and an attacker can exploit that gap without ever sending an invalid request.

Draw the system boundary before claiming security

In ML security, the system is not only the running POST /predict handler. The deployed image-classification model exists because a training pipeline produced a model artifact from data and labels, and both the serving path and training path shape what an adversary can influence.

Use this boundary for the running system. A client sends an image to the API, the serving code preprocesses it, the model produces logits or probabilities, and the API returns a label. Upstream, training code consumes datasets and labels, produces a new model artifact, and that artifact is what serving loads.

What counts as the ML system you are defending usually includes more than the endpoint.

• Data sources, labeling processes, and stored datasets

• Feature and preprocessing code used in both training and serving

• Training configuration, training code, and model artifacts

• The serving stack that loads the model and answers requests

• Monitoring, evaluation sets, and any feedback loop from production data

Boundary rule
A security claim is incomplete until it states which components and assets are in scope and which evidence supports the claim.

The evidence you typically have also differs. Instead of only relying on unit tests and integration tests, you often rely on evaluation datasets, distribution and drift metrics, slice-based performance reports, logs of requests, and documentation like model cards that describe intended use and known limitations. With that evidence in hand, the next question is what kind of problem you’re actually looking at because “the model got something wrong” can mean three very different things.

Error, shift, and attack are different hypotheses

When a classification API behaves badly, three common explanations lead to different next steps. Treat them as competing hypotheses rather than immediately calling the event an attack.

Ordinary model error is the baseline. The model may consistently confuse two visually similar classes because the training set did not cover enough variation, or because labels were wrong in a way that taught the model the wrong pattern. The fix usually involves data curation, better labels, architecture changes, or better evaluation coverage.

Natural distribution shift happens when the world changes while the code stays the same. A camera upgrade changes image noise, seasonal lighting changes the background, or users start uploading a new style of product photo. The fix often involves monitoring shift signals, adding updated data, and validating performance on the new distribution before deployment.

Adversarial behavior is different because an attacker chooses inputs or interactions to push the system toward a specific outcome. For the same POST /predict endpoint, an attacker might craft images to systematically trigger a chosen label, or might probe the API to learn which perturbations flip outputs. Even without operational details, the key is that intent and control over inputs change what you must defend.

A practical rule of thumb keeps the conversation disciplined.

Attack check
When you say attack, state the objective being harmed and the interaction channel being used.

That single sentence forces you to separate malformed-request security from model-behavior security, and it sets up the next step in thinking about ML systems. With that evidence in hand, the next question is what kind of problem you’re actually looking at because “the model got something wrong” can mean three very different things.