Threat Modeling an Inference-Time API Attacker
Explore how to build a precise threat model for an inference-time API attacker by defining system boundaries, attacker goals, access rights, and constraints. Understand how API design choices affect the attack surface and evaluate the feasibility of different attacks based on observable outputs and request limits. This lesson helps you create actionable security models that guide defenses against adversaries interacting through ML prediction APIs.
A small API contract decision changes the attacker’s world. If your image-classification prediction API returns only a top label, the response leaks less than an API that returns a full probability vector, even when both APIs run the same model.
To keep the threat model honest, lock the attacker to the interface. This attacker cannot read model weights, cannot see training code, and cannot run code on your servers. The only interaction is sending an image and receiving whatever fields the API returns, plus side effects like latency ...