Guardrails, Safety Controls, and Failure Modes
Explore the principles of guardrails as layered defenses in AI systems, including input screening, output screening, and tool-call authorization. Understand the importance of fail-secure designs and identify risks distinctive to large language models, such as prompt injection and data leakage. Learn how to defend AI systems against these risks to maintain safe and compliant operations.
The previous chapter assumed a system already works and asked how to prove it, compare it, and keep it efficient. This chapter asks a different question: Is the system safe to run at all, and what happens the moment something goes wrong? A system can pass every accuracy and cost metric built in the last chapter and still be unsafe if a malicious input can redirect it, if a tool call can execute with more authority than the situation warrants, or if a screening component’s own failure quietly becomes a pass rather than a block. Guardrails are what stand between a capable system and the specific ways that capability can be misused, and this lesson builds the vocabulary and design habits for placing them correctly.
In this lesson, we will cover:
Guardrails as layered defense: input screening, output screening, and tool-call authorization working together, since no single layer can stand in for all three.
Failing securely rather than openly, and why a guardrail’s own failure mode matters as much as what it is built to catch.
The risks and failure modes distinctive to LLM systems: hallucination, prompt injection, jailbreak, unbounded delegation, and data leakage.
Defending against prompt injection, direct and indirect, by treating retrieved and tool-sourced ...