The Big Picture: A Map of Responsible AI

AI governance is the emerging discipline that addresses how AI systems should be developed, deployed, and controlled. It spans technical safeguards, organizational policies, and legal frameworks.

AI systems fail in ways traditional software does not. A model can be mathematically correct and still cause harm through biased outputs, unpredictable behavior, or objectives that align with instructions but not with the intended outcomes. These failures are not fixable with unit tests and require a different mental model.

This course focuses on building that model. But first, we need a shared vocabulary.

The umbrella: Responsible AI

Responsible AI (also referred to as trustworthy AI or ethical AI) represents a broad, overarching objective. It provides a conceptual and normative framework for the design, development, and deployment of artificial intelligence systems in ways that benefit individuals and society while respecting human rights and fundamental values.

Responsible AI is the umbrella that covers all ethical and technical considerations, including:

While the primary focus of this course is safety, it is essential to distinguish it from related concepts to ensure terminological precision.

• AI safety: Preventing unintentional harm (accidents/malfunctions).

• AI security: Preventing intentional harm (attacks/adversaries).

• AI fairness: Ensuring the system does not discriminate against specific demographic groups.

• Privacy: Protecting the personal data used to train or query the system.

• Accountability: Establishing clear lines of responsibility for system outcomes.

• Transparency/explainability: The ability to understand why a system made a decision.

AI safety

AI safety is one of the core pillars that hold up the responsible AI umbrella.

For this course, we will use a very precise definition: AI safety is the discipline of preventing unintentional harm. This is the accident problem. The harm comes from internal flaws, design errors, unforeseen failures, or misaligned objectives.

This course focuses specifically on AI safety. But safety doesn’t exist in isolation; it’s deeply connected to fairness and requires explainability as a diagnostic tool. Let’s explore those connections.

AI fairness

So, where does fairness fit in?

While often treated as a distinct ethical domain, we categorize fairness under safety in this course. Why? Because unintended bias is a form of system malfunction. If a model injures a specific demographic because of skewed training data, it has failed to function robustly.

When an AI model for screening job applications is biased against a certain group, or a loan model is biased against a specific demographic, this is a malfunction. It is an accident where the system is causing unintended discriminatory outcomes.

Therefore, we cannot have a truly safe system if it is not also fair. This is why fairness and mitigating bias are a core part of AI safety.

Explainability (XAI) / Interpretability

Finally, what is explainability (or interpretability)?

Explainability (XAI) is not a goal in itself; it is the essential tool we use to achieve safety and fairness.

These are the methods for inspecting opaque models. As engineers, we must be able to understand why an AI model made a specific decision.

• How can we know if a model is fair if we can’t inspect why it denied a loan?

• How can we know if a model is safe if we can’t debug why it malfunctioned?

Next, we will do a deep dive into the two foundational pillars: AI safety (accidents) vs. AI security (attackers).

As engineers, this is the single most important distinction for us to understand. Why? Because confusing them leads to using the wrong tools for the job. You can’t fix an attack with a tool designed to prevent an accident.

The Accident problem: What is AI safety?

First, let’s be precise about AI safety. As we’ve said, this is the accident problem. The harm is unintentional. It’s caused by internal flaws, design errors, the system not truly understanding our goals, or failing in unforeseen ways when it encounters a new situation.

This leads to our formal definition:

AI safety: It is the property of an AI system to avoid causing unintended harmful outcomes to individuals, environments, or institutions, despite uncertainties in its goals, data, or the environment it's operating in.  

The critical question AI safety seeks to answer is: Does this system work as it should, even in a complex, unpredictable, nonadversarial world?

The “attacker” problem: What is AI security?

Now, let’s look at AI security. This is the attacker problem.  

AI security assumes a hostile world. It assumes there is an intelligent, deliberate person (a malicious actor) on the other side of the keyboard actively trying to compromise, manipulate, or steal from your system.  

This leads to our second formal definition:

AI security: It is the property of an AI system to remain resilient against intentional attacks on its data, algorithms, or operations, preserving its confidentiality, integrity, and availability in the presence of adversarial actors.  

The key question AI security answers is: Can an attacker force this system to fail, even if it’s ‘safe’ in normal operation?

An analogy for engineers

This difference isn’t just academic. It determines the tools you use. The best way to understand this is with an analogy from classic computer science: sending a message.

• The AI safety problem (an accident): Imagine you send a message over a noisy network. A random burst of static flips a bit and corrupts the file. This is an accident.

  • The safety solution (checksum): You add a checksum (like a CRC). The receiver runs the checksum. If it doesn’t match, they know the file was corrupted by random noise and ask you to resend it.

• The AI security problem (an attacker): Now, imagine an attacker is listening. They intercept your message, “Pay Bob $100.”

  • Why the safety tool fails: The attacker changes the message to “Pay Eve $10,000.” Then, they simply re-calculate a new, valid checksum for their malicious message. The receiver gets it, the checksum passes, and the money is stolen. The safety tool (checksum) was completely useless against an intelligent attacker who can adapt.  

  • The security solution (MAC): You use a Message Authentication Code (MAC), which utilizes a secret key known only to you and the receiver. The attacker can’t forge this because they don’t have the key.  

This is exactly the mental model we must apply to AI.

Let’s connect this back to our map and show how this course is structured to provide you with a safety toolbox.

Our focus: The AI safety toolbox

This is an AI safety course.

Our primary mission is to give you the engineering skills to solve the accident problem: How do we build systems that are robust, reliable, and aligned with our intentions, even when no one is actively attacking them?  

This means we will focus on the AI safety toolbox. While we will touch on security topics (like using red teaming to find failures ), our core curriculum is built around these five pillars of preventing unintentional harm:  

1. Alignment (e.g., RLHF, constitutional AI): This is our steering toolkit. It’s the technical process of closing the alignment gap, the difference between the simple goal we give the AI (like “maximize this score”) and the complex goal we actually want (like “be helpful and harmless”). This is our main tool for preventing the unintentional misalignment that leads to King MidasGetting exactly what you asked for, but realizing it's fatal. or paperclipAn AI destroying the world just to optimize a trivial goal like producing paperclips. scenarios. We’ll learn hands-on by building a reinforcement learning from human feedback (RLHF) loop.  

2. Robustness (e.g., adversarial attacks): This is our sturdiness toolkit. In this course, we use adversarial attacks as a stress test to find brittle points in our model (safety). While we use attacks to find these bugs, fixing them requires safety techniques like robust training, which are different from the cybersecurity defenses used to stop active hackers .

3. Interpretability (e.g., LIME, SHAP): This is our diagnostic toolkit. It contains the methods for inspecting opaque models to understand why a model made a certain decision. This is how we connect our tools to our goals. We use interpretability tools to audit our systems for fairness, because we can’t fix an unintentional bias if we can’t find its source.  

4. Evaluation (e.g., HarmBenchA standardized dataset for testing refusals., safety by measurement): This is our quality assurance toolkit. Instead of just hoping our system is safe, evaluation is the active, systematic process of testing it for potential accidents and malfunctions. We’ll learn advanced frameworks like “Safety by Measurement” to measure a model’s dangerous capabilities (what’s the worst it can do?) and its propensities (what does it tend to do by default?).  

5. Governance (e.g., safety cases, runtime monitoring): This is our production toolkit. It’s the whole-system framework that holds everything else together. We’ll learn how to build an AI safety case, a formal argument, backed by evidence from our evaluations, that proves our system is safe to deploy. And we’ll learn to design runtime monitoring systems (like the MI9 frameworkA framework for monitoring autonomous agents.) to watch and control live AI agents and catch accidents in real-time.

Let’s pause and summarize the key concepts you’ve learned.

This map of terms is the most important foundation for the rest of the course. Everything we do from here on will build on these core ideas.

Key takeaways

1. Responsible AI is the umbrella. This is the high-level, overarching goal of building AI that benefits society. It includes all other concepts, from safety and security to privacy and accountability.

2. Safety vs. security is the core distinction. This is the most critical concept for us as engineers.

   1. AI safety is about preventing unintentional harm (accidents, malfunctions, misaligned goals).

   2. AI security is about preventing intentional harm (attackers, misuse, manipulation).

   3. We used the “Checksum vs. MAC” analogy to illustrate why a safety tool (such as a checksum) is ineffective against a security threat (an intelligent attacker).  

3. Fairness is a component of AI safety. For the purposes of this course, we treat algorithmic bias as a form of unintentional harm (a malfunction). Therefore, in our framework, you cannot have a truly safe system if it is not also fair.

4. Explainability (XAI) is our tool. Explainability is not the end goal; it’s the practical tool we use to achieve our goals. It allows us to inspect the opaque model to identify the sources of unintentional bias (ensuring fairness) and other unintentional malfunctions (ensuring safety).  

5. This is an AI safety course. Our primary mission is to provide you with an AI safety toolbox for preventing accidents. We will focus on the five pillars we outlined: Alignment, Robustness, Interpretability, Evaluation, and Governance.

We now have the complete conceptual framework.

In our next lesson, we will use this framework to explore the map of specific harms we’re trying to prevent.