Content Moderation: Problem Framing & Requirements

Every day, platforms like Meta, YouTube, and Reddit process billions of posts, comments, images, and videos. A single piece of harmful content that slips through can go viral in minutes, causing real-world damage before any human reviewer even sees it. Designing the system that catches that content is not a classification exercise you can solve with a fine-tuned model and a threshold. It is a full ML system design challenge that spans policy operationalization, multimodal signal fusion, human-in-the-loop review, and real-time serving under strict latency budgets.

In an interview setting, you might be asked to design this system end to end. This lesson equips you with the foundational framing you need before touching any model architecture. We cover four pillars that structure the rest of the case study: policy scope definition, asymmetric cost metrics, latency mode selection, and level-appropriate scoping. Each pillar feeds directly into the data, modeling, and serving decisions that follow in subsequent lessons.

Defining policy scope for ML systems

Before writing a single line of training code, you must answer a deceptively hard question: what exactly counts as a violation? Platforms maintain detailed community guidelines that enumerate harm categories such as hate speech, violence and graphic content, nudity, spam, misinformation, and self-harm. These documents are written for humans. Translating them into machine-actionable label taxonomies is a process called policy operationalizationThe systematic conversion of human-readable policy documents into structured label definitions, inclusion rules, and exclusion rules that annotators and models can consistently apply..

A single general-purpose classifier cannot effectively cover the full diversity of violation types. Hate speech detection relies on linguistic nuance and cultural context, while graphic violence detection depends on visual features entirely absent from text. This motivates category-specific architecturesSystem designs where separate classifiers or model heads handle distinct violation types, each with its own label taxonomy, training data, and decision threshold.. In practice, platforms like Meta run dozens of specialized models rather than one monolithic classifier.

The hardest part of policy ...