Design an AI Content Moderation System

Hundreds of millions of posts, messages, images, and videos can arrive each day, and the platform still has to stop harmful content before users see it. Sending every submission to a large language model or a human reviewer breaks both latency and cost budgets, so the system must make most decisions cheaply and reserve expensive judgment for the minority of ambiguous cases.

This lesson builds a hybrid moderation system that combines rules, lightweight classifiers, LLM adjudicationUsing an LLM with a policy prompt to judge ambiguous cases., and human review in one production pipeline. In an interview, that means framing trust and safety as a system with measurable targets rather than as a vague policy problem.

The scope starts with category-specific precision and recall, pre-publish and post-publish latency budgets, and submission volume. A moderation system also makes asymmetric decisions. Missing a credible terror threat is not the same as overblocking satire, and the acceptable trade-off changes by category.

We can think of the design like an airport security line. Most bags pass quick screening, a few go through deeper scans, and only the uncertain ones reach manual inspection.

By the end, you should be able to explain how uncertain machine decisions are routed through structured review and policy layers, a pattern that carries directly into the next lesson on extraction and decision pipelines.

System requirements

Let's start by defining the product behavior before drawing boxes. The system receives text, images, short videos, audio transcripts, and links, then decides whether to allow, block, downrank, age-gate, or send the submission for review.

Let's use the following categories to anchor the design discussion.

• Terrorism and violent threats: These categories usually optimize for recall first because legal and safety exposure from false negatives is unacceptable.

• Hate speech and harassment: These categories often use balanced thresholds, with precision increasing on speech-sensitive surfaces.

• Misinformation and self-harm: These categories often require policy nuance, escalation, and review-heavy handling.

• Spam, scams, and phishing: These categories support aggressive automated blocking because repeated patterns make them easier to detect.

Terrorist content needs special handling outside the normal cascade. Known material is detected mainly through hash matching against industry-shared databases, and platforms often face legal duties to remove it within hours and to report credible threats to the authorities. The evidence is kept out of third-party LLMs and general reviewer queues. Matches go to a restricted evidence vault and a small specialist team, and the escalation and reporting workflow runs outside the standard pipeline.

For latency, we split the system into pre-publish and post-publish paths. Urgent blocking on uploads or public comments may need sub-200 ms p99, which only Tier 1 and Tier 2 decisions can meet. Items that need the LLM or human tier are held back for severe categories, or published with limited reach and reviewed afterward for lower-risk ones, while slower judgments on nuanced misinformation can tolerate seconds.

Functional requirements

• ...