Content Moderation: Evaluation & Fairness

A content moderation system built on a tiered ensemble with an active learning flywheel can achieve impressive aggregate metrics, 95% F1, perhaps even higher. Yet that single number can mask catastrophic failures. A model might catch 99% of English-language spam while missing 30% of hate speech in Hindi. It might wrongfully remove benign political commentary from one community at triple the rate of another. In an ML system design interview, reporting a single F1 score and stopping there signals a junior-level understanding. Interviewers at L5 and above expect you to decompose evaluation along policy-relevant dimensions and proactively surface fairness concerns before being asked.

This lesson covers four evaluation pillars that separate strong interview answers from average ones. First, asymmetric cost-aware metrics that assign different thresholds per violation category. Second, slice-based evaluation that detects hidden disparities across languages and communities. Third, counterfactual fairness testing that checks whether moderation decisions depend on author identity rather than content behavior. Fourth, human review queue prioritization framed as its own ML ranking problem.

Aggregate accuracy is necessary but insufficient. Let’s break down each pillar.

Precision vs. recall under asymmetric policy costs

In most classification problems, false positives and false negatives carry roughly equal weight. Content moderation is fundamentally different. A missed child safety violation, a false negative, carries extreme legal liability and direct human harm. A wrongfully removed spam post, a false positive, causes mild user annoyance. Treating these errors identically produces a system that optimizes for the wrong objective.

Cost-sensitive thresholds per violation category

Rather than selecting a single operating point on the ROC curve for all violation types, each harm category receives its own decision threshold derived from a policy-defined cost ratio. Child safety might operate at a 100:1 ratio of false negative cost to false positive cost, meaning the system tolerates 100 unnecessary escalations to avoid missing a single genuine violation. Hate speech might use 10:1, and spam 2:1.

To operationalize this, you multiply each cell of the confusion matrix by its respective cost and minimize total weighted cost. The objective becomes $\min_\theta \sum_i \left[ c_{FN} \cdot FN_i(\theta) + c_{FP} \cdot FP_i(\theta) \right]$ ...