LLM A/B Testing
Understand how to effectively design A/B tests comparing LLM variants by controlling confounding factors and selecting relevant metrics. Learn to interpret results considering faithfulness as a critical release criterion, measure relevance and helpfulness tradeoffs, and avoid false positives by using guardrails and multiple evaluation runs.
Two variants are on the table for the grounded Q&A feature, and spot checks do not agree. Variant A sticks closely to retrieved text but sometimes misses what the user asked, while Variant B reads as more helpful but occasionally invents specifics.
A defensible shipping decision needs a definition of winning that matches product risk. Treat faithfulness as a release blocker for grounded Q&A because a single invented claim can be worse than a slightly less relevant answer, then treat relevance and helpfulness as tuning knobs that can trade off against each other. With that policy, anecdotes are not evidence because they mix different questions, different retrieved passages, and different levels of randomness into one impression.
This lesson's scope covers experiment design and interpretation for comparing prompt and model variants on the existing eval set and automated scoring.
Keep the trial identical across A and B
With a win condition set, the next step is to define what one comparison means so each A vs B pair measures only the change we intended. The experimental unit is one eval item that flows through the full pipeline, which includes the user query, the retrieval results, and the generated output that our scorers grade.
For offline evaluation, replay the same eval inputs through both variants so each item produces an A output and a B output under the same conditions. For any online slice, randomize assignment at the user request level so one request is scored once, not influenced by previous answers.
Before we compare, pin every non-tested setting because any drift turns the test into A/B/C without telling us. Hold constant: