LLM A/B Testing
Explore how to design and interpret A/B tests for large language models by focusing on faithfulness as a key release criterion. Learn to keep environment variables constant during testing, select appropriate metrics, interpret result distributions, and conduct repeat trials to separate signal from noise. Gain skills to make defensible deployment decisions and avoid overfitting by freezing evaluation sets and monitoring guardrails.
Two variants are on the table for the grounded Q&A feature, and spot checks do not agree. Variant A sticks closely to retrieved text but sometimes misses what the user asked, while Variant B reads as more helpful but occasionally invents specifics.
A defensible shipping decision needs a definition of winning that matches product risk. Treat faithfulness as a release blocker for grounded Q&A because a single invented claim can be worse than a slightly less relevant answer, then treat relevance and helpfulness as tuning knobs that can trade off against each other. With that policy, anecdotes are not evidence because they mix different questions, different retrieved passages, and different levels of randomness into one impression.
Scope covers experiment design and interpretation for comparing prompt and model variants on the existing eval set and automated scoring.
Keep the trial identical across A and B
With a win condition set, the next step is to define what one comparison means so each A vs B pair measures only the change we intended. The experimental unit is one eval item that flows through the full pipeline, which includes the user query, the retrieval results, and the generated output that our scorers grade.
For offline evaluation, replay the same eval inputs through both variants so each item produces an A output and a B output under the same conditions. For any online slice, randomize assignment at the user request level so one request is scored once, not influenced by previous answers.
Before we compare, pin every non-tested setting because any drift turns the test into A/B/C without telling us. Hold constant:
retriever index and filters ...