Online Evaluation, Experiment Design, and Drift
Explore how to measure GenAI system performance in production by designing reliable online experiments, tracking key metrics, and detecting behavioral drift. Understand rollout strategies like canary deployments, A/B tests, and shadow traffic. Learn to integrate offline and online evaluations to ensure system robustness amidst dynamic user behavior and model updates.
We'll cover the following...
Passing every test in staging doesn’t guarantee that a generative application will perform well under live user traffic. Production systems encounter changing user behavior, updated knowledge sources, model provider changes, and non-deterministic outputs that static tests may not capture.
Offline evaluation helps catch known regressions before release, but production evaluation answers a different question: how does the system perform for real users over time? This lesson focuses on measuring online outcomes, designing reliable experiments, and detecting changes in system behavior.
Production evaluation as a release system
A production GenAI stack routes a request through retrieval, prompt assembly, model inference, post-processing, and sometimes tool execution. Evaluation has to sit across that whole path, not only at the model layer. The release unit is therefore the entire serving configuration.
A simple rule works well in interviews. If a change can alter the answer, the refusal, the tool call, the latency, or the cost, it must enter the same release pipeline.
The operating life cycle usually contains these control points.
Predeploy gate: The candidate configuration runs against a versioned regression suite before any user sees it.
Exposure control: Traffic is mirrored, sampled, or gradually shifted so the candidate sees realistic load without immediate full blast radius.
Online measurement: The serving layer emits metrics and traces sliced by version and cohort.
Rollback ownership: The on-call or release owner has preapproved thresholds and can revert without waiting for a debate. ...