When the Evals Disagree With the Customer
Respond to evaluation results that conflict with customer expectations by explaining the evidence and negotiating launch conditions. Decline immediate advisor exposure while proposing a conditional pilot that addresses access enforcement and cached-data handling failures. Separate scripted test counts from production performance, and distinguish reranker exposure from cache-revocation failures. Require repaired controls, independent logs, representative holdout tests, expert review, and approved workflows before advisor use. Document pilot boundaries, unresolved source decisions, operating responsibilities, and stop conditions while preserving the research goals and safeguarding sensitive information.
Sooner or later a customer will want to launch on a result you know is weak. Aster’s sponsor wants the research assistant in front of six advisors for 30 requests, citing 15 of 20 cases passed and $0.020 per task. Saying no is easy. The skill the interviewer is looking for is saying no to this launch while offering a next step the sponsor can accept. Your answer has to separate what the passing cases show from what they don’t, and keep a proposed pilot clearly apart from permission to expose advisors now.
OpenAI’s interview guide says interviews examine how ...