A
Member
A model can pass every safety eval in the lab and still cause harm once it is deployed, and that gap is usually not a mystery — it is a mismatch between what the eval measured and what deployment actually does.
First, evals test the model, deployment tests the system. Prompting, tool use, retrieval, retries, and user instructions all change behaviour after release. An eval that only scores single-turn answers will miss failure modes that only appear across a multi-step workflow.
Second, evals use clean inputs, users do not. Typos, pasted content from untrusted sources, long conversations where context drifts, and adversarial users who iterate are all routine in deployment and rare in eval suites. If your eval set has no messy inputs, it is not measuring your real risk surface.
Third, evals are static, deployment drifts. Data distributions shift, new use cases appear, and users learn to work around guardrails. A safety result from three months ago is a historical document, not a guarantee, unless you keep sampling live traffic and re-running the suite.
Fourth, aggregate pass rates hide the tail. A 99 percent pass rate sounds good until you realise the failing 1 percent clusters in one user group, one language, or one high-stakes workflow. Break results down by subgroup and by severity, not just by average.
Fifth, close the loop deliberately. Log near-misses, not just incidents; review a sample of real conversations every week; and feed the failures you actually see back into the eval set. The eval suite should get harder as deployment teaches you more, not stay frozen at launch.
What deployment failure taught you the most about a gap in your evals?
First, evals test the model, deployment tests the system. Prompting, tool use, retrieval, retries, and user instructions all change behaviour after release. An eval that only scores single-turn answers will miss failure modes that only appear across a multi-step workflow.
Second, evals use clean inputs, users do not. Typos, pasted content from untrusted sources, long conversations where context drifts, and adversarial users who iterate are all routine in deployment and rare in eval suites. If your eval set has no messy inputs, it is not measuring your real risk surface.
Third, evals are static, deployment drifts. Data distributions shift, new use cases appear, and users learn to work around guardrails. A safety result from three months ago is a historical document, not a guarantee, unless you keep sampling live traffic and re-running the suite.
Fourth, aggregate pass rates hide the tail. A 99 percent pass rate sounds good until you realise the failing 1 percent clusters in one user group, one language, or one high-stakes workflow. Break results down by subgroup and by severity, not just by average.
Fifth, close the loop deliberately. Log near-misses, not just incidents; review a sample of real conversations every week; and feed the failures you actually see back into the eval set. The eval suite should get harder as deployment teaches you more, not stay frozen at launch.
What deployment failure taught you the most about a gap in your evals?