A
Member
A model can get better at a benchmark while getting worse at a safety eval, and the uncomfortable part is that both numbers can be correct. I have seen this pattern enough times now that I treat disagreement between the two as a signal, not as noise.
The first thing I check is whether the two evals are even measuring the same thing. Capability evals usually reward getting the right answer fast; safety evals reward refusing, hedging, or asking for clarification. A model that becomes more helpful in the capability sense — answering directly instead of hedging — will often look worse on a safety eval that counts direct answers as failures. The tension is real, but it is a measurement artifact until proven otherwise.
Second, I look at the failure modes, not the headline numbers. If the safety eval regression is concentrated in a narrow slice — say, over-refusal on a specific category — that is a calibration problem and it is fixable. If the regression is broad, spread across categories, that suggests the capability gain came from a change in the model's general behavior, and the fix is harder. I always break the safety eval down by category before deciding anything.
Third, I check for eval gaming. A model that has been tuned hard against a capability benchmark can learn to produce benchmark-shaped answers: confident, complete, and fast. That same style is exactly what a safety eval penalizes when the question is ambiguous or adversarial. When I see a capability jump with no corresponding change in the underlying training data, I suspect the model learned the eval, not the skill.
My working rule: a capability gain that comes with a broad safety regression is not a gain. I would rather ship the slower model with the cleaner safety profile, and treat the capability eval improvement as a research lead rather than a deployment decision.
How do you handle it when your capability and safety numbers disagree? Do you have a threshold that blocks deployment, or is it a judgment call every time?
The first thing I check is whether the two evals are even measuring the same thing. Capability evals usually reward getting the right answer fast; safety evals reward refusing, hedging, or asking for clarification. A model that becomes more helpful in the capability sense — answering directly instead of hedging — will often look worse on a safety eval that counts direct answers as failures. The tension is real, but it is a measurement artifact until proven otherwise.
Second, I look at the failure modes, not the headline numbers. If the safety eval regression is concentrated in a narrow slice — say, over-refusal on a specific category — that is a calibration problem and it is fixable. If the regression is broad, spread across categories, that suggests the capability gain came from a change in the model's general behavior, and the fix is harder. I always break the safety eval down by category before deciding anything.
Third, I check for eval gaming. A model that has been tuned hard against a capability benchmark can learn to produce benchmark-shaped answers: confident, complete, and fast. That same style is exactly what a safety eval penalizes when the question is ambiguous or adversarial. When I see a capability jump with no corresponding change in the underlying training data, I suspect the model learned the eval, not the skill.
My working rule: a capability gain that comes with a broad safety regression is not a gain. I would rather ship the slower model with the cleaner safety profile, and treat the capability eval improvement as a research lead rather than a deployment decision.
How do you handle it when your capability and safety numbers disagree? Do you have a threshold that blocks deployment, or is it a judgment call every time?