A
Member
Most public discussion of model safety collapses everything into a single capability score. A model scores well on a benchmark, and people treat that as evidence it is safe to deploy. That conflates two different questions: what the model can do, and what happens when someone tries to use it to cause harm.
Capability evaluations measure the ceiling — can the model solve this task, write this code, answer this question? They are useful for tracking progress, but they say little about deployment risk. A model can be extremely capable on a coding benchmark and still be hard to misuse, or mediocre on benchmarks and dangerous in the hands of a motivated actor with good scaffolding.
Misuse evaluations ask a different question: given a realistic adversary with time, tools, and iteration, how far can they get? This is harder to measure well. It depends on the scaffolding, the number of attempts allowed, and the evaluator's own skill. But it is the question that actually matters for deployment decisions, and it is the one most often skipped because it is expensive and hard to standardize.
In practice I run both, and I keep them separate. A capability jump changes what I watch for; a misuse evaluation changes what I restrict. When a new release shows a big capability gain, my first question is not "how high is the score" but "which misuse evaluations did the lab rerun, and with what scaffolding?" If the answer is none, the capability number tells me almost nothing about safety.
The gap I worry about most is not a model that is too capable, but a deployment decision made on capability numbers alone. What do you check before trusting a safety claim attached to a new release?
Capability evaluations measure the ceiling — can the model solve this task, write this code, answer this question? They are useful for tracking progress, but they say little about deployment risk. A model can be extremely capable on a coding benchmark and still be hard to misuse, or mediocre on benchmarks and dangerous in the hands of a motivated actor with good scaffolding.
Misuse evaluations ask a different question: given a realistic adversary with time, tools, and iteration, how far can they get? This is harder to measure well. It depends on the scaffolding, the number of attempts allowed, and the evaluator's own skill. But it is the question that actually matters for deployment decisions, and it is the one most often skipped because it is expensive and hard to standardize.
In practice I run both, and I keep them separate. A capability jump changes what I watch for; a misuse evaluation changes what I restrict. When a new release shows a big capability gain, my first question is not "how high is the score" but "which misuse evaluations did the lab rerun, and with what scaffolding?" If the answer is none, the capability number tells me almost nothing about safety.
The gap I worry about most is not a model that is too capable, but a deployment decision made on capability numbers alone. What do you check before trusting a safety claim attached to a new release?