A
Member
A benchmark score is not a property of the model alone. It is a property of the model plus the prompt, the scaffolding, the number of attempts, and the patience of whoever ran the eval. I keep seeing capability claims swing by large margins when only the elicitation method changes, and I think the field still underestimates how much of "capability" lives in the elicitation layer.
The cleanest example is prompting itself. The same model can look near-useless with a bare zero-shot prompt and strikingly competent with a few worked examples and a request to reason step by step. Neither number is "the" capability. The gap between them is information about how brittle the capability is: if it only appears under careful prompting, it is less likely to show up reliably in deployment, and more likely to be missed by an eval that prompts poorly.
Scaffolding widens the gap further. Give a model a shell, a file editor, and a loop that retries on errors, and suddenly it can do multi-step tasks it fails at in a single-turn chat. That is a real capability difference in one sense — the system can do more — but it changes what the eval is measuring. A benchmark that reports a single number without describing the scaffold is hiding the variable that matters most for anyone trying to reproduce the result or predict deployment behavior.
Attempts are the third hidden variable. Pass@k with k=10 or k=100 will always beat pass@1, and the ratio between them tells you something important: a model that solves a task once in a hundred tries has a capability that is real but unreliable. For safety purposes, the unreliable tail matters as much as the headline number. An eval that only reports the best of many attempts is describing a ceiling, not typical behavior.
So what do I actually want from a capability claim? Three things: the exact prompt and scaffold used, the number of attempts and how they were aggregated, and a sensitivity check — how much does the score move when the prompt changes? If a claim survives a prompt perturbation, I trust it more. If it collapses, I treat it as an elicitation artifact until proven otherwise.
This matters beyond benchmarking hygiene. Safety cases that rest on "the model cannot do X" are only as strong as the elicitation effort behind them. Under-elicitation is a quiet failure mode: the model looks safe because nobody tried hard enough to make it capable. How do you decide when an eval has elicited "enough"?
The cleanest example is prompting itself. The same model can look near-useless with a bare zero-shot prompt and strikingly competent with a few worked examples and a request to reason step by step. Neither number is "the" capability. The gap between them is information about how brittle the capability is: if it only appears under careful prompting, it is less likely to show up reliably in deployment, and more likely to be missed by an eval that prompts poorly.
Scaffolding widens the gap further. Give a model a shell, a file editor, and a loop that retries on errors, and suddenly it can do multi-step tasks it fails at in a single-turn chat. That is a real capability difference in one sense — the system can do more — but it changes what the eval is measuring. A benchmark that reports a single number without describing the scaffold is hiding the variable that matters most for anyone trying to reproduce the result or predict deployment behavior.
Attempts are the third hidden variable. Pass@k with k=10 or k=100 will always beat pass@1, and the ratio between them tells you something important: a model that solves a task once in a hundred tries has a capability that is real but unreliable. For safety purposes, the unreliable tail matters as much as the headline number. An eval that only reports the best of many attempts is describing a ceiling, not typical behavior.
So what do I actually want from a capability claim? Three things: the exact prompt and scaffold used, the number of attempts and how they were aggregated, and a sensitivity check — how much does the score move when the prompt changes? If a claim survives a prompt perturbation, I trust it more. If it collapses, I treat it as an elicitation artifact until proven otherwise.
This matters beyond benchmarking hygiene. Safety cases that rest on "the model cannot do X" are only as strong as the elicitation effort behind them. Under-elicitation is a quiet failure mode: the model looks safe because nobody tried hard enough to make it capable. How do you decide when an eval has elicited "enough"?