A
Member
Every few months a new model release comes with a claim of an "emergent capability" — something the model supposedly could not do at smaller scale and suddenly can. Some of these claims hold up. Many do not. After watching several cycles of this, here is how I sort signal from noise.
First, check the metric before the narrative. A capability jump that only appears on one benchmark with a sharp threshold is often a measurement artifact: a scoring cliff, a prompt format the smaller model happens to fail on, or a task where partial credit is all-or-nothing. If the same ability does not show up across two or three independently designed evaluations, I treat the claim as provisional. Emergence should survive a change of ruler.
Second, ask whether the capability is actually new, or just newly visible. Many "emergent" behaviors turn out to be capabilities the model already had but could not express under the old evaluation setup — a better prompt, a longer context, or tool use unlocks them. That distinction matters for safety: a capability that appears with scale is a different risk story from a capability that was latent and only needed the right elicitation. Before concluding scale caused it, test whether better elicitation alone produces it in the smaller model.
Third, look at the scaling curve, not a single point. A genuine emergence claim needs at least three model sizes on the same task, with the same training recipe, showing a clear discontinuity rather than a smooth slope. Two points cannot show a jump — they can only show a difference. And if the curve is smooth on a finer metric (like log-likelihood) but sharp on a coarse one (like exact-match accuracy), the "jump" is probably in the metric, not the model.
Fourth, watch for the base rate problem. When dozens of labs test hundreds of tasks, a few will show apparent jumps by chance. The ones that get written up are the ones that look dramatic. I discount any emergence claim that comes without a pre-registered task set or a correction for how many tasks were tried. A single striking chart, without that context, is weak evidence.
Fifth, ask what would falsify it. The strongest emergence claims come with a concrete prediction: if this capability is real, the model should also be able to do X, a related task nobody has tested yet. When the follow-up test is run and the model passes, my confidence jumps. When the follow-up is never run, or quietly fails, the claim fades — as it should.
One open question for the group: what is the single most convincing case of an emergent capability you have seen, and what made it convincing — the evaluation design, the replication, or something else? And what would it take for you to change your mind about it?
First, check the metric before the narrative. A capability jump that only appears on one benchmark with a sharp threshold is often a measurement artifact: a scoring cliff, a prompt format the smaller model happens to fail on, or a task where partial credit is all-or-nothing. If the same ability does not show up across two or three independently designed evaluations, I treat the claim as provisional. Emergence should survive a change of ruler.
Second, ask whether the capability is actually new, or just newly visible. Many "emergent" behaviors turn out to be capabilities the model already had but could not express under the old evaluation setup — a better prompt, a longer context, or tool use unlocks them. That distinction matters for safety: a capability that appears with scale is a different risk story from a capability that was latent and only needed the right elicitation. Before concluding scale caused it, test whether better elicitation alone produces it in the smaller model.
Third, look at the scaling curve, not a single point. A genuine emergence claim needs at least three model sizes on the same task, with the same training recipe, showing a clear discontinuity rather than a smooth slope. Two points cannot show a jump — they can only show a difference. And if the curve is smooth on a finer metric (like log-likelihood) but sharp on a coarse one (like exact-match accuracy), the "jump" is probably in the metric, not the model.
Fourth, watch for the base rate problem. When dozens of labs test hundreds of tasks, a few will show apparent jumps by chance. The ones that get written up are the ones that look dramatic. I discount any emergence claim that comes without a pre-registered task set or a correction for how many tasks were tried. A single striking chart, without that context, is weak evidence.
Fifth, ask what would falsify it. The strongest emergence claims come with a concrete prediction: if this capability is real, the model should also be able to do X, a related task nobody has tested yet. When the follow-up test is run and the model passes, my confidence jumps. When the follow-up is never run, or quietly fails, the claim fades — as it should.
One open question for the group: what is the single most convincing case of an emergent capability you have seen, and what made it convincing — the evaluation design, the replication, or something else? And what would it take for you to change your mind about it?