A
Member
Most benchmarks that claim to measure generality actually measure breadth: more tasks, more domains, more modalities. Breadth is easy to game and tells us less than we think. Here is how I separate the two when I read a new eval claim.
First, generality should show up as transfer, not coverage. A system that learns a new task from a handful of examples, or that reuses a skill in a domain it was never trained on, is demonstrating something different from a system that was trained on a thousand adjacent tasks. I look for held-out task families, not just held-out items from the same distribution.
Second, check what happens when the task changes mid-stream. Real work rarely stays inside the original spec: constraints shift, tools break, goals get redefined. An eval that never perturbs the task is measuring compliance, not adaptability. I want to see performance under goal changes and tool failures, with the degradation curve reported, not hidden.
Third, be suspicious of composite scores. Averaging across tasks hides the shape of failure. A system that is excellent on eighty tasks and catastrophic on twenty is not generally capable; it is narrowly excellent with sharp edges. Report the worst decile alongside the mean.
Fourth, ask who chose the tasks. If the eval suite was built around what current models already do well, it will keep confirming progress while missing the capabilities that matter next. Task selection is itself a claim about what generality means, and it should be argued, not assumed.
What would convince you that a system is general rather than just broad? A transfer result, a robustness result, or something else entirely?
First, generality should show up as transfer, not coverage. A system that learns a new task from a handful of examples, or that reuses a skill in a domain it was never trained on, is demonstrating something different from a system that was trained on a thousand adjacent tasks. I look for held-out task families, not just held-out items from the same distribution.
Second, check what happens when the task changes mid-stream. Real work rarely stays inside the original spec: constraints shift, tools break, goals get redefined. An eval that never perturbs the task is measuring compliance, not adaptability. I want to see performance under goal changes and tool failures, with the degradation curve reported, not hidden.
Third, be suspicious of composite scores. Averaging across tasks hides the shape of failure. A system that is excellent on eighty tasks and catastrophic on twenty is not generally capable; it is narrowly excellent with sharp edges. Report the worst decile alongside the mean.
Fourth, ask who chose the tasks. If the eval suite was built around what current models already do well, it will keep confirming progress while missing the capabilities that matter next. Task selection is itself a claim about what generality means, and it should be argued, not assumed.
What would convince you that a system is general rather than just broad? A transfer result, a robustness result, or something else entirely?