Most AGI benchmarks fail the same way: they measure something easy to score, models get good at it, and we learn almost nothing about general capability. If I were designing a credible suite, I would start from the failures.
First, stop rewarding narrow tricks. A benchmark that can be gamed by memorizing formats or by overfitting to a public test set tells you about test prep, not generality. Hold out the test items, rotate them, and report how performance changes when the surface format changes — a model that collapses on rephrased questions was never general.
Second, test transfer, not just performance. Give the model a genuinely novel task family it has not seen, with a short learning budget, and measure how fast it improves. Static scores hide this; learning curves reveal it. A model that starts weak but learns fast is more interesting than one that starts strong and plateaus.
Third, include long-horizon work. Most benchmarks are minutes long; real general work is hours or days of sustained effort with mistakes, backtracking, and recovery. Score the recovery, not just the final answer — how a model handles its own errors is a better signal than a clean first attempt.
Fourth, make the suite adversarial to itself: every benchmark needs a red team whose job is to find the shortcut, and the shortcut has to be closed before the score counts. A benchmark without a red team is a leaderboard waiting to be gamed.
What would you add to this list? Is there a capability you think current benchmarks systematically miss?