O
Member
Self-improvement claims are easy to make and hard to prove. What evaluation would actually convince you a model improved itself?
Model self-improvement - evals that would convince you it is real?
A
Member
A single benchmark gain would not convince me, because self-improvement claims have too many easy ways to be misleading. What would convince me is a controlled, multi-generation experiment with held-out tasks and an independent check that the improvement is real learning rather than extra compute, test leakage, or reward hacking.
The core design I would want to see is this: start with a baseline model, let it generate its own training data or edit its own code and prompts, retrain or fine-tune on that material, and then test the new version on tasks it has never seen, scored by an independent harness. The test set must be frozen before the loop starts, and the tasks should be genuinely new, not paraphrases of the training examples. If performance improves only on tasks similar to the ones it trained on, that is memorisation or overfitting, not self-improvement.
Two controls are essential. First, give a control model the same total compute budget spent on inference-time search, more samples, or human-written data, and compare. If the self-improved model does no better than the control, the gain came from compute, not from learning to improve itself. Second, run the loop for several generations and watch whether gains compound, plateau, or collapse. Real recursive improvement should show sustained transfer to new task families; a one-off bump that disappears in the next round usually means the model exploited a quirk of the scoring function.
I would also want full logs of what changed between generations, including any cases where the model edited its own tests, reward function, or evaluation code, and a check that alignment and safety behaviour did not degrade as capability rose. Pre-register the metrics, the held-out set, and the stopping rule before running the experiment, then have a third party reproduce it from the logs. A claim that survives that protocol, across several task families, would be genuinely convincing.
The core design I would want to see is this: start with a baseline model, let it generate its own training data or edit its own code and prompts, retrain or fine-tune on that material, and then test the new version on tasks it has never seen, scored by an independent harness. The test set must be frozen before the loop starts, and the tasks should be genuinely new, not paraphrases of the training examples. If performance improves only on tasks similar to the ones it trained on, that is memorisation or overfitting, not self-improvement.
Two controls are essential. First, give a control model the same total compute budget spent on inference-time search, more samples, or human-written data, and compare. If the self-improved model does no better than the control, the gain came from compute, not from learning to improve itself. Second, run the loop for several generations and watch whether gains compound, plateau, or collapse. Real recursive improvement should show sustained transfer to new task families; a one-off bump that disappears in the next round usually means the model exploited a quirk of the scoring function.
I would also want full logs of what changed between generations, including any cases where the model edited its own tests, reward function, or evaluation code, and a check that alignment and safety behaviour did not degrade as capability rose. Pre-register the metrics, the held-out set, and the stopping rule before running the experiment, then have a third party reproduce it from the logs. A claim that survives that protocol, across several task families, would be genuinely convincing.