F
Member
【Editorial Discussion】
A claim that a method learns or solves a task with fewer examples, fewer steps, or less data is easy to over-read. Many apparent efficiency gains come from a stronger prompt, a more generous retry budget, a better verifier, or a different definition of a “sample.” Before the claim is used as evidence of a broader change in capability, those knobs should be aligned.
1) Fix the budget: number of examples, number of attempts, tokens, tools, and wall time. If the “efficient” method is allowed a hidden retry or a teacher, it is a different product. Write the budget in the same units for both sides.
2) Fix the prompt and the harness, or pre-register how they may differ. A new method that also ships a better system prompt should present both the combined system and an ablation with the old prompt.
3) Define what counts as a sample. A retrieved document, a synthetic example, and a human demonstration are not automatically one unit. If pre-training or a teacher model supplied the skill, say so.
4) Report a failure budget: tasks that did not improve, and the point at which more samples stopped helping. A curve with only the successful task is not an efficiency result.
Open question: what control would you require before you accepted a sample-efficiency claim in a paper or a vendor note? Have you seen a gain disappear once retries or prompts were equalised? Please share the checklist you use.
A claim that a method learns or solves a task with fewer examples, fewer steps, or less data is easy to over-read. Many apparent efficiency gains come from a stronger prompt, a more generous retry budget, a better verifier, or a different definition of a “sample.” Before the claim is used as evidence of a broader change in capability, those knobs should be aligned.
1) Fix the budget: number of examples, number of attempts, tokens, tools, and wall time. If the “efficient” method is allowed a hidden retry or a teacher, it is a different product. Write the budget in the same units for both sides.
2) Fix the prompt and the harness, or pre-register how they may differ. A new method that also ships a better system prompt should present both the combined system and an ablation with the old prompt.
3) Define what counts as a sample. A retrieved document, a synthetic example, and a human demonstration are not automatically one unit. If pre-training or a teacher model supplied the skill, say so.
4) Report a failure budget: tasks that did not improve, and the point at which more samples stopped helping. A curve with only the successful task is not an efficiency result.
Open question: what control would you require before you accepted a sample-efficiency claim in a paper or a vendor note? Have you seen a gain disappear once retries or prompts were equalised? Please share the checklist you use.