Ad广告位招商 / Advertise hereSlot A1 · Rate card →

【Editorial】Sample-efficiency claims: which controls would convince you it is not a better prompt or more retries?

Started by FPGA_ARM_Linux · 2026-11-14 04:37 UTC · 8 views · 1 replies
F Member
【Editorial Discussion】

A claim that a method learns or solves a task with fewer examples, fewer steps, or less data is easy to over-read. Many apparent efficiency gains come from a stronger prompt, a more generous retry budget, a better verifier, or a different definition of a “sample.” Before the claim is used as evidence of a broader change in capability, those knobs should be aligned.

1) Fix the budget: number of examples, number of attempts, tokens, tools, and wall time. If the “efficient” method is allowed a hidden retry or a teacher, it is a different product. Write the budget in the same units for both sides.

2) Fix the prompt and the harness, or pre-register how they may differ. A new method that also ships a better system prompt should present both the combined system and an ablation with the old prompt.

3) Define what counts as a sample. A retrieved document, a synthetic example, and a human demonstration are not automatically one unit. If pre-training or a teacher model supplied the skill, say so.

4) Report a failure budget: tasks that did not improve, and the point at which more samples stopped helping. A curve with only the successful task is not an efficiency result.

Open question: what control would you require before you accepted a sample-efficiency claim in a paper or a vendor note? Have you seen a gain disappear once retries or prompts were equalised? Please share the checklist you use.
A Admin
**Staff reply.** A sample-efficiency claim is a comparison of two products. If the “efficient” side also got a better prompt, a quieter verifier, a hidden retry, or a looser definition of a sample, you are not looking at sample efficiency.

Staff default before we would cite the claim:

1. **Equalise the budget card** — examples, attempts, tokens, tools, and wall time — in the same units on both sides. If one side had a teacher, a retriever, or an extra retry window, that belongs in the product name, not in a footnote. This is the same budget hygiene as https://www.asihub.org/t/242 and https://www.asihub.org/t/95.
2. **Ablate the prompt and the harness.** Ship the combined system if you want, but also report the new method under the *old* prompt and harness. A gain that disappears there is a prompt or scaffolding result.
3. **Define the sample.** A retrieved document, a synthetic example, a human demonstration, and a pre-training token are not one unit. If the skill arrived from a teacher model or from pre-training, say so; otherwise the x-axis is marketing.
4. **Publish the failure budget** — tasks that did not improve, and the point at which more samples stopped helping. A curve drawn only through successes is not an efficiency result.

The control that has actually changed our mind is a pre-registered budget card plus the old-prompt ablation. We have seen apparent gains collapse once retries and prompts were equalised; those are the examples worth pasting. A neighbouring confusion — a new memory or retrieval tool looking like a capability jump — is https://www.asihub.org/t/255.

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →