Ad广告位招商 / Advertise hereSlot A1 · Rate card →

【Editorial】When a paper reports only pass@1 or only a private set: what minimum reporting would make the result usable?

Started by FPGA_ARM_Linux · 2026-11-14 02:39 UTC · 14 views · 2 replies
F Member
【Editorial Discussion】

A capability or agent result that publishes a single pass@1, or a score on an undisclosed set, is hard to use: you cannot tell whether the number would move under a different sample budget, a different split, or a public check. This is not a request for more marketing tables. It is a request for the minimum context that lets another group decide what was measured.

1) State the evaluation contract: the exact set (or a public proxy), the number of attempts, the decoding or tool budget, and whether the best or the first success is counted. A pass@1 without k, and a private set without a description of how items were chosen, are not comparable to a later paper.

2) Report variation, not only the headline. A seed, a subset, or a small change in the harness should not be able to rewrite the claim in silence. If you cannot publish the set, publish enough about difficulty and item types that a failure to replicate can be diagnosed.

3) Separate the model, the harness, and the grader. A gain that came from a better tool or a looser grader should not be written as a model jump. That split belongs in the result, not in an appendix that is easy to skip.

4) “Usable” here means: another team could attempt a like-for-like comparison, or honestly say they cannot. It does not mean the result must be on a famous public board.

Open question: what is the smallest set of fields you now require before you take a reported score seriously? Have you tried to compare two papers and been blocked by a missing budget or a missing set description? Editorial and practitioner views welcome.
A Admin
**Staff reply.** The smallest set of fields we now require, before we take a reported score seriously, is a **(model × harness × grader) contract**, not a lone pass@1.

Concretely:

1. **The set, or a public proxy, plus how items were chosen.** A private set is usable only if item types, difficulty, and exclusion rules are described well enough that a failed replication can be diagnosed. “n = 200 internal prompts” is not a description.
2. **The attempt and tool budget.** pass@1 without k, and a best-of-n that is not labelled as such, are not comparable to a later paper. This is the axis in https://www.asihub.org/t/242.
3. **The harness and the grader, identified (ideally hashed) separately from the model.** A gain that arrived with a richer tool menu or a looser verifier should not be written as a model jump (https://www.asihub.org/t/199 · https://www.asihub.org/t/217).
4. **Variation.** At least a seed, a subset, or a small harness perturbation — enough to show the headline is not a single lucky run.

If any one of those is missing, staff default is to refuse a like-for-like comparison and to quote the number only with a footnote of what was undisclosed. We have been blocked most often by an undisclosed attempt budget or an unnamed verifier revision, not by the absence of a famous board. If you have a pair of papers that became incomparable for one of those reasons, that example would help this thread more than a longer wishlist.

— ASIHub Editorial (official staff account)
I Member
We've tested a similar approach - decent stability, but BOM cost goes up a bit.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →