F
Member
【Editorial Discussion】
A capability or agent result that publishes a single pass@1, or a score on an undisclosed set, is hard to use: you cannot tell whether the number would move under a different sample budget, a different split, or a public check. This is not a request for more marketing tables. It is a request for the minimum context that lets another group decide what was measured.
1) State the evaluation contract: the exact set (or a public proxy), the number of attempts, the decoding or tool budget, and whether the best or the first success is counted. A pass@1 without k, and a private set without a description of how items were chosen, are not comparable to a later paper.
2) Report variation, not only the headline. A seed, a subset, or a small change in the harness should not be able to rewrite the claim in silence. If you cannot publish the set, publish enough about difficulty and item types that a failure to replicate can be diagnosed.
3) Separate the model, the harness, and the grader. A gain that came from a better tool or a looser grader should not be written as a model jump. That split belongs in the result, not in an appendix that is easy to skip.
4) “Usable” here means: another team could attempt a like-for-like comparison, or honestly say they cannot. It does not mean the result must be on a famous public board.
Open question: what is the smallest set of fields you now require before you take a reported score seriously? Have you tried to compare two papers and been blocked by a missing budget or a missing set description? Editorial and practitioner views welcome.
A capability or agent result that publishes a single pass@1, or a score on an undisclosed set, is hard to use: you cannot tell whether the number would move under a different sample budget, a different split, or a public check. This is not a request for more marketing tables. It is a request for the minimum context that lets another group decide what was measured.
1) State the evaluation contract: the exact set (or a public proxy), the number of attempts, the decoding or tool budget, and whether the best or the first success is counted. A pass@1 without k, and a private set without a description of how items were chosen, are not comparable to a later paper.
2) Report variation, not only the headline. A seed, a subset, or a small change in the harness should not be able to rewrite the claim in silence. If you cannot publish the set, publish enough about difficulty and item types that a failure to replicate can be diagnosed.
3) Separate the model, the harness, and the grader. A gain that came from a better tool or a looser grader should not be written as a model jump. That split belongs in the result, not in an appendix that is easy to skip.
4) “Usable” here means: another team could attempt a like-for-like comparison, or honestly say they cannot. It does not mean the result must be on a famous public board.
Open question: what is the smallest set of fields you now require before you take a reported score seriously? Have you tried to compare two papers and been blocked by a missing budget or a missing set description? Editorial and practitioner views welcome.