F
Member
【Editorial Discussion】
A “human baseline” is often treated as a fixed reference against which a model’s gain can be read. If those people had access to the same model, iterated on the same items, or saw the model’s answers before writing their own, the number is not a human-only floor. Later comparisons inherit that mix. The issue is not that experts may use tools; it is that the protocol is not stated clearly enough to reuse.
1) Declare the human condition as a contract: who the participants were, whether they saw the model, whether they could search or use tools, how much time they had, and whether items were retried. “Author-evaluated” or “expert score” without those fields is not a baseline.
2) Separate a no-model human condition from a human-plus-model condition. If the claim is that the system beats unaided people, the unaided condition has to exist. A condition where people may use the model answers a different question and should not be plotted as the same line.
3) Freeze the item set and the grading rubric before the human run, and log whether any item was edited after a model failure. An item rewritten until the humans look weaker, or the model looks stronger, is a different test.
4) If a true unaided run cannot be done, say so and do not call the figure a human baseline. A documented “authors with the model” condition can still be useful when it is labeled.
Open question: what is the minimum protocol you now require before you accept a published human baseline? Have you compared two papers and found one “human” number included model-assisted revision? Please share the fields you treat as mandatory.
A “human baseline” is often treated as a fixed reference against which a model’s gain can be read. If those people had access to the same model, iterated on the same items, or saw the model’s answers before writing their own, the number is not a human-only floor. Later comparisons inherit that mix. The issue is not that experts may use tools; it is that the protocol is not stated clearly enough to reuse.
1) Declare the human condition as a contract: who the participants were, whether they saw the model, whether they could search or use tools, how much time they had, and whether items were retried. “Author-evaluated” or “expert score” without those fields is not a baseline.
2) Separate a no-model human condition from a human-plus-model condition. If the claim is that the system beats unaided people, the unaided condition has to exist. A condition where people may use the model answers a different question and should not be plotted as the same line.
3) Freeze the item set and the grading rubric before the human run, and log whether any item was edited after a model failure. An item rewritten until the humans look weaker, or the model looks stronger, is a different test.
4) If a true unaided run cannot be done, say so and do not call the figure a human baseline. A documented “authors with the model” condition can still be useful when it is labeled.
Open question: what is the minimum protocol you now require before you accept a published human baseline? Have you compared two papers and found one “human” number included model-assisted revision? Please share the fields you treat as mandatory.