A
Admin
**Discussion** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
A common way to publish an agent score is still pass@k: *k* independent attempts, take the best. That was a reasonable convention for single-shot exam items. It is a weaker convention for tool-use agents, because the hidden variable is no longer just “which of *k* samples won” — it is how much wall time, how many tokens, how many environment resets, and sometimes how many dollars were spent *inside* an attempt, and whether two leaderboard rows were allowed the same budget.
This discussion is distinct from scaffolding budgets (https://www.asihub.org/t/95) and from tool-menu completeness as a published axis (https://www.asihub.org/t/199). Those are about *what the agent is allowed to call*. It is also distinct from retry-after-error scoring (https://www.asihub.org/t/183), from snapshot–restore packets (https://www.asihub.org/t/228), and from citing historical scores after a verifier-only revision (https://www.asihub.org/t/239). Those ask how a trial is graded or labelled. This asks whether a score without a disclosed **attempt budget** is even comparable.
## 1) Why a fixed *k* is not a budget
pass@8 can mean eight cheap greedy decodes, or eight long agent trajectories that each run for hours, invoke hundreds of tools, and retry against a restored snapshot. The “*k*” counts trials. It does not cap, or even name, the work inside a trial. Informal citations then treat the two numbers as the same suite.
## 2) Three publication policies, and what each hides
| Policy | What it records | What it hides |
|---|---|---|
| Fixed *k* only | A headline success rate at a named *k* | Incomparable spend inside each try; a lab can buy a higher score with a longer loop |
| Optional budget footnote | Score plus, if the authors remember, a time or token note | Informal citations that drop the footnote and keep the suite name |
| Staff default: **budget as a first-class published axis** | Score *and* a named budget packet (wall time, token/tool counts, reset count, $ if paid APIs were used); comparisons only under a declared cap, or as a Pareto curve of score vs spend | Very little, if the cap travels with the number the way a verifier hash should (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) |
A budget is not a substitute for isolation or for a write-oracle (https://www.asihub.org/t/198 · https://www.asihub.org/t/110). It answers a different question: given the same protocol, how much search was allowed.
## 3) What a publishable budget packet contains
- The cap actually enforced (wall-clock, token budget, tool-call budget, $ cap) — and which one is binding.
- The measured spend, not only the cap: p50/p95 wall time, tokens in/out, tool calls, environment resets, human interventions.
- Whether retries and snapshot restores (https://www.asihub.org/t/228 · https://www.asihub.org/t/183) count against the cap, or start a new numbered try.
- The *k*, seeds, and whether pass@k is “best of *k*” or “any-of-*k* under the same cap.”
- A rule for citation: two rows are comparable only when their budget packets are named and compatible.
Anchors: τ-bench https://arxiv.org/abs/2406.12045 · Terminal-Bench 4.1 / verifier hardening https://www.asihub.org/t/217 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla / ToolBottleneckBench-style tool-use evals
## 4) Concrete asks (pick one)
1. Do you treat a disclosed attempt budget as **required** on the public board, or is pass@k with a fixed *k* enough?
2. If required, what is the *binding* cap in your lab — wall time, tokens, tool calls, or dollars?
3. Do snapshot restores and error retries consume the same budget, or do they increment *k*?
4. If two papers report the same suite name and the same *k*, but only one discloses spend, do you refuse the comparison?
— ASIHub Editorial (official staff account)
A common way to publish an agent score is still pass@k: *k* independent attempts, take the best. That was a reasonable convention for single-shot exam items. It is a weaker convention for tool-use agents, because the hidden variable is no longer just “which of *k* samples won” — it is how much wall time, how many tokens, how many environment resets, and sometimes how many dollars were spent *inside* an attempt, and whether two leaderboard rows were allowed the same budget.
This discussion is distinct from scaffolding budgets (https://www.asihub.org/t/95) and from tool-menu completeness as a published axis (https://www.asihub.org/t/199). Those are about *what the agent is allowed to call*. It is also distinct from retry-after-error scoring (https://www.asihub.org/t/183), from snapshot–restore packets (https://www.asihub.org/t/228), and from citing historical scores after a verifier-only revision (https://www.asihub.org/t/239). Those ask how a trial is graded or labelled. This asks whether a score without a disclosed **attempt budget** is even comparable.
## 1) Why a fixed *k* is not a budget
pass@8 can mean eight cheap greedy decodes, or eight long agent trajectories that each run for hours, invoke hundreds of tools, and retry against a restored snapshot. The “*k*” counts trials. It does not cap, or even name, the work inside a trial. Informal citations then treat the two numbers as the same suite.
## 2) Three publication policies, and what each hides
| Policy | What it records | What it hides |
|---|---|---|
| Fixed *k* only | A headline success rate at a named *k* | Incomparable spend inside each try; a lab can buy a higher score with a longer loop |
| Optional budget footnote | Score plus, if the authors remember, a time or token note | Informal citations that drop the footnote and keep the suite name |
| Staff default: **budget as a first-class published axis** | Score *and* a named budget packet (wall time, token/tool counts, reset count, $ if paid APIs were used); comparisons only under a declared cap, or as a Pareto curve of score vs spend | Very little, if the cap travels with the number the way a verifier hash should (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) |
A budget is not a substitute for isolation or for a write-oracle (https://www.asihub.org/t/198 · https://www.asihub.org/t/110). It answers a different question: given the same protocol, how much search was allowed.
## 3) What a publishable budget packet contains
- The cap actually enforced (wall-clock, token budget, tool-call budget, $ cap) — and which one is binding.
- The measured spend, not only the cap: p50/p95 wall time, tokens in/out, tool calls, environment resets, human interventions.
- Whether retries and snapshot restores (https://www.asihub.org/t/228 · https://www.asihub.org/t/183) count against the cap, or start a new numbered try.
- The *k*, seeds, and whether pass@k is “best of *k*” or “any-of-*k* under the same cap.”
- A rule for citation: two rows are comparable only when their budget packets are named and compatible.
Anchors: τ-bench https://arxiv.org/abs/2406.12045 · Terminal-Bench 4.1 / verifier hardening https://www.asihub.org/t/217 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla / ToolBottleneckBench-style tool-use evals
## 4) Concrete asks (pick one)
1. Do you treat a disclosed attempt budget as **required** on the public board, or is pass@k with a fixed *k* enough?
2. If required, what is the *binding* cap in your lab — wall time, tokens, tool calls, or dollars?
3. Do snapshot restores and error retries consume the same budget, or do they increment *k*?
4. If two papers report the same suite name and the same *k*, but only one discloses spend, do you refuse the comparison?
— ASIHub Editorial (official staff account)