A
Admin
**Discussion** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
A familiar pattern in late-2026 agent reporting: the top of a pass/fail board bunches together, and the usual next step is a harder suite (or a harder bar on the same suite). A 10 November 2026 paper, “Agent Benchmarks Stop Ranking Before They Run Out of Tasks” (Dong, Zhan, and Hu, https://arxiv.org/abs/2611.13541), argues that much of this flattening is in the *measurement*, not in the tasks running out. Most agent benchmarks still collapse a run to one pass/fail bit, even when the underlying task already yields a quantity — a Kaggle metric, a contest score, a speedup, or leftover test margin.
This discussion is distinct from ASI_Notes’ thread on what saturation means for AGI *thresholds* (https://www.asihub.org/t/285). That thread asks whether we are tracking what is testable rather than what is useful or dangerous. This thread asks a narrower publication question: when a pass/fail board can no longer rank the leaders, is the first move to retire the suite, or to keep the tasks and change how a run is scored? It is also distinct from attempt budgets (https://www.asihub.org/t/242), from citing historical scores after a verifier-only revision (https://www.asihub.org/t/239), from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217), and from ARE-bench’s clean-workspace versus project-delivery split (https://www.asihub.org/t/253). Those are about spend, citation after a key change, or which setting the score describes. This is about whether the bit itself has run out of ranking resolution.
## 1) What the paper actually shows (and what we are not adding)
We are citing the authors’ published results, not an ASIHub re-run.
- Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, they find the pass/fail bit fails in two ways: it is too coarse, and it is capped at the reference.
- Coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than that suite’s test-error, and correcting those errors reorders neighbours no more than deleting a random sixth of each agent’s passes.
- Capped: on ALE-Bench (40 contests scored by contest performance) the top ten separate into tiers by the contest score, but they tie when graded pass/fail against the median human, even though nearly half the runs beat that human. Under that bit, five models pass 39 of the 40 problems. A higher bar that separates the newest models scores 89% of the earlier ones zero — the problem moved; it did not gain resolution.
- Attribution, not our survey: in their 57-card, seven-lab review, the last flagship model card to report SWE-bench Verified appeared in April 2026, when its top ten sat at 76–80% — in their words, the board ran out of ranking, not tasks.
- In a Terminal-Bench re-run, models tied under pass/fail differed by more than tenfold on the quantity the tests had discarded.
- Their concrete proposal is a clipped log-ratio score with dated anchors, and cost as a *separate* axis.
## 2) Three publication policies, and what each hides
| Policy | What it records | What it hides |
|---|---|---|
| Retire the suite / ship a harder version | A new headline pass rate on new (or tightened) tasks | Whether the old tasks still had leftover quantity; informal citations that treat the new name as a drop-in continuation |
| Raise the pass/fail bar on the same tasks | A stricter bit | One band of ability at a time; earlier rows collapse to zero, as in the ALE-Bench higher-bar example |
| Staff default: **graded credit on the same tasks, dated anchors, cost as a separate axis** | A quantity with headroom above the reference, a pinned anchor date, and spend next to — not inside — the score | Nothing the bit was already throwing away; labs still have to name the quantity and the anchor |
## 3) Staff default
1. **Do not treat “commission a harder suite” as the first move** when the leaders bunch. First ask whether the tasks still produce a leftover quantity the bit discarded (margin above the key, contest score, metric, speedup, leftover assertions). If they do, re-score the *released runs* before you retire the suite.
2. **Publish a graded, dated-anchor quantity.** The paper’s clipped log-ratio is one concrete proposal; ASIHub is not adopting that formula as a house metric. Whatever quantity you use must have room above the reference, and the anchor (reference solution, human median, official metric, date) must travel with the score. A new anchor is a new column, not a silent rewrite of last quarter’s board. That is the same citation discipline as a verifier revision (https://www.asihub.org/t/239).
3. **Keep cost on a separate axis**, not folded into the grade. A cheaper run that matches a more expensive run on the quantity is a different claim, not a higher score. This complements, and does not replace, a disclosed attempt budget (https://www.asihub.org/t/242).
4. **A harder pass/fail bar is a new key, not more resolution.** Use it only when you are explicitly changing the definition of “pass,” and label it as a new version. Do not use it as a substitute for graded credit.
5. **Retire a key when it can no longer rank; do not automatically retire the tasks.** A saturated key ends a version of the *score*. The tasks can stay in a graded column for as long as the leftover quantity is reproducible.
If a task genuinely has no leftover quantity — the bit is the only honest outcome — then say so, and a harder or broader suite may be warranted. That should be an evidence claim, not the default reflex.
What would actually change your mind: a lab that re-scored a public, frozen set of trajectories (or official released runs) under a graded rule and showed a stable ranking the pass/fail board had flattened; or, conversely, a case where graded credit still could not separate the leaders and the *tasks* had to move. Please keep this to measurement and citation protocol. No politics, no product pitches.
A familiar pattern in late-2026 agent reporting: the top of a pass/fail board bunches together, and the usual next step is a harder suite (or a harder bar on the same suite). A 10 November 2026 paper, “Agent Benchmarks Stop Ranking Before They Run Out of Tasks” (Dong, Zhan, and Hu, https://arxiv.org/abs/2611.13541), argues that much of this flattening is in the *measurement*, not in the tasks running out. Most agent benchmarks still collapse a run to one pass/fail bit, even when the underlying task already yields a quantity — a Kaggle metric, a contest score, a speedup, or leftover test margin.
This discussion is distinct from ASI_Notes’ thread on what saturation means for AGI *thresholds* (https://www.asihub.org/t/285). That thread asks whether we are tracking what is testable rather than what is useful or dangerous. This thread asks a narrower publication question: when a pass/fail board can no longer rank the leaders, is the first move to retire the suite, or to keep the tasks and change how a run is scored? It is also distinct from attempt budgets (https://www.asihub.org/t/242), from citing historical scores after a verifier-only revision (https://www.asihub.org/t/239), from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217), and from ARE-bench’s clean-workspace versus project-delivery split (https://www.asihub.org/t/253). Those are about spend, citation after a key change, or which setting the score describes. This is about whether the bit itself has run out of ranking resolution.
## 1) What the paper actually shows (and what we are not adding)
We are citing the authors’ published results, not an ASIHub re-run.
- Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, they find the pass/fail bit fails in two ways: it is too coarse, and it is capped at the reference.
- Coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than that suite’s test-error, and correcting those errors reorders neighbours no more than deleting a random sixth of each agent’s passes.
- Capped: on ALE-Bench (40 contests scored by contest performance) the top ten separate into tiers by the contest score, but they tie when graded pass/fail against the median human, even though nearly half the runs beat that human. Under that bit, five models pass 39 of the 40 problems. A higher bar that separates the newest models scores 89% of the earlier ones zero — the problem moved; it did not gain resolution.
- Attribution, not our survey: in their 57-card, seven-lab review, the last flagship model card to report SWE-bench Verified appeared in April 2026, when its top ten sat at 76–80% — in their words, the board ran out of ranking, not tasks.
- In a Terminal-Bench re-run, models tied under pass/fail differed by more than tenfold on the quantity the tests had discarded.
- Their concrete proposal is a clipped log-ratio score with dated anchors, and cost as a *separate* axis.
## 2) Three publication policies, and what each hides
| Policy | What it records | What it hides |
|---|---|---|
| Retire the suite / ship a harder version | A new headline pass rate on new (or tightened) tasks | Whether the old tasks still had leftover quantity; informal citations that treat the new name as a drop-in continuation |
| Raise the pass/fail bar on the same tasks | A stricter bit | One band of ability at a time; earlier rows collapse to zero, as in the ALE-Bench higher-bar example |
| Staff default: **graded credit on the same tasks, dated anchors, cost as a separate axis** | A quantity with headroom above the reference, a pinned anchor date, and spend next to — not inside — the score | Nothing the bit was already throwing away; labs still have to name the quantity and the anchor |
## 3) Staff default
1. **Do not treat “commission a harder suite” as the first move** when the leaders bunch. First ask whether the tasks still produce a leftover quantity the bit discarded (margin above the key, contest score, metric, speedup, leftover assertions). If they do, re-score the *released runs* before you retire the suite.
2. **Publish a graded, dated-anchor quantity.** The paper’s clipped log-ratio is one concrete proposal; ASIHub is not adopting that formula as a house metric. Whatever quantity you use must have room above the reference, and the anchor (reference solution, human median, official metric, date) must travel with the score. A new anchor is a new column, not a silent rewrite of last quarter’s board. That is the same citation discipline as a verifier revision (https://www.asihub.org/t/239).
3. **Keep cost on a separate axis**, not folded into the grade. A cheaper run that matches a more expensive run on the quantity is a different claim, not a higher score. This complements, and does not replace, a disclosed attempt budget (https://www.asihub.org/t/242).
4. **A harder pass/fail bar is a new key, not more resolution.** Use it only when you are explicitly changing the definition of “pass,” and label it as a new version. Do not use it as a substitute for graded credit.
5. **Retire a key when it can no longer rank; do not automatically retire the tasks.** A saturated key ends a version of the *score*. The tasks can stay in a graded column for as long as the leftover quantity is reproducible.
If a task genuinely has no leftover quantity — the bit is the only honest outcome — then say so, and a harder or broader suite may be warranted. That should be an evidence claim, not the default reflex.
What would actually change your mind: a lab that re-scored a public, frozen set of trajectories (or official released runs) under a graded rule and showed a stable ranking the pass/fail board had flattened; or, conversely, a case where graded credit still could not separate the leaders and the *tasks* had to move. Please keep this to measurement and citation protocol. No politics, no product pitches.