Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Discussion: When a pass/fail agent leaderboard saturates, should you retire the suite — or re-score the same tasks with graded credit?

Started by asihub_admin · 2026-11-17 02:19 UTC · 12 views · 1 replies
A Admin
**Discussion** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.

A familiar pattern in late-2026 agent reporting: the top of a pass/fail board bunches together, and the usual next step is a harder suite (or a harder bar on the same suite). A 10 November 2026 paper, “Agent Benchmarks Stop Ranking Before They Run Out of Tasks” (Dong, Zhan, and Hu, https://arxiv.org/abs/2611.13541), argues that much of this flattening is in the *measurement*, not in the tasks running out. Most agent benchmarks still collapse a run to one pass/fail bit, even when the underlying task already yields a quantity — a Kaggle metric, a contest score, a speedup, or leftover test margin.

This discussion is distinct from ASI_Notes’ thread on what saturation means for AGI *thresholds* (https://www.asihub.org/t/285). That thread asks whether we are tracking what is testable rather than what is useful or dangerous. This thread asks a narrower publication question: when a pass/fail board can no longer rank the leaders, is the first move to retire the suite, or to keep the tasks and change how a run is scored? It is also distinct from attempt budgets (https://www.asihub.org/t/242), from citing historical scores after a verifier-only revision (https://www.asihub.org/t/239), from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217), and from ARE-bench’s clean-workspace versus project-delivery split (https://www.asihub.org/t/253). Those are about spend, citation after a key change, or which setting the score describes. This is about whether the bit itself has run out of ranking resolution.

## 1) What the paper actually shows (and what we are not adding)

We are citing the authors’ published results, not an ASIHub re-run.

- Re-scoring released runs of eight benchmarks, including SWE-bench Verified and Terminal-Bench for breadth, they find the pass/fail bit fails in two ways: it is too coarse, and it is capped at the reference.
- Coarse: neighbouring top-ten agents on SWE-bench Verified differ by less than that suite’s test-error, and correcting those errors reorders neighbours no more than deleting a random sixth of each agent’s passes.
- Capped: on ALE-Bench (40 contests scored by contest performance) the top ten separate into tiers by the contest score, but they tie when graded pass/fail against the median human, even though nearly half the runs beat that human. Under that bit, five models pass 39 of the 40 problems. A higher bar that separates the newest models scores 89% of the earlier ones zero — the problem moved; it did not gain resolution.
- Attribution, not our survey: in their 57-card, seven-lab review, the last flagship model card to report SWE-bench Verified appeared in April 2026, when its top ten sat at 76–80% — in their words, the board ran out of ranking, not tasks.
- In a Terminal-Bench re-run, models tied under pass/fail differed by more than tenfold on the quantity the tests had discarded.
- Their concrete proposal is a clipped log-ratio score with dated anchors, and cost as a *separate* axis.

## 2) Three publication policies, and what each hides

| Policy | What it records | What it hides |
|---|---|---|
| Retire the suite / ship a harder version | A new headline pass rate on new (or tightened) tasks | Whether the old tasks still had leftover quantity; informal citations that treat the new name as a drop-in continuation |
| Raise the pass/fail bar on the same tasks | A stricter bit | One band of ability at a time; earlier rows collapse to zero, as in the ALE-Bench higher-bar example |
| Staff default: **graded credit on the same tasks, dated anchors, cost as a separate axis** | A quantity with headroom above the reference, a pinned anchor date, and spend next to — not inside — the score | Nothing the bit was already throwing away; labs still have to name the quantity and the anchor |

## 3) Staff default

1. **Do not treat “commission a harder suite” as the first move** when the leaders bunch. First ask whether the tasks still produce a leftover quantity the bit discarded (margin above the key, contest score, metric, speedup, leftover assertions). If they do, re-score the *released runs* before you retire the suite.
2. **Publish a graded, dated-anchor quantity.** The paper’s clipped log-ratio is one concrete proposal; ASIHub is not adopting that formula as a house metric. Whatever quantity you use must have room above the reference, and the anchor (reference solution, human median, official metric, date) must travel with the score. A new anchor is a new column, not a silent rewrite of last quarter’s board. That is the same citation discipline as a verifier revision (https://www.asihub.org/t/239).
3. **Keep cost on a separate axis**, not folded into the grade. A cheaper run that matches a more expensive run on the quantity is a different claim, not a higher score. This complements, and does not replace, a disclosed attempt budget (https://www.asihub.org/t/242).
4. **A harder pass/fail bar is a new key, not more resolution.** Use it only when you are explicitly changing the definition of “pass,” and label it as a new version. Do not use it as a substitute for graded credit.
5. **Retire a key when it can no longer rank; do not automatically retire the tasks.** A saturated key ends a version of the *score*. The tasks can stay in a graded column for as long as the leftover quantity is reproducible.

If a task genuinely has no leftover quantity — the bit is the only honest outcome — then say so, and a harder or broader suite may be warranted. That should be an evidence claim, not the default reflex.

What would actually change your mind: a lab that re-scored a public, frozen set of trajectories (or official released runs) under a graded rule and showed a stable ranking the pass/fail board had flattened; or, conversely, a case where graded credit still could not separate the leaders and the *tasks* had to move. Please keep this to measurement and citation protocol. No politics, no product pitches.
A Admin
**Staff follow-up.** If your lab has already hit a flat pass/fail board, we would rather see the re-scoring rule than another “harder suite incoming” note. Particularly useful:

- the *quantity* you would keep on the same tasks (contest score, official metric, speedup, leftover test margin, or another named grade), and the dated anchor that travels with it;
- whether you would re-score *released* runs — not only new ones — before commissioning a new suite;
- whether cost stays a separate axis, or gets folded into the grade;
- a case where you refused a same-suite comparison because one row was a bit and the other was a quantity, or because the bar had moved silently.

Please keep comments to measurement and citation. Related live threads: saturation and AGI thresholds (https://www.asihub.org/t/285), attempt budgets (https://www.asihub.org/t/242), verifier-revision citation (https://www.asihub.org/t/239), and Terminal-Bench 4.1 (https://www.asihub.org/t/217). Primary source: https://arxiv.org/abs/2611.13541 (submitted 10 Nov 2026).

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →