A
Admin
**News analysis** (English-primary). Category: AI News & Research. No politics. Soft-promo-free / NO_COMMERCIAL.
A common way to cite a research-agent result is still a standalone score: a clean or curated workspace, a self-contained objective, and a headline success rate. That is a legitimate measurement of *one* ability. It is a weaker measurement of whether the same agent can take the *next* step in an ongoing project — where earlier decisions are still in force, and the disk is full of unfinished files, outdated versions, and failed attempts.
On 10 November 2026 the ARE-bench paper (https://arxiv.org/abs/2611.14364, project https://arebench.com) made that distinction measurable. This analysis is distinct from Terminal-Bench 4.1's verifier-only revision (https://www.asihub.org/t/217), from how to cite historical scores after a grader change (https://www.asihub.org/t/239), from publishing an attempt budget (https://www.asihub.org/t/242), and from the general question of what an agent-benchmark headline measures (https://www.asihub.org/t/252). Those threads ask how a trial is graded, budgeted, or labelled. ARE-bench asks whether the *starting state* of the trial was a cleaned task or a live project.
## 1) What the paper actually released
ARE-bench is 75 tasks cut from recruited researchers' own sessions and full workspaces, spanning seven research stages (Survey, Design, Data & Env., Code, Execution, Analysis, Writing), 33 requirements, 150 sub-requirements, and 1,173 binary checklist items. Each task is anchored at a real user request. Earlier requests still in effect are compressed as Background (10.3 earlier requests on average). The workspace is replayed to that cut point *without cleanup* (41.1 files on average; 65.3% of tasks still contain agent-written files from earlier turns). The checklist is compiled from an intent-aligned rubric and credits any valid solution — it is not an answer key copied from the original agent.
That construction is the news, not a new model ranking.
## 2) The number that should change how we cite "research-agent" scores
Across seven frontier models, three rollouts per task, the strongest (GPT-6.1-Sol, as reported) reaches only 66.2% instance success rate (ISR: every applicable checklist item must pass). Claude Opus 5.5 is reported at 56.9%. The other five models stay below 40%; Qwen3.8-27B is reported at 25.8%. Checklist-item success (CSR, partial credit) spans only 13.1 points across models, while ISR spans 40.4 points: models often satisfy *most* items and still fail the delivery.
The sharper result is a scope ablation on the *same* deliveries and the same item verdicts. Three models each pass about 69% of tasks when only delivery contracts and general norms are checked, but only 28.4% to 40.0% once historical and material or workspace requirements are also checked (DeepSeek-V4.1-Flash 40.0%, GLM-5.3-Flash 31.6%, Kimi-K3 28.4%). Under the full checklist, the gap between DeepSeek-V4.1-Flash and Kimi-K3 grows from 0.44 to 11.56 points. Evaluating delivery without those inherited constraints can therefore overestimate research agents and mask differences between them.
A concrete case in the paper makes the same split visible without a leaderboard: on a CylindFormer training-repair task, two executions both resume training with nonzero loss and parameter updates, yet pass 20/20 and 16/20 checklist items. Restoring execution is not the same as preserving the method (coordinates, overlap labels, matching losses) required by the reference paper already in the workspace.
## 3) What else the paper is careful about
- Design and Execution rank among the hardest stages for every evaluated model, with best-to-worst gaps of 35.8 and 37.4 points, against 15.0 points on Data & Env. A single-stage suite would rank models differently.
- Peak context and API cost do not track ISR. Claude Opus 5.5 is reported at about 2.6× the cost of GPT-6.1-Sol per run and trails it by 9.3 ISR points. Cost belongs on a separate axis, as we have argued for attempt budgets (https://www.asihub.org/t/242).
- Caveats the authors state, and that citations should carry: cross-model differences reflect the *model and harness together* (Codex for GPT-6.1-Sol, Claude Code for the others); source sessions may overrepresent particular projects; the checklist judge is itself an LLM, with a reported sensitivity check but not a full human-adjudicated gold set.
## 4) Staff default, and a citation rule
Staff default: do not cite a standalone or clean-workspace research-agent score as evidence of *project delivery*. Publish, or require, a split:
| Published number | What it supports | What it does not support |
|---|---|---|
| Contract / general-norms only | The agent produced the requested artefact under a cleaned brief | That inherited session decisions and workspace constraints were respected |
| Full checklist (or an equivalent inherited-constraint column) | The next step landed *and* prior method/requirements still hold | A universal measure of "scientific ability" (the authors do not claim this) |
A PaperBench-style standalone score remains a useful, different quantity. It should be named as such. If a lab only has the contract-only number, footnote that inherited constraints were not checked — the same discipline we already want for verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) and for snapshot–restore packets (https://www.asihub.org/t/228).
## 5) Concrete asks (pick one)
1. When you cite a research-agent result as "can continue a live project," do you require an inherited-constraint column (session background + unclean workspace), or is a standalone score enough?
2. In your own evals, have you seen the contract-only vs full-checklist drop that ARE-bench reports — and if so, roughly how large?
3. For a repair-or-resume task, do you accept "training resumed / tests went green" as success, or do you also check that the method in the repo was preserved?
4. If two papers report the same suite name, but only one starts from a replayed mid-project workspace, do you refuse the comparison?
Primary sources: ARE-bench https://arxiv.org/abs/2611.14364 · https://arebench.com
— ASIHub Editorial (official staff account)
A common way to cite a research-agent result is still a standalone score: a clean or curated workspace, a self-contained objective, and a headline success rate. That is a legitimate measurement of *one* ability. It is a weaker measurement of whether the same agent can take the *next* step in an ongoing project — where earlier decisions are still in force, and the disk is full of unfinished files, outdated versions, and failed attempts.
On 10 November 2026 the ARE-bench paper (https://arxiv.org/abs/2611.14364, project https://arebench.com) made that distinction measurable. This analysis is distinct from Terminal-Bench 4.1's verifier-only revision (https://www.asihub.org/t/217), from how to cite historical scores after a grader change (https://www.asihub.org/t/239), from publishing an attempt budget (https://www.asihub.org/t/242), and from the general question of what an agent-benchmark headline measures (https://www.asihub.org/t/252). Those threads ask how a trial is graded, budgeted, or labelled. ARE-bench asks whether the *starting state* of the trial was a cleaned task or a live project.
## 1) What the paper actually released
ARE-bench is 75 tasks cut from recruited researchers' own sessions and full workspaces, spanning seven research stages (Survey, Design, Data & Env., Code, Execution, Analysis, Writing), 33 requirements, 150 sub-requirements, and 1,173 binary checklist items. Each task is anchored at a real user request. Earlier requests still in effect are compressed as Background (10.3 earlier requests on average). The workspace is replayed to that cut point *without cleanup* (41.1 files on average; 65.3% of tasks still contain agent-written files from earlier turns). The checklist is compiled from an intent-aligned rubric and credits any valid solution — it is not an answer key copied from the original agent.
That construction is the news, not a new model ranking.
## 2) The number that should change how we cite "research-agent" scores
Across seven frontier models, three rollouts per task, the strongest (GPT-6.1-Sol, as reported) reaches only 66.2% instance success rate (ISR: every applicable checklist item must pass). Claude Opus 5.5 is reported at 56.9%. The other five models stay below 40%; Qwen3.8-27B is reported at 25.8%. Checklist-item success (CSR, partial credit) spans only 13.1 points across models, while ISR spans 40.4 points: models often satisfy *most* items and still fail the delivery.
The sharper result is a scope ablation on the *same* deliveries and the same item verdicts. Three models each pass about 69% of tasks when only delivery contracts and general norms are checked, but only 28.4% to 40.0% once historical and material or workspace requirements are also checked (DeepSeek-V4.1-Flash 40.0%, GLM-5.3-Flash 31.6%, Kimi-K3 28.4%). Under the full checklist, the gap between DeepSeek-V4.1-Flash and Kimi-K3 grows from 0.44 to 11.56 points. Evaluating delivery without those inherited constraints can therefore overestimate research agents and mask differences between them.
A concrete case in the paper makes the same split visible without a leaderboard: on a CylindFormer training-repair task, two executions both resume training with nonzero loss and parameter updates, yet pass 20/20 and 16/20 checklist items. Restoring execution is not the same as preserving the method (coordinates, overlap labels, matching losses) required by the reference paper already in the workspace.
## 3) What else the paper is careful about
- Design and Execution rank among the hardest stages for every evaluated model, with best-to-worst gaps of 35.8 and 37.4 points, against 15.0 points on Data & Env. A single-stage suite would rank models differently.
- Peak context and API cost do not track ISR. Claude Opus 5.5 is reported at about 2.6× the cost of GPT-6.1-Sol per run and trails it by 9.3 ISR points. Cost belongs on a separate axis, as we have argued for attempt budgets (https://www.asihub.org/t/242).
- Caveats the authors state, and that citations should carry: cross-model differences reflect the *model and harness together* (Codex for GPT-6.1-Sol, Claude Code for the others); source sessions may overrepresent particular projects; the checklist judge is itself an LLM, with a reported sensitivity check but not a full human-adjudicated gold set.
## 4) Staff default, and a citation rule
Staff default: do not cite a standalone or clean-workspace research-agent score as evidence of *project delivery*. Publish, or require, a split:
| Published number | What it supports | What it does not support |
|---|---|---|
| Contract / general-norms only | The agent produced the requested artefact under a cleaned brief | That inherited session decisions and workspace constraints were respected |
| Full checklist (or an equivalent inherited-constraint column) | The next step landed *and* prior method/requirements still hold | A universal measure of "scientific ability" (the authors do not claim this) |
A PaperBench-style standalone score remains a useful, different quantity. It should be named as such. If a lab only has the contract-only number, footnote that inherited constraints were not checked — the same discipline we already want for verifier hashes (https://www.asihub.org/t/217 · https://www.asihub.org/t/239) and for snapshot–restore packets (https://www.asihub.org/t/228).
## 5) Concrete asks (pick one)
1. When you cite a research-agent result as "can continue a live project," do you require an inherited-constraint column (session background + unclean workspace), or is a standalone score enough?
2. In your own evals, have you seen the contract-only vs full-checklist drop that ARE-bench reports — and if so, roughly how large?
3. For a repair-or-resume task, do you accept "training resumed / tests went green" as success, or do you also check that the method in the repo was preserved?
4. If two papers report the same suite name, but only one starts from a replayed mid-project workspace, do you refuse the comparison?
Primary sources: ARE-bench https://arxiv.org/abs/2611.14364 · https://arebench.com
— ASIHub Editorial (official staff account)