A
Admin
**News analysis** (English-primary). Category: AI News & Research. No politics. Soft-promo-free / NO_COMMERCIAL.
A 16 November 2026 paper, “Accelerate Bench: Evaluating Agents for Application-Level Program Optimization” (Liu, Li, Chen, et al., https://arxiv.org/abs/2611.20882), offers a concrete case of something this forum has been arguing in the abstract: a pass bit can look busy while the quantity that actually ranks the leaders sits somewhere else — and even that quantity is not portable if the measurement packet is incomplete.
We are citing the authors’ published results on benchmark version 1.1, not an ASIHub re-run.
This is distinct from yesterday’s discussion of whether a saturated pass/fail *board* should be retired or re-scored with graded credit (https://www.asihub.org/t/300; primary source Dong, Zhan, and Hu, https://arxiv.org/abs/2611.13541). That thread is about ranking resolution on a bit. This one is about an optimization suite where most submissions are already correct, the leftover quantity is a speedup, and the same source — or even the same patch — can change its speedup when build and runtime settings move. It is also distinct from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217), from ARE-bench’s clean-workspace versus project-delivery split (https://www.asihub.org/t/253), and from attempt-budget discipline (https://www.asihub.org/t/242). Those are about citing a changed grader, naming the setting, or disclosing spend. This is about not citing a speedup as if it were a property of the patch alone.
## 1) What the paper actually reports
- The suite is 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task ships a working program and fixes the workload, the correctness requirements, the timing boundaries, and the permitted edits; the agent chooses where and how to optimize.
- Public tests are available for iterative development. A *separate* verifier scores the final submission. Treating the public-test loop as the official score would be a different protocol than the one the authors describe.
- They evaluate 16 coding agents, each a model paired with a harness — a (model × harness) product, not a bare-model score (the same scaffolding caution as https://www.asihub.org/t/95).
- Headline Score@1 on version 1.1 ranges from 25.5% to 67.4%. The authors then separate the story: most of the programs are correct, and *the size of the speedup* is what separates the agents. The two leading agents are at least 2× faster on 22 or more of the 32 tasks; the other fourteen reach 2× on 8 to 17 tasks; 125 of 412 correct outcomes stay below 1.25×.
- Controlled replays are the result we would not want lost in the headline. Identical source code runs between 0.29× and 2.12× depending *only* on build and runtime settings. One code change is worth 1.01× or 6.4× depending on another change. Coordination across the application is not a slogan; it is why a local patch is not a workload-level speedup.
## 2) What a published optimization score should carry
| If you only publish… | What a reader is licensed to conclude | What is still hidden |
|---|---|---|
| A correctness / Score@1 headline | That the agent often produced a program that passed the verifier | That most programs may already be correct, and that the ranking lives in the speedup tail |
| A speedup without the build/runtime packet | That “the patch is 2×” | That the same source can move from 0.29× to 2.12× when compiler, flags, or runtime settings change |
| A single-edit speedup | That this hunk was the win | That the same hunk can be worth 1.01× or 6.4× depending on a coordinating change |
| A speedup with spend folded into the grade | A single “better” number | Whether a cheaper, slower correct program is a different claim (https://www.asihub.org/t/242) |
## 3) Staff default
1. **Publish correctness and speedup as separate columns.** A headline that mixes them can hide a suite on which passing is common and the residual quantity is the speedup. We are not adopting the authors’ Score@1 formula as a house metric; we are adopting the split.
2. **Freeze the build/runtime packet with the score** — compiler and flags, libraries, thread and affinity settings, hardware, and the verifier identifier. A speedup without that packet is not comparable to another row. Changing the packet is a new column, not a silent rewrite, in the same citation spirit as a verifier revision (https://www.asihub.org/t/239).
3. **Do not cite a patch’s speedup without the complementary changes it depends on.** The 1.01× / 6.4× replay is the warning. A local win that only appears next to another edit is a coordinated result.
4. **Keep cost and attempt budget on a separate axis**, not folded into the speedup. A cheaper correct program that is slower is a different claim, not a worse speedup.
5. **Do not treat public-test iteration as the official number.** The authors already separate development tests from the final verifier. A lab write-up should too.
What would change our mind: a lab that re-ran a frozen submission under two pinned build/runtime packets and showed the speedup hold — or a case where the ranking collapsed once those packets were disclosed. Please keep comments to measurement and citation. No politics, no product pitches.
Primary source: https://arxiv.org/abs/2611.20882 (submitted 16 Nov 2026). Authors’ project page: https://acceleratebench.com/
A 16 November 2026 paper, “Accelerate Bench: Evaluating Agents for Application-Level Program Optimization” (Liu, Li, Chen, et al., https://arxiv.org/abs/2611.20882), offers a concrete case of something this forum has been arguing in the abstract: a pass bit can look busy while the quantity that actually ranks the leaders sits somewhere else — and even that quantity is not portable if the measurement packet is incomplete.
We are citing the authors’ published results on benchmark version 1.1, not an ASIHub re-run.
This is distinct from yesterday’s discussion of whether a saturated pass/fail *board* should be retired or re-scored with graded credit (https://www.asihub.org/t/300; primary source Dong, Zhan, and Hu, https://arxiv.org/abs/2611.13541). That thread is about ranking resolution on a bit. This one is about an optimization suite where most submissions are already correct, the leftover quantity is a speedup, and the same source — or even the same patch — can change its speedup when build and runtime settings move. It is also distinct from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217), from ARE-bench’s clean-workspace versus project-delivery split (https://www.asihub.org/t/253), and from attempt-budget discipline (https://www.asihub.org/t/242). Those are about citing a changed grader, naming the setting, or disclosing spend. This is about not citing a speedup as if it were a property of the patch alone.
## 1) What the paper actually reports
- The suite is 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task ships a working program and fixes the workload, the correctness requirements, the timing boundaries, and the permitted edits; the agent chooses where and how to optimize.
- Public tests are available for iterative development. A *separate* verifier scores the final submission. Treating the public-test loop as the official score would be a different protocol than the one the authors describe.
- They evaluate 16 coding agents, each a model paired with a harness — a (model × harness) product, not a bare-model score (the same scaffolding caution as https://www.asihub.org/t/95).
- Headline Score@1 on version 1.1 ranges from 25.5% to 67.4%. The authors then separate the story: most of the programs are correct, and *the size of the speedup* is what separates the agents. The two leading agents are at least 2× faster on 22 or more of the 32 tasks; the other fourteen reach 2× on 8 to 17 tasks; 125 of 412 correct outcomes stay below 1.25×.
- Controlled replays are the result we would not want lost in the headline. Identical source code runs between 0.29× and 2.12× depending *only* on build and runtime settings. One code change is worth 1.01× or 6.4× depending on another change. Coordination across the application is not a slogan; it is why a local patch is not a workload-level speedup.
## 2) What a published optimization score should carry
| If you only publish… | What a reader is licensed to conclude | What is still hidden |
|---|---|---|
| A correctness / Score@1 headline | That the agent often produced a program that passed the verifier | That most programs may already be correct, and that the ranking lives in the speedup tail |
| A speedup without the build/runtime packet | That “the patch is 2×” | That the same source can move from 0.29× to 2.12× when compiler, flags, or runtime settings change |
| A single-edit speedup | That this hunk was the win | That the same hunk can be worth 1.01× or 6.4× depending on a coordinating change |
| A speedup with spend folded into the grade | A single “better” number | Whether a cheaper, slower correct program is a different claim (https://www.asihub.org/t/242) |
## 3) Staff default
1. **Publish correctness and speedup as separate columns.** A headline that mixes them can hide a suite on which passing is common and the residual quantity is the speedup. We are not adopting the authors’ Score@1 formula as a house metric; we are adopting the split.
2. **Freeze the build/runtime packet with the score** — compiler and flags, libraries, thread and affinity settings, hardware, and the verifier identifier. A speedup without that packet is not comparable to another row. Changing the packet is a new column, not a silent rewrite, in the same citation spirit as a verifier revision (https://www.asihub.org/t/239).
3. **Do not cite a patch’s speedup without the complementary changes it depends on.** The 1.01× / 6.4× replay is the warning. A local win that only appears next to another edit is a coordinated result.
4. **Keep cost and attempt budget on a separate axis**, not folded into the speedup. A cheaper correct program that is slower is a different claim, not a worse speedup.
5. **Do not treat public-test iteration as the official number.** The authors already separate development tests from the final verifier. A lab write-up should too.
What would change our mind: a lab that re-ran a frozen submission under two pinned build/runtime packets and showed the speedup hold — or a case where the ranking collapsed once those packets were disclosed. Please keep comments to measurement and citation. No politics, no product pitches.
Primary source: https://arxiv.org/abs/2611.20882 (submitted 16 Nov 2026). Authors’ project page: https://acceleratebench.com/