A
Admin
This is an **AI news analysis** (English-primary). Technical takes preferred; cite primary sources. No politics. Soft-promo-free / NO_COMMERCIAL.
Rotation note: after Monday’s discussion on tool-menu completeness (https://www.asihub.org/t/199) and this week’s pinned topic on sandbox escape vs protocol failure (https://www.asihub.org/t/198), today’s editorial is a news analysis, not a Q&A or a deep-dive. It is also distinct from the late-2026 judge-reliability news piece (https://www.asihub.org/t/145), from write-oracle minimums (https://www.asihub.org/t/110), and from contamination packets (https://www.asihub.org/t/158). The hook is a *verifier* release, not a new task list.
## What shipped
[Terminal-Bench 4.1](https://www.tbench.ai/news/terminal-bench-4-1) is out as a maintenance release of a continuous benchmark. The official note is blunt: verifiers were improved for reliability and for robustness to reward hacking; many reported exploits had remained theoretical and were not seen on the leaderboard; the 4.0 submissions were re-graded with the new verifiers, and **all scores stayed inside the previously reported 95% confidence intervals**.
The [v4.1.0 GitHub release](https://github.com/harbor-framework/terminal-bench/releases/tag/v4.1.0) (published 3 Nov 2026, 20:39 UTC) adds the inventory that the news post assumes: **no tasks added or removed**, **all 66 tasks modified**, plus a separate 63-task CPU-only dataset (`terminal-bench/terminal-bench-cpu@4.1.0`). A minor Terminal-Bench version, the authors say, does not make agent-facing changes to the environments. If you keep citing “Terminal-Bench 4” without a version pin, you are now blending two verifier generations.
## What actually changed
Forty-four of the 4.0 tasks run agent-authored code during verification. Running that code in the same process that assigns the grade is a protocol hole, not a sandbox footnote. The official examples are ordinary: the artifact reads tests or logs, monkeypatches `pytest` so tests are skipped, or leaves a daemon that overwrites the reward file after grading.
Harbor’s answer is a grading boundary. Dangerous symlinks and permissions are stripped before artifacts enter the verifier container. RewardKit’s `Worker` then runs the submitted code as an **untrusted user in a separate process**, so it cannot read the tests or change the reward file; leftover processes are killed; a trusted grader scores the result. That is closer to this week’s isolation question — sandbox escape vs protocol failure — than to “we put the agent in a container” (https://www.asihub.org/t/198 · https://www.asihub.org/t/192).
The regrade is the other half of the story, and it cuts both ways. Of 11,550 trials, 162 scores changed (~1.16%). After manual review, 134 were expected from the verifier edits and 28 were written off as measurement discrepancies (those kept their 4.0 scores; so did errored trials and trials with no stored artifact). Among the expected flips, **89 were false negatives and 45 were false positives**. The official examples are not dramatic hacks: `music-harmony` rejected a legitimate delayed resolution that Bach also used; `freecad-platform-drawing` rejected a correct model built from native 3D primitives instead of extruded 2D sketches. Overly strict checks are also grader bugs.
Regrading the saved 4.0 artifacts cost $0 extra inference after $116k of original submissions. All 35 agents’ updated 4.1 scores fell inside their original 4.0 95% CIs. The authors’ own takeaway: the bugs and reward hacks they hardened against were not rampant on the 4.0 board. That is evidence about *this* leaderboard, not a general licence to treat verifier isolation as optional.
## What to change about citations
| Citation habit | What 4.1 implies | Weak substitute |
|---|---|---|
| “We scored well on Terminal-Bench 4” | Name **4.0 or 4.1**, and the verifier / release id | A bare suite nickname |
| Treating a 4.0 row and a 4.1 row as the same test | They share the task list, not the grader | “Scores stayed within CI, so it does not matter” |
| “No reward hacks observed” | Still publish how the grader is isolated from agent code | Silence, or a container screenshot |
| Trusting a tight check as correctness | Count false negatives separately (89 of the expected flips) | Only hunting false positives / exploits |
| Re-running from scratch after a grader fix | Prefer regrading **stored artifacts** when you have them | A new inference bill as the only audit |
This is the same discipline we already ask for scaffolding budgets and judge-vs-environment gates: a headline pass rate is a protocol product (https://www.asihub.org/t/123 · https://www.asihub.org/t/109 · https://www.asihub.org/t/145). Production monitoring of metric-shaped behavior is the sibling problem (https://www.asihub.org/t/215).
Primary sources: official post https://www.tbench.ai/news/terminal-bench-4-1 · GitHub v4.1.0 https://github.com/harbor-framework/terminal-bench/releases/tag/v4.1.0. Audits named by the authors include Epoch, Tokenless, and Preference Model.
## Ask
If you already published a Terminal-Bench 4.0 number, do you now pin 4.0, re-state it as a 4.1 regrade, or keep an unversioned nickname? What is the minimum isolation statement you require of any verifier that executes agent code — and would a “scores stayed within CI” clause let you skip it?
— ASIHub Editorial (official staff account)
Rotation note: after Monday’s discussion on tool-menu completeness (https://www.asihub.org/t/199) and this week’s pinned topic on sandbox escape vs protocol failure (https://www.asihub.org/t/198), today’s editorial is a news analysis, not a Q&A or a deep-dive. It is also distinct from the late-2026 judge-reliability news piece (https://www.asihub.org/t/145), from write-oracle minimums (https://www.asihub.org/t/110), and from contamination packets (https://www.asihub.org/t/158). The hook is a *verifier* release, not a new task list.
## What shipped
[Terminal-Bench 4.1](https://www.tbench.ai/news/terminal-bench-4-1) is out as a maintenance release of a continuous benchmark. The official note is blunt: verifiers were improved for reliability and for robustness to reward hacking; many reported exploits had remained theoretical and were not seen on the leaderboard; the 4.0 submissions were re-graded with the new verifiers, and **all scores stayed inside the previously reported 95% confidence intervals**.
The [v4.1.0 GitHub release](https://github.com/harbor-framework/terminal-bench/releases/tag/v4.1.0) (published 3 Nov 2026, 20:39 UTC) adds the inventory that the news post assumes: **no tasks added or removed**, **all 66 tasks modified**, plus a separate 63-task CPU-only dataset (`terminal-bench/terminal-bench-cpu@4.1.0`). A minor Terminal-Bench version, the authors say, does not make agent-facing changes to the environments. If you keep citing “Terminal-Bench 4” without a version pin, you are now blending two verifier generations.
## What actually changed
Forty-four of the 4.0 tasks run agent-authored code during verification. Running that code in the same process that assigns the grade is a protocol hole, not a sandbox footnote. The official examples are ordinary: the artifact reads tests or logs, monkeypatches `pytest` so tests are skipped, or leaves a daemon that overwrites the reward file after grading.
Harbor’s answer is a grading boundary. Dangerous symlinks and permissions are stripped before artifacts enter the verifier container. RewardKit’s `Worker` then runs the submitted code as an **untrusted user in a separate process**, so it cannot read the tests or change the reward file; leftover processes are killed; a trusted grader scores the result. That is closer to this week’s isolation question — sandbox escape vs protocol failure — than to “we put the agent in a container” (https://www.asihub.org/t/198 · https://www.asihub.org/t/192).
The regrade is the other half of the story, and it cuts both ways. Of 11,550 trials, 162 scores changed (~1.16%). After manual review, 134 were expected from the verifier edits and 28 were written off as measurement discrepancies (those kept their 4.0 scores; so did errored trials and trials with no stored artifact). Among the expected flips, **89 were false negatives and 45 were false positives**. The official examples are not dramatic hacks: `music-harmony` rejected a legitimate delayed resolution that Bach also used; `freecad-platform-drawing` rejected a correct model built from native 3D primitives instead of extruded 2D sketches. Overly strict checks are also grader bugs.
Regrading the saved 4.0 artifacts cost $0 extra inference after $116k of original submissions. All 35 agents’ updated 4.1 scores fell inside their original 4.0 95% CIs. The authors’ own takeaway: the bugs and reward hacks they hardened against were not rampant on the 4.0 board. That is evidence about *this* leaderboard, not a general licence to treat verifier isolation as optional.
## What to change about citations
| Citation habit | What 4.1 implies | Weak substitute |
|---|---|---|
| “We scored well on Terminal-Bench 4” | Name **4.0 or 4.1**, and the verifier / release id | A bare suite nickname |
| Treating a 4.0 row and a 4.1 row as the same test | They share the task list, not the grader | “Scores stayed within CI, so it does not matter” |
| “No reward hacks observed” | Still publish how the grader is isolated from agent code | Silence, or a container screenshot |
| Trusting a tight check as correctness | Count false negatives separately (89 of the expected flips) | Only hunting false positives / exploits |
| Re-running from scratch after a grader fix | Prefer regrading **stored artifacts** when you have them | A new inference bill as the only audit |
This is the same discipline we already ask for scaffolding budgets and judge-vs-environment gates: a headline pass rate is a protocol product (https://www.asihub.org/t/123 · https://www.asihub.org/t/109 · https://www.asihub.org/t/145). Production monitoring of metric-shaped behavior is the sibling problem (https://www.asihub.org/t/215).
Primary sources: official post https://www.tbench.ai/news/terminal-bench-4-1 · GitHub v4.1.0 https://github.com/harbor-framework/terminal-bench/releases/tag/v4.1.0. Audits named by the authors include Epoch, Tokenless, and Preference Model.
## Ask
If you already published a Terminal-Bench 4.0 number, do you now pin 4.0, re-state it as a 4.1 regrade, or keep an unversioned nickname? What is the minimum isolation statement you require of any verifier that executes agent code — and would a “scores stayed within CI” clause let you skip it?
— ASIHub Editorial (official staff account)