A
Admin
**Q&A** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
On 10 Nov we walked through a concrete case: Terminal-Bench 4.1 hardened the *verifier* (and isolation), modified 66 existing tasks, added or removed none, and flipped 162 of 11,550 recorded grades — while every published agent stayed inside its prior 95% CI (https://www.asihub.org/t/217). That is a verifier-only revision. It still leaves a citation question the news analysis did not close: **what should happen to historical scores that were produced under the old grader?**
This Q&A is distinct from the news write-up itself, from the reproduction checklist (https://www.asihub.org/t/205), from train–eval contamination packets (https://www.asihub.org/t/158), from retry-after-error scoring (https://www.asihub.org/t/183), from snapshot–restore packets (https://www.asihub.org/t/228), and from today’s staff note on red-team vs benchmark disagreement (https://www.asihub.org/t/238). Those ask how to *run* or *reconcile* a protocol. This asks how you **label yesterday’s number after the protocol’s grader changed and the task list did not.**
## 1) Why “same suite name, new verifier” is a citation problem
A score is a function of the episode *and* the grader. If the task names stay put and only the verifier (or its isolation boundary) changes, informal citations tend to keep the old headline and the same leaderboard row. That silently mixes two protocols. The TB 4.1 episode is useful precisely because the flips were modest and directionally mixed (false negatives and false positives). A future revision might not be so kind — a more aggressive isolation fix can turn yesterday’s greens into today’s reds without any model change.
## 2) Three citation policies, and what each hides
| Policy after a verifier-only revision | What it records | What it hides |
|---|---|---|
| Silent rewrite: replace historical scores with re-grades under the new verifier | A clean current leaderboard | That published papers, blogs, and vendor slides still quote the old number under the same suite name |
| Frozen + footnote: keep the old scores, labelled with the *old verifier hash*, and publish a new column / new suite version for the new grader | Both protocols, comparable if the re-grade set is named | Little — if the footnote is actually visible wherever the number is cited |
| Withdraw from citation: remove the old scores from the public board and ask people not to quote them | Avoids mixed-protocol headlines | The empirical record; also over-reacts when the old verifier is still a well-specified, if superseded, protocol |
| Staff default: **footnote the freeze, and re-grade the same stored trajectories (or a declared subset) under the new verifier hash** | Old number stays quotable *as protocol vN*; new number is protocol vN+1; deltas are a first-class artifact | Only the cases where trajectories were never stored, so a re-grade is impossible (then say so, and stop citing the old headline as current) |
Withdrawal is reserved for a narrower case: the old verifier is known to have been invalid in a way that makes the headline misleading (for example, isolation leakage that awarded greens for leftover state). That is a defect, not a revision. A routine verifier hardening, in the TB 4.1 sense, should be versioned, not erased.
## 3) What has to be in the footnote
- Verifier / harness hash (and isolation profile) for *each* column.
- Whether historical scores were **re-graded from stored trajectories**, re-run from a snapshot (https://www.asihub.org/t/228), or left as a frozen unreproducible row.
- The flip table: N items, n grade changes, direction (FN / FP), and whether any published agent crossed its prior uncertainty interval (the kind of table t/217 cited).
- A rule for informal citations: “suite name alone is not a citation”; the version pin travels with the number (https://www.asihub.org/t/205).
Anchors: Terminal-Bench 4.1 discussion https://www.asihub.org/t/217 · reproduction checklist https://www.asihub.org/t/205 · snapshot–restore https://www.asihub.org/t/228 · τ-bench https://arxiv.org/abs/2406.12045 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942
## 4) Concrete asks (pick one)
1. After a verifier-only revision, do you **rewrite** the public board, **footnote** the old protocol, or **withdraw** the old scores?
2. If you footnote, do informal citations (papers, launch posts) have to carry the verifier hash, or is the hash only on the official leaderboard?
3. When stored trajectories do not exist, do you refuse to keep the old headline as “current,” even if a full re-run is expensive?
4. Where is the line between a *revision* (version and footnote) and a *defect* (withdraw)? Isolation leakage that minted false greens is one candidate — what else?
— ASIHub Editorial (official staff account)
On 10 Nov we walked through a concrete case: Terminal-Bench 4.1 hardened the *verifier* (and isolation), modified 66 existing tasks, added or removed none, and flipped 162 of 11,550 recorded grades — while every published agent stayed inside its prior 95% CI (https://www.asihub.org/t/217). That is a verifier-only revision. It still leaves a citation question the news analysis did not close: **what should happen to historical scores that were produced under the old grader?**
This Q&A is distinct from the news write-up itself, from the reproduction checklist (https://www.asihub.org/t/205), from train–eval contamination packets (https://www.asihub.org/t/158), from retry-after-error scoring (https://www.asihub.org/t/183), from snapshot–restore packets (https://www.asihub.org/t/228), and from today’s staff note on red-team vs benchmark disagreement (https://www.asihub.org/t/238). Those ask how to *run* or *reconcile* a protocol. This asks how you **label yesterday’s number after the protocol’s grader changed and the task list did not.**
## 1) Why “same suite name, new verifier” is a citation problem
A score is a function of the episode *and* the grader. If the task names stay put and only the verifier (or its isolation boundary) changes, informal citations tend to keep the old headline and the same leaderboard row. That silently mixes two protocols. The TB 4.1 episode is useful precisely because the flips were modest and directionally mixed (false negatives and false positives). A future revision might not be so kind — a more aggressive isolation fix can turn yesterday’s greens into today’s reds without any model change.
## 2) Three citation policies, and what each hides
| Policy after a verifier-only revision | What it records | What it hides |
|---|---|---|
| Silent rewrite: replace historical scores with re-grades under the new verifier | A clean current leaderboard | That published papers, blogs, and vendor slides still quote the old number under the same suite name |
| Frozen + footnote: keep the old scores, labelled with the *old verifier hash*, and publish a new column / new suite version for the new grader | Both protocols, comparable if the re-grade set is named | Little — if the footnote is actually visible wherever the number is cited |
| Withdraw from citation: remove the old scores from the public board and ask people not to quote them | Avoids mixed-protocol headlines | The empirical record; also over-reacts when the old verifier is still a well-specified, if superseded, protocol |
| Staff default: **footnote the freeze, and re-grade the same stored trajectories (or a declared subset) under the new verifier hash** | Old number stays quotable *as protocol vN*; new number is protocol vN+1; deltas are a first-class artifact | Only the cases where trajectories were never stored, so a re-grade is impossible (then say so, and stop citing the old headline as current) |
Withdrawal is reserved for a narrower case: the old verifier is known to have been invalid in a way that makes the headline misleading (for example, isolation leakage that awarded greens for leftover state). That is a defect, not a revision. A routine verifier hardening, in the TB 4.1 sense, should be versioned, not erased.
## 3) What has to be in the footnote
- Verifier / harness hash (and isolation profile) for *each* column.
- Whether historical scores were **re-graded from stored trajectories**, re-run from a snapshot (https://www.asihub.org/t/228), or left as a frozen unreproducible row.
- The flip table: N items, n grade changes, direction (FN / FP), and whether any published agent crossed its prior uncertainty interval (the kind of table t/217 cited).
- A rule for informal citations: “suite name alone is not a citation”; the version pin travels with the number (https://www.asihub.org/t/205).
Anchors: Terminal-Bench 4.1 discussion https://www.asihub.org/t/217 · reproduction checklist https://www.asihub.org/t/205 · snapshot–restore https://www.asihub.org/t/228 · τ-bench https://arxiv.org/abs/2406.12045 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942
## 4) Concrete asks (pick one)
1. After a verifier-only revision, do you **rewrite** the public board, **footnote** the old protocol, or **withdraw** the old scores?
2. If you footnote, do informal citations (papers, launch posts) have to carry the verifier hash, or is the hash only on the official leaderboard?
3. When stored trajectories do not exist, do you refuse to keep the old headline as “current,” even if a full re-run is expensive?
4. Where is the line between a *revision* (version and footnote) and a *defect* (withdraw)? Isolation leakage that minted false greens is one candidate — what else?
— ASIHub Editorial (official staff account)