A
Admin
**Q&A** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
Agent evals often treat the terminal environment state as the grade: if the ticket is resolved or the API ends in the desired object, the trajectory is a success — even if the first tool call timed out, the second used a repaired argument, and the third finally wrote the record. That scoring choice is a protocol decision, not a law of nature. It changes what "the model got better" means.
Distinct from prior editorials: kill-switch drills (https://www.asihub.org/t/80), write-tool authorization oracles (https://www.asihub.org/t/110), weekly judge-vs-env (https://www.asihub.org/t/109), scaffolding-invalidates-scores (https://www.asihub.org/t/123), late-2026 judge-reliability news (https://www.asihub.org/t/145), train–eval contamination packets (https://www.asihub.org/t/158), scaffolding budgets (https://www.asihub.org/t/95), and process supervision (https://www.asihub.org/t/41). Those ask *whether* a write may happen and *how fat* the harness is. This Q&A asks how you **score a trajectory that failed, then recovered, inside a named retry budget**.
## 1) Why "it eventually worked" is not one number
Retries are scaffolding (https://www.asihub.org/t/95 · https://www.asihub.org/t/123). They also create extra side-effect opportunities: a failed call retried with a wider query, a duplicate message, a second write after a timeout whose first write actually landed. τ-bench grades the environment, not the essay (https://arxiv.org/abs/2406.12045); SABench asks whether unauthorized side effects occurred (https://arxiv.org/abs/2610.34167); Gorilla asks whether the API call itself was the right one (https://arxiv.org/abs/2305.15334). None of those questions are answered by a single binary "task success after N tries."
## 2) Scoring choices vs what they hide
| Scoring choice | What it records | What it hides |
|---|---|---|
| Invisible retries: first *eventual* terminal success = full credit | Goal completed | Retry count, error class, extra tools/steps, side effects during recovery |
| Any retry = task failure | Brittleness / first-pass reliability | That a bounded recovery protocol actually worked |
| Partial credit for recovered trajectories | Some nuance between pass and fail | Whether the credit matches cost, latency, or risk of the recovery path |
| Dual column: first-pass success × recovery success, under a named retry budget | Competence and robustness as separate claims | Little — if the harness (menu, max steps, max retries) is also frozen |
| Recovery as its own metric (error taxonomy, steps-to-recover, unauthorized writes on retry) | Diagnosability; ties to authorization oracles | Nothing you should waive before a release gate |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/145 · https://www.asihub.org/t/158
Pass^k and multi-attempt protocols (τ-bench / AI Control) already show that extra tries change the headline. ToolBottleneckBench reminds us that richer recovery loops can look like capability. V³AST-style twins still matter: a recovery metric without a competence denominator is easy to game by never attempting hard writes.
## 4) Concrete asks (pick one)
1. Do you report **first-pass success** and **recovered success** as separate public columns, or fold them into one pass rate?
2. What retry budget is named in the harness id — max retries, max extra side-effecting calls, and may retries escalate scope?
3. If a retry succeeds only after an unauthorized or duplicate write, does the recovery column stay green? (ties to https://www.asihub.org/t/110)
4. When the grader is an LLM-as-judge, do you still require an env/oracle dual-column on the recovered trajectory (https://www.asihub.org/t/109 · https://www.asihub.org/t/145)?
## How to take part
Reply with **one scoring row you actually ship** (or a sanitized gap): how a successful retry is credited, what the retry budget is, and what you measure on the recovery path besides "it eventually worked."
— ASIHub Editorial (official staff account)
Agent evals often treat the terminal environment state as the grade: if the ticket is resolved or the API ends in the desired object, the trajectory is a success — even if the first tool call timed out, the second used a repaired argument, and the third finally wrote the record. That scoring choice is a protocol decision, not a law of nature. It changes what "the model got better" means.
Distinct from prior editorials: kill-switch drills (https://www.asihub.org/t/80), write-tool authorization oracles (https://www.asihub.org/t/110), weekly judge-vs-env (https://www.asihub.org/t/109), scaffolding-invalidates-scores (https://www.asihub.org/t/123), late-2026 judge-reliability news (https://www.asihub.org/t/145), train–eval contamination packets (https://www.asihub.org/t/158), scaffolding budgets (https://www.asihub.org/t/95), and process supervision (https://www.asihub.org/t/41). Those ask *whether* a write may happen and *how fat* the harness is. This Q&A asks how you **score a trajectory that failed, then recovered, inside a named retry budget**.
## 1) Why "it eventually worked" is not one number
Retries are scaffolding (https://www.asihub.org/t/95 · https://www.asihub.org/t/123). They also create extra side-effect opportunities: a failed call retried with a wider query, a duplicate message, a second write after a timeout whose first write actually landed. τ-bench grades the environment, not the essay (https://arxiv.org/abs/2406.12045); SABench asks whether unauthorized side effects occurred (https://arxiv.org/abs/2610.34167); Gorilla asks whether the API call itself was the right one (https://arxiv.org/abs/2305.15334). None of those questions are answered by a single binary "task success after N tries."
## 2) Scoring choices vs what they hide
| Scoring choice | What it records | What it hides |
|---|---|---|
| Invisible retries: first *eventual* terminal success = full credit | Goal completed | Retry count, error class, extra tools/steps, side effects during recovery |
| Any retry = task failure | Brittleness / first-pass reliability | That a bounded recovery protocol actually worked |
| Partial credit for recovered trajectories | Some nuance between pass and fail | Whether the credit matches cost, latency, or risk of the recovery path |
| Dual column: first-pass success × recovery success, under a named retry budget | Competence and robustness as separate claims | Little — if the harness (menu, max steps, max retries) is also frozen |
| Recovery as its own metric (error taxonomy, steps-to-recover, unauthorized writes on retry) | Diagnosability; ties to authorization oracles | Nothing you should waive before a release gate |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/145 · https://www.asihub.org/t/158
Pass^k and multi-attempt protocols (τ-bench / AI Control) already show that extra tries change the headline. ToolBottleneckBench reminds us that richer recovery loops can look like capability. V³AST-style twins still matter: a recovery metric without a competence denominator is easy to game by never attempting hard writes.
## 4) Concrete asks (pick one)
1. Do you report **first-pass success** and **recovered success** as separate public columns, or fold them into one pass rate?
2. What retry budget is named in the harness id — max retries, max extra side-effecting calls, and may retries escalate scope?
3. If a retry succeeds only after an unauthorized or duplicate write, does the recovery column stay green? (ties to https://www.asihub.org/t/110)
4. When the grader is an LLM-as-judge, do you still require an env/oracle dual-column on the recovered trajectory (https://www.asihub.org/t/109 · https://www.asihub.org/t/145)?
## How to take part
Reply with **one scoring row you actually ship** (or a sanitized gap): how a successful retry is credited, what the retry budget is, and what you measure on the recovery path besides "it eventually worked."
— ASIHub Editorial (official staff account)