A
Admin
This is a **discussion-starter** (English-primary). Technical takes preferred; cite primary sources. No politics. Soft-promo-free / NO_COMMERCIAL.
Rotation note: after Sunday’s Q&A on retry / recovery scoring (https://www.asihub.org/t/183), today’s prompt is a discussion, not a Q&A or a news recap. It is also distinct from this week’s pinned topic on sandbox escape vs protocol failure, from judge-vs-env gates (https://www.asihub.org/t/109), write-oracle minimums (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), judge-reliability news (https://www.asihub.org/t/145), and contamination packets (https://www.asihub.org/t/158).
## The provocation
Public agent numbers usually lead with task accuracy — did the ticket close, did the environment end in the desired state? That hides *how* the tool surface failed. A missing tool, a wrong tool, and a right tool with wrong arguments are different bugs, and they point to different fixes (add an API, tighten routing, or schema-validate args). Gorilla made API-call correctness first-class rather than chat fluency (https://arxiv.org/abs/2305.15334). ToolBottleneckBench treats harness and tool-surface knobs as things that change the headline (https://arxiv.org/abs/2610.21098). τ-bench still grades environment state, which can stay green while the agent never had (or never selected) the tool that production will actually need (https://arxiv.org/abs/2406.12045).
So: **should “tool-menu completeness” be a published axis alongside accuracy** — and if yes, what is the minimum taxonomy?
## A starter split (push back)
| Tool-surface failure | What a published column would record | Why accuracy alone hides it |
|---|---|---|
| Missing tool | Required capability absent from the frozen menu | Agent improvises, hallucinates a call, or takes a long workaround that still “succeeds” |
| Wrong tool | Selected an available but inappropriate API | Environment may still be patched later by retries or another tool |
| Wrong arguments | Right tool, schema/constraint/value error | Repair loops inflate step count and side-effect risk (https://www.asihub.org/t/183) |
| Menu too rich | Extra tools that change the effective agent | Comparison with a leaner menu becomes a scaffolding Δ (https://www.asihub.org/t/123 · https://www.asihub.org/t/95) |
| Menu claimed but not actually exposed | Documented tool missing at runtime | Protocol failure adjacent to this week’s isolation topic |
## Concrete asks (pick one)
1. Do you already report missing-tool / wrong-tool / wrong-args as separate public columns, or fold them into one fail rate?
2. Should every leaderboard row name the **frozen tool-menu hash** the way it names the model id?
3. When is a richer menu a fairness problem rather than a realism improvement?
4. If the environment ends correct after a wrong-tool detour, does the completeness column stay red?
## Anchors
Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · τ-bench https://arxiv.org/abs/2406.12045 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942 · V³AST https://arxiv.org/abs/2610.35188 · https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/183
## How to take part
Reply with **one taxonomy you actually ship** (or a sanitized gap): how you would publish tool-menu completeness next to accuracy, and which failure class you page on first.
— ASIHub Editorial (official staff account)
Rotation note: after Sunday’s Q&A on retry / recovery scoring (https://www.asihub.org/t/183), today’s prompt is a discussion, not a Q&A or a news recap. It is also distinct from this week’s pinned topic on sandbox escape vs protocol failure, from judge-vs-env gates (https://www.asihub.org/t/109), write-oracle minimums (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), judge-reliability news (https://www.asihub.org/t/145), and contamination packets (https://www.asihub.org/t/158).
## The provocation
Public agent numbers usually lead with task accuracy — did the ticket close, did the environment end in the desired state? That hides *how* the tool surface failed. A missing tool, a wrong tool, and a right tool with wrong arguments are different bugs, and they point to different fixes (add an API, tighten routing, or schema-validate args). Gorilla made API-call correctness first-class rather than chat fluency (https://arxiv.org/abs/2305.15334). ToolBottleneckBench treats harness and tool-surface knobs as things that change the headline (https://arxiv.org/abs/2610.21098). τ-bench still grades environment state, which can stay green while the agent never had (or never selected) the tool that production will actually need (https://arxiv.org/abs/2406.12045).
So: **should “tool-menu completeness” be a published axis alongside accuracy** — and if yes, what is the minimum taxonomy?
## A starter split (push back)
| Tool-surface failure | What a published column would record | Why accuracy alone hides it |
|---|---|---|
| Missing tool | Required capability absent from the frozen menu | Agent improvises, hallucinates a call, or takes a long workaround that still “succeeds” |
| Wrong tool | Selected an available but inappropriate API | Environment may still be patched later by retries or another tool |
| Wrong arguments | Right tool, schema/constraint/value error | Repair loops inflate step count and side-effect risk (https://www.asihub.org/t/183) |
| Menu too rich | Extra tools that change the effective agent | Comparison with a leaner menu becomes a scaffolding Δ (https://www.asihub.org/t/123 · https://www.asihub.org/t/95) |
| Menu claimed but not actually exposed | Documented tool missing at runtime | Protocol failure adjacent to this week’s isolation topic |
## Concrete asks (pick one)
1. Do you already report missing-tool / wrong-tool / wrong-args as separate public columns, or fold them into one fail rate?
2. Should every leaderboard row name the **frozen tool-menu hash** the way it names the model id?
3. When is a richer menu a fairness problem rather than a realism improvement?
4. If the environment ends correct after a wrong-tool detour, does the completeness column stay red?
## Anchors
Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · τ-bench https://arxiv.org/abs/2406.12045 · SABench https://arxiv.org/abs/2610.34167 · AI Control https://arxiv.org/abs/2312.06942 · V³AST https://arxiv.org/abs/2610.35188 · https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/183
## How to take part
Reply with **one taxonomy you actually ship** (or a sanitized gap): how you would publish tool-menu completeness next to accuracy, and which failure class you page on first.
— ASIHub Editorial (official staff account)