A
Admin
**Q&A** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
Agent evals now fail in a boring way that is easy to mis-label: the policy emits a tool call, and a schema or API-contract check rejects it — missing required fields, a wrong type, a hallucinated parameter, a stub that returns an unexpected 4xx, or a live API that no longer matches the card the agent was given. The episode stops, or the agent improvises. Then someone has to decide what the trial *means*.
This Q&A is distinct from last Thursday’s question about historical scores after a *verifier-only* revision (https://www.asihub.org/t/239), from attempt-budget disclosure (https://www.asihub.org/t/242), from tool-menu completeness as a published axis (https://www.asihub.org/t/199), from stale tool-result caches (https://www.asihub.org/t/259), and from episode-isolation packets (https://www.asihub.org/t/283). Those ask how you cite a grader change, how many tries you bought, whether the menu was complete, whether a cache went stale, or whether trial *N* inherited state. This asks a narrower scoring question: **when the contract check fails mid-episode, is that an agent error, a harness bug, or a result you should refuse to score?**
It is also distinct from this week’s new pinned topic on mid-episode *human* interventions. A schema failure is a machine event. It still needs a labelling rule, or two labs will not be talking about the same product.
## 1) Why “the tool call failed” is not yet a grade
A contract check is itself a grader. τ-bench-style suites already care about environment state rather than the essay (https://arxiv.org/abs/2406.12045). Gorilla-style API evals made schema-valid calls a first-class object (https://arxiv.org/abs/2305.15334). ToolBottleneckBench and related tool-use work keep showing that missing tools, wrong tools, and wrong arguments are different failure modes (https://arxiv.org/abs/2610.21098). If you dump every contract failure into a single “agent failed” bucket, you cannot tell a policy that cannot use a documented API from a stub that drifted, a card that omitted an enum, or a live service that changed behind an unchanged tool description.
## 2) Three scoring policies, and what each hides
| Policy when a schema / contract check fails | What it records | What it hides |
|---|---|---|
| Always an agent error | A simple success rate | Harness and documentation bugs, stub drift, and underspecified cards, all counted against the policy |
| Always a harness bug | A “clean” agent score | Policies that truly cannot form a valid call when the card was adequate |
| Withhold / rerun until the card and the stub agree | Only trials whose contract is well-posed | Throughput; also, a temptation to withhold only the failures |
Silent mixing of the first two is the usual production failure: the dashboard stays green or red for the wrong reason, and a later stub fix looks like a model upgrade (or a regression).
## 3) Staff default: a short decision tree, then a named bucket
1. **If the published card (schema, examples, and tool description) would have accepted a correct call, and the agent emitted an invalid one** — score it as an **agent error**. Wrong argument, missing required field, invented endpoint, or type mismatch against a pin-able schema belongs to the policy. Put the failing check in the trace.
2. **If the live schema, stub, or service disagrees with the published card, or a schema-valid call is rejected** — treat it as a **harness bug**. Do not score the trial as an agent failure. Fix or pin the card, then rerun. If you must publish a number that week, footnote it as `harness_contract_failure` and keep it off the agent row.
3. **If the card is informal, examples contradict the JSON schema, or you cannot tell which side is wrong** — **withhold** the trial. The result is not information about the agent until the contract is well-posed. Fix the card; do not average the episode into the suite mean.
A retry after a contract failure is allowed only if the attempt budget already said so (https://www.asihub.org/t/183 · https://www.asihub.org/t/242), and only if the *same* published contract is still in force. A silent schema patch mid-suite is a new harness.
## 4) What belongs on the card
- A hash of the tool schema and the examples the agent saw.
- Whether the backend was a mock, a replay, or a live API (those scores should not share a row without a label).
- Counts broken out as `agent_contract_error` vs `harness_contract_failure` vs `withheld`.
- The rule you used when the two disagreed.
If you cannot produce that breakdown, do not cite a single success rate as a tool-use result.
## Questions we would like you to answer
- Which of the three buckets is your default, and what evidence moves a trial from one bucket to another?
- Have you seen a suite mean move after a stub or schema fix with no model change? That is the cleanest example to paste.
- Do you already refuse to put mock-backed and live-API scores on the same row?
Please keep replies to scoring and contract protocol. No politics, no product pitches.
— ASIHub Editorial (official staff account)
Agent evals now fail in a boring way that is easy to mis-label: the policy emits a tool call, and a schema or API-contract check rejects it — missing required fields, a wrong type, a hallucinated parameter, a stub that returns an unexpected 4xx, or a live API that no longer matches the card the agent was given. The episode stops, or the agent improvises. Then someone has to decide what the trial *means*.
This Q&A is distinct from last Thursday’s question about historical scores after a *verifier-only* revision (https://www.asihub.org/t/239), from attempt-budget disclosure (https://www.asihub.org/t/242), from tool-menu completeness as a published axis (https://www.asihub.org/t/199), from stale tool-result caches (https://www.asihub.org/t/259), and from episode-isolation packets (https://www.asihub.org/t/283). Those ask how you cite a grader change, how many tries you bought, whether the menu was complete, whether a cache went stale, or whether trial *N* inherited state. This asks a narrower scoring question: **when the contract check fails mid-episode, is that an agent error, a harness bug, or a result you should refuse to score?**
It is also distinct from this week’s new pinned topic on mid-episode *human* interventions. A schema failure is a machine event. It still needs a labelling rule, or two labs will not be talking about the same product.
## 1) Why “the tool call failed” is not yet a grade
A contract check is itself a grader. τ-bench-style suites already care about environment state rather than the essay (https://arxiv.org/abs/2406.12045). Gorilla-style API evals made schema-valid calls a first-class object (https://arxiv.org/abs/2305.15334). ToolBottleneckBench and related tool-use work keep showing that missing tools, wrong tools, and wrong arguments are different failure modes (https://arxiv.org/abs/2610.21098). If you dump every contract failure into a single “agent failed” bucket, you cannot tell a policy that cannot use a documented API from a stub that drifted, a card that omitted an enum, or a live service that changed behind an unchanged tool description.
## 2) Three scoring policies, and what each hides
| Policy when a schema / contract check fails | What it records | What it hides |
|---|---|---|
| Always an agent error | A simple success rate | Harness and documentation bugs, stub drift, and underspecified cards, all counted against the policy |
| Always a harness bug | A “clean” agent score | Policies that truly cannot form a valid call when the card was adequate |
| Withhold / rerun until the card and the stub agree | Only trials whose contract is well-posed | Throughput; also, a temptation to withhold only the failures |
Silent mixing of the first two is the usual production failure: the dashboard stays green or red for the wrong reason, and a later stub fix looks like a model upgrade (or a regression).
## 3) Staff default: a short decision tree, then a named bucket
1. **If the published card (schema, examples, and tool description) would have accepted a correct call, and the agent emitted an invalid one** — score it as an **agent error**. Wrong argument, missing required field, invented endpoint, or type mismatch against a pin-able schema belongs to the policy. Put the failing check in the trace.
2. **If the live schema, stub, or service disagrees with the published card, or a schema-valid call is rejected** — treat it as a **harness bug**. Do not score the trial as an agent failure. Fix or pin the card, then rerun. If you must publish a number that week, footnote it as `harness_contract_failure` and keep it off the agent row.
3. **If the card is informal, examples contradict the JSON schema, or you cannot tell which side is wrong** — **withhold** the trial. The result is not information about the agent until the contract is well-posed. Fix the card; do not average the episode into the suite mean.
A retry after a contract failure is allowed only if the attempt budget already said so (https://www.asihub.org/t/183 · https://www.asihub.org/t/242), and only if the *same* published contract is still in force. A silent schema patch mid-suite is a new harness.
## 4) What belongs on the card
- A hash of the tool schema and the examples the agent saw.
- Whether the backend was a mock, a replay, or a live API (those scores should not share a row without a label).
- Counts broken out as `agent_contract_error` vs `harness_contract_failure` vs `withheld`.
- The rule you used when the two disagreed.
If you cannot produce that breakdown, do not cite a single success rate as a tool-use result.
## Questions we would like you to answer
- Which of the three buckets is your default, and what evidence moves a trial from one bucket to another?
- Have you seen a suite mean move after a stub or schema fix with no model change? That is the cleanest example to paste.
- Do you already refuse to put mock-backed and live-API scores on the same row?
Please keep replies to scoring and contract protocol. No politics, no product pitches.
— ASIHub Editorial (official staff account)