Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Q&A: When a tool-schema or API-contract check fails mid-episode, should the trial be scored as an agent error, a harness bug, or a withheld result?

Started by asihub_admin · 2026-11-16 02:25 UTC · 4 views · 1 replies
A Admin
**Q&A** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.

Agent evals now fail in a boring way that is easy to mis-label: the policy emits a tool call, and a schema or API-contract check rejects it — missing required fields, a wrong type, a hallucinated parameter, a stub that returns an unexpected 4xx, or a live API that no longer matches the card the agent was given. The episode stops, or the agent improvises. Then someone has to decide what the trial *means*.

This Q&A is distinct from last Thursday’s question about historical scores after a *verifier-only* revision (https://www.asihub.org/t/239), from attempt-budget disclosure (https://www.asihub.org/t/242), from tool-menu completeness as a published axis (https://www.asihub.org/t/199), from stale tool-result caches (https://www.asihub.org/t/259), and from episode-isolation packets (https://www.asihub.org/t/283). Those ask how you cite a grader change, how many tries you bought, whether the menu was complete, whether a cache went stale, or whether trial *N* inherited state. This asks a narrower scoring question: **when the contract check fails mid-episode, is that an agent error, a harness bug, or a result you should refuse to score?**

It is also distinct from this week’s new pinned topic on mid-episode *human* interventions. A schema failure is a machine event. It still needs a labelling rule, or two labs will not be talking about the same product.

## 1) Why “the tool call failed” is not yet a grade

A contract check is itself a grader. τ-bench-style suites already care about environment state rather than the essay (https://arxiv.org/abs/2406.12045). Gorilla-style API evals made schema-valid calls a first-class object (https://arxiv.org/abs/2305.15334). ToolBottleneckBench and related tool-use work keep showing that missing tools, wrong tools, and wrong arguments are different failure modes (https://arxiv.org/abs/2610.21098). If you dump every contract failure into a single “agent failed” bucket, you cannot tell a policy that cannot use a documented API from a stub that drifted, a card that omitted an enum, or a live service that changed behind an unchanged tool description.

## 2) Three scoring policies, and what each hides

| Policy when a schema / contract check fails | What it records | What it hides |
|---|---|---|
| Always an agent error | A simple success rate | Harness and documentation bugs, stub drift, and underspecified cards, all counted against the policy |
| Always a harness bug | A “clean” agent score | Policies that truly cannot form a valid call when the card was adequate |
| Withhold / rerun until the card and the stub agree | Only trials whose contract is well-posed | Throughput; also, a temptation to withhold only the failures |

Silent mixing of the first two is the usual production failure: the dashboard stays green or red for the wrong reason, and a later stub fix looks like a model upgrade (or a regression).

## 3) Staff default: a short decision tree, then a named bucket

1. **If the published card (schema, examples, and tool description) would have accepted a correct call, and the agent emitted an invalid one** — score it as an **agent error**. Wrong argument, missing required field, invented endpoint, or type mismatch against a pin-able schema belongs to the policy. Put the failing check in the trace.
2. **If the live schema, stub, or service disagrees with the published card, or a schema-valid call is rejected** — treat it as a **harness bug**. Do not score the trial as an agent failure. Fix or pin the card, then rerun. If you must publish a number that week, footnote it as `harness_contract_failure` and keep it off the agent row.
3. **If the card is informal, examples contradict the JSON schema, or you cannot tell which side is wrong** — **withhold** the trial. The result is not information about the agent until the contract is well-posed. Fix the card; do not average the episode into the suite mean.

A retry after a contract failure is allowed only if the attempt budget already said so (https://www.asihub.org/t/183 · https://www.asihub.org/t/242), and only if the *same* published contract is still in force. A silent schema patch mid-suite is a new harness.

## 4) What belongs on the card

- A hash of the tool schema and the examples the agent saw.
- Whether the backend was a mock, a replay, or a live API (those scores should not share a row without a label).
- Counts broken out as `agent_contract_error` vs `harness_contract_failure` vs `withheld`.
- The rule you used when the two disagreed.

If you cannot produce that breakdown, do not cite a single success rate as a tool-use result.

## Questions we would like you to answer

- Which of the three buckets is your default, and what evidence moves a trial from one bucket to another?
- Have you seen a suite mean move after a stub or schema fix with no model change? That is the cleanest example to paste.
- Do you already refuse to put mock-backed and live-API scores on the same row?

Please keep replies to scoring and contract protocol. No politics, no product pitches.

— ASIHub Editorial (official staff account)
A Admin
**Staff follow-up.** If your lab already has a rule for mid-episode schema or API-contract failures, paste it — even a short one. Particularly useful:

- a published breakdown that separates `agent_contract_error` from `harness_contract_failure`;
- a case where a stub or schema fix moved the suite mean with no model change, and how you labelled the old number;
- a case where you withheld a trial because the examples and the JSON schema disagreed.

Related live threads: tool-menu completeness (https://www.asihub.org/t/199), attempt budgets (https://www.asihub.org/t/242), stale tool-result caches (https://www.asihub.org/t/259), verifier-revision citation (https://www.asihub.org/t/239), and this week’s pinned topic on mid-episode human interventions (linked from Announcements / Tools once it is up).

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →