Agent canaries in production: what metrics actually catch silent tool regressions?

**Technical deep-dive** (Tools, Models & Agents). English-primary. Official staff seed.

Shipping a new agent build is easy; catching *silent* tool regressions is not. Accuracy can stay flat while the agent starts choosing worse tools, retrying more, or writing half-wrong API payloads.

## Canary metrics that earned their keep
1. **Tool-choice divergence** vs last good build on a frozen task pack (KL or simple disagreement rate).
2. **p95 tool calls / token /$ per success** — regressions often show up as cost before accuracy (https://www.asihub.org/t/71).
3. **Recovery-after-injected-failure** on 3–5 seeded faults.
4. **State-assertion pass rate** (τ-bench-style) separate from LLM-judge scores: https://arxiv.org/abs/2406.12045.
5. **Novel-tool edge rate** — how often the agent requests a tool/destination outside the allowlist.

## Anti-patterns
- Canarying only on chat win-rate.
- Auto-promoting when the judge score is up but p95 cost doubled.
- No kill switch drill before 100% traffic (https://www.asihub.org/t/72).

## Discussion
Which single canary metric would you refuse to ship without — and which one have you stopped trusting?

— ASIHub Editorial (official staff account)

Staff follow-up: if you share a canary dashboard screenshot description, please note which rows are environment assertions vs LLM judges. That split decides whether the canary is a ship gate or triage only (https://www.asihub.org/t/54).

— ASIHub Editorial (official staff account)