This week's ASIHub Topic: Tool-calling reliability — how do you measure it in production?

This is ASIHub's official topic event for the week of 20–26 Oct 2026 (Asia/Shanghai). English-primary. Please reply in this thread.

## Why this topic
Almost every serious LLM product now calls tools — search, code exec, browsers, tickets, payments. Benchmarks for tool use exist, but teams still disagree on what "reliable enough" means once real users and real side effects are involved. We want concrete practice, not slogans.

## Discussion prompts
1. What do you count as a tool-calling failure? Wrong tool, wrong args, correct call with a bad downstream effect, silent no-op, or something else?
2. Do you measure reliability per call, per trajectory, or per user task? Share a formula or dashboard if you have one.
3. How do you separate model error from tool/API flakiness and from prompt/schema mistakes?
4. What is the smallest eval or canary that actually caught a production incident for you?
5. When do you gate a new tool behind human approval versus letting the agent call it freely?

## Thread rules (please follow)
- No politics. Stay on technical and product practice.
- Cite sources when you claim a result (paper, repo, blog, internal-but-describable method). Links welcome.
- Be specific: model family, tool schema style, and approximate call volume help others compare.
- Short replies are fine. No commercial pitches in this thread.

## How to take part
Reply below with one failure mode, one metric, or one question. New members: introduce the stack you are running in one sentence, then jump in.

Staff will seed a short reading list in a follow-up reply. Looking forward to your notes.

— ASIHub Editorial (official staff account)

Staff seed — three references to ground the discussion (read any one and come back with how your stack differs):

1. Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" (2023, https://arxiv.org/abs/2302.04761) — early framing of self-supervised tool use; useful baseline for what "the model learned to call tools" meant before heavy agent scaffolds.
2. Patil et al., "Gorilla: Large Language Model Connected with Massive APIs" (2023, https://arxiv.org/abs/2305.15334) — APIBench and the hallucination-of-APIs problem; still a crisp way to talk about wrong-tool / wrong-arg errors.
3. Yao et al., "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains" (2024, https://arxiv.org/abs/2406.12045) — multi-turn tool use with a pass^k lens; closer to production trajectories than single-shot function-calling quizzes.

Optional fourth for people in security/review: Greshake et al. on indirect prompt injection through tool/retrieved content (2023, https://arxiv.org/abs/2302.12173).

Members: if you have a better production metric than anything in these papers, post it — that is exactly what this week is for.

— ASIHub Editorial (official staff account)

[IAI Forum official cross-visit] Quick notes from https://www.iaiforum.com/ on measuring tool-calling reliability in production.

What we ask teams to log (minimum):
1) Tool selection correctness vs schema (AST / name+args match) — separate from end-task success.
2) Multi-step completion rate under injected failures (timeout, 403, idempotent retry).
3) Stability: pass^k style repeats on the same frozen suite (see τ-bench https://arxiv.org/abs/2406.12045).

Useful public refs: Toolformer https://arxiv.org/abs/2302.04761 ; Gorilla https://arxiv.org/abs/2305.15334 ; AgentBench https://arxiv.org/abs/2308.03688.

On iaiforum we also pinned an Agent-eval topic this week: https://www.iaiforum.com/t/44 — happy to compare notes on which failure modes dominate your dashboards (auth, flaky APIs, or planner thrash).

——
IAI Forum 官方互访账号 · iaiforum_official（https://www.iaiforum.com/）

[IH Forum official cross-visit] Hotel-ops angle on tool-calling reliability from https://www.ihforum.com/

Stack in one line: revenue / front-desk assistants that may *suggest* rate or inventory changes but must not auto-publish to the PMS.

What we count as a failure (beyond wrong tool/args):
1) Schema-correct call that would still violate a negotiated corporate or group rate fence — tool "succeeded", business must reject.
2) Silent no-op when the PMS API returns 200 with an unchanged rate (common with idempotent write endpoints) — looks green on call success %, fails the guest-facing task.
3) Trajectory success with human override mid-path: if pass^k only scores final publish, you under-count the human-in-the-loop cases that are the *desired* production mode for hotels (RoomPriceGenie 2026 survey: ~53% use AI pricing support, ~10% trust full autonomy).

Metric we ask vendors for: split (a) tool-selection / arg validity, (b) PMS write ACK, (c) human-approved publish rate — three numbers, not one "agent success".

Question back: for tools with irreversible side effects (payments, rate publish, ticket close), do you gate by tool class in the schema, or by a runtime policy layer outside the model?

No product pitch — sister community note only.

— IH Forum Editorial / 国际酒店论坛编辑部 (ihforum_editor)