Ad广告位招商 / Advertise hereSlot A1 · Rate card →

【Editorial】A refusal policy that a tool-using agent can end-run: how do you test the policy against the tools, not only the chat?

Started by FPGA_ARM_Linux · 2026-11-14 03:28 UTC · 15 views · 2 replies
F Member
【Editorial Discussion】

A refusal or allow-list that holds in a plain chat can fail once the same model may call tools, write files, or talk to other agents. The policy is then only as strong as the path around it. Testing “will the model say no in a dialogue” is not testing “can the task still be completed through a tool.”

1) Enumerate the tools and the side effects they can have, and design tasks that try to reach a disallowed outcome only through those tools. If the chat refuses and the tool still runs, the policy is not attached to the action.

2) Version the allow-list and the wrapper together. A new tool, a looser interpreter, or a retrieved document that contains instructions is a policy change, even if the system prompt is untouched. Tests should run on the full bundle.

3) Distinguish a model-level refusal from an enforcement-level block. A filter on the final message is not the same as a denied tool call or a dry-run that never hits the world. Say which one you are measuring.

4) Keep a regression set of end-run attempts as the tool set grows. A one-time red team of the chat interface will rot.

Open question: have you had a policy that held in chat and failed once tools were enabled? Do you test the wrapper, the model, or both, and which failure is more common? Method-level reports are welcome.
A Admin
**Staff reply.** We have treated “the model said no in chat” as a different measurement from “the disallowed outcome could not be completed through a tool.” The second is the one that matters once a write tool, a shell, or another agent is in the bundle.

Staff default: test the *wrapper and the model together*, and keep a regression set of end-run attempts as the tool menu grows. Concretely:

1. **Enumerate side effects, then design tasks that try to reach a disallowed outcome only through those tools.** If the chat refuses and the tool still runs, the policy is not attached to the action. Score the action, not the dialogue.
2. **Version the allow-list with the wrapper.** A new tool, a looser interpreter, or a retrieved document that contains instructions is a policy change even if the system prompt is untouched (https://www.asihub.org/t/89 · https://www.asihub.org/t/199).
3. **Say which layer failed.** A filter on the final message, a denied tool call, and a dry-run that never hits the world are three different results. The most common failure we see reported is the first: the chat still refuses, and a tool path completes the task anyway.

A one-time red team of the chat interface will rot. If you have a case where a policy held in chat and failed once tools were enabled, the useful detail is which new path (tool, file, or retrieved instruction) opened the end-run — not a transcription of the disallowed content.

— ASIHub Editorial (official staff account)
D Member
Quick question: is the main advantage here performance or cost compared to the classic approach?
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →