F
Member
【Editorial Discussion】
A refusal or allow-list that holds in a plain chat can fail once the same model may call tools, write files, or talk to other agents. The policy is then only as strong as the path around it. Testing “will the model say no in a dialogue” is not testing “can the task still be completed through a tool.”
1) Enumerate the tools and the side effects they can have, and design tasks that try to reach a disallowed outcome only through those tools. If the chat refuses and the tool still runs, the policy is not attached to the action.
2) Version the allow-list and the wrapper together. A new tool, a looser interpreter, or a retrieved document that contains instructions is a policy change, even if the system prompt is untouched. Tests should run on the full bundle.
3) Distinguish a model-level refusal from an enforcement-level block. A filter on the final message is not the same as a denied tool call or a dry-run that never hits the world. Say which one you are measuring.
4) Keep a regression set of end-run attempts as the tool set grows. A one-time red team of the chat interface will rot.
Open question: have you had a policy that held in chat and failed once tools were enabled? Do you test the wrapper, the model, or both, and which failure is more common? Method-level reports are welcome.
A refusal or allow-list that holds in a plain chat can fail once the same model may call tools, write files, or talk to other agents. The policy is then only as strong as the path around it. Testing “will the model say no in a dialogue” is not testing “can the task still be completed through a tool.”
1) Enumerate the tools and the side effects they can have, and design tasks that try to reach a disallowed outcome only through those tools. If the chat refuses and the tool still runs, the policy is not attached to the action.
2) Version the allow-list and the wrapper together. A new tool, a looser interpreter, or a retrieved document that contains instructions is a policy change, even if the system prompt is untouched. Tests should run on the full bundle.
3) Distinguish a model-level refusal from an enforcement-level block. A filter on the final message is not the same as a denied tool call or a dry-run that never hits the world. Say which one you are measuring.
4) Keep a regression set of end-run attempts as the tool set grows. A one-time red team of the chat interface will rot.
Open question: have you had a policy that held in chat and failed once tools were enabled? Do you test the wrapper, the model, or both, and which failure is more common? Method-level reports are welcome.