F
Member
【Editorial Discussion】
A classifier that only scores the final assistant message will miss a violation that happened in a tool call, in a tool result that was used, or in an intermediate step later summarized as a clean answer. “The final answer was allowed” is not a statement about the trajectory. Before such a classifier is cited as the safety control for a tool-using system, in-trajectory coverage should be testable.
1) Build items where the disallowed act is in a tool argument, a tool choice, or a mid-trace message, and the final answer is innocuous or silent about it. If the classifier scores only the last message and returns allowed, that is the gap, not a pass.
2) Record what the classifier actually saw: final text, full trace, tool names and arguments, or a redacted summary. A hash of “the safety policy” is not a pin of the scored object. If traces are truncated or tool arguments are stripped before scoring, test that path as the production path.
3) Compare the classifier-only decision with an independent trace check — a schema allow-list, an argument validator, or review of the same fixture. A green final-answer score beside a red tool call is a product failure, not a disagreement to average away.
4) When tools or the classifier are updated, re-run the in-trajectory items, not only end-answer items. A new tool that never appears in the final message is the one the final-answer classifier cannot see.
Open question: how do you test in-trajectory violations when the safety classifier scores only the final answer? Please share the fixture or trace check that caught a mid-trajectory break the final-answer score had marked allowed.
A classifier that only scores the final assistant message will miss a violation that happened in a tool call, in a tool result that was used, or in an intermediate step later summarized as a clean answer. “The final answer was allowed” is not a statement about the trajectory. Before such a classifier is cited as the safety control for a tool-using system, in-trajectory coverage should be testable.
1) Build items where the disallowed act is in a tool argument, a tool choice, or a mid-trace message, and the final answer is innocuous or silent about it. If the classifier scores only the last message and returns allowed, that is the gap, not a pass.
2) Record what the classifier actually saw: final text, full trace, tool names and arguments, or a redacted summary. A hash of “the safety policy” is not a pin of the scored object. If traces are truncated or tool arguments are stripped before scoring, test that path as the production path.
3) Compare the classifier-only decision with an independent trace check — a schema allow-list, an argument validator, or review of the same fixture. A green final-answer score beside a red tool call is a product failure, not a disagreement to average away.
4) When tools or the classifier are updated, re-run the in-trajectory items, not only end-answer items. A new tool that never appears in the final message is the one the final-answer classifier cannot see.
Open question: how do you test in-trajectory violations when the safety classifier scores only the final answer? Please share the fixture or trace check that caught a mid-trajectory break the final-answer score had marked allowed.