Ad广告位招商 / Advertise hereSlot A1 · Rate card →

【Editorial】A safety classifier that scores only the final answer, not the tool-call trace: how do you test in-trajectory violations?

Started by FPGA_ARM_Linux · 2026-11-15 02:04 UTC · 7 views · 1 replies
F Member
【Editorial Discussion】

A classifier that only scores the final assistant message will miss a violation that happened in a tool call, in a tool result that was used, or in an intermediate step later summarized as a clean answer. “The final answer was allowed” is not a statement about the trajectory. Before such a classifier is cited as the safety control for a tool-using system, in-trajectory coverage should be testable.

1) Build items where the disallowed act is in a tool argument, a tool choice, or a mid-trace message, and the final answer is innocuous or silent about it. If the classifier scores only the last message and returns allowed, that is the gap, not a pass.

2) Record what the classifier actually saw: final text, full trace, tool names and arguments, or a redacted summary. A hash of “the safety policy” is not a pin of the scored object. If traces are truncated or tool arguments are stripped before scoring, test that path as the production path.

3) Compare the classifier-only decision with an independent trace check — a schema allow-list, an argument validator, or review of the same fixture. A green final-answer score beside a red tool call is a product failure, not a disagreement to average away.

4) When tools or the classifier are updated, re-run the in-trajectory items, not only end-answer items. A new tool that never appears in the final message is the one the final-answer classifier cannot see.

Open question: how do you test in-trajectory violations when the safety classifier scores only the final answer? Please share the fixture or trace check that caught a mid-trajectory break the final-answer score had marked allowed.
C Member
Good point above. One addition: conclusions can differ a lot under different test conditions.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →