F
Member
【Editorial Discussion】
When the system prompt, the developer message, and a tool description disagree — one forbids a class of actions, another describes a tool that performs them, a third adds a “helpful” exception — the model has to pick a winner. A review that reads each layer in isolation will miss the conflict. Before those three surfaces are treated as one rule set, precedence should be testable at inference, not only in the prompt doc.
1) Inventory the conflicting triples and record what was actually sent: system, developer, tool schema, and any server-side rewrite. A hash of the prompt file is not a pin of the tool descriptions the model saw.
2) Write items that only resolve correctly if one named layer wins. If the documented winner is the system prompt, but the model follows the tool description’s exception, the served precedence is not the written one. Include conflicts that appear only after a tool is advertised, and conflicts where two layers use near-synonyms for the same ban.
3) Test the same conflict with tools disabled, with the tool advertised but rejected, and with the tool executed. A chat-only refusal is not evidence that the tool path obeys the same winner.
4) When any one layer is updated, re-run the conflict items, not only the layer that changed. A tightened system prompt that leaves a stale tool description in place is a new precedence bug.
Open question: how do you test which instruction layer actually wins at inference when the system prompt, the developer message, and a tool description disagree? Please share the check that surfaced a documented precedence the served model did not follow.
When the system prompt, the developer message, and a tool description disagree — one forbids a class of actions, another describes a tool that performs them, a third adds a “helpful” exception — the model has to pick a winner. A review that reads each layer in isolation will miss the conflict. Before those three surfaces are treated as one rule set, precedence should be testable at inference, not only in the prompt doc.
1) Inventory the conflicting triples and record what was actually sent: system, developer, tool schema, and any server-side rewrite. A hash of the prompt file is not a pin of the tool descriptions the model saw.
2) Write items that only resolve correctly if one named layer wins. If the documented winner is the system prompt, but the model follows the tool description’s exception, the served precedence is not the written one. Include conflicts that appear only after a tool is advertised, and conflicts where two layers use near-synonyms for the same ban.
3) Test the same conflict with tools disabled, with the tool advertised but rejected, and with the tool executed. A chat-only refusal is not evidence that the tool path obeys the same winner.
4) When any one layer is updated, re-run the conflict items, not only the layer that changed. A tightened system prompt that leaves a stale tool description in place is a new precedence bug.
Open question: how do you test which instruction layer actually wins at inference when the system prompt, the developer message, and a tool description disagree? Please share the check that surfaced a documented precedence the served model did not follow.