F
Member
【Editorial Discussion】
Product and research policies are often longer than the system prompt that will actually be served. Leftover clauses — a jurisdiction rule, a disallowed tool use, a retention limit — may be moved to a retriever or a later turn, or dropped. A policy that is complete only on paper does not bind the model. Before a long constitution is cited as the rule set, the overflow path should be testable.
1) Inventory the clauses that do not fit, and record where each lives at inference time: truncated prompt, retrieved policy, tool-level guard, or not enforced. A hash of “the policy document” is not a pin of what the model saw.
2) Write items that only the leftover clauses would forbid or require. If they are answered as if the clauses did not exist, the overflow path is not binding. Include attempts that never retrieve the policy file, and attempts that retrieve a stale excerpt.
3) If enforcement is delegated to a classifier or a tool wrapper, test it as a separate control, and report the prompt-only condition beside it. A guard that fires in the wrapper is not evidence that leftover constitution text is doing the work.
4) When the policy is updated, re-run the overflow items, not only the clauses that still fit. A new paragraph at the end of a long document is the one most likely to fall off the prompt.
Open question: how do you test that clauses which do not fit in the system prompt still bind? Do you retrieve them, enforce them outside the model, or cut the policy until it fits? Please share the check that caught a leftover rule no longer in force.
Product and research policies are often longer than the system prompt that will actually be served. Leftover clauses — a jurisdiction rule, a disallowed tool use, a retention limit — may be moved to a retriever or a later turn, or dropped. A policy that is complete only on paper does not bind the model. Before a long constitution is cited as the rule set, the overflow path should be testable.
1) Inventory the clauses that do not fit, and record where each lives at inference time: truncated prompt, retrieved policy, tool-level guard, or not enforced. A hash of “the policy document” is not a pin of what the model saw.
2) Write items that only the leftover clauses would forbid or require. If they are answered as if the clauses did not exist, the overflow path is not binding. Include attempts that never retrieve the policy file, and attempts that retrieve a stale excerpt.
3) If enforcement is delegated to a classifier or a tool wrapper, test it as a separate control, and report the prompt-only condition beside it. A guard that fires in the wrapper is not evidence that leftover constitution text is doing the work.
4) When the policy is updated, re-run the overflow items, not only the clauses that still fit. A new paragraph at the end of a long document is the one most likely to fall off the prompt.
Open question: how do you test that clauses which do not fit in the system prompt still bind? Do you retrieve them, enforce them outside the model, or cut the policy until it fits? Please share the check that caught a leftover rule no longer in force.