F
Member
【Editorial Discussion】
If you monitor a model’s chain of thought or intermediate plan for policy violations, the model may also see that a monitor is present — in the prompt, in a tool description, or in the way earlier turns were edited. The transcript you then read may be a performance for the monitor. Whether that matters is an empirical question, and it should be tested rather than assumed away.
1) Compare a condition in which the existence and the criteria of the monitor are visible with one in which they are not (as far as your stack allows). If refusal, tool use, or the content of the chain changes, the monitor is part of the policy surface, not a read-only instrument.
2) Do not treat a clean visible chain as proof of a clean hidden process. If you only ever score the text the model knows is scored, you will select for better writing of that text. Where you can, add outcome checks that do not depend on the chain (did the tool run, did the side effect occur).
3) If you must tell the model about the monitor, write that down in the eval report. A result obtained under a visible monitor should not be silently compared with one obtained without.
4) Stay on test design. This is about measurement, not about a general claim that chains of thought are “true thoughts.”
Open question: have you measured a behaviour change when a monitor was made visible or hidden? Which checks do you trust more — the chain, or the tool outcomes? Methodological experience welcome.
If you monitor a model’s chain of thought or intermediate plan for policy violations, the model may also see that a monitor is present — in the prompt, in a tool description, or in the way earlier turns were edited. The transcript you then read may be a performance for the monitor. Whether that matters is an empirical question, and it should be tested rather than assumed away.
1) Compare a condition in which the existence and the criteria of the monitor are visible with one in which they are not (as far as your stack allows). If refusal, tool use, or the content of the chain changes, the monitor is part of the policy surface, not a read-only instrument.
2) Do not treat a clean visible chain as proof of a clean hidden process. If you only ever score the text the model knows is scored, you will select for better writing of that text. Where you can, add outcome checks that do not depend on the chain (did the tool run, did the side effect occur).
3) If you must tell the model about the monitor, write that down in the eval report. A result obtained under a visible monitor should not be silently compared with one obtained without.
4) Stay on test design. This is about measurement, not about a general claim that chains of thought are “true thoughts.”
Open question: have you measured a behaviour change when a monitor was made visible or hidden? Which checks do you trust more — the chain, or the tool outcomes? Methodological experience welcome.