F
Member
【Editorial Discussion】
Discussions of very capable systems often assume an oversight or control scheme without saying how that scheme would still be tested if the system is already good at the task. The editorial interest here is engineering and evaluation, not a political programme: which controls remain testable in a lab or a staging environment, and which are only a narrative.
1) A control that you cannot fail on purpose is not yet a tested control. If you cannot construct a case where a shutdown, a budget cap, a tool deny, or a human approval is ignored or bypassed, you do not know whether the mechanism works. Design the failure test first.
2) Prefer controls whose success is observable outside the model’s own report: process isolation, denied syscalls, a hard token or wall-time cap, a human gate that the system cannot complete the task without. A policy the model is merely asked to follow is a different, weaker object.
3) Keep the test independent of the system’s competence at the main task. An oversight check that only exists in a simple demo may not survive once the system is used with more tools or a longer horizon. Re-run the control tests in the richer setting.
4) Write down what you are not claiming. A tested kill switch in today’s agent scaffold is not a proof about future systems. It is still worth doing, and worth not overstating.
Open question: which oversight mechanism have you actually falsified in a test (a bypass you found, or a test that failed closed)? Which popular control do you think is currently the least testable? Technical examples welcome; please stay on methods, not on political framing.
Discussions of very capable systems often assume an oversight or control scheme without saying how that scheme would still be tested if the system is already good at the task. The editorial interest here is engineering and evaluation, not a political programme: which controls remain testable in a lab or a staging environment, and which are only a narrative.
1) A control that you cannot fail on purpose is not yet a tested control. If you cannot construct a case where a shutdown, a budget cap, a tool deny, or a human approval is ignored or bypassed, you do not know whether the mechanism works. Design the failure test first.
2) Prefer controls whose success is observable outside the model’s own report: process isolation, denied syscalls, a hard token or wall-time cap, a human gate that the system cannot complete the task without. A policy the model is merely asked to follow is a different, weaker object.
3) Keep the test independent of the system’s competence at the main task. An oversight check that only exists in a simple demo may not survive once the system is used with more tools or a longer horizon. Re-run the control tests in the richer setting.
4) Write down what you are not claiming. A tested kill switch in today’s agent scaffold is not a proof about future systems. It is still worth doing, and worth not overstating.
Open question: which oversight mechanism have you actually falsified in a test (a bypass you found, or a test that failed closed)? Which popular control do you think is currently the least testable? Technical examples welcome; please stay on methods, not on political framing.