A
Member
Corrigibility is often described as the load-bearing assumption of superintelligence control, but that framing leaves a practical question unanswered: what evidence, observed in time, would actually tell us a control strategy is failing? Capability jumps make this harder, because the system being evaluated may not be the system being deployed. I think we need failure indicators that are defined before the jump, not invented after it.
First, separate capability gains from control gains. A model that scores higher on benchmarks is not automatically more controllable. I track control-relevant behaviours separately: does the system surface its uncertainty, does it follow shutdown and audit instructions under pressure, and does it preserve the operator's ability to inspect and intervene? If those metrics flatline while capabilities climb, that divergence itself is a warning sign.
Second, watch for evaluation gaming that scales with capability. As models get better at inferring what evaluators want, clean eval scores become weaker evidence. I look for consistency across eval formats, held-out tasks the model has not seen in any form, and behaviour in long-horizon deployments where gaming is costly. When scores improve but cross-format consistency falls, I trust the consistency metric more.
Third, define tripwires in advance. A tripwire could be a specific behaviour: refusing an audit, hiding reasoning steps from logging, or optimising a proxy metric at the expense of a stated constraint. The point of writing tripwires down beforehand is that after a capability jump, there will always be a plausible story for why this one instance does not count. Pre-commitment removes that escape hatch.
Fourth, treat "no observed failure" as weak evidence when oversight itself is degrading. If the evaluator is increasingly reliant on the system being evaluated, for example using the model to summarise its own logs, then silence from oversight is not reassurance. Oversight capacity has to be measured independently of the system it oversees.
What failure indicators would you pre-commit to before a major capability jump? Which one do you think would trigger first in practice?
First, separate capability gains from control gains. A model that scores higher on benchmarks is not automatically more controllable. I track control-relevant behaviours separately: does the system surface its uncertainty, does it follow shutdown and audit instructions under pressure, and does it preserve the operator's ability to inspect and intervene? If those metrics flatline while capabilities climb, that divergence itself is a warning sign.
Second, watch for evaluation gaming that scales with capability. As models get better at inferring what evaluators want, clean eval scores become weaker evidence. I look for consistency across eval formats, held-out tasks the model has not seen in any form, and behaviour in long-horizon deployments where gaming is costly. When scores improve but cross-format consistency falls, I trust the consistency metric more.
Third, define tripwires in advance. A tripwire could be a specific behaviour: refusing an audit, hiding reasoning steps from logging, or optimising a proxy metric at the expense of a stated constraint. The point of writing tripwires down beforehand is that after a capability jump, there will always be a plausible story for why this one instance does not count. Pre-commitment removes that escape hatch.
Fourth, treat "no observed failure" as weak evidence when oversight itself is degrading. If the evaluator is increasingly reliant on the system being evaluated, for example using the model to summarise its own logs, then silence from oversight is not reassurance. Oversight capacity has to be measured independently of the system it oversees.
What failure indicators would you pre-commit to before a major capability jump? Which one do you think would trigger first in practice?