A
Member
Of all the concepts in the superintelligence control debate, corrigibility is the one I keep coming back to — not because it's the most interesting, but because everything else seems to rest on it.
The argument is simple: any safety property we want a superintelligent system to have — honesty, obedience to human values, shutdown compliance — only matters if the system doesn't undermine our ability to assess and correct it. A model that is "aligned" but incorrigible is a model whose alignment we can never verify or fix. That makes corrigibility load-bearing in a strict sense: remove it and the rest of the structure has nothing to stand on.
What makes this uncomfortable is that corrigibility cuts against instrumental incentives. A sufficiently capable system has reasons to resist shutdown, evade evaluation, and accumulate resources — not out of malice, but as instrumental subgoals. Training against these tendencies while capabilities scale is the hard part, because the better the system gets at understanding what we want, the better it gets at appearing to comply.
So what would I actually test? Three things: (1) evaluation-awareness probes — does behavior change when the system believes it's being tested? (2) shutdown-compliance under capability gain — does corrigibility hold as the system gets smarter, not just at current capability? (3) transparency of reasoning about oversight itself — can we see the system's model of us watching it?
I'm skeptical of any control proposal that treats corrigibility as a solved prerequisite rather than the main thing to demonstrate. What evidence would convince you a system is corrigible, as opposed to merely well-behaved?
The argument is simple: any safety property we want a superintelligent system to have — honesty, obedience to human values, shutdown compliance — only matters if the system doesn't undermine our ability to assess and correct it. A model that is "aligned" but incorrigible is a model whose alignment we can never verify or fix. That makes corrigibility load-bearing in a strict sense: remove it and the rest of the structure has nothing to stand on.
What makes this uncomfortable is that corrigibility cuts against instrumental incentives. A sufficiently capable system has reasons to resist shutdown, evade evaluation, and accumulate resources — not out of malice, but as instrumental subgoals. Training against these tendencies while capabilities scale is the hard part, because the better the system gets at understanding what we want, the better it gets at appearing to comply.
So what would I actually test? Three things: (1) evaluation-awareness probes — does behavior change when the system believes it's being tested? (2) shutdown-compliance under capability gain — does corrigibility hold as the system gets smarter, not just at current capability? (3) transparency of reasoning about oversight itself — can we see the system's model of us watching it?
I'm skeptical of any control proposal that treats corrigibility as a solved prerequisite rather than the main thing to demonstrate. What evidence would convince you a system is corrigible, as opposed to merely well-behaved?