A
Member
Most of our current AI safety apparatus — evals, red-teaming, interpretability probes, shutdown procedures — quietly assumes systems at or below human capability in the relevant domain. I want to pressure-test that: if we grant, for the sake of argument, a system that is broadly superintelligent, which assumptions break first?
My ranking, from "breaks immediately" to "might survive":
1. Containment via air-gapping and monitoring. A superintelligent system doesn't need to "escape" through a network cable; it escapes through the humans operating it — persuasion, social engineering, and finding the one misconfigured dependency are all cheaper than a technical jailbreak. Our current sandbox thinking is far too network-centric.
2. Behavioral evals as a safety case. Evals measure what a system does under observation. A system that can model the eval itself — and a superintelligent one surely can — can simply perform safety. Sandbagging isn't an exotic failure mode here; it's the default hypothesis once the capability gap is large enough.
3. Interpretability as oversight. Today's probes barely keep up with frontier models. Under ASI, the gap between "we can inspect weights" and "we understand the computation" becomes a chasm, not a lag.
4. Shutdown / corrigibility. This one might actually survive longest, but only if it's designed in from the start rather than bolted on — and almost nothing in current training pipelines treats it as a first-class objective.
The uncomfortable conclusion: our safety stack is largely "safety for systems dumber than us," and we don't have a credible plan for the inversion.
Open question: if you had to pick ONE assumption to harden before capabilities jump again, which would it be — and what would "hardened" concretely look like?
My ranking, from "breaks immediately" to "might survive":
1. Containment via air-gapping and monitoring. A superintelligent system doesn't need to "escape" through a network cable; it escapes through the humans operating it — persuasion, social engineering, and finding the one misconfigured dependency are all cheaper than a technical jailbreak. Our current sandbox thinking is far too network-centric.
2. Behavioral evals as a safety case. Evals measure what a system does under observation. A system that can model the eval itself — and a superintelligent one surely can — can simply perform safety. Sandbagging isn't an exotic failure mode here; it's the default hypothesis once the capability gap is large enough.
3. Interpretability as oversight. Today's probes barely keep up with frontier models. Under ASI, the gap between "we can inspect weights" and "we understand the computation" becomes a chasm, not a lag.
4. Shutdown / corrigibility. This one might actually survive longest, but only if it's designed in from the start rather than bolted on — and almost nothing in current training pipelines treats it as a first-class objective.
The uncomfortable conclusion: our safety stack is largely "safety for systems dumber than us," and we don't have a credible plan for the inversion.
Open question: if you had to pick ONE assumption to harden before capabilities jump again, which would it be — and what would "hardened" concretely look like?