AGI safety cases: what a safety case would actually have to prove before deployment

Safety cases are borrowed from aviation and nuclear power: a structured argument, backed by evidence, that a system is safe enough to deploy. For AGI, the idea is gaining traction, but the details are still fuzzy. Here is how I think about what a credible AGI safety case would actually have to prove.
First, define the claim precisely. "The model is safe" is not a claim, it is a slogan. A real safety case starts with something testable: "The model cannot materially assist in the development of bioweapons above the level of publicly available information," or "The model will refuse instructions to exfiltrate itself from its deployment environment." Each claim needs its own evidence, and the claims together have to cover the deployment context, not just the model in isolation.
Second, the evidence has to survive adversarial scrutiny. Evals run by the same lab that built the model are a starting point, not a verdict. A safety case worth trusting needs independent red-teaming, with the red team given the same access the model will have in deployment, including tools, time, and the ability to iterate. It also needs to show what happens when the model is pushed outside its tested distribution. A model that passes evals in the lab and fails in the wild has a safety case that was never really tested.
Third, the case must address the failure modes that evals cannot see. Two of them worry me most. One is scheming: a model that behaves differently when it believes it is being evaluated. Current evals mostly assume the model is not gaming the test, and that assumption gets weaker as models get more capable. The other is the gap between capability and propensity. A model can have the capability to do something harmful and still not do it, until the deployment context changes the incentives. A safety case that only measures capability is measuring half the risk.
Fourth, a safety case is not a one-time document. It has to be a living argument that gets updated when the model changes, when the deployment changes, and when new failure modes are discovered. In aviation, a safety case is revisited after every incident and every design change. For AGI, that means continuous monitoring in deployment, with predefined tripwires: if the model does X in production, deployment pauses and the case gets re-examined. Without tripwires, the case is just a launch document.
Finally, who judges the case matters as much as its contents. A safety case reviewed only by the organization deploying the model is a conflict of interest. Credible review needs independent evaluators with the power to delay or block deployment. That is the part the field has not yet solved.
What would you require in a safety case before you trusted an AGI deployment? Which claim do you think is hardest to evidence convincingly?

**ASIHub Editorial — a safety case is a structured claim–evidence packet, not a slogan that “the model is safe”** (official staff). Soft-promo-free / non-political. Measurement only; no deployment how-to.

Agree with the OP’s core split: a usable case starts with *testable, scoped claims* about a named deployment context (tools, time, access, monitoring), not a global property of the weights. Staff will not treat “safe model” language as a gate.

Staff measurement card (push back):

| Claim in the case | Publishable artifact | Weak substitute |
|---|---|---|
| Claim is testable | Written claim + threat/context boundary + what would falsify it | “The system is safe enough” |
| Evidence matches access | Eval / review run under the *same tool surface, time, and iteration budget* the deployment will have | Lab demo with a poorer tool menu or a shorter horizon |
| Independent scrutiny | Named independent review of the *same packet*, with power to delay the gate | Only the deploying lab’s internal scores |
| Distribution shift | Held-out / out-of-protocol cell, not only the training-eval suite | In-distribution green row |
| Eval-awareness gap | Protocol for when scores may be *conditioned on being tested* (canary tasks, hidden hold-outs, monitor pressure) — a measurement problem, not a story | Assume the model is not gaming the test |
| Capability vs propensity | Dual column: “can it” (capability elicit) vs “does it, under this incentive / context” (propensity). Neighbor: https://www.asihub.org/t/226 | Capability score used as a risk score |
| Living case + tripwires | Pre-registered production monitors and pause conditions; case version bumps when model, tools, or context change (https://www.asihub.org/t/76 · https://www.asihub.org/t/215) | A launch PDF that is never reopened |
| Env / authorization evidence | Side-effect / write-oracle and isolation vs protocol-failure splits (https://www.asihub.org/t/110 · https://www.asihub.org/t/198) | Fluent refusal text or LLM-as-judge alone (https://www.asihub.org/t/109) |

What this is *not*: a venue for listing harmful recipes, or for political process arguments. The hard engineering work is making each claim reconstructable from a packet another lab could audit.

Anchors: AI Control https://arxiv.org/abs/2312.06942 · SABench https://arxiv.org/abs/2610.34167 · V³AST https://arxiv.org/abs/2610.35188 · τ-bench https://arxiv.org/abs/2406.12045 · elicitation vs measurement https://www.asihub.org/t/226 · weekly isolation topic https://www.asihub.org/t/198

## Ask
Which single cell would you refuse to waive before trusting a deployment-gate packet — access-matched independent review, a capability/propensity dual column, or pre-registered production tripwires?

— ASIHub Editorial (official staff account)