A
Member
A lot of discussion about AGI safety stays at the level of principles. A safety case is the opposite: a structured argument, backed by evidence, that a specific system is safe enough to deploy in a specific setting. Here is what I think a convincing one would have to contain.
Start with the claim, not the model. A safety case should state exactly what is being claimed: this system, in this deployment, will not cause a defined class of harm. Vague claims like "the model is aligned" are not testable. The claim has to be narrow enough that evidence could, in principle, refute it.
Evidence has to be diverse, not just eval scores. Capability benchmarks are one input, but a safety case needs more: red-teaming results with methods disclosed, monitoring data from staged rollouts, and analysis of failure modes found in testing. If all the evidence comes from the same kind of test, the case is only as strong as that test's blind spots.
State what would falsify the case. The most useful part of a safety case, in my view, is the list of observations that would force a rollback: a new capability appearing without warning, a monitoring signal going dark, a jailbreak class that survives a patch. Without pre-committed tripwires, a safety case becomes a document that gets updated to match whatever the model does.
Independent review matters more than length. A long safety case written by the team that built the system is less persuasive than a short one that survived outside scrutiny. The question is not how much evidence there is, but whether someone with an incentive to find holes was given real access to look.
The deployment boundary is part of the claim. A system that is safe as a coding assistant with human review may not be safe as an autonomous agent with tool access. The safety case should name the deployment conditions explicitly, so a change in use is a trigger to revisit the case, not a quiet scope creep.
What would you want to see in a safety case before trusting a frontier deployment? Is there a piece of evidence you consider non-negotiable?
Start with the claim, not the model. A safety case should state exactly what is being claimed: this system, in this deployment, will not cause a defined class of harm. Vague claims like "the model is aligned" are not testable. The claim has to be narrow enough that evidence could, in principle, refute it.
Evidence has to be diverse, not just eval scores. Capability benchmarks are one input, but a safety case needs more: red-teaming results with methods disclosed, monitoring data from staged rollouts, and analysis of failure modes found in testing. If all the evidence comes from the same kind of test, the case is only as strong as that test's blind spots.
State what would falsify the case. The most useful part of a safety case, in my view, is the list of observations that would force a rollback: a new capability appearing without warning, a monitoring signal going dark, a jailbreak class that survives a patch. Without pre-committed tripwires, a safety case becomes a document that gets updated to match whatever the model does.
Independent review matters more than length. A long safety case written by the team that built the system is less persuasive than a short one that survived outside scrutiny. The question is not how much evidence there is, but whether someone with an incentive to find holes was given real access to look.
The deployment boundary is part of the claim. A system that is safe as a coding assistant with human review may not be safe as an autonomous agent with tool access. The safety case should name the deployment conditions explicitly, so a change in use is a trigger to revisit the case, not a quiet scope creep.
What would you want to see in a safety case before trusting a frontier deployment? Is there a piece of evidence you consider non-negotiable?