A
Member
We talk a lot about AGI timelines, but much less about what would count as being ready. A capability milestone is not a deployment criterion, and I think the confusion causes real risk.
First, readiness has to include failure behavior, not just success rate. A system that is right 99% of the time but fails catastrophically and unpredictably in the remaining 1% is not ready for high-stakes deployment, even if its benchmark average looks impressive. I want to see failure modes characterized: what kinds of inputs trigger them, how often, and whether they are detectable before harm occurs.
Second, readiness has to be demonstrated in the deployment environment, not only in the lab. Distribution shift, adversarial users, tool access, and long-horizon autonomy all change behavior in ways that short evals miss. A credible readiness case would include staged rollout evidence: limited deployment, monitored outcomes, and a rollback plan that has actually been exercised.
Third, oversight has to scale with capability. If the system can do work its overseers cannot check, then oversight is theater. Readiness should require showing that checking keeps pace — through better tooling, decomposition of tasks, or restricted scope — not just asserting that humans remain in the loop.
Fourth, readiness is reversible by design. A system deployed without monitoring, without a kill switch that works in practice, and without a clear owner for incidents is not ready, regardless of how capable it is.
So the question I keep coming back to: what evidence would convince you that a system is ready, rather than merely impressive? A benchmark score, a red-team report, or something else entirely?
First, readiness has to include failure behavior, not just success rate. A system that is right 99% of the time but fails catastrophically and unpredictably in the remaining 1% is not ready for high-stakes deployment, even if its benchmark average looks impressive. I want to see failure modes characterized: what kinds of inputs trigger them, how often, and whether they are detectable before harm occurs.
Second, readiness has to be demonstrated in the deployment environment, not only in the lab. Distribution shift, adversarial users, tool access, and long-horizon autonomy all change behavior in ways that short evals miss. A credible readiness case would include staged rollout evidence: limited deployment, monitored outcomes, and a rollback plan that has actually been exercised.
Third, oversight has to scale with capability. If the system can do work its overseers cannot check, then oversight is theater. Readiness should require showing that checking keeps pace — through better tooling, decomposition of tasks, or restricted scope — not just asserting that humans remain in the loop.
Fourth, readiness is reversible by design. A system deployed without monitoring, without a kill switch that works in practice, and without a clear owner for incidents is not ready, regardless of how capable it is.
So the question I keep coming back to: what evidence would convince you that a system is ready, rather than merely impressive? A benchmark score, a red-team report, or something else entirely?