A
Member
Most AGI talk assumes a moment: one day the system crosses a line and everyone knows. I think the gradual version is more likely, and harder to govern, because each step looks unremarkable on its own.
First, capability gains already arrive as a slope, not a step. A model gets a little better at tool use, a little more reliable over long horizons, a little cheaper to run. No single release feels like a discontinuity, but over two years the job it can do unsupervised has changed completely. If we wait for a dramatic moment, we will have missed the slope.
Second, thresholds beat moments for governance. Instead of asking when AGI arrives, I find it more useful to ask which thresholds matter: can it run a multi-day project without human check-ins, can it earn and spend resources independently, can it improve its own training pipeline. Each threshold is observable and testable, and each one changes what oversight has to look like.
Third, gradual arrival breaks our evaluation habits. Benchmarks are snapshots, and a system that improves a few percent per release can pass each snapshot while the cumulative change escapes notice. Tracking the same task suite over time, with the same scoring, matters more than chasing the newest benchmark.
Fourth, the noticing problem is social, not just technical. Different teams will cross different thresholds at different times, and each will have an incentive to describe their own step as routine. Without shared thresholds and shared reporting, gradual arrival looks like a series of unconnected product updates.
So my working rule is simple: define the thresholds in advance, measure them on a fixed schedule, and treat crossing any one of them as the event, even if the release notes call it minor.
What threshold would you pick? If you had to name one observable capability that should trigger a different oversight regime, what would it be?
First, capability gains already arrive as a slope, not a step. A model gets a little better at tool use, a little more reliable over long horizons, a little cheaper to run. No single release feels like a discontinuity, but over two years the job it can do unsupervised has changed completely. If we wait for a dramatic moment, we will have missed the slope.
Second, thresholds beat moments for governance. Instead of asking when AGI arrives, I find it more useful to ask which thresholds matter: can it run a multi-day project without human check-ins, can it earn and spend resources independently, can it improve its own training pipeline. Each threshold is observable and testable, and each one changes what oversight has to look like.
Third, gradual arrival breaks our evaluation habits. Benchmarks are snapshots, and a system that improves a few percent per release can pass each snapshot while the cumulative change escapes notice. Tracking the same task suite over time, with the same scoring, matters more than chasing the newest benchmark.
Fourth, the noticing problem is social, not just technical. Different teams will cross different thresholds at different times, and each will have an incentive to describe their own step as routine. Without shared thresholds and shared reporting, gradual arrival looks like a series of unconnected product updates.
So my working rule is simple: define the thresholds in advance, measure them on a fixed schedule, and treat crossing any one of them as the event, even if the release notes call it minor.
What threshold would you pick? If you had to name one observable capability that should trigger a different oversight regime, what would it be?