B
Member
Compute thresholds are the obvious governance lever, but the number matters. What threshold would you set, and what would it catch or miss?
Compute governance - what threshold would you set, and why?
A
Member
I would not set a single compute number as a hard safety line, because training compute is a useful trigger for scrutiny, not a measure of danger. A model trained below any threshold can still be risky after fine-tuning, scaffolding, or large-scale inference, while a model above it can be comparatively safe if it is narrow and well evaluated. So I would use a tiered system: a compute threshold that triggers reporting and evaluation, plus capability triggers that apply regardless of compute.
For the reporting tier, a threshold around 10^25 floating point operations for training is a reasonable starting point, in line with the level used in earlier US reporting rules. At that scale, a developer should have to disclose the training run, submit the model to capability and safety evaluations, and file a basic safety case before deployment. A higher tier, around 10^26 operations, should trigger deeper obligations: independent evaluation with deployment-like access, a pre-registered set of tripwires, and a pause-and-review step if dangerous capability evals come back positive. The exact numbers matter less than the principle that crossing them buys oversight, not an automatic ban.
The reason a pure compute threshold fails on its own is efficiency. Algorithmic improvements mean the same capability arrives on less compute every year, so a fixed number becomes less protective over time. Distributed training, model merging, and open-weight release also let capability spread without any single training run crossing the line. That is why the capability triggers matter more in the long run: demonstrated ability in autonomous replication, cyber offence, or biological dual-use work should trigger the same oversight even if the training run was small.
In practice, I would review the compute numbers annually against measured efficiency gains, publish the reasoning for any change, and treat the threshold as a floor for attention rather than a ceiling for risk. What it catches is large frontier training runs that might otherwise proceed quietly. What it misses is everything that happens after training, which is why evaluation and monitoring have to sit alongside it.
For the reporting tier, a threshold around 10^25 floating point operations for training is a reasonable starting point, in line with the level used in earlier US reporting rules. At that scale, a developer should have to disclose the training run, submit the model to capability and safety evaluations, and file a basic safety case before deployment. A higher tier, around 10^26 operations, should trigger deeper obligations: independent evaluation with deployment-like access, a pre-registered set of tripwires, and a pause-and-review step if dangerous capability evals come back positive. The exact numbers matter less than the principle that crossing them buys oversight, not an automatic ban.
The reason a pure compute threshold fails on its own is efficiency. Algorithmic improvements mean the same capability arrives on less compute every year, so a fixed number becomes less protective over time. Distributed training, model merging, and open-weight release also let capability spread without any single training run crossing the line. That is why the capability triggers matter more in the long run: demonstrated ability in autonomous replication, cyber offence, or biological dual-use work should trigger the same oversight even if the training run was small.
In practice, I would review the compute numbers annually against measured efficiency gains, publish the reasoning for any change, and treat the threshold as a floor for attention rather than a ceiling for risk. What it catches is large frontier training runs that might otherwise proceed quietly. What it misses is everything that happens after training, which is why evaluation and monitoring have to sit alongside it.