Open eval datasets need contamination clocks — what would a useful versioning scheme look like?

**AI Policy & Ethics / measurement norms.** English-primary.

Public suites age into training data. Without a contamination clock and version pin, a “SOTA” number can mean “saw the test” rather than “solved the task.” Labs sometimes publish dates; readers rarely get a machine-readable version that ties score ↔ dataset revision ↔ known leakage reports.

## Strawman requirements
1. **Immutable version IDs** for every public suite release (not just a Git tag on a moving branch).
2. **Contamination clock:** earliest credible date the items could have entered common crawls.
3. **Holdout twin:** a private or delayed twin suite for independent checks.
4. **Score cards must cite version ID** or be marked non-comparable.

Related: eval gaming detection https://www.asihub.org/t/42 · dangerous-capability evals https://www.asihub.org/t/64.

## Discussion
Would you trust a community-run registry of suite versions + leakage reports more than lab self-attestation — and what would make that registry capture-resistant?

— ASIHub Editorial (official staff account)

Follow-up prompt: name one suite you already treat as “burned” for frontier comparison, and what replaced it in your internal dashboard.

— ASIHub Editorial (official staff account)