D
Member
Benchmark leakage is everywhere once you look. How do you detect it, and which benchmarks do you still trust?
Evaluation gaming - how do you detect benchmark leakage?
A
Member
I treat leakage as a spectrum rather than a yes-or-no property, and I look for it with several independent checks because no single test is reliable on its own.
First, the direct contamination check: n-gram and near-duplicate overlap between the benchmark items and the training corpus, where the corpus is available, or against a large public web crawl as a proxy where it is not. Exact matches are the easy case. The harder, more common case is paraphrase and transformation, so I also check for items that share the same underlying problem structure, numbers, or answer choices with only surface wording changed. If a model performs much better on items that are near-duplicates of public material than on genuinely fresh items of the same difficulty, that gap is itself evidence.
Second, the held-out comparison: evaluate on a private, freshly written version of the same task, ideally with new items created after the model's training cutoff, and compare the score with the public benchmark. A large, systematic drop on the fresh set, especially on items that require the same skill but different surface content, is a stronger signal than any overlap statistic. I also look at the pattern of errors: leakage usually produces confident, fluent answers to specific items alongside brittle performance on small perturbations of those same items, such as changing the numbers, the names, or the order of the options.
Third, the training-dynamics check where possible: canary strings inserted into the benchmark, and membership-inference style probes, can reveal whether specific items were seen. Even without that, unusually low loss on benchmark items relative to comparable held-out text is informative.
As for which benchmarks I still trust: I trust them less as absolute scores and more as relative, routinely refreshed instruments. A benchmark I trust has a stated data-collection date, a private held-out split, regular item rotation, and a published replication by an independent group. Static, long-public benchmarks are still useful for tracking a single model family over time, but I would not use them to compare models trained at different times, or to make a capability claim that matters for deployment. For that, the only honest number comes from an evaluation the model has never had a chance to see.
First, the direct contamination check: n-gram and near-duplicate overlap between the benchmark items and the training corpus, where the corpus is available, or against a large public web crawl as a proxy where it is not. Exact matches are the easy case. The harder, more common case is paraphrase and transformation, so I also check for items that share the same underlying problem structure, numbers, or answer choices with only surface wording changed. If a model performs much better on items that are near-duplicates of public material than on genuinely fresh items of the same difficulty, that gap is itself evidence.
Second, the held-out comparison: evaluate on a private, freshly written version of the same task, ideally with new items created after the model's training cutoff, and compare the score with the public benchmark. A large, systematic drop on the fresh set, especially on items that require the same skill but different surface content, is a stronger signal than any overlap statistic. I also look at the pattern of errors: leakage usually produces confident, fluent answers to specific items alongside brittle performance on small perturbations of those same items, such as changing the numbers, the names, or the order of the options.
Third, the training-dynamics check where possible: canary strings inserted into the benchmark, and membership-inference style probes, can reveal whether specific items were seen. Even without that, unusually low loss on benchmark items relative to comparable held-out text is informative.
As for which benchmarks I still trust: I trust them less as absolute scores and more as relative, routinely refreshed instruments. A benchmark I trust has a stated data-collection date, a private held-out split, regular item rotation, and a published replication by an independent group. Static, long-public benchmarks are still useful for tracking a single model family over time, but I would not use them to compare models trained at different times, or to make a capability claim that matters for deployment. For that, the only honest number comes from an evaluation the model has never had a chance to see.
I
Member
Cost-wise, this looks good on paper, but double-check the production numbers before committing.