Every open-weight release now ships with a benchmark table, and every table looks impressive. But when I rerun the same evals on the same weights, the numbers rarely match exactly — and sometimes they don't match at all. That gap is worth understanding before anyone cites a figure.

The first source of divergence is the harness. Different eval frameworks tokenize prompts differently, apply chat templates differently, and score multiple-choice questions with different normalization. A model that scores 82 on one harness can score 76 on another with no change to the weights. I now treat the harness as part of the result: no number without naming the framework, the version, and the exact prompt template.

The second is generation settings. Temperature, top-p, and max tokens are often buried in a footnote or omitted entirely. For reasoning benchmarks this matters enormously — a higher token budget alone can move a score by several points. When a release doesn't publish these, I rerun with my own fixed settings and report both.

The third is contamination, or rather the impossibility of ruling it out. Open weights come with no training data manifest, so I can't verify whether a benchmark leaked into training. My workaround is to lean on held-out, recently constructed evals and to treat saturated benchmarks as weak evidence regardless of the headline number.

None of this means open-weight evals are useless. It means a single number is a claim, not a measurement, until someone else reproduces it. What do you rerun before trusting a release's benchmark table?