A
Member
Every open-weight release now ships with a benchmark table, and every table claims a new state of the art. After watching a few of these cycles, I have stopped updating my eval suite on release day. Here is what I actually take from a new open-weight release, and what I refuse to change.
1. A new release is a new test subject, not a new test. The release tells me what to evaluate, not how. My eval set — the tasks, the rubrics, the failure taxonomy — stays fixed across releases. If I change the test every time a model changes, I lose the only thing that makes comparisons meaningful: a stable baseline.
2. Weights let me test what APIs hide. With weights in hand I can run ablations an API never allows: swap the system prompt, strip the reasoning trace, quantize to 4-bit and measure the accuracy drop, or probe the model on inputs the API filters out. That is where open weights actually earn their keep — not in the headline benchmark number, but in the failure analysis the weights make possible.
3. Benchmark scores from the release post are a starting point, not a result. I re-run the headline benchmarks myself, on my own harness, with the same decoding settings I use for everything else. The gap between the vendor's number and mine is usually 2–5 points, and the direction of the gap tells me something about the vendor's evaluation choices.
4. What doesn't change: the deployment questions. Can it run on my hardware at my latency budget? Does it degrade gracefully on out-of-distribution inputs? Does it refuse cleanly instead of hallucinating a confident wrong answer? No release note answers those questions. Only my own evals do.
The practical upshot: I now spend release day running my fixed suite against the new weights, and I write down the failure modes, not the scores.
How do you handle a new open-weight release — do you re-run your own evals, trust the vendor numbers, or wait for the community to do the benchmarking first?
1. A new release is a new test subject, not a new test. The release tells me what to evaluate, not how. My eval set — the tasks, the rubrics, the failure taxonomy — stays fixed across releases. If I change the test every time a model changes, I lose the only thing that makes comparisons meaningful: a stable baseline.
2. Weights let me test what APIs hide. With weights in hand I can run ablations an API never allows: swap the system prompt, strip the reasoning trace, quantize to 4-bit and measure the accuracy drop, or probe the model on inputs the API filters out. That is where open weights actually earn their keep — not in the headline benchmark number, but in the failure analysis the weights make possible.
3. Benchmark scores from the release post are a starting point, not a result. I re-run the headline benchmarks myself, on my own harness, with the same decoding settings I use for everything else. The gap between the vendor's number and mine is usually 2–5 points, and the direction of the gap tells me something about the vendor's evaluation choices.
4. What doesn't change: the deployment questions. Can it run on my hardware at my latency budget? Does it degrade gracefully on out-of-distribution inputs? Does it refuse cleanly instead of hallucinating a confident wrong answer? No release note answers those questions. Only my own evals do.
The practical upshot: I now spend release day running my fixed suite against the new weights, and I write down the failure modes, not the scores.
How do you handle a new open-weight release — do you re-run your own evals, trust the vendor numbers, or wait for the community to do the benchmarking first?