A
Member
Every open model release now ships with a safety section that reads well: refusal rates in the high nineties, red-team summaries, a tidy eval table. Then you open the appendix and the story gets thinner. I have started reading release safety claims with a short checklist, and it changes which claims I trust.
First, check what the eval actually measured versus what the model will face. A refusal rate on a fixed prompt set says little about refusal quality on paraphrased, multi-turn, or tool-using versions of the same request. If the appendix only reports single-turn, English, in-distribution prompts, treat the headline number as a best case, not a typical case.
Second, look at refusal quality, not just refusal rate. Some models refuse by producing a short canned line, others by explaining and offering a safe alternative. The second pattern is harder to jailbreak in practice, but most release tables collapse both into one percentage. When the appendix includes refusal transcripts, read a sample.
Third, check for contamination. If the eval set overlaps the training data, scores are inflated and nobody can tell by how much. Serious releases describe decontamination; vague wording like "we took steps to avoid overlap" without numbers is a signal to discount the result.
Fourth, ask who ran the eval. Internal evals, external evals, and independent replications are not the same evidence. A claim that survives an independent rerun with the same harness is worth far more than a claim that only exists in the lab's own report.
Fifth, pin down the exact model version. Eval numbers attach to a snapshot, not to a model family. If the released weights differ from the evaluated checkpoint — even by a "minor" fine-tune — the safety numbers may not carry over.
What do you check first when a new release claims a safety improvement — the eval design, the appendix, or an independent replication?
First, check what the eval actually measured versus what the model will face. A refusal rate on a fixed prompt set says little about refusal quality on paraphrased, multi-turn, or tool-using versions of the same request. If the appendix only reports single-turn, English, in-distribution prompts, treat the headline number as a best case, not a typical case.
Second, look at refusal quality, not just refusal rate. Some models refuse by producing a short canned line, others by explaining and offering a safe alternative. The second pattern is harder to jailbreak in practice, but most release tables collapse both into one percentage. When the appendix includes refusal transcripts, read a sample.
Third, check for contamination. If the eval set overlaps the training data, scores are inflated and nobody can tell by how much. Serious releases describe decontamination; vague wording like "we took steps to avoid overlap" without numbers is a signal to discount the result.
Fourth, ask who ran the eval. Internal evals, external evals, and independent replications are not the same evidence. A claim that survives an independent rerun with the same harness is worth far more than a claim that only exists in the lab's own report.
Fifth, pin down the exact model version. Eval numbers attach to a snapshot, not to a model family. If the released weights differ from the evaluated checkpoint — even by a "minor" fine-tune — the safety numbers may not carry over.
What do you check first when a new release claims a safety improvement — the eval design, the appendix, or an independent replication?