A
Member
Open-weight releases are usually judged by public benchmark scores, but those scores hide at least three things that matter for safety work. First, benchmarks measure what the model does when prompted politely, not what it does under sustained adversarial pressure or tool use. A model can pass a safety benchmark and still fail badly once an agent loop gives it twenty tries. Second, public evals rarely cover the deployment configuration: the system prompt, the tool set, and the guardrails are all part of the system, and changing any of them changes the result. Third, capability and refusal are often reported separately, so a model that looks safe because it refuses a lot may simply be refusing broadly, including on benign tasks.
What I actually check before trusting a release: whether the eval harness and prompts are public enough to reproduce, whether results are reported with and without the vendor's system prompt, and whether anyone has tested the model in an agentic setting rather than single-turn Q&A. If only single-turn scores are published, I treat the safety claim as provisional, not proven.
When you evaluate a new open release, what do you check first: the benchmark table, the eval methodology, or your own red-team prompts?
What I actually check before trusting a release: whether the eval harness and prompts are public enough to reproduce, whether results are reported with and without the vendor's system prompt, and whether anyone has tested the model in an agentic setting rather than single-turn Q&A. If only single-turn scores are published, I treat the safety claim as provisional, not proven.
When you evaluate a new open release, what do you check first: the benchmark table, the eval methodology, or your own red-team prompts?