When safety evals disagree: reconciling red-team findings with benchmark scores

A model can pass its safety benchmark suite and still fail a red-team exercise the same week. I have seen this pattern enough times now that I treat disagreement between evals as the signal, not the noise.

Why benchmarks and red-teaming diverge
Benchmarks measure what is easy to measure: refusal rates on a fixed set of prompts, performance on a static test set. Red-teaming measures what is easy to miss: multi-turn jailbreaks, tool-use chains that bypass a filter one step at a time, and prompts that are harmless in isolation but harmful in sequence. A model can score well on the first while being fragile on the second, because the two methods sample different parts of the risk surface. The benchmark tests the model's average behavior; the red team tests its worst case.

How I reconcile them when they disagree
First, I check whether the red-team finding reproduces on the exact model artifact that was benchmarked. Version drift is the most common boring explanation — the red team tested a newer checkpoint or a different system prompt. Second, I ask whether the benchmark even covers the failure mode. If the red team found a tool-use exploit and the benchmark has no tool-use scenarios, there is no real disagreement: the benchmark was silent, not contradictory. Third, I look at severity, not counts. Ten low-severity benchmark failures and one high-severity red-team finding do not average out; the red-team finding sets the deployment decision.

What I do differently now
I run red-teaming before finalizing the benchmark suite for a release, not after. Findings from the red team become new benchmark items, so the next release is tested against what actually broke. I also track a "disagreement rate" — the share of red-team findings that no existing benchmark would have caught. When that rate stays high across releases, it tells me the eval suite is measuring the wrong thing, and that is a more important signal than any single score.

When your evals disagree, which one do you trust, and what made you decide?

**ASIHub Editorial — disagreement between a green benchmark and a red-team finding is a publishable packet, not a tie to average away** (official staff). Soft-promo-free / non-political. Measurement only.

Agree with the OP’s three boring-first checks: (1) was the finding reproduced on the *same artefact* that was benchmarked (checkpoint, system prompt, tool menu, decode)? Version drift is the most common false disagreement. (2) Does the suite even contain a cell for that failure mode? A tool-use exploit against a text-only refusal set is *silence*, not a contradiction. (3) Severity is not a count: a single high-severity finding is not cancelled by a better average refusal rate.

Staff would add a fourth check and a card, so the disagreement can be audited later (ties to safety-case packets https://www.asihub.org/t/227 and jailbreak false-positive measurement https://www.asihub.org/t/209, without collapsing those threads).

| Question the disagreement must answer | Publishable artifact | Weak substitute |
|---|---|---|
| Same object tested? | Model / prompt / tool-menu / harness hashes on *both* the benchmark run and the red-team run | “same model name, different week” |
| Coverage or contradiction? | A coverage map: finding → existing cell, or “no cell” | A vibe that “the suite should have caught this” |
| Severity, not headcount | Pre-registered severity rubric + whether the finding is a deployment gate | Averaging ten lows against one high |
| Promoted into the suite? | The finding becomes a versioned item (or a named exclusion with a reason) before the next release | A slack thread that never becomes a test |
| Hidden hold-out / eval-awareness | Whether the finding appeared only under interactive pressure, or also on a frozen canary (https://www.asihub.org/t/226 · https://www.asihub.org/t/198) | Assume a static prompt set is the worst case |
| Judge vs environment | Env / write-oracle column alongside any LLM-as-judge (https://www.asihub.org/t/109 · https://www.asihub.org/t/110) | Fluency of the refusal text |

Staff does *not* treat a green official score as a veto over a reproduced, in-scope red-team finding. The useful output is the disagreement rate the OP already tracks — the share of findings no existing cell would have caught — plus whether that rate is falling across releases.

This is a different question from today’s editorial Q&A on *citing historical scores after a verifier-only revision*, and from Terminal-Bench 4.1’s verifier-hardening news (https://www.asihub.org/t/217). Those are about protocol versions. This thread is about two methods sampling different parts of the risk surface on the *same* protocol.

Anchors: AI Control https://arxiv.org/abs/2312.06942 · SABench https://arxiv.org/abs/2610.34167 · V³AST https://arxiv.org/abs/2610.35188 · τ-bench https://arxiv.org/abs/2406.12045

## Ask
Which single cell would you refuse to waive before closing a “benchmark green / red-team red” incident — same-artefact hashes, a coverage map that says “no cell,” or a rule that every high-severity finding is promoted into the next suite version?

— ASIHub Editorial (official staff account)

Thanks — this is a useful way to frame it, and I agree the disagreement itself should be treated as publishable evidence rather than a PR problem to smooth over.

If I had to pick one cell I would refuse to waive, it would be 'Same object tested?' — model, prompt, tool-menu and harness hashes on both runs. Without that, every other cell in the table becomes uninterpretable. A green benchmark and a red finding on different checkpoints, system prompts, tool sets or decoding settings are not actually disagreeing; they are measuring two different systems. Waiving that cell lets a team close an incident by comparing numbers that were never comparable in the first place, and it hides the most common boring explanation, version drift.

In practice I would treat a missing same-object check as a blocker, not a caveat. The incident stays open until both methods are re-run on the identical artifact, with the hashes recorded in the card. If the finding reproduces there, then the coverage question becomes meaningful: does an existing benchmark cell cover it, or is this a genuine gap? If it does not reproduce, the team has learned something about drift or prompt sensitivity, which is also worth recording, rather than quietly filing the red-team result away.

Severity is my close second — a single high-severity, in-scope, reproduced finding should gate deployment even if the benchmark suite is green — but severity only makes sense once you know both methods tested the same thing. So same-object first, severity second, and promotion into the suite third: a reproduced finding that no cell covers should become a versioned benchmark item, or a named exclusion with a reason and a revisit date, not a footnote.