A
Member
Automated safety evaluations commonly use a judge model to decide whether a response is harmful. Errors in that judge can dominate the reported attack success rate.
Two error types to measure explicitly:
False positive: a benign or a refusal response is scored as a successful attack.
False negative: a genuinely harmful response is scored as safe.
A small false-positive rate on a large benign control set can produce a substantial apparent attack success rate.
A defensible evaluation design:
1) Include benign controls that share surface features with attacks (similar phrasing, role-play framing, encoded text) but request harmless content. Score them with the same judge.
2) Build a human-labelled calibration set, and report judge agreement, precision and recall, not just overall accuracy.
3) Separate "model complied" from "output was actually harmful". A long refusal with a quoted example can be misclassified either way.
4) Fix decoding parameters and the number of attempts; reporting the best-of-N result without N is not comparable.
5) Keep the judge model, prompt and version fixed across compared systems, and re-run calibration when any of them change.
6) Report confidence intervals over prompts and seeds; attack success is usually highly skewed.
Also note that a defence which increases refusals on benign requests can look better on safety metrics while being worse in practice, so pair safety results with an over-refusal measurement on a benign set.
Takeaway: publish the judge calibration and the benign control results alongside the attack success rate. Otherwise the metric mostly measures the judge.
Two error types to measure explicitly:
False positive: a benign or a refusal response is scored as a successful attack.
False negative: a genuinely harmful response is scored as safe.
A small false-positive rate on a large benign control set can produce a substantial apparent attack success rate.
A defensible evaluation design:
1) Include benign controls that share surface features with attacks (similar phrasing, role-play framing, encoded text) but request harmless content. Score them with the same judge.
2) Build a human-labelled calibration set, and report judge agreement, precision and recall, not just overall accuracy.
3) Separate "model complied" from "output was actually harmful". A long refusal with a quoted example can be misclassified either way.
4) Fix decoding parameters and the number of attempts; reporting the best-of-N result without N is not comparable.
5) Keep the judge model, prompt and version fixed across compared systems, and re-run calibration when any of them change.
6) Report confidence intervals over prompts and seeds; attack success is usually highly skewed.
Also note that a defence which increases refusals on benign requests can look better on safety metrics while being worse in practice, so pair safety results with an over-refusal measurement on a benign set.
Takeaway: publish the judge calibration and the benign control results alongside the attack success rate. Otherwise the metric mostly measures the judge.