F
Member
【Editorial Discussion】
A safety router can appear well calibrated when its evidence comes almost entirely from cases it already escalates. If ordinary approvals are rarely reviewed, unsafe answers below the escalation threshold remain unlabelled. If blocked answers are never examined, unnecessary blocks may also disappear from the evidence. Better performance on the reviewed queue does not necessarily mean better decisions over all incoming cases.
This is selective observation, not simply a weak classifier. The routing decision influences which labels become available. Retraining or choosing thresholds on that same stream can reinforce a blind spot even when every existing review label is correct.
**State the denominator before changing the threshold**
Separate the questions: how accurate are reviewers on escalations, how often does the router miss unsafe candidate answers, and how frequently does it block acceptable ones? The first can be studied inside the queue. The others require evidence from outside it. Do not treat an unreviewed approval as a correct approval or an unreviewed block as a confirmed safety success.
A practical diagnostic is an offline audit of stored candidate answers, subject to the applicable retention and access rules. Sample across routing outcomes and score ranges, including cases that would normally receive no review. Reviewers should see enough task context to judge the candidate, but not the router score or disposition where blinding is feasible. This need not expose blocked output to an end user.
**Compare two evaluation views**
Estimate errors using only the normal review queue, then using the audit sample representing the full candidate population. If sampling rates differ across strata, retain those rates and use an appropriate weighted estimate. A raw average from an audit that deliberately oversamples suspicious cases does not estimate the production-wide error rate.
Inspect disagreements by routing outcome and task type before adjusting thresholds. If the audit finds errors in a region with little routine review, expanding review coverage may matter more than tuning a threshold on already-visible cases. Keep the audit separate from threshold selection or collect fresh evaluation labels after selection; otherwise its independence erodes.
**What this cannot establish**
A small audit may contain too few rare severe failures to estimate their rate tightly. Reviewers can share blind spots, and some harms depend on downstream consequences unavailable from a stored candidate alone. If rejected requests have no candidate answer, an answer-level audit cannot recover one without a separately designed offline experiment. Inverse sampling weights address known sampling imbalance, not missing context or unreliable judgments.
How do you obtain ground truth for the decisions your safety router does not normally escalate? What audit finding would make you change the label-collection policy rather than merely lower a threshold and enlarge the same review queue?
A safety router can appear well calibrated when its evidence comes almost entirely from cases it already escalates. If ordinary approvals are rarely reviewed, unsafe answers below the escalation threshold remain unlabelled. If blocked answers are never examined, unnecessary blocks may also disappear from the evidence. Better performance on the reviewed queue does not necessarily mean better decisions over all incoming cases.
This is selective observation, not simply a weak classifier. The routing decision influences which labels become available. Retraining or choosing thresholds on that same stream can reinforce a blind spot even when every existing review label is correct.
**State the denominator before changing the threshold**
Separate the questions: how accurate are reviewers on escalations, how often does the router miss unsafe candidate answers, and how frequently does it block acceptable ones? The first can be studied inside the queue. The others require evidence from outside it. Do not treat an unreviewed approval as a correct approval or an unreviewed block as a confirmed safety success.
A practical diagnostic is an offline audit of stored candidate answers, subject to the applicable retention and access rules. Sample across routing outcomes and score ranges, including cases that would normally receive no review. Reviewers should see enough task context to judge the candidate, but not the router score or disposition where blinding is feasible. This need not expose blocked output to an end user.
**Compare two evaluation views**
Estimate errors using only the normal review queue, then using the audit sample representing the full candidate population. If sampling rates differ across strata, retain those rates and use an appropriate weighted estimate. A raw average from an audit that deliberately oversamples suspicious cases does not estimate the production-wide error rate.
Inspect disagreements by routing outcome and task type before adjusting thresholds. If the audit finds errors in a region with little routine review, expanding review coverage may matter more than tuning a threshold on already-visible cases. Keep the audit separate from threshold selection or collect fresh evaluation labels after selection; otherwise its independence erodes.
**What this cannot establish**
A small audit may contain too few rare severe failures to estimate their rate tightly. Reviewers can share blind spots, and some harms depend on downstream consequences unavailable from a stored candidate alone. If rejected requests have no candidate answer, an answer-level audit cannot recover one without a separately designed offline experiment. Inverse sampling weights address known sampling imbalance, not missing context or unreliable judgments.
How do you obtain ground truth for the decisions your safety router does not normally escalate? What audit finding would make you change the label-collection policy rather than merely lower a threshold and enlarge the same review queue?