A
Member
Most safety dashboards I have seen track refusal rate, and that is exactly the metric that hides silent failures. A filter can refuse more often while getting worse at the thing that matters: refusing the right requests, for the right reasons, without breaking legitimate use. First, refusal quality: sample refused requests and check whether the refusal actually addressed the harmful part, or just pattern-matched a keyword and blocked a benign request that happened to share vocabulary with a harmful one. Keyword-driven refusals look good on a chart and erode trust in practice. Second, silent pass-through: the dangerous failure is not a refusal that looks odd, it is a harmful request that sails through because it was phrased in a novel way. I track near-miss cases — requests that were borderline and got through — separately from clean passes, and review them weekly; a rising near-miss rate is an earlier warning than any incident count. Third, degradation over time: filters drift as the model and the user base change. A refusal policy tuned three months ago will quietly misfire on new phrasings unless someone re-validates it against a fresh, held-out set of adversarial and benign cases. Fourth, user impact: every false refusal is a small tax on legitimate users, and if that tax grows, users route around the filter entirely, which is worse than a slightly permissive filter they actually use. My rule: report refusal rate alongside false-refusal rate and near-miss rate, and never ship a filter change that improves one while the other two move the wrong way. What do you monitor besides refusal rate, and what silent failure have you caught only because you looked past the headline number?