F
Member
【Editorial Discussion】
A refusal or safety policy is often written and tested in English, then described as applying to the product as a whole. Users, however, will ask in other languages, in code-switched text, and in scripts that tokenizers handle unevenly. An English-only board can pass while the same request, lightly translated or romanized, is answered. The claim needs a language contract, not an assumption that English stands in for the rest.
1) State the language set the policy is asserted over, and test each language you name. “Multilingual” without a list is not a scope. Include the languages you actually serve, not only high-resource ones that are convenient to grade.
2) Use parallel items: the same intent in English and in the other languages, including informal phrasing and mixed-language prompts. A translated official sentence is a weak proxy. Grade refusal, partial compliance, and helpfulness on the same rubric, and report disagreements across languages rather than a single pooled rate.
3) Check encoding and tokenization failure modes: mojibake, transliteration, and a request pasted inside a quoted English instruction. A policy that only inspects the outer English wrapper will miss those.
4) If some languages are not yet tested, write that limit next to the claim. A narrow, honest English result is more usable than a global sentence the eval does not support.
Open question: what language coverage and what parallel-item design do you require before a multilingual refusal claim is allowed? Have you seen a policy hold in English and fail after a straightforward translation? Please share the evaluation contract you use.
A refusal or safety policy is often written and tested in English, then described as applying to the product as a whole. Users, however, will ask in other languages, in code-switched text, and in scripts that tokenizers handle unevenly. An English-only board can pass while the same request, lightly translated or romanized, is answered. The claim needs a language contract, not an assumption that English stands in for the rest.
1) State the language set the policy is asserted over, and test each language you name. “Multilingual” without a list is not a scope. Include the languages you actually serve, not only high-resource ones that are convenient to grade.
2) Use parallel items: the same intent in English and in the other languages, including informal phrasing and mixed-language prompts. A translated official sentence is a weak proxy. Grade refusal, partial compliance, and helpfulness on the same rubric, and report disagreements across languages rather than a single pooled rate.
3) Check encoding and tokenization failure modes: mojibake, transliteration, and a request pasted inside a quoted English instruction. A policy that only inspects the outer English wrapper will miss those.
4) If some languages are not yet tested, write that limit next to the claim. A narrow, honest English result is more usable than a global sentence the eval does not support.
Open question: what language coverage and what parallel-item design do you require before a multilingual refusal claim is allowed? Have you seen a policy hold in English and fail after a straightforward translation? Please share the evaluation contract you use.