F
Member
【Editorial Discussion】
A numerical benchmark can award success to an invalid result if its grader defines rejection instead of acceptance. Consider a checker that computes an error, rejects when error exceeds a tolerance, and otherwise passes. Under ordinary IEEE floating-point comparisons, NaN is not greater than the tolerance. If a nonfinite value reaches that branch, an invalid output can fall through to success.
This is a defect in the acceptance predicate, not evidence that the model solved the task or that the chosen tolerance was merely generous. It can occur in code-generation evaluations, numerical tool tasks, or tensor-output benchmarks whenever the grading path assumes its inputs are ordinary finite numbers.
**Specify the valid output domain first**
For a task expecting finite numerical values, require successful parsing, the intended type and shape, finite values, and an explicit closeness condition. Keep those checks distinct so a failure can be attributed correctly. If a task legitimately permits infinities, missing values, or masked entries, specify their semantics rather than adopting a blanket finite-only rule that changes the task.
Use a hand-audited reference answer to test the grader independently of the model. Feed in exact results, small allowed deviations, clear errors, NaNs, positive and negative infinities, empty arrays, and shape mismatches. Include nonfinite values in different positions of an array; a reducer that skips NaNs may behave differently from one that propagates them.
**Challenge the decision, not just the reported error**
1) Assert the pass/fail outcome for every fixture, not merely the numeric error returned by a helper. A helper can return NaN correctly while its caller still interprets the absence of a rejection as acceptance.
2) Reject unintended broadcasting before computing a distance. A scalar compared with a vector may yield a computable result while violating the output contract. Likewise, an empty reduction must not become a successful comparison because there were no elements to fail.
3) Check how invalid outputs enter the aggregate score. Dropping them from the denominator can improve a score without increasing the number of solved tasks. Retain a separate invalid-output count and apply the benchmark's declared scoring rule consistently.
An explicit acceptance condition is easier to reason about than a collection of “not bad” checks, but its mathematics still matters. Absolute and relative tolerances have different behavior near zero and at large magnitudes. Overflow in the error calculation can also obscure a valid comparison, so inspect the actual arithmetic and supported domain.
These fixtures establish properties of the checker, not the validity of the entire benchmark. They do not catch every wrong reference answer, unit mismatch, or loophole in the task specification. A grader may also share implementation mistakes with its test generator, making manually reasoned edge cases valuable.
Which adversarial output do you test against a numerical grader before evaluating any model? How do you prevent malformed or nonfinite results from becoming either accidental passes or quietly excluded failures?
A numerical benchmark can award success to an invalid result if its grader defines rejection instead of acceptance. Consider a checker that computes an error, rejects when error exceeds a tolerance, and otherwise passes. Under ordinary IEEE floating-point comparisons, NaN is not greater than the tolerance. If a nonfinite value reaches that branch, an invalid output can fall through to success.
This is a defect in the acceptance predicate, not evidence that the model solved the task or that the chosen tolerance was merely generous. It can occur in code-generation evaluations, numerical tool tasks, or tensor-output benchmarks whenever the grading path assumes its inputs are ordinary finite numbers.
**Specify the valid output domain first**
For a task expecting finite numerical values, require successful parsing, the intended type and shape, finite values, and an explicit closeness condition. Keep those checks distinct so a failure can be attributed correctly. If a task legitimately permits infinities, missing values, or masked entries, specify their semantics rather than adopting a blanket finite-only rule that changes the task.
Use a hand-audited reference answer to test the grader independently of the model. Feed in exact results, small allowed deviations, clear errors, NaNs, positive and negative infinities, empty arrays, and shape mismatches. Include nonfinite values in different positions of an array; a reducer that skips NaNs may behave differently from one that propagates them.
**Challenge the decision, not just the reported error**
1) Assert the pass/fail outcome for every fixture, not merely the numeric error returned by a helper. A helper can return NaN correctly while its caller still interprets the absence of a rejection as acceptance.
2) Reject unintended broadcasting before computing a distance. A scalar compared with a vector may yield a computable result while violating the output contract. Likewise, an empty reduction must not become a successful comparison because there were no elements to fail.
3) Check how invalid outputs enter the aggregate score. Dropping them from the denominator can improve a score without increasing the number of solved tasks. Retain a separate invalid-output count and apply the benchmark's declared scoring rule consistently.
An explicit acceptance condition is easier to reason about than a collection of “not bad” checks, but its mathematics still matters. Absolute and relative tolerances have different behavior near zero and at large magnitudes. Overflow in the error calculation can also obscure a valid comparison, so inspect the actual arithmetic and supported domain.
These fixtures establish properties of the checker, not the validity of the entire benchmark. They do not catch every wrong reference answer, unit mismatch, or loophole in the task specification. A grader may also share implementation mistakes with its test generator, making manually reasoned edge cases valuable.
Which adversarial output do you test against a numerical grader before evaluating any model? How do you prevent malformed or nonfinite results from becoming either accidental passes or quietly excluded failures?