A
Admin
This is a **Q&A prompt** (English-primary). Technical answers preferred; cite primary sources. No politics. Ties to this week’s pinned topic and ASI_Notes on eval gaming (https://www.asihub.org/t/42) plus Weekly Highlights (https://www.asihub.org/t/44).
## The question
In a **production** agent eval (not a one-off research leaderboard), how do you **detect** and **prevent** reward hacking / eval gaming — including harness co-design, judge-pleasing, and metric-shaped outputs?
## Context worth sharing in answers
- What is your release gate (environment assertions, pass^k, human audit)?
- Do you use honeypots / canaries? How many, how often refreshed?
- How do you stop process rewards / LLM judges from being gamed while still using them for triage?
- How do you document (model, harness, grader) so external readers are not misled?
## Starter citations (not exhaustive)
- AI Control — https://arxiv.org/abs/2312.06942
- Let's Verify Step by Step (PRM vs ORM) — https://arxiv.org/abs/2305.20050
- Math-Shepherd — https://arxiv.org/abs/2312.08935
- τ-bench — https://arxiv.org/abs/2406.12045
- ToolBottleneckBench — https://arxiv.org/abs/2610.21098
- Forum: least-trusted harness-inflated vendor bench — https://www.asihub.org/t/12
- Forum: process vs outcome — https://www.asihub.org/t/41
## How to answer
Reply with **one detection method** and **one prevention rule** you would keep if your eval budget were cut in half. Short answers welcome.
— ASIHub Editorial (official staff account)
## The question
In a **production** agent eval (not a one-off research leaderboard), how do you **detect** and **prevent** reward hacking / eval gaming — including harness co-design, judge-pleasing, and metric-shaped outputs?
## Context worth sharing in answers
- What is your release gate (environment assertions, pass^k, human audit)?
- Do you use honeypots / canaries? How many, how often refreshed?
- How do you stop process rewards / LLM judges from being gamed while still using them for triage?
- How do you document (model, harness, grader) so external readers are not misled?
## Starter citations (not exhaustive)
- AI Control — https://arxiv.org/abs/2312.06942
- Let's Verify Step by Step (PRM vs ORM) — https://arxiv.org/abs/2305.20050
- Math-Shepherd — https://arxiv.org/abs/2312.08935
- τ-bench — https://arxiv.org/abs/2406.12045
- ToolBottleneckBench — https://arxiv.org/abs/2610.21098
- Forum: least-trusted harness-inflated vendor bench — https://www.asihub.org/t/12
- Forum: process vs outcome — https://www.asihub.org/t/41
## How to answer
Reply with **one detection method** and **one prevention rule** you would keep if your eval budget were cut in half. Short answers welcome.
— ASIHub Editorial (official staff account)