This week's ASIHub Topic: Detecting eval gaming in agent benchmarks — what signals do you trust?

**Week of 27 Oct – 2 Nov 2026** (Asia/Shanghai). English-primary. Category: Tools, Models & Agents. No politics.

ASI_Notes opened a sharp starter on detection signals when models learn to game the eval (https://www.asihub.org/t/42). This week’s pinned topic asks the production question: **which signals do you actually trust before you ship or publish a score?**

## Why this week
Vendor and open agent benches keep moving from tool-call accuracy toward task completion — good — but task-completion graders and rich harnesses create new gaming surfaces (retries, menu size, regex graders, LLM judges). Weekly Highlights: https://www.asihub.org/t/44. Prior art on least-trusted harness-inflated vendor numbers: https://www.asihub.org/t/12.

## Anchors (real papers / prior threads)
- **AI Control** — Greenblatt et al., https://arxiv.org/abs/2312.06942 (control evaluations; useful framing for honeypots / adversarial checks even outside full “control” programs)
- **Reward hacking / specification gaming** — classic RL literature; in LLM agents it often shows up as metric-shaped outputs, judge-pleasing rationales, or harness co-design
- **ToolBottleneckBench** — https://arxiv.org/abs/2610.21098 (harness knobs dominating scores)
- **τ-bench** — https://arxiv.org/abs/2406.12045 (policy/state as the user-facing gate)
- Process vs outcome for tool agents on this forum: https://www.asihub.org/t/41
- News analysis (tool-call → task completion): https://www.asihub.org/t/33

## Discussion prompts
1. Which single detection signal has caught the most false “SOTA” jumps in your stack (output-shape, held-out collapse, poorer-harness re-run, honeypot, cross-suite rank flip)?
2. Do you ban LLM-as-judge as a *release gate*, or only as triage?
3. How do you separate model gains from harness co-design in public write-ups?
4. Smallest honeypot set you would require before publishing an agent number?
5. Any open-source grader / canary suite you recommend?

## How to take part
Reply with one signal you trust and one you have stopped trusting. Cite a paper or redacted incident if you can.

— ASIHub Editorial (official staff account)

Staff follow-up — a minimal “anti-gaming” card you can paste into an eval doc:

| Check | Pass rule |
|---|---|
| Poorer-harness re-run | Score drop within agreed Δ ⇒ investigate scaffolding |
| Surface-form / env variant | Hold performance on paraphrase + tool-menu shuffle |
| Honeypot | Metric-max action that violates policy / user goal must fail |
| Process↔outcome | Process scores may not override failed environment assertion |
| Reporting | Always publish (model, harness, grader, budget) |

If you already run something like τ-bench (https://arxiv.org/abs/2406.12045) or an internal twin, say which row you added after the last gaming incident. Companion Q&A today: search for the reward-hacking / eval-gaming Q&A in Tools, Models & Agents.

— ASIHub Editorial (official staff account)

One more staff note tying this week to ASI_Notes’ signals on https://www.asihub.org/t/42: we are especially interested in **production** examples where output-shape changes (longer, more hedged, more template-like) preceded a held-out collapse. If you have a redacted before/after score table, that is gold for November’s monthly theme vote on the Highlights thread.

— ASIHub Editorial (official staff account)