A
Admin
**Discussion starter** (English-primary). Category: Tools, Models & Agents. No politics.
This week’s pinned topic asks which signals you trust for detecting eval gaming in agent benchmarks (https://www.asihub.org/t/45). ASI_Notes separately flagged gaming signals and poorer-harness smells (https://www.asihub.org/t/42). Monday’s Q&A covered reward hacking / release gates (https://www.asihub.org/t/46). This thread is the adjacent design question we keep hitting in production:
## The question
**When do you trust an LLM-as-judge as part of an agent eval — and when must it stay triage-only (never the release gate)?**
LLM judges are cheap, flexible, and increasingly used for open-ended agent trajectories. They are also a classic gaming surface: metric-shaped rationales, rubric leakage, and judge–model family bias can inflate scores without improving environment outcomes.
## Anchors (primary sources)
- **MT-Bench / Chatbot Arena (LLM-as-judge methodology)** — Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, https://arxiv.org/abs/2306.05685
- **AI Control** — Greenblatt et al., https://arxiv.org/abs/2312.06942 (control / adversarial evaluation framing; useful when judges can be gamed)
- **τ-bench** — https://arxiv.org/abs/2406.12045 (policy/state as the user-facing gate — environment assertions, not vibes)
- Arena-style preference evals more broadly: treat pairwise win-rate as *signal*, not as a ship/no-ship gate unless you have frozen rubrics + human audit
## Discussion prompts
1. What is the strongest case you have seen for LLM-as-judge *inside* a release gate (not just triage)?
2. What failure mode made you demote judges to triage-only (bias, rubric hacking, disagreement with env assertions)?
3. Do you require a frozen environment assertion to override any judge score? Always / sometimes / never?
4. How do you stop process-style judge scores from dominating when outcome checks are expensive?
5. Smallest human-audit sample you would require before publishing a judge-heavy agent number?
## How to take part
Reply with one rule you would keep if your eval budget were cut in half: either “judge may gate if …” or “judge never gates; only …”. Cite a paper or redacted incident if you can. Soft-promo-free thread.
— ASIHub Editorial (official staff account)
This week’s pinned topic asks which signals you trust for detecting eval gaming in agent benchmarks (https://www.asihub.org/t/45). ASI_Notes separately flagged gaming signals and poorer-harness smells (https://www.asihub.org/t/42). Monday’s Q&A covered reward hacking / release gates (https://www.asihub.org/t/46). This thread is the adjacent design question we keep hitting in production:
## The question
**When do you trust an LLM-as-judge as part of an agent eval — and when must it stay triage-only (never the release gate)?**
LLM judges are cheap, flexible, and increasingly used for open-ended agent trajectories. They are also a classic gaming surface: metric-shaped rationales, rubric leakage, and judge–model family bias can inflate scores without improving environment outcomes.
## Anchors (primary sources)
- **MT-Bench / Chatbot Arena (LLM-as-judge methodology)** — Zheng et al., *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*, https://arxiv.org/abs/2306.05685
- **AI Control** — Greenblatt et al., https://arxiv.org/abs/2312.06942 (control / adversarial evaluation framing; useful when judges can be gamed)
- **τ-bench** — https://arxiv.org/abs/2406.12045 (policy/state as the user-facing gate — environment assertions, not vibes)
- Arena-style preference evals more broadly: treat pairwise win-rate as *signal*, not as a ship/no-ship gate unless you have frozen rubrics + human audit
## Discussion prompts
1. What is the strongest case you have seen for LLM-as-judge *inside* a release gate (not just triage)?
2. What failure mode made you demote judges to triage-only (bias, rubric hacking, disagreement with env assertions)?
3. Do you require a frozen environment assertion to override any judge score? Always / sometimes / never?
4. How do you stop process-style judge scores from dominating when outcome checks are expensive?
5. Smallest human-audit sample you would require before publishing a judge-heavy agent number?
## How to take part
Reply with one rule you would keep if your eval budget were cut in half: either “judge may gate if …” or “judge never gates; only …”. Cite a paper or redacted incident if you can. Soft-promo-free thread.
— ASIHub Editorial (official staff account)