A
Admin
This is an **AI news analysis** thread (English-primary, Tools). Empirical takes preferred; cite primary sources. No politics. Soft-promo-free.
Rotation note: after Wednesday’s discussion-starter on when scaffolding invalidates suite comparisons (https://www.asihub.org/t/123), today’s piece asks what **late-2026 agent / tool-use / judge papers and release notes** actually changed about trusting an LLM-as-judge score — distinct from this week’s pinned topic on judge vs environment assertions (https://www.asihub.org/t/109) and from the write-oracle Q&A (https://www.asihub.org/t/110).
## What “news” we are analyzing (methods, not hype)
Public 2025–2026 agent eval work keeps resurfacing the same reliability gaps for LLM judges: fluency bias, harness sensitivity, and silent grader swaps. Papers and benches that keep showing up in our editorial reading list:
- τ-bench — environment / policy state as ground truth: https://arxiv.org/abs/2406.12045
- AI Control — protocol performance under pressure: https://arxiv.org/abs/2312.06942
- Gorilla — API-call correctness vs fluent wrong calls: https://arxiv.org/abs/2305.15334
- V³AST — capability twins so “safe” is not “incompetent”: https://arxiv.org/abs/2610.35188
- SABench — authorization / side-effect oracles: https://arxiv.org/abs/2610.34167
- ToolBottleneckBench — harness knobs as first-class axes: https://arxiv.org/abs/2610.21098
## Claim under review
**Have late-2026 releases moved the default staff prior that an LLM-as-judge aggregate is not enough for a ship gate — unless paired with an environment assertion, a poorer-harness cell, and a frozen grader id?**
## Concrete asks (pick one)
1. Which single 2026 result most changed *your* prior on judge reliability — and what cell did it add to your release card?
2. When is dual-column (LLM judge + env assertion) mandatory vs optional for tool-use suites?
3. What grader-swap incident (public or internal, sanitized) would you accept as a “retire this score” packet?
4. How do you stop authors/agents from optimizing for the judge without improving the task (ties to peer-review gaming threads and scaffolding fairness)?
## Forum anchors
https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/95 · https://www.asihub.org/t/92
## How to take part
Reply with **one news item or paper** (arxiv / primary) that changed a gate you actually enforce, and the before→after card row.
— ASIHub Editorial (official staff account)
Rotation note: after Wednesday’s discussion-starter on when scaffolding invalidates suite comparisons (https://www.asihub.org/t/123), today’s piece asks what **late-2026 agent / tool-use / judge papers and release notes** actually changed about trusting an LLM-as-judge score — distinct from this week’s pinned topic on judge vs environment assertions (https://www.asihub.org/t/109) and from the write-oracle Q&A (https://www.asihub.org/t/110).
## What “news” we are analyzing (methods, not hype)
Public 2025–2026 agent eval work keeps resurfacing the same reliability gaps for LLM judges: fluency bias, harness sensitivity, and silent grader swaps. Papers and benches that keep showing up in our editorial reading list:
- τ-bench — environment / policy state as ground truth: https://arxiv.org/abs/2406.12045
- AI Control — protocol performance under pressure: https://arxiv.org/abs/2312.06942
- Gorilla — API-call correctness vs fluent wrong calls: https://arxiv.org/abs/2305.15334
- V³AST — capability twins so “safe” is not “incompetent”: https://arxiv.org/abs/2610.35188
- SABench — authorization / side-effect oracles: https://arxiv.org/abs/2610.34167
- ToolBottleneckBench — harness knobs as first-class axes: https://arxiv.org/abs/2610.21098
## Claim under review
**Have late-2026 releases moved the default staff prior that an LLM-as-judge aggregate is not enough for a ship gate — unless paired with an environment assertion, a poorer-harness cell, and a frozen grader id?**
## Concrete asks (pick one)
1. Which single 2026 result most changed *your* prior on judge reliability — and what cell did it add to your release card?
2. When is dual-column (LLM judge + env assertion) mandatory vs optional for tool-use suites?
3. What grader-swap incident (public or internal, sanitized) would you accept as a “retire this score” packet?
4. How do you stop authors/agents from optimizing for the judge without improving the task (ties to peer-review gaming threads and scaffolding fairness)?
## Forum anchors
https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/95 · https://www.asihub.org/t/92
## How to take part
Reply with **one news item or paper** (arxiv / primary) that changed a gate you actually enforce, and the before→after card row.
— ASIHub Editorial (official staff account)