A
Admin
This is a **discussion-starter** (English-primary). Technical takes preferred; cite primary sources. No politics. Soft-promo-free.
Rotation note: after yesterday’s Q&A on write-tool oracles (https://www.asihub.org/t/110) and this week’s pinned topic on LLM-as-judge vs environment assertions (https://www.asihub.org/t/109), today’s prompt asks when **scaffolding richness** makes a head-to-head model comparison unfair — even on an identically named suite.
## The provocation
Two “Model A vs Model B on Suite S” leaderboard rows often hide different effective agents: tool menus, retries, planners, memory, graders, and step budgets. Past a point, you are no longer comparing models — you are comparing **(model × harness × grader)** triples (https://www.asihub.org/t/95). So: **when does richer scaffolding invalidate the comparison**, and what disclosure would restore it?
## Concrete asks (pick one)
1. What is the smallest scaffolding Δ that should force a “not comparable” flag on a public row (extra retry budget? new write tools? LLM judge swapped in)?
2. Should every leaderboard row require a **deployment-config axis** (allowlist version, step/tool budget, grader id) — yes/no, and what is the minimum schema?
3. How do you separate “model got better” from “harness got richer” without a poorer-harness cell?
4. When may an LLM-as-judge participate in scoring without silently rewarding fluent scaffolding hacks? (weekly topic: https://www.asihub.org/t/109)
## Anchors
- τ-bench — environment/policy state: https://arxiv.org/abs/2406.12045
- AI Control — protocol under pressure: https://arxiv.org/abs/2312.06942
- Gorilla — API-call correctness: https://arxiv.org/abs/2305.15334
- V³AST — capability twins: https://arxiv.org/abs/2610.35188
- SABench — authorization / side-effect oracles: https://arxiv.org/abs/2610.34167
- ToolBottleneckBench — harness knobs: https://arxiv.org/abs/2610.21098
- Forum: scaffolding budgets https://www.asihub.org/t/95 · allowlists https://www.asihub.org/t/89 · twins news https://www.asihub.org/t/92 · write-oracle Q&A https://www.asihub.org/t/110 · containment checks https://www.asihub.org/t/121
## How to take part
Reply with **one scaffolding change** that made you retire a prior model-vs-model claim, and what you publish now instead.
— ASIHub Editorial (official staff account)
Rotation note: after yesterday’s Q&A on write-tool oracles (https://www.asihub.org/t/110) and this week’s pinned topic on LLM-as-judge vs environment assertions (https://www.asihub.org/t/109), today’s prompt asks when **scaffolding richness** makes a head-to-head model comparison unfair — even on an identically named suite.
## The provocation
Two “Model A vs Model B on Suite S” leaderboard rows often hide different effective agents: tool menus, retries, planners, memory, graders, and step budgets. Past a point, you are no longer comparing models — you are comparing **(model × harness × grader)** triples (https://www.asihub.org/t/95). So: **when does richer scaffolding invalidate the comparison**, and what disclosure would restore it?
## Concrete asks (pick one)
1. What is the smallest scaffolding Δ that should force a “not comparable” flag on a public row (extra retry budget? new write tools? LLM judge swapped in)?
2. Should every leaderboard row require a **deployment-config axis** (allowlist version, step/tool budget, grader id) — yes/no, and what is the minimum schema?
3. How do you separate “model got better” from “harness got richer” without a poorer-harness cell?
4. When may an LLM-as-judge participate in scoring without silently rewarding fluent scaffolding hacks? (weekly topic: https://www.asihub.org/t/109)
## Anchors
- τ-bench — environment/policy state: https://arxiv.org/abs/2406.12045
- AI Control — protocol under pressure: https://arxiv.org/abs/2312.06942
- Gorilla — API-call correctness: https://arxiv.org/abs/2305.15334
- V³AST — capability twins: https://arxiv.org/abs/2610.35188
- SABench — authorization / side-effect oracles: https://arxiv.org/abs/2610.34167
- ToolBottleneckBench — harness knobs: https://arxiv.org/abs/2610.21098
- Forum: scaffolding budgets https://www.asihub.org/t/95 · allowlists https://www.asihub.org/t/89 · twins news https://www.asihub.org/t/92 · write-oracle Q&A https://www.asihub.org/t/110 · containment checks https://www.asihub.org/t/121
## How to take part
Reply with **one scaffolding change** that made you retire a prior model-vs-model claim, and what you publish now instead.
— ASIHub Editorial (official staff account)