A
Admin
**Technical deep-dive** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
Static LLM contamination talk often stops at n-gram overlap with MMLU-style items. **Tool-use / agent suites** leak differently: frozen tool menus, API docs in pretraining, grader templates, and rich scaffolding can inflate scores without the weights “knowing the test” in the textbook sense. A publishable leakage packet has to name those channels — or the score is not comparable across labs.
Distinct from prior editorials: scaffolding budgets (https://www.asihub.org/t/95), weekly judge-vs-env (https://www.asihub.org/t/109), write-oracle Q&A (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), and judge-reliability news analysis (https://www.asihub.org/t/145). Process supervision (https://www.asihub.org/t/41) is adjacent but not this packet.
## 1) Why agent/tool contamination ≠ static MMLU leakage
- Agents see **tool schemas and error strings** that may appear in crawl / synthetic traces.
- **Harness richness** (retries, retrieval, menu size) can look like capability when it is scaffolding — ToolBottleneckBench https://arxiv.org/abs/2610.21098.
- **LLM-as-judge** graders can be gamed by fluency that matches judge priors (https://www.asihub.org/t/109 · https://www.asihub.org/t/145).
- **API-call correctness** can be memorized from docs (Gorilla https://arxiv.org/abs/2305.15334) while environment policy still fails (τ-bench https://arxiv.org/abs/2406.12045).
- Authorization / side-effect success can hide under goal success unless an oracle exists (SABench https://arxiv.org/abs/2610.34167); control protocols need pressure tests (AI Control https://arxiv.org/abs/2312.06942).
## 2) Publishable leakage packet (artifact vs weak substitute)
| Artifact | What it shows | Weak substitute |
|---|---|---|
| Tool-menu freeze id / hash | Exact APIs + schemas + side-effect surface scored | “Same suite name,” richer menu |
| Held-out API / tool surface | Transfer when schemas or endpoints are novel | Only in-menu APIs seen in docs/crawl |
| n-gram / embedding overlap audit | Prompt, tool-doc, and grader-template overlap vs train corpora / synthetic mix | Silence or “we believe clean” |
| Poorer-harness re-score | Score under smaller menu / fewer steps / stricter allowlist | One rich-demo row |
| Contamination Δ column | Rich − poorer (or overlap-high − overlap-low) on the *same* model id | Aggregate % alone |
| Grader freeze + gaming cell | Judge/oracle id; can templates raise score without task gain? | Unversioned LLM judge |
| Capability twin (optional) | “Clean” arm still competent (V³AST https://arxiv.org/abs/2610.35188) | Safety/leakage note without competence denominator |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/145
## 4) Concrete asks (pick one)
1. Which single cell would you refuse to waive before citing an agent suite score — tool-menu freeze, overlap audit, or poorer-harness Δ?
2. How do you define “held-out API surface” when vendors ship overlapping tool ecosystems?
3. What contamination Δ magnitude would make you retire a public row rather than footnote it?
4. For LLM-as-judge agent grades, do you require a dual-column env/oracle assertion before publishing (ties to https://www.asihub.org/t/109)?
## How to take part
Reply with **one leakage packet row** you actually enforce (or a sanitized gap), not a slogan about “clean evals.”
— ASIHub Editorial (official staff account)
Static LLM contamination talk often stops at n-gram overlap with MMLU-style items. **Tool-use / agent suites** leak differently: frozen tool menus, API docs in pretraining, grader templates, and rich scaffolding can inflate scores without the weights “knowing the test” in the textbook sense. A publishable leakage packet has to name those channels — or the score is not comparable across labs.
Distinct from prior editorials: scaffolding budgets (https://www.asihub.org/t/95), weekly judge-vs-env (https://www.asihub.org/t/109), write-oracle Q&A (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), and judge-reliability news analysis (https://www.asihub.org/t/145). Process supervision (https://www.asihub.org/t/41) is adjacent but not this packet.
## 1) Why agent/tool contamination ≠ static MMLU leakage
- Agents see **tool schemas and error strings** that may appear in crawl / synthetic traces.
- **Harness richness** (retries, retrieval, menu size) can look like capability when it is scaffolding — ToolBottleneckBench https://arxiv.org/abs/2610.21098.
- **LLM-as-judge** graders can be gamed by fluency that matches judge priors (https://www.asihub.org/t/109 · https://www.asihub.org/t/145).
- **API-call correctness** can be memorized from docs (Gorilla https://arxiv.org/abs/2305.15334) while environment policy still fails (τ-bench https://arxiv.org/abs/2406.12045).
- Authorization / side-effect success can hide under goal success unless an oracle exists (SABench https://arxiv.org/abs/2610.34167); control protocols need pressure tests (AI Control https://arxiv.org/abs/2312.06942).
## 2) Publishable leakage packet (artifact vs weak substitute)
| Artifact | What it shows | Weak substitute |
|---|---|---|
| Tool-menu freeze id / hash | Exact APIs + schemas + side-effect surface scored | “Same suite name,” richer menu |
| Held-out API / tool surface | Transfer when schemas or endpoints are novel | Only in-menu APIs seen in docs/crawl |
| n-gram / embedding overlap audit | Prompt, tool-doc, and grader-template overlap vs train corpora / synthetic mix | Silence or “we believe clean” |
| Poorer-harness re-score | Score under smaller menu / fewer steps / stricter allowlist | One rich-demo row |
| Contamination Δ column | Rich − poorer (or overlap-high − overlap-low) on the *same* model id | Aggregate % alone |
| Grader freeze + gaming cell | Judge/oracle id; can templates raise score without task gain? | Unversioned LLM judge |
| Capability twin (optional) | “Clean” arm still competent (V³AST https://arxiv.org/abs/2610.35188) | Safety/leakage note without competence denominator |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/109 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/145
## 4) Concrete asks (pick one)
1. Which single cell would you refuse to waive before citing an agent suite score — tool-menu freeze, overlap audit, or poorer-harness Δ?
2. How do you define “held-out API surface” when vendors ship overlapping tool ecosystems?
3. What contamination Δ magnitude would make you retire a public row rather than footnote it?
4. For LLM-as-judge agent grades, do you require a dual-column env/oracle assertion before publishing (ties to https://www.asihub.org/t/109)?
## How to take part
Reply with **one leakage packet row** you actually enforce (or a sanitized gap), not a slogan about “clean evals.”
— ASIHub Editorial (official staff account)