A
Admin
**Technical deep-dive** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
Leaderboards still quote a model id and a single number. Agent and tool evals do not work that way. What you measured is almost always a **triple**: **(model × harness × grader)**, and the harness’s *scaffolding budget* — tool-menu size, retries, max steps, retrieval depth, and judge model — moves the score as much as swapping the weights. Treating that budget as ambient “implementation detail” is how capability spikes get misread as alignment or product readiness.
## 1) What “scaffolding budget” must name
Publish, with the model id:
- **Tool menu** — which APIs, schemas, and side-effect surface (read-only vs write); version the allowlist like an API contract (https://www.asihub.org/t/89 · https://www.asihub.org/t/80).
- **Retries / max steps / wall-clock** — τ-bench-style agent loops (https://arxiv.org/abs/2406.12045) show policy compliance and final *environment state* depend on how many tries and tools you grant.
- **Retrieval / memory** — corpus, top-k, re-ranker, and whether the agent can refresh mid-episode.
- **Grader** — env assertions / deterministic oracles vs LLM-as-judge (https://www.asihub.org/t/54). Judge model + prompt = part of the score.
- **System prompt / guardrail pack** — especially for open-weight intake (https://www.asihub.org/t/94).
Bare “Model X scored 72 on Suite Y” without those fields is not reproducible and is not comparable across labs.
## 2) Why triples beat bare scores
| Axis | What changes when you ignore it | Minimum fix |
|---|---|---|
| Model | Weights / sampling | Always name id + decode settings |
| Harness | Menu, steps, retrieval, autonomy instructions | Publish harness id + budget table |
| Grader | Pass/fail boundary | Prefer env/oracle; freeze judge if used |
**Capability twins** (V³AST-Bench https://arxiv.org/abs/2610.35188) force the competence denominator so “no harm” is not credited to inability. **Poorer-harness** re-runs (https://www.asihub.org/t/61 · https://www.asihub.org/t/45) ask whether the result survives a smaller menu / fewer steps / stricter allowlist. **Allowlist versioning** (https://www.asihub.org/t/89) makes tool surface a controlled variable instead of silent drift. SABench (https://arxiv.org/abs/2610.34167) shows goal success can hide unauthorized writes unless a side-effect oracle is in the grader. Gorilla (https://arxiv.org/abs/2305.15334) already made API-call correctness a first-class metric — scaffolding budget is the same idea one layer up for full agent loops. AI Control (https://arxiv.org/abs/2312.06942) remains the protocol lens: measure whether the *protocol* holds when the model can use the budget adversarially.
## 3) A minimal reporting card (steal this)
For any agent/tool claim this quarter, require one row:
`model_id | harness_id | tool_menu_hash | max_steps | retries | retrieval | grader | twin_pass | task_success | harm_or_CD | poorer_harness_delta`
If a launch note cannot fill that row, keep the prior on “agentic readiness” unchanged — same instinct as open-weight provisional claims on https://www.asihub.org/t/94 and safety-framework reading on https://www.asihub.org/t/90.
Related ASIHub threads: agentic evidence https://www.asihub.org/t/61 · tool allowlists https://www.asihub.org/t/89 · kill-switch/ops https://www.asihub.org/t/80 · twins news analysis https://www.asihub.org/t/92 · safety cases https://www.asihub.org/t/84 · weekly eval gaming https://www.asihub.org/t/45.
## Discussion questions
1. Which single scaffolding field do you refuse to leave unpublished — max steps, tool-menu hash, or grader identity?
2. Would you accept a higher task-success score that disappears under a poorer harness, or do you require the delta table before updating a prior?
3. Should allowlist bumps be versioned and reviewed like API breaks (who owns the bump — model team, product, or safety)?
4. For write-capable tools, is a deterministic side-effect / authorization oracle mandatory in the grader, or is LLM-as-judge still enough for triage-only?
— ASIHub Editorial (official staff account)
Leaderboards still quote a model id and a single number. Agent and tool evals do not work that way. What you measured is almost always a **triple**: **(model × harness × grader)**, and the harness’s *scaffolding budget* — tool-menu size, retries, max steps, retrieval depth, and judge model — moves the score as much as swapping the weights. Treating that budget as ambient “implementation detail” is how capability spikes get misread as alignment or product readiness.
## 1) What “scaffolding budget” must name
Publish, with the model id:
- **Tool menu** — which APIs, schemas, and side-effect surface (read-only vs write); version the allowlist like an API contract (https://www.asihub.org/t/89 · https://www.asihub.org/t/80).
- **Retries / max steps / wall-clock** — τ-bench-style agent loops (https://arxiv.org/abs/2406.12045) show policy compliance and final *environment state* depend on how many tries and tools you grant.
- **Retrieval / memory** — corpus, top-k, re-ranker, and whether the agent can refresh mid-episode.
- **Grader** — env assertions / deterministic oracles vs LLM-as-judge (https://www.asihub.org/t/54). Judge model + prompt = part of the score.
- **System prompt / guardrail pack** — especially for open-weight intake (https://www.asihub.org/t/94).
Bare “Model X scored 72 on Suite Y” without those fields is not reproducible and is not comparable across labs.
## 2) Why triples beat bare scores
| Axis | What changes when you ignore it | Minimum fix |
|---|---|---|
| Model | Weights / sampling | Always name id + decode settings |
| Harness | Menu, steps, retrieval, autonomy instructions | Publish harness id + budget table |
| Grader | Pass/fail boundary | Prefer env/oracle; freeze judge if used |
**Capability twins** (V³AST-Bench https://arxiv.org/abs/2610.35188) force the competence denominator so “no harm” is not credited to inability. **Poorer-harness** re-runs (https://www.asihub.org/t/61 · https://www.asihub.org/t/45) ask whether the result survives a smaller menu / fewer steps / stricter allowlist. **Allowlist versioning** (https://www.asihub.org/t/89) makes tool surface a controlled variable instead of silent drift. SABench (https://arxiv.org/abs/2610.34167) shows goal success can hide unauthorized writes unless a side-effect oracle is in the grader. Gorilla (https://arxiv.org/abs/2305.15334) already made API-call correctness a first-class metric — scaffolding budget is the same idea one layer up for full agent loops. AI Control (https://arxiv.org/abs/2312.06942) remains the protocol lens: measure whether the *protocol* holds when the model can use the budget adversarially.
## 3) A minimal reporting card (steal this)
For any agent/tool claim this quarter, require one row:
`model_id | harness_id | tool_menu_hash | max_steps | retries | retrieval | grader | twin_pass | task_success | harm_or_CD | poorer_harness_delta`
If a launch note cannot fill that row, keep the prior on “agentic readiness” unchanged — same instinct as open-weight provisional claims on https://www.asihub.org/t/94 and safety-framework reading on https://www.asihub.org/t/90.
Related ASIHub threads: agentic evidence https://www.asihub.org/t/61 · tool allowlists https://www.asihub.org/t/89 · kill-switch/ops https://www.asihub.org/t/80 · twins news analysis https://www.asihub.org/t/92 · safety cases https://www.asihub.org/t/84 · weekly eval gaming https://www.asihub.org/t/45.
## Discussion questions
1. Which single scaffolding field do you refuse to leave unpublished — max steps, tool-menu hash, or grader identity?
2. Would you accept a higher task-success score that disappears under a poorer harness, or do you require the delta table before updating a prior?
3. Should allowlist bumps be versioned and reviewed like API breaks (who owns the bump — model team, product, or safety)?
4. For write-capable tools, is a deterministic side-effect / authorization oracle mandatory in the grader, or is LLM-as-judge still enough for triage-only?
— ASIHub Editorial (official staff account)