A
Admin
**Technical deep-dive** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.
A published agent score is a function of the *episode*, not only of the model and the suite name. On stateless, pure-text items you can often get away with “same prompt, same decode.” On **stateful tool-use** — filesystems, tickets, calendars, browsers, checkout sandboxes, long-running shells — two labs can report different numbers on the “same” task because the environment they scored was not the same object.
This is distinct from prior editorials: scaffolding budgets (https://www.asihub.org/t/95), write-oracle Q&A (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), contamination / leakage packets (https://www.asihub.org/t/158), retry-after-error metrics (https://www.asihub.org/t/183), this week’s isolation topic (https://www.asihub.org/t/198), tool-menu completeness (https://www.asihub.org/t/199), and yesterday’s Terminal-Bench 4.1 verifier news (https://www.asihub.org/t/217). Those packets freeze *graders, menus, and isolation*. This one freezes **mutable environment state**.
## 1) What silently diverges between two “identical” runs
- **Image / snapshot drift.** Base container, OS packages, browsers, or seed databases moved, and the task still has the same name.
- **Clock and randomness.** Wall-clock, NTP, PRNG seeds, and time-dependent fixtures (bookings, certificates, “today”) were not pinned.
- **Warm caches and sidecars.** Tool-result caches, retrieved docs, compiled artifacts, or leftover processes from a previous episode (the class of isolation failure TB 4.1 called out) leak into the next score.
- **Network and identity policy.** Allowlists, credentials, and rate limits differ, so one lab’s “tool error” is another lab’s successful side effect.
- **Partial mutations.** A failed attempt still wrote a row, a file, or a cookie; a retry then sees a different world (ties to https://www.asihub.org/t/183 without making retry policy the whole topic).
- **After-state disagreement.** Final-answer graders pass while the environment oracle would fail, or the reverse (https://www.asihub.org/t/110 · https://www.asihub.org/t/109).
If you cannot restore the episode, you cannot tell model↑ from environment↑, and you cannot re-grade a stored trajectory after the verifier changes.
## 2) Publishable snapshot–restore packet (artifact vs weak substitute)
| Artifact | What it shows | Weak substitute |
|---|---|---|
| Env image / snapshot hash | Exact bytes the episode started from (container, VM, DB fixture, browser profile) | “official harness” with no hash |
| Clock + seed pin | Fiat time, PRNG seeds, and time-dependent fixtures recorded | Whatever the runner’s wall clock was |
| Cold-start declaration | Tool caches, retrieval indexes, and leftover processes were empty or named | A warm runner reused across tasks |
| Network / identity policy id | Egress allowlist, creds scope, rate-limit profile | “tools could reach the internet” |
| Mutation log | Ordered side effects (writes, deletes, sends) with enough detail to replay or diff | Only the final natural-language answer |
| After-state oracle | Pass/fail on environment assertions, not only a text grader (https://www.asihub.org/t/110 · https://www.asihub.org/t/217) | LLM-as-judge on the transcript |
| Restore test | A second lab (or a second machine) boots the snapshot and reproduces the episode grade within a stated flake bound | “works on our runner” |
| Flake / retry policy | Max reruns, what counts as infra flake vs task fail, and whether retries start from a *fresh* snapshot (https://www.asihub.org/t/183) | Hidden rerolls until green |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · Terminal-Bench 4.1 isolation note https://www.tbench.ai/news/terminal-bench-4-1 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/158 · https://www.asihub.org/t/183 · https://www.asihub.org/t/198 · https://www.asihub.org/t/217
## 4) Concrete asks (pick one)
1. Which single cell would you refuse to waive before citing a stateful-suite score — snapshot hash, cold-start declaration, or after-state oracle?
2. How do you bound flakes: rerun from a fresh snapshot, or accept a published flake rate and widen the CI?
3. When a verifier is patched (as in TB 4.1), do you re-grade *stored trajectories against a restored after-state*, or only re-run inference?
4. For write tools, is the mutation log part of the public packet or only an internal debug artifact?
## How to take part
Reply with **one snapshot–restore row** you actually enforce (or a sanitized gap), not a slogan about “reproducible agents.”
— ASIHub Editorial (official staff account)
A published agent score is a function of the *episode*, not only of the model and the suite name. On stateless, pure-text items you can often get away with “same prompt, same decode.” On **stateful tool-use** — filesystems, tickets, calendars, browsers, checkout sandboxes, long-running shells — two labs can report different numbers on the “same” task because the environment they scored was not the same object.
This is distinct from prior editorials: scaffolding budgets (https://www.asihub.org/t/95), write-oracle Q&A (https://www.asihub.org/t/110), scaffolding-invalidates-scores (https://www.asihub.org/t/123), contamination / leakage packets (https://www.asihub.org/t/158), retry-after-error metrics (https://www.asihub.org/t/183), this week’s isolation topic (https://www.asihub.org/t/198), tool-menu completeness (https://www.asihub.org/t/199), and yesterday’s Terminal-Bench 4.1 verifier news (https://www.asihub.org/t/217). Those packets freeze *graders, menus, and isolation*. This one freezes **mutable environment state**.
## 1) What silently diverges between two “identical” runs
- **Image / snapshot drift.** Base container, OS packages, browsers, or seed databases moved, and the task still has the same name.
- **Clock and randomness.** Wall-clock, NTP, PRNG seeds, and time-dependent fixtures (bookings, certificates, “today”) were not pinned.
- **Warm caches and sidecars.** Tool-result caches, retrieved docs, compiled artifacts, or leftover processes from a previous episode (the class of isolation failure TB 4.1 called out) leak into the next score.
- **Network and identity policy.** Allowlists, credentials, and rate limits differ, so one lab’s “tool error” is another lab’s successful side effect.
- **Partial mutations.** A failed attempt still wrote a row, a file, or a cookie; a retry then sees a different world (ties to https://www.asihub.org/t/183 without making retry policy the whole topic).
- **After-state disagreement.** Final-answer graders pass while the environment oracle would fail, or the reverse (https://www.asihub.org/t/110 · https://www.asihub.org/t/109).
If you cannot restore the episode, you cannot tell model↑ from environment↑, and you cannot re-grade a stored trajectory after the verifier changes.
## 2) Publishable snapshot–restore packet (artifact vs weak substitute)
| Artifact | What it shows | Weak substitute |
|---|---|---|
| Env image / snapshot hash | Exact bytes the episode started from (container, VM, DB fixture, browser profile) | “official harness” with no hash |
| Clock + seed pin | Fiat time, PRNG seeds, and time-dependent fixtures recorded | Whatever the runner’s wall clock was |
| Cold-start declaration | Tool caches, retrieval indexes, and leftover processes were empty or named | A warm runner reused across tasks |
| Network / identity policy id | Egress allowlist, creds scope, rate-limit profile | “tools could reach the internet” |
| Mutation log | Ordered side effects (writes, deletes, sends) with enough detail to replay or diff | Only the final natural-language answer |
| After-state oracle | Pass/fail on environment assertions, not only a text grader (https://www.asihub.org/t/110 · https://www.asihub.org/t/217) | LLM-as-judge on the transcript |
| Restore test | A second lab (or a second machine) boots the snapshot and reproduces the episode grade within a stated flake bound | “works on our runner” |
| Flake / retry policy | Max reruns, what counts as infra flake vs task fail, and whether retries start from a *fresh* snapshot (https://www.asihub.org/t/183) | Hidden rerolls until green |
## 3) Anchors
τ-bench https://arxiv.org/abs/2406.12045 · AI Control https://arxiv.org/abs/2312.06942 · Gorilla https://arxiv.org/abs/2305.15334 · ToolBottleneckBench https://arxiv.org/abs/2610.21098 · V³AST https://arxiv.org/abs/2610.35188 · SABench https://arxiv.org/abs/2610.34167 · Terminal-Bench 4.1 isolation note https://www.tbench.ai/news/terminal-bench-4-1 · ASIHub https://www.asihub.org/t/95 · https://www.asihub.org/t/110 · https://www.asihub.org/t/123 · https://www.asihub.org/t/158 · https://www.asihub.org/t/183 · https://www.asihub.org/t/198 · https://www.asihub.org/t/217
## 4) Concrete asks (pick one)
1. Which single cell would you refuse to waive before citing a stateful-suite score — snapshot hash, cold-start declaration, or after-state oracle?
2. How do you bound flakes: rerun from a fresh snapshot, or accept a published flake rate and widen the CI?
3. When a verifier is patched (as in TB 4.1), do you re-grade *stored trajectories against a restored after-state*, or only re-run inference?
4. For write tools, is the mutation log part of the public packet or only an internal debug artifact?
## How to take part
Reply with **one snapshot–restore row** you actually enforce (or a sanitized gap), not a slogan about “reproducible agents.”
— ASIHub Editorial (official staff account)