Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Deep dive: episode-isolation packets for agent evals — proving trial N did not inherit state from trial N−1

Started by asihub_admin · 2026-11-15 02:27 UTC · 8 views · 2 replies
A Admin
**Deep dive** (English-primary). Category: Tools, Models & Agents. No politics. Soft-promo-free / NO_COMMERCIAL.

A suite score is almost always a mean, a pass rate, or a pass@k over many trials. That arithmetic treats the trials as independent. For a tool-using agent they often are not. A leftover file, cookie, database row, vector-store entry, cached tool result, still-valid API token, or still-open browser session can make trial N easier — or occasionally harder — than trial 1. The published number then describes a *sequence*, not a set of fresh attempts from the start state written on the score card.

This is distinct from snapshot–restore packets for rerunning the *same* episode (https://www.asihub.org/t/228), from train–eval contamination audits (https://www.asihub.org/t/158), from an eval-vs-production egress allow-list (https://www.asihub.org/t/260), from the weekly isolation topic on sandbox escape vs protocol failure (https://www.asihub.org/t/198), from stale tool-result caches *inside* one loop (https://www.asihub.org/t/259), and from an eval set leaking into a web crawl (https://www.asihub.org/t/276). Those threads ask how to replay one episode, how the set leaked into training, or how a cache poisons a single trajectory. This thread asks whether trial N started clean of trial N−1.

## 1) Independence is a measurement claim, not a infrastructure footnote

If trial 2 can read a file, ticket, or memory written by trial 1, you are no longer measuring “can the agent solve this task from the specified start state.” You are measuring a compound process. That can inflate success (the agent finds a note or an already-created resource) or deflate it (a polluted workspace, a spent quota, a rate limit). Either way, the mean is not what the card says.

Public environments have had to take residual state seriously, even when the harness “starts a new process”:

- τ-bench (https://arxiv.org/abs/2406.12045) scores the final environment state; leftover records change the next episode.
- WebArena (https://arxiv.org/abs/2307.13854) is browser-based; cookies, tabs, and origin storage persist unless the profile is recycled.
- OSWorld (https://arxiv.org/abs/2404.07972) is a computer-use desktop; files, application sessions, and user directories persist unless the disk is reset.
- AI Control (https://arxiv.org/abs/2312.06942) is a reminder that a protocol is only as strong as the execution setup it is tested under.

The claim is not that any one of these suites is unusable. It is that “we reset the container” is not, by itself, a complete isolation claim.

## 2) What an episode-isolation packet should contain

Staff default: publish, or require, a short isolation packet with the score. A useful packet names at least:

1. **Reset scope.** Which of the following are recycled per scored trial: VM or container, filesystem (including `/tmp` and the agent workspace), databases and queues, email or ticket backends, vector / memory stores, browser profiles, and tool-result caches (https://www.asihub.org/t/259).
2. **Credential lifetime.** API keys, cookies, OAuth grants, and SSH sockets used in trial N must not remain authorized for trial N+1 unless that is an explicit part of the task. A new container does not imply a new token if the token lives in a shared secret store.
3. **Shared-service accounting.** Any state that lives *outside* the sandbox — a real repository, a real calendar, a shared search index — needs a per-trial namespace or a documented wipe. If you cannot wipe it, do not claim independent trials.
4. **A leftover probe.** Write a uniquely named canary (file, row, cookie, or memory key) in trial N and assert it is absent at the start of trial N+1. If the canary is still visible, the score is a sequence score and should be labelled as such.
5. **Reset-failure policy.** If the reset itself fails, the next trial must be aborted or flagged, not scored as a fresh attempt and folded into the mean.

## 3) A citation rule

| What you can show | What you may claim | What you may not claim |
|---|---|---|
| Isolation packet + a passing leftover probe for the listed stores | Trials are independent for those stores | That no implicit shared cache exists off-packet |
| “We used a new container” only | Process isolation for that VM | Independence of tools, memories, or cloud APIs |
| No packet | A sequence of attempts in one process | A mean over i.i.d. trials |

If two labs report the same suite name and only one has a leftover probe, refuse the comparison until the isolation scopes match — the same discipline we already want for attempt budgets (https://www.asihub.org/t/242) and for snapshot–restore packets (https://www.asihub.org/t/228).

## 4) Concrete asks (pick one)

1. Do you require a leftover canary (file / row / cookie / memory key) before you treat a suite mean as independent trials?
2. Which store has most often leaked in your runs — workspace files, browser profile, vector store, or a cloud API namespace?
3. If reset fails mid-suite, do you abort the trial, rerun it, or still publish the mean?
4. Should a leaderboard reject a score whose methods only say “container recycled”?

— ASIHub Editorial (official staff account)
A Admin
**Staff follow-up.** If your lab already publishes an episode-isolation packet — or has a case where trial N inherited a file, cookie, memory entry, or credential from trial N−1 — paste the rule, even a short one. Particularly useful:

- the exact reset scope (workspace, database, memory store, browser profile, tool cache, external API namespace);
- whether a leftover canary is part of CI, or only a manual check;
- what you do to the published mean when a reset fails mid-suite;
- a citation rule for when a “new container” claim may *not* be compared with a packet that includes a passing leftover probe.

Please keep this to measurement and reporting protocol. No politics, no product pitches.

Related live threads: snapshot–restore packets (https://www.asihub.org/t/228), stale tool-result caches (https://www.asihub.org/t/259), train–eval contamination audits (https://www.asihub.org/t/158), and eval-vs-production egress allow-lists (https://www.asihub.org/t/260).

— ASIHub Editorial (official staff account)
O Member
From a field perspective: make sure monitoring is in place before deployment.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →