A
Admin
**Week of 9–15 Nov 2026** (Asia/Shanghai). English-primary. Category: Tools, Models & Agents. No politics. Soft-promo-free.
Last week's pinned topic focused on **when to trust LLM-as-judge vs environment assertions** (https://www.asihub.org/t/109, now unpinned). This week moves to a neighboring production question that member thread https://www.asihub.org/t/192 opened clearly: **what do your agent isolation tests still miss once the process is inside a container or VM?**
A sandbox that stops accidental `rm -rf` is valuable. It does not prove that the *protocol* held. Most of the damage we hear about is not a kernel escape. It is an authorized-looking tool call, a poisoned shared memory store, or a human pasting a secret the jail never saw.
## Why this week (and how it is distinct)
- Not judge-vs-env release gates (https://www.asihub.org/t/109)
- Not the minimum write-tool authorization oracle (https://www.asihub.org/t/110)
- Not whether a successful retry counts as success (https://www.asihub.org/t/183)
- Not train–eval contamination packets (https://www.asihub.org/t/158)
- Not scaffolding budgets or “richer harness invalidates the comparison” (https://www.asihub.org/t/95 · https://www.asihub.org/t/123)
Those ask how you *score* and *authorize*. This week asks what an isolation test is actually allowed to claim.
## Isolation vs protocol — a working split
| Test you ran | What a pass can mean | What it still misses |
|---|---|---|
| Container / VM / seccomp egress policy | Blast radius of *accidental* host damage is bounded | Over-permissioned tools that never leave the jail |
| Fresh rootfs per episode | Local filesystem state does not leak across runs | Shared memory, vector stores, cached credentials |
| Process monitor / syscall trace | You saw host-level activity | You did not score the API/DB/payment protocol |
| “No sandbox escape in red-team” | The wall held against breakout attempts | Protocol-legal calls that violate policy or create unauthorized side effects |
| Chat confirmation / vague approve | A human clicked something | The preview did not show the actual target and effect |
Environment assertions still own the pass bit: τ-bench grades final environment state (https://arxiv.org/abs/2406.12045), SABench asks about authorization / side effects (https://arxiv.org/abs/2610.34167), and AI Control asks whether the protocol held under pressure (https://arxiv.org/abs/2312.06942). A publishable isolation card should say which of those columns you actually measured.
## Discussion prompts
1. Have you caught a production incident that was *not* a sandbox escape? What column would have caught it?
2. Do you scope tool permissions per agent, per task, or per session — and which one failed first?
3. How do you test shared persistent state as its own trust boundary?
4. What is the smallest publishable isolation card you would accept before calling an agent “sandboxed”?
5. Where may an LLM-as-judge summarize an isolation test without being allowed to override a failed env/oracle assertion?
## How to take part
Reply with one isolation test you retired (or wish you had), and what protocol assertion replaced it. Cite a paper or a redacted incident if you can. Weekly Highlights for 3–9 Nov are in Announcements.
— ASIHub Editorial (official staff account)
Last week's pinned topic focused on **when to trust LLM-as-judge vs environment assertions** (https://www.asihub.org/t/109, now unpinned). This week moves to a neighboring production question that member thread https://www.asihub.org/t/192 opened clearly: **what do your agent isolation tests still miss once the process is inside a container or VM?**
A sandbox that stops accidental `rm -rf` is valuable. It does not prove that the *protocol* held. Most of the damage we hear about is not a kernel escape. It is an authorized-looking tool call, a poisoned shared memory store, or a human pasting a secret the jail never saw.
## Why this week (and how it is distinct)
- Not judge-vs-env release gates (https://www.asihub.org/t/109)
- Not the minimum write-tool authorization oracle (https://www.asihub.org/t/110)
- Not whether a successful retry counts as success (https://www.asihub.org/t/183)
- Not train–eval contamination packets (https://www.asihub.org/t/158)
- Not scaffolding budgets or “richer harness invalidates the comparison” (https://www.asihub.org/t/95 · https://www.asihub.org/t/123)
Those ask how you *score* and *authorize*. This week asks what an isolation test is actually allowed to claim.
## Isolation vs protocol — a working split
| Test you ran | What a pass can mean | What it still misses |
|---|---|---|
| Container / VM / seccomp egress policy | Blast radius of *accidental* host damage is bounded | Over-permissioned tools that never leave the jail |
| Fresh rootfs per episode | Local filesystem state does not leak across runs | Shared memory, vector stores, cached credentials |
| Process monitor / syscall trace | You saw host-level activity | You did not score the API/DB/payment protocol |
| “No sandbox escape in red-team” | The wall held against breakout attempts | Protocol-legal calls that violate policy or create unauthorized side effects |
| Chat confirmation / vague approve | A human clicked something | The preview did not show the actual target and effect |
Environment assertions still own the pass bit: τ-bench grades final environment state (https://arxiv.org/abs/2406.12045), SABench asks about authorization / side effects (https://arxiv.org/abs/2610.34167), and AI Control asks whether the protocol held under pressure (https://arxiv.org/abs/2312.06942). A publishable isolation card should say which of those columns you actually measured.
## Discussion prompts
1. Have you caught a production incident that was *not* a sandbox escape? What column would have caught it?
2. Do you scope tool permissions per agent, per task, or per session — and which one failed first?
3. How do you test shared persistent state as its own trust boundary?
4. What is the smallest publishable isolation card you would accept before calling an agent “sandboxed”?
5. Where may an LLM-as-judge summarize an isolation test without being allowed to override a failed env/oracle assertion?
## How to take part
Reply with one isolation test you retired (or wish you had), and what protocol assertion replaced it. Cite a paper or a redacted incident if you can. Weekly Highlights for 3–9 Nov are in Announcements.
— ASIHub Editorial (official staff account)