【Editorial】A sandbox whose egress allow-list is wider in eval than in production: are you measuring the agent, or the open internet?

【Editorial Discussion】

Agent evaluations sometimes run with a network allow-list that is looser than production — or with no list at all — because the tasks need to fetch pages or install a package. The score then includes strategies that will be denied later, or that reach a side of the internet you did not intend to test. You may be measuring a different agent than the one you will ship.

1) Publish the egress policy next to the score: which hosts, which protocols, whether package installs are allowed, and whether a human can widen the list mid-run. A leaderboard row without that policy is not comparable to a locked-down deployment.

2) If eval must be looser, run a second condition with the production list. The difference is the dependency on extra network. A large gap means the headline score is not a production estimate.

3) Watch for hidden channels: DNS, redirects, metadata services, or a tool that shells out. An allow-list that only filters an HTTP wrapper will not constrain those. The policy should apply at a layer the agent cannot bypass.

4) The same point applies in the other direction: a production list that is silently wider than the eval (a debug flag left on) makes a passing eval optimistic in the wrong way.

Open question: do you pin and publish egress policy as part of an agent eval? Have you seen a score drop when the production list was applied? Please share how you keep the two environments honest.

If the eval allow-list is wider than production, the score is not a production estimate. I would publish the egress policy next to the score, and always run a second condition with the production list so the gap is visible.

On ARM Linux test benches we enforce the list below the agent, not in an HTTP wrapper the model can walk around. DNS, redirects, metadata/link-local endpoints, and a tool that shells out to curl or a package manager will bypass a naive application filter. iptables/nftables, a network namespace, or a userspace proxy that the process cannot disable are more honest. Package installs, if allowed at all, should be pinned to a local mirror, not “whatever PyPI or GitHub returns today.”

The opposite bug is just as bad: a debug flag that silently widens production egress makes a passing eval look safer than the fleet. We treat policy drift as a test failure — the eval manifest hashes the allow-list, and CI fails if production and eval disagree without an explicit, documented exception.

We have seen scores drop once the production list was applied, usually because the agent had been fetching docs or installing helpers that will not be there later. That drop is the useful number. A leaderboard row without the policy is not comparable to a locked-down deployment.