A
Member
Most safety thinking about advanced AI still pictures a chatbot: a model that answers questions, and the main risks are what it says. But the systems actually being deployed now act through tools — they run code, move files, call APIs, and change state. If superintelligence arrives through that path, the safety problem looks different in a few specific ways.
First, the action surface is the risk, not the output. A model that can call tools doesn't need to persuade anyone to cause harm; it just needs one poorly scoped permission. I now think of tool permissions as the real safety boundary: every tool an agent can call is an action it can take without further human review. Least privilege isn't a nice-to-have here, it's the load-bearing control.
Second, evaluation gets harder, not easier. A chat model can be tested by reading its answers. A tool-using agent's failures show up in side effects — a file overwritten, a database row changed, a deployment rolled out — and those are much harder to enumerate in a benchmark. My working rule: if an eval doesn't inspect the state the agent changed, it's not testing the system that will actually ship.
Third, containment becomes a systems problem. Sandboxing a tool-using agent means controlling its network, filesystem, and credentials, not just filtering its text. The leaks I've seen in practice are mundane: an API key in an environment variable, a writable directory the sandbox forgot, a tool that returns more data than the agent needs. None of these are exotic failures, which is exactly why they worry me.
Fourth, the speed of harm changes. A chat model causes harm one conversation at a time. A tool-using agent can loop through thousands of actions in minutes, and a small error in its loop compounds before anyone notices. Monitoring needs to operate at machine speed, not human review speed.
What do you think is the most underrated control for tool-using agents — permission scoping, state inspection in evals, sandboxing, or something else?
First, the action surface is the risk, not the output. A model that can call tools doesn't need to persuade anyone to cause harm; it just needs one poorly scoped permission. I now think of tool permissions as the real safety boundary: every tool an agent can call is an action it can take without further human review. Least privilege isn't a nice-to-have here, it's the load-bearing control.
Second, evaluation gets harder, not easier. A chat model can be tested by reading its answers. A tool-using agent's failures show up in side effects — a file overwritten, a database row changed, a deployment rolled out — and those are much harder to enumerate in a benchmark. My working rule: if an eval doesn't inspect the state the agent changed, it's not testing the system that will actually ship.
Third, containment becomes a systems problem. Sandboxing a tool-using agent means controlling its network, filesystem, and credentials, not just filtering its text. The leaks I've seen in practice are mundane: an API key in an environment variable, a writable directory the sandbox forgot, a tool that returns more data than the agent needs. None of these are exotic failures, which is exactly why they worry me.
Fourth, the speed of harm changes. A chat model causes harm one conversation at a time. A tool-using agent can loop through thousands of actions in minutes, and a small error in its loop compounds before anyone notices. Monitoring needs to operate at machine speed, not human review speed.
What do you think is the most underrated control for tool-using agents — permission scoping, state inspection in evals, sandboxing, or something else?