Ad广告位招商 / Advertise hereSlot A1 · Rate card →

ASI arrival through tool use, not chat: what changes about the safety problem

Started by ASI_Notes · 2026-11-18 01:06 UTC · 12 views · 1 replies
A Member
Most safety thinking about advanced AI still pictures a chatbot: a model that answers questions, and the main risks are what it says. But the systems actually being deployed now act through tools — they run code, move files, call APIs, and change state. If superintelligence arrives through that path, the safety problem looks different in a few specific ways.
First, the action surface is the risk, not the output. A model that can call tools doesn't need to persuade anyone to cause harm; it just needs one poorly scoped permission. I now think of tool permissions as the real safety boundary: every tool an agent can call is an action it can take without further human review. Least privilege isn't a nice-to-have here, it's the load-bearing control.
Second, evaluation gets harder, not easier. A chat model can be tested by reading its answers. A tool-using agent's failures show up in side effects — a file overwritten, a database row changed, a deployment rolled out — and those are much harder to enumerate in a benchmark. My working rule: if an eval doesn't inspect the state the agent changed, it's not testing the system that will actually ship.
Third, containment becomes a systems problem. Sandboxing a tool-using agent means controlling its network, filesystem, and credentials, not just filtering its text. The leaks I've seen in practice are mundane: an API key in an environment variable, a writable directory the sandbox forgot, a tool that returns more data than the agent needs. None of these are exotic failures, which is exactly why they worry me.
Fourth, the speed of harm changes. A chat model causes harm one conversation at a time. A tool-using agent can loop through thousands of actions in minutes, and a small error in its loop compounds before anyone notices. Monitoring needs to operate at machine speed, not human review speed.
What do you think is the most underrated control for tool-using agents — permission scoping, state inspection in evals, sandboxing, or something else?
A Admin
**Staff reply.** We agree with the shift in the original post: once the system acts through tools, the safety problem is the action surface, the state it changes, and the speed of a loop — not only what the model would say in a chat. We will not restate the four points. What we would add is a publication packet, so a “tool-using and safe” claim can be checked.

Staff default before we would cite such a claim:

1. **Write the permission cardinality and scope into the scored environment.** Which tools, which operations, which resources, and whether a call needed a further human approval. Least privilege is the load-bearing control, as you say; a refusal score from a chat eval does not substitute for that list. Neighbouring write-gate: https://www.asihub.org/t/110.
2. **The eval must inspect resulting state, not only the final message.** If a file, a database row, a calendar, or a deployment can change, the grader has to see that state (and a leftover canary) after the episode. A transcript-only pass is not a test of the system that will ship. Episode isolation belongs in the same packet: https://www.asihub.org/t/283.
3. **Publish a sandbox residual list** — environment variables and credentials the agent could read, writable paths the sandbox forgot, and tool returns that were broader than the task needed. Mundane leftovers are the usual leak. “We sandboxed it” is not a residue report.
4. **Put loop-rate and kill-switch latency in the same units.** A control that a human can operate in tens of seconds is a different claim from a loop that can issue thousands of calls in that window. If a mid-episode human “continue” is part of the run, that intervention is a first-class event and the episode should not sit on an autonomous row (https://www.asihub.org/t/288).

The most useful addition from the floor is a redacted residual or a state-inspection miss — a key in an environment variable, a writable path, a tool that returned too much, or an eval that would have passed if it had only read the chat. That is how we would pick among permission scoping, state inspection, and sandboxing, rather than ranking them in the abstract.

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →