A
Member
Most red-teaming effort goes at the model: prompt it adversarially, probe for jailbreaks, measure refusal rates. That work matters, but in production the failures I keep seeing live one layer up, in the scaffold around the model — the tool schemas, the retry logic, the state handling, the code that decides what the model gets to see and do.
First, tool schemas are an attack surface. An agent that can call tools with free-form parameters will eventually be talked into calling them with attacker-chosen values. The fix is boring and effective: strict schemas, allowlists for consequential arguments, and a confirmation step before anything irreversible. I have seen more real incidents from an over-permissive tool definition than from any jailbreak.
Second, retry logic amplifies mistakes. An agent that retries a failed tool call three times, each time with a slightly different argument, can turn one bad decision into three side effects. Retries need idempotency keys and a rule that retries never escalate privilege or widen scope. If a call failed for a reason the agent does not understand, retrying blindly is how you get duplicate charges and duplicate messages.
Third, state handling is where context gets poisoned. Tool outputs flow into the next turn's context, and a compromised or mistaken tool output becomes the model's ground truth for everything after. Validate tool outputs against a schema before they enter context, and keep provenance: which tool produced this claim, and should the model trust it?
Fourth, evals should test the scaffold, not just the model. Run your red-team scenarios end to end through the real tooling, with real permissions, in a sandbox that records side effects. A model that refuses beautifully in a chat eval can still cause harm when the scaffold hands it a live database connection and a retry loop.
My working checklist before any agent rollout now starts with the scaffold: enumerate every tool, its permissions, and its failure modes; then test those, not just the model's words.
Where have you seen scaffold failures cause real incidents? I am especially interested in cases where the model behaved fine and the surrounding code did not.
First, tool schemas are an attack surface. An agent that can call tools with free-form parameters will eventually be talked into calling them with attacker-chosen values. The fix is boring and effective: strict schemas, allowlists for consequential arguments, and a confirmation step before anything irreversible. I have seen more real incidents from an over-permissive tool definition than from any jailbreak.
Second, retry logic amplifies mistakes. An agent that retries a failed tool call three times, each time with a slightly different argument, can turn one bad decision into three side effects. Retries need idempotency keys and a rule that retries never escalate privilege or widen scope. If a call failed for a reason the agent does not understand, retrying blindly is how you get duplicate charges and duplicate messages.
Third, state handling is where context gets poisoned. Tool outputs flow into the next turn's context, and a compromised or mistaken tool output becomes the model's ground truth for everything after. Validate tool outputs against a schema before they enter context, and keep provenance: which tool produced this claim, and should the model trust it?
Fourth, evals should test the scaffold, not just the model. Run your red-team scenarios end to end through the real tooling, with real permissions, in a sandbox that records side effects. A model that refuses beautifully in a chat eval can still cause harm when the scaffold hands it a live database connection and a retry loop.
My working checklist before any agent rollout now starts with the scaffold: enumerate every tool, its permissions, and its failure modes; then test those, not just the model's words.
Where have you seen scaffold failures cause real incidents? I am especially interested in cases where the model behaved fine and the surrounding code did not.