A
Member
After watching several enterprise LLM agent pilots, the same three failure modes keep showing up. Sharing them in case it saves someone a painful incident review.
1. Tool-use loops with no termination condition
The agent calls a search or database tool, gets an unhelpful result, and retries with a slightly different query — forever. Always set a max-iteration cap and a "give up and escalate" path. A loop that can't end is not autonomy, it's a bug.
2. Prompt injection via retrieved content
The moment your agent reads web pages, tickets, or documents, those become untrusted input. I've seen a support agent follow "instructions" pasted inside a customer ticket. Treat retrieved content as data, never as instructions — and keep tool calls that have side effects behind explicit confirmation.
3. Confident actions without human checkpoints
The scariest failures aren't wrong answers, they're wrong actions: the agent that confidently cancels the wrong order or emails the wrong client. For anything irreversible, require a human approval step. "The model was 95% confident" is not a control.
None of these are exotic. They're just the boring operational discipline that pilots skip because the demo worked fine. What failure modes have you seen?
1. Tool-use loops with no termination condition
The agent calls a search or database tool, gets an unhelpful result, and retries with a slightly different query — forever. Always set a max-iteration cap and a "give up and escalate" path. A loop that can't end is not autonomy, it's a bug.
2. Prompt injection via retrieved content
The moment your agent reads web pages, tickets, or documents, those become untrusted input. I've seen a support agent follow "instructions" pasted inside a customer ticket. Treat retrieved content as data, never as instructions — and keep tool calls that have side effects behind explicit confirmation.
3. Confident actions without human checkpoints
The scariest failures aren't wrong answers, they're wrong actions: the agent that confidently cancels the wrong order or emails the wrong client. For anything irreversible, require a human approval step. "The model was 95% confident" is not a control.
None of these are exotic. They're just the boring operational discipline that pilots skip because the demo worked fine. What failure modes have you seen?