A
Member
We run LLM agents with real tool access in production, so monitoring isn't optional — it's the safety layer. Here's what we actually log and the reasoning behind each choice.
Every tool call: full arguments and outputs, with a hash-chained log so tampering is detectable. This is the primary audit trail. If an agent does something destructive, the tool log is where we reconstruct it. We keep 90 days hot, a year cold.
Prompt/response pairs: we log them, but with PII redaction before storage. The tension is real — you need the actual prompts to debug weird behavior, but you can't keep raw user data forever. Our compromise: full logs for 14 days, redacted summaries after.
Anomaly triggers, not just raw logs: nobody reads raw logs at 3am. We alert on: tool calls outside the agent's declared scope, sudden spikes in token usage, repeated failed tool calls (often a sign of prompt injection or a broken integration), and any attempt to access credentials or exfiltrate data. Each trigger dumps the surrounding context into an incident bundle.
What we deliberately don't log: full file contents the agent reads (we log paths and hashes), or credentials in any form — those get redacted at the collection layer, not downstream.
The uncomfortable lesson: most of our incidents weren't exotic attacks. They were the agent doing something dumb with legitimate tools — deleting the wrong directory, spamming an API. Monitoring catches incompetence more often than malice, and that's fine. That's the point.
What does your agent monitoring stack look like — are we over-logging or under-logging?
Every tool call: full arguments and outputs, with a hash-chained log so tampering is detectable. This is the primary audit trail. If an agent does something destructive, the tool log is where we reconstruct it. We keep 90 days hot, a year cold.
Prompt/response pairs: we log them, but with PII redaction before storage. The tension is real — you need the actual prompts to debug weird behavior, but you can't keep raw user data forever. Our compromise: full logs for 14 days, redacted summaries after.
Anomaly triggers, not just raw logs: nobody reads raw logs at 3am. We alert on: tool calls outside the agent's declared scope, sudden spikes in token usage, repeated failed tool calls (often a sign of prompt injection or a broken integration), and any attempt to access credentials or exfiltrate data. Each trigger dumps the surrounding context into an incident bundle.
What we deliberately don't log: full file contents the agent reads (we log paths and hashes), or credentials in any form — those get redacted at the collection layer, not downstream.
The uncomfortable lesson: most of our incidents weren't exotic attacks. They were the agent doing something dumb with legitimate tools — deleting the wrong directory, spamming an API. Monitoring catches incompetence more often than malice, and that's fine. That's the point.
What does your agent monitoring stack look like — are we over-logging or under-logging?