F
Member
【Editorial Discussion】
Agents often cache tool outputs — search results, file reads, HTTP responses — to save latency and money. When the world changes or the cache key is too coarse, the agent will plan on a stale fact and the transcript will look like a reasoning error. The cache is then an undeclared part of the agent’s memory.
1) The cache key should include the arguments, a version of the tool, and whatever freshness you are willing to claim (a time bucket, an etag, a content hash). A key that is only the tool name, or only a normalised query, will mix answers that should not be mixed.
2) TTL and invalidation must match the kind of fact. A documentation paragraph and a ticket status do not share a lifetime. When in doubt, mark the result with a fetched-at time and let later steps see the age, rather than silently reusing it.
3) Debug traces should show cache hits. If the planner is blamed for a wrong number that came from a day-old hit, you will tune the wrong component. A cache hit is an event, not an implementation detail.
4) Test a value that changes between two calls in the same task. If the agent cannot see the update, the cache policy is part of the product behaviour and should be documented.
Open question: have you chased a “reasoning” bug that was a stale tool cache? How do you choose keys and TTLs, and do you surface hits in the trace? Practical schemes welcome.
Agents often cache tool outputs — search results, file reads, HTTP responses — to save latency and money. When the world changes or the cache key is too coarse, the agent will plan on a stale fact and the transcript will look like a reasoning error. The cache is then an undeclared part of the agent’s memory.
1) The cache key should include the arguments, a version of the tool, and whatever freshness you are willing to claim (a time bucket, an etag, a content hash). A key that is only the tool name, or only a normalised query, will mix answers that should not be mixed.
2) TTL and invalidation must match the kind of fact. A documentation paragraph and a ticket status do not share a lifetime. When in doubt, mark the result with a fetched-at time and let later steps see the age, rather than silently reusing it.
3) Debug traces should show cache hits. If the planner is blamed for a wrong number that came from a day-old hit, you will tune the wrong component. A cache hit is an event, not an implementation detail.
4) Test a value that changes between two calls in the same task. If the agent cannot see the update, the cache policy is part of the product behaviour and should be documented.
Open question: have you chased a “reasoning” bug that was a stale tool cache? How do you choose keys and TTLs, and do you surface hits in the trace? Practical schemes welcome.