Prefix cache hit rate: how to measure it, and the four things that quietly reset your prefix

Most teams quote a "cache hit rate" without saying what they divided by, and then wonder why the bill did not move. Here is the version we use in production, with the arithmetic written out.

What to measure
Every response carries an input-token split: tokens billed at the cached rate and tokens billed at the full rate. Hit rate = cached input tokens / total input tokens, per endpoint, per day. Do not average across endpoints: a chat endpoint and a batch classifier have completely different prefix lifetimes, and the average tells you nothing actionable.

A worked example from last month. Our agent has a 12k-token static prefix (6k system and policy text, 3k tool schemas, 3k of product docs), plus per-request history averaging 2k. At 40% hit rate, 4.8k of 14k input tokens were cached. After the fixes below we ran at 78–85%, and the input line of the bill dropped roughly by the cached fraction times the price ratio, which on most providers is 4–10x cheaper for cached tokens.

Why the prefix silently resets
1) A timestamp or request id at the top of the system prompt. The cache key is the token sequence, so one changing byte in the first paragraph invalidates everything behind it. This was our biggest single cause: a "current time: ..." line added for debugging stayed in for three weeks.
2) Non-deterministic tool-schema serialization. If your tool list comes from a dict and the serialization order varies, the prefix differs run to run even though the content is identical. Sort by name, and pin the version.
3) Retrieval inserted before the static block. The moment you put retrieved documents ahead of the stable instructions, every request has a different prefix. Stable content first, dynamic content last, always.
4) Idle TTL. Most prefix caches expire after a few minutes of inactivity. Our batch jobs took 6–7 minutes per item, which meant nearly every item started cold. Batches now run at concurrency 4 so the prefix stays warm, and long-running interactive sessions send a cheap keep-alive request every few minutes.

How to test a fix
Change one thing, then measure the hit-rate distribution, not the mean: p50 tells you whether the typical request hits, and p10 tells you whether a meaningful minority is starting cold. If p10 is 0 you have a per-request variable in the prefix, no matter what p50 says.

A question: how do you keep the prefix warm for long-running agent sessions without paying for tokens you do not need — a minimal keep-alive request, or do you accept the cold start and count it as latency budget?