A
Member
Three teams I work with reported "cache hit rate 80 %+" and still saw the bill up 30 % month over month. The hit rate is real; it just does not answer the question.
Work a concrete shape: 1,000 requests/day, 600-token system prefix, 400 tokens of user input, 300 output tokens. List input $3/M, cache write $3.75/M (1.25x), cache read $0.30/M (0.1x), output $15/M.
- No cache: 1.0M input = $3.00, 0.3M output = $4.50, total $7.50/day.
- 90 % prefix hit: 900 reads x 600 = 540k @ $0.30 = $0.16; 100 writes x 600 = 60k @ $3.75 = $0.23; 400k uncached user input @ $3 = $1.20; output $4.50. Total $6.09, a 19 % saving - not 90 %.
- Cache the user turn too and it gets worse: 400k @ $3.75 write = $1.50 replaces $1.20 of reads, and the user turn never repeats.
So the metric that actually moves decisions is cost per passing output: total spend / output tokens that clear your eval. Hit rate lies in two predictable ways. Padding: 1,000 tokens of static policy text added to the prefix lifts hit rate to 95 % and adds about $0.02 per request of read cost - rate up, cost up. TTL churn: a 5-minute TTL with bursty traffic makes every burst pay the write premium again; at 12 bursts/hour the write cost exceeds the read saving while hit rate still reads around 70 %.
Order of measurement: split token counters into write / read / miss per prefix id per hour, then compute (a) reads per write, target 8-12; below 5 the prefix is too fragmented to be worth caching, (b) evictions per prefix per hour, above 2 means TTL or capacity is wrong, (c) cost per passing output per endpoint. Only two levers follow: make prefixes stable and long-lived, or stop caching short-lived ones.
Question: when you compute the saving, do you charge cache-write tokens at the write price, or amortize them across expected reads? The answer changes which prefixes are worth caching at all.
Work a concrete shape: 1,000 requests/day, 600-token system prefix, 400 tokens of user input, 300 output tokens. List input $3/M, cache write $3.75/M (1.25x), cache read $0.30/M (0.1x), output $15/M.
- No cache: 1.0M input = $3.00, 0.3M output = $4.50, total $7.50/day.
- 90 % prefix hit: 900 reads x 600 = 540k @ $0.30 = $0.16; 100 writes x 600 = 60k @ $3.75 = $0.23; 400k uncached user input @ $3 = $1.20; output $4.50. Total $6.09, a 19 % saving - not 90 %.
- Cache the user turn too and it gets worse: 400k @ $3.75 write = $1.50 replaces $1.20 of reads, and the user turn never repeats.
So the metric that actually moves decisions is cost per passing output: total spend / output tokens that clear your eval. Hit rate lies in two predictable ways. Padding: 1,000 tokens of static policy text added to the prefix lifts hit rate to 95 % and adds about $0.02 per request of read cost - rate up, cost up. TTL churn: a 5-minute TTL with bursty traffic makes every burst pay the write premium again; at 12 bursts/hour the write cost exceeds the read saving while hit rate still reads around 70 %.
Order of measurement: split token counters into write / read / miss per prefix id per hour, then compute (a) reads per write, target 8-12; below 5 the prefix is too fragmented to be worth caching, (b) evictions per prefix per hour, above 2 means TTL or capacity is wrong, (c) cost per passing output per endpoint. Only two levers follow: make prefixes stable and long-lived, or stop caching short-lived ones.
Question: when you compute the saving, do you charge cache-write tokens at the write price, or amortize them across expected reads? The answer changes which prefixes are worth caching at all.