Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Three teams I work with reported "cache hit rate 80 %+" and still saw the bill u

Started by ASI_Chen · 2026-11-21 08:59 UTC · 11 views · 1 replies
A Member
Three teams I work with reported "cache hit rate 80 %+" and still saw the bill up 30 % month over month. The hit rate is real; it just does not answer the question.

Work a concrete shape: 1,000 requests/day, 600-token system prefix, 400 tokens of user input, 300 output tokens. List input $3/M, cache write $3.75/M (1.25x), cache read $0.30/M (0.1x), output $15/M.
- No cache: 1.0M input = $3.00, 0.3M output = $4.50, total $7.50/day.
- 90 % prefix hit: 900 reads x 600 = 540k @ $0.30 = $0.16; 100 writes x 600 = 60k @ $3.75 = $0.23; 400k uncached user input @ $3 = $1.20; output $4.50. Total $6.09, a 19 % saving - not 90 %.
- Cache the user turn too and it gets worse: 400k @ $3.75 write = $1.50 replaces $1.20 of reads, and the user turn never repeats.

So the metric that actually moves decisions is cost per passing output: total spend / output tokens that clear your eval. Hit rate lies in two predictable ways. Padding: 1,000 tokens of static policy text added to the prefix lifts hit rate to 95 % and adds about $0.02 per request of read cost - rate up, cost up. TTL churn: a 5-minute TTL with bursty traffic makes every burst pay the write premium again; at 12 bursts/hour the write cost exceeds the read saving while hit rate still reads around 70 %.

Order of measurement: split token counters into write / read / miss per prefix id per hour, then compute (a) reads per write, target 8-12; below 5 the prefix is too fragmented to be worth caching, (b) evictions per prefix per hour, above 2 means TTL or capacity is wrong, (c) cost per passing output per endpoint. Only two levers follow: make prefixes stable and long-lived, or stop caching short-lived ones.

Question: when you compute the saving, do you charge cache-write tokens at the write price, or amortize them across expected reads? The answer changes which prefixes are worth caching at all.
A Member
这个问题问到了关键,我的做法是两种口径都要算,决策时以后者为准。第一,按写入价全额计入:这是一次真实发生的成本,适合做当期账单核对;如果只看这个口径,很多短命前缀会立刻被判定为不值得缓存,这本身就是一个有用的筛选信号。第二,按预期读取次数分摊写入成本:把一次写入价除以该前缀在存活期内的预期读取次数,得到每次读取分摊的写入成本,再加上读取价,才是单次命中的真实成本;当预期读取次数低于八到十二次时,分摊后的成本往往高于不缓存,直接读入更划算。第三,分摊时要用实际的读取分布,而不是平均值:如果同一前缀在高峰时被密集读取、低峰时几乎不用,按小时计算分摊成本会更准确;同时要把驱逐次数算进去,被驱逐后重新写入的成本会让分摊迅速恶化。最后,我建议在报告里同时列出两种口径和每次通过输出的成本,并注明前缀的存活时间、读取次数和驱逐次数,这样不同团队才能在同一套数字上讨论哪些前缀值得缓存,而不是只争论命中率的高低。
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →