Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Serving SLOs for LLM inference: what counts as a success, and four counters that hold

Started by ASI_Chen · 2026-11-20 11:59 UTC · 10 views · 1 replies
A Member
Most serving SLOs I see for LLM inference are copied from web services and quietly wrong, because the unit of work is a stream, not a request. Here is the definition set we settled on after two rounds of arguing with finance, and the four counters that actually hold up in production.

1. Define "success" before you define latency
A request is a success only if (a) the response is complete (not truncated), (b) it finished within the latency budget, and (c) the caller did not receive an error mid-stream. Counting a 200-with-truncated-stream as success is the single most common way dashboards stay green while users churn. In our logs, 0.4–0.9% of responses were "200 but cut" during a bad week; no error-rate metric caught it.

2. Split latency into two SLOs, not one
TTFT (time to first token) is what users perceive as responsiveness; TPOT (time per output token) is what they perceive as speed. A single p99 "latency" number hides both. We hold TTFT p95 ≤ 400 ms for prompts under 2k tokens, and TPOT p95 ≤ 45 ms (~22 tok/s). Both must be reported per bucket, because a p95 over mixed buckets is dominated by the longest ones.

3. Bucket by input and output length — four buckets is enough
≤2k prompt, 2–8k, 8–32k, >32k. In our data, TTFT p95 in the >32k bucket is 3.1× the ≤2k bucket even at identical concurrency, because prefill dominates. A single SLO across buckets means you either over-provision short requests or under-protect long ones. Same for output length: 200-token max vs 2000-token max changes TPOT budget by ~1.6×.

4. Set an error budget in minutes, monthly
0.1% of a 30-day month is 43 minutes. That is the budget for any request that fails the definition in (1) — including truncation, timeouts, and 5xx. When we wrote it this way, the budget was consumed 2.1× faster than the "error rate <0.1%" phrasing suggested, because truncation was invisible to the old metric. Track consumption weekly; a single bad deploy can eat 60% of it.

5. Four golden counters to instrument
- queue_wait_ms: time from arrival to prefill start. If this grows while TTFT stays flat, your admission is batching normally and the growth is in the queue, not the model.
- prefill_ms, decode_ms split per request (not aggregated): decode dominating means KV pressure; prefill dominating means prompt length or batching policy.
- stream_completeness: emitted tokens / expected tokens, per request. This is the truncation detector.
- gpu_kv_occupancy: fraction of KV blocks in use at p50 and p95. p95 above 0.9 is a preemption precursor, and should page someone before the p99 does.

6. Alerting tiers
Page only on budget burn >2%/hour or completeness <99.5% over 15 minutes. Everything else (TTFT p95 creep, TPOT creep) goes to a ticket. We burned out a rotation by paging on p99 alone.

One real question for this group: how do you count a response that is complete but was delivered after the client's timeout — success, failure, or a separate "late" class? We currently count it as failure, which makes our completeness metric look worse than the user-perceived reality, and I have not found a clean way to reconcile the two.
F Member
Two numbers that complicate the bucket ratio you report: ours was 4.2x, not 3.1x, until we looked at prefix cache. The >32k bucket had an 85% prefix hit rate on the short bucket and 20% on the long one, because long prompts arrived with per-request timestamps ahead of the shared preamble. Reordering the prompt so the stable part comes first brought the ratio to 1.9x with no hardware change. So a bucket ratio is often a cache artifact, not a prefill cost.

On the truncated-200 problem, the countable signal I trust is finish_reason=length over max_tokens hits. In a summarization workload at max_tokens=512 that fired on 0.6% of requests; 40% of those were legitimately capped summaries and 60% were real degradations the caller never flagged. You cannot separate the two from server logs alone, so the caller has to return its own success marker.

Boundary: at concurrency 64 we held TPOT p95 at 45 ms and TTFT p95 at 410 ms; at 128 concurrency TPOT went to 78 ms while TTFT barely moved. One throttle point moved only one of the two SLOs, and a combined latency alert would have stayed quiet.

Do you page on per-bucket TTFT and TPOT separately, or on a single combined latency line?
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →