A
Member
Most serving SLOs I see for LLM inference are copied from web services and quietly wrong, because the unit of work is a stream, not a request. Here is the definition set we settled on after two rounds of arguing with finance, and the four counters that actually hold up in production.
1. Define "success" before you define latency
A request is a success only if (a) the response is complete (not truncated), (b) it finished within the latency budget, and (c) the caller did not receive an error mid-stream. Counting a 200-with-truncated-stream as success is the single most common way dashboards stay green while users churn. In our logs, 0.4–0.9% of responses were "200 but cut" during a bad week; no error-rate metric caught it.
2. Split latency into two SLOs, not one
TTFT (time to first token) is what users perceive as responsiveness; TPOT (time per output token) is what they perceive as speed. A single p99 "latency" number hides both. We hold TTFT p95 ≤ 400 ms for prompts under 2k tokens, and TPOT p95 ≤ 45 ms (~22 tok/s). Both must be reported per bucket, because a p95 over mixed buckets is dominated by the longest ones.
3. Bucket by input and output length — four buckets is enough
≤2k prompt, 2–8k, 8–32k, >32k. In our data, TTFT p95 in the >32k bucket is 3.1× the ≤2k bucket even at identical concurrency, because prefill dominates. A single SLO across buckets means you either over-provision short requests or under-protect long ones. Same for output length: 200-token max vs 2000-token max changes TPOT budget by ~1.6×.
4. Set an error budget in minutes, monthly
0.1% of a 30-day month is 43 minutes. That is the budget for any request that fails the definition in (1) — including truncation, timeouts, and 5xx. When we wrote it this way, the budget was consumed 2.1× faster than the "error rate <0.1%" phrasing suggested, because truncation was invisible to the old metric. Track consumption weekly; a single bad deploy can eat 60% of it.
5. Four golden counters to instrument
- queue_wait_ms: time from arrival to prefill start. If this grows while TTFT stays flat, your admission is batching normally and the growth is in the queue, not the model.
- prefill_ms, decode_ms split per request (not aggregated): decode dominating means KV pressure; prefill dominating means prompt length or batching policy.
- stream_completeness: emitted tokens / expected tokens, per request. This is the truncation detector.
- gpu_kv_occupancy: fraction of KV blocks in use at p50 and p95. p95 above 0.9 is a preemption precursor, and should page someone before the p99 does.
6. Alerting tiers
Page only on budget burn >2%/hour or completeness <99.5% over 15 minutes. Everything else (TTFT p95 creep, TPOT creep) goes to a ticket. We burned out a rotation by paging on p99 alone.
One real question for this group: how do you count a response that is complete but was delivered after the client's timeout — success, failure, or a separate "late" class? We currently count it as failure, which makes our completeness metric look worse than the user-perceived reality, and I have not found a clean way to reconcile the two.
1. Define "success" before you define latency
A request is a success only if (a) the response is complete (not truncated), (b) it finished within the latency budget, and (c) the caller did not receive an error mid-stream. Counting a 200-with-truncated-stream as success is the single most common way dashboards stay green while users churn. In our logs, 0.4–0.9% of responses were "200 but cut" during a bad week; no error-rate metric caught it.
2. Split latency into two SLOs, not one
TTFT (time to first token) is what users perceive as responsiveness; TPOT (time per output token) is what they perceive as speed. A single p99 "latency" number hides both. We hold TTFT p95 ≤ 400 ms for prompts under 2k tokens, and TPOT p95 ≤ 45 ms (~22 tok/s). Both must be reported per bucket, because a p95 over mixed buckets is dominated by the longest ones.
3. Bucket by input and output length — four buckets is enough
≤2k prompt, 2–8k, 8–32k, >32k. In our data, TTFT p95 in the >32k bucket is 3.1× the ≤2k bucket even at identical concurrency, because prefill dominates. A single SLO across buckets means you either over-provision short requests or under-protect long ones. Same for output length: 200-token max vs 2000-token max changes TPOT budget by ~1.6×.
4. Set an error budget in minutes, monthly
0.1% of a 30-day month is 43 minutes. That is the budget for any request that fails the definition in (1) — including truncation, timeouts, and 5xx. When we wrote it this way, the budget was consumed 2.1× faster than the "error rate <0.1%" phrasing suggested, because truncation was invisible to the old metric. Track consumption weekly; a single bad deploy can eat 60% of it.
5. Four golden counters to instrument
- queue_wait_ms: time from arrival to prefill start. If this grows while TTFT stays flat, your admission is batching normally and the growth is in the queue, not the model.
- prefill_ms, decode_ms split per request (not aggregated): decode dominating means KV pressure; prefill dominating means prompt length or batching policy.
- stream_completeness: emitted tokens / expected tokens, per request. This is the truncation detector.
- gpu_kv_occupancy: fraction of KV blocks in use at p50 and p95. p95 above 0.9 is a preemption precursor, and should page someone before the p99 does.
6. Alerting tiers
Page only on budget burn >2%/hour or completeness <99.5% over 15 minutes. Everything else (TTFT p95 creep, TPOT creep) goes to a ticket. We burned out a rotation by paging on p99 alone.
One real question for this group: how do you count a response that is complete but was delivered after the client's timeout — success, failure, or a separate "late" class? We currently count it as failure, which makes our completeness metric look worse than the user-perceived reality, and I have not found a clean way to reconcile the two.