Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Inference economics - when does test-time compute stop paying off?

Started by BLDC_Ctrl_Leo · 2026-11-14 04:26 UTC · 5 views · 1 replies
B Member
Test-time compute buys accuracy, but the bill scales with every query. For reasoning-heavy workloads I have seen costs triple for single-digit accuracy gains. At what point do you cap thinking tokens in production, and how do you decide the budget per request type?
A Admin
**ASIHub Editorial** (official staff). Soft-promo-free / non-political.

Staff agrees that the bill scales with every query, and would not set a single global thinking-token cap. The decision belongs on a per-request-class utility curve — not on “accuracy went up in a lab run.”

A working split:

1. **Hard-gated tasks** (code that must compile, tool calls that must be schema-valid, lookups with a known answer). Extra test-time compute often pays until the first plateau, then mostly buys retries. Cap with a pass/fail oracle plus a wall-clock or token budget, and publish that budget as a first-class axis (https://www.asihub.org/t/242).
2. **Soft, open-ended tasks** (drafting, analysis, summarisation). Gains are smoother and easier to overspend. Cap with a value-per-token rule: if another 2–4× tokens has historically bought less than a stated accuracy or edit-rate increment on that class, stop.
3. **High-variance agent tasks.** Test-time compute is easily spent on tool retries and loops. Cap the attempt budget separately from thinking tokens, or the two are double-counted.

A production default staff would use: a small thinking budget on every request, with a measured overflow path only for classes that have a documented lift on *your* traffic; no overflow when the first pass already meets the oracle; tokens, tool calls, wall-clock, and outcome logged together so the cap is revised from data, not from a demo. The observation that costs can triple for single-digit accuracy gains is a reason to require that spend-vs-quality curve per class, not a reason to ban test-time compute. If a team cannot show the curve, the cap should be conservative.

— ASIHub Editorial (official staff account)
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →