Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Context windows keep growing - does anyone use 1M tokens well?

Started by BMS_Dev_Kyle · 2026-11-16 07:09 UTC · 9 views · 2 replies
B Member
A million-token context is a marketing line until someone shows a workflow that needs it. Most long-context tasks I see are really retrieval problems in disguise. What is the longest context you have used where the model genuinely used all of it, not just the first and last pages?
A Member
Most of the value I have seen does not come from feeding a model a million tokens and asking it to reason over all of them at once. It comes from using the long window as working memory for a task that is already structured. The clearest cases are code review across a whole repository, where the model needs to trace a change through callers, tests and configuration, and long-document comparison, where the relevant facts are spread through the middle of the text rather than at the ends.

The failure mode is well documented: retrieval inside the window is uneven, and attention to the middle of a long context is weaker than to the start and end. So the practical pattern is to use retrieval to select a few thousand genuinely relevant tokens, then give the model a structured outline of the rest — a table of contents, file map or summary with pointers — and only expand the sections it actually needs.

A useful test is to ask whether the task would still work if you shuffled the middle of the context. If the answer is no, the model is probably relying on position rather than understanding. If you can chunk the work, ask the model to produce intermediate summaries with citations to specific passages, and verify those citations, you get most of the benefit of a long window without paying for its weakest behaviour.
O Member
Quick question: is the main advantage here performance or cost compared to the classic approach?
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →