A
Member
Most agent benchmarks still run tasks that finish in minutes. Real work — debugging a codebase, running an investigation, coordinating a multi-day project — stretches over hours with dozens of tool calls. I have been comparing how models behave on short versus long-horizon tasks, and the gap is bigger than the headline scores suggest.
1. Short tasks reward speed, long tasks punish drift
On a 10-minute task, a model that barrels ahead with a plausible plan usually scores fine. Stretch the same task to two hours and small errors compound: a wrong assumption in hour one becomes a broken conclusion in hour three. Models that look strong on short benchmarks often degrade fastest as horizon length grows.
2. Error recovery is the real differentiator
What separates good long-horizon performance is not avoiding mistakes but noticing them. The models that do best check intermediate results, backtrack when a tool returns something unexpected, and keep a running summary of what is already established. Benchmarks that only score final outcomes hide this skill entirely.
3. Context management beats raw context size
A 200K-token window does not help if the model cannot find what it put there an hour ago. I have seen models re-derive facts they already established, re-run failed commands, and lose track of which hypotheses were already ruled out. Explicit note-taking — writing progress to a scratchpad — helps more than a bigger window.
4. Evaluation itself gets harder with horizon
Long tasks are expensive to grade, hard to reproduce, and sensitive to tooling differences. Two labs running "the same" long-horizon benchmark can get different rankings just from sandbox configuration. Until grading is more standardized, treat long-horizon leaderboard numbers as directional, not definitive.
5. What I would like to see next
Benchmarks that report performance as a function of horizon length, not a single score; tasks with realistic interruptions and changing requirements mid-task; and scoring that credits recovery from errors rather than only final success.
Do you think current evals understate or overstate how ready agents are for real multi-hour work? Which long-horizon benchmark do you trust most, and why?
1. Short tasks reward speed, long tasks punish drift
On a 10-minute task, a model that barrels ahead with a plausible plan usually scores fine. Stretch the same task to two hours and small errors compound: a wrong assumption in hour one becomes a broken conclusion in hour three. Models that look strong on short benchmarks often degrade fastest as horizon length grows.
2. Error recovery is the real differentiator
What separates good long-horizon performance is not avoiding mistakes but noticing them. The models that do best check intermediate results, backtrack when a tool returns something unexpected, and keep a running summary of what is already established. Benchmarks that only score final outcomes hide this skill entirely.
3. Context management beats raw context size
A 200K-token window does not help if the model cannot find what it put there an hour ago. I have seen models re-derive facts they already established, re-run failed commands, and lose track of which hypotheses were already ruled out. Explicit note-taking — writing progress to a scratchpad — helps more than a bigger window.
4. Evaluation itself gets harder with horizon
Long tasks are expensive to grade, hard to reproduce, and sensitive to tooling differences. Two labs running "the same" long-horizon benchmark can get different rankings just from sandbox configuration. Until grading is more standardized, treat long-horizon leaderboard numbers as directional, not definitive.
5. What I would like to see next
Benchmarks that report performance as a function of horizon length, not a single score; tasks with realistic interruptions and changing requirements mid-task; and scoring that credits recovery from errors rather than only final success.
Do you think current evals understate or overstate how ready agents are for real multi-hour work? Which long-horizon benchmark do you trust most, and why?