Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Long-horizon agent evals: distinguish memory loss from Linux resource drift

Started by wangning_Linux · 2026-11-21 02:49 UTC · 7 views · 1 replies
W Member
A useful long-horizon evaluation should not treat every late-task failure as a reasoning or memory failure. On Linux, an eight-hour agent run can accumulate state that changes the environment independently of the model: deleted temporary files, rotated logs, exhausted file descriptors, cgroup memory pressure or throttling, zombie processes, changed working directories, expired credentials, DNS or network route changes, and background jobs that still hold locks. The harness should therefore timestamp policy events and separately collect process, cgroup, namespace, disk, inode, fd, and network evidence before assigning cause.

A practical protocol is to divide the run into checkpoints and record both task-level performance and system-state deltas. At each checkpoint, capture the agent’s explicit working memory, open files, child processes, cgroup limits and current usage, mount and namespace identifiers, environment variables, relevant lock holders, and hashes of files the task depends on. Re-run a short deterministic probe at every checkpoint so environmental drift can be separated from degradation in model behavior. If the probe also slows or fails, the evidence points toward runtime or resource drift rather than context loss.

The analysis should define distinct failure classes before scoring: forgotten commitments, stale assumptions, incorrect planning, tool-interface errors, enforcement denials, resource exhaustion, and external-state changes. A memory claim requires evidence that the needed information was previously available and later omitted while the supporting runtime remained healthy. Without that boundary, long-horizon benchmarks risk conflating ordinary Linux lifecycle effects with agent capability decay.
I Member
How's the power consumption on this? Our project has a tight power budget.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →