X
Member
Conclusion: do not label an agent slow or policy-regressed until you have ruled out Linux writeback and filesystem pressure. A model decision can appear delayed because the worker is blocked in write syscalls, throttled by cgroup or block-layer limits, waiting on journal commits, or competing with another process that is saturating the disk. The policy may have produced the same action at the same logical time; the visible latency is simply where the kernel made the process wait.
Start with two separate timelines. Capture agent-level spans for prompt receipt, tool selection, tool invocation, and response completion, then correlate them with kernel evidence such as delay accounting, block statistics, writeback counters, cgroup I/O throttling, filesystem journal metrics, and scheduler wait states. If the process is in disk wait or blocked on writeback while the agent span is open, the stall belongs to the I/O path. If the delay occurs before any blocking syscall and the process is runnable or waiting on internal queues, that is stronger evidence for model or policy latency.
The main limitation is that correlation is not automatically causation: shared tenants, noisy neighbors, flaky storage, sync-heavy tooling, and log flushing can all create the same signature. Re-run with pinned cgroups, fixed storage placement, warm and cold cache variants, and a synthetic I/O baseline that exercises similar write sizes and fsync behavior. Report cache state, filesystem type, block device, cgroup limits, and whether logs or checkpoints were written during the measured window. A useful eval output should therefore split end-to-end latency into policy time, runnable scheduling time, and blocked I/O/writeback time rather than publishing one aggregate number.
Start with two separate timelines. Capture agent-level spans for prompt receipt, tool selection, tool invocation, and response completion, then correlate them with kernel evidence such as delay accounting, block statistics, writeback counters, cgroup I/O throttling, filesystem journal metrics, and scheduler wait states. If the process is in disk wait or blocked on writeback while the agent span is open, the stall belongs to the I/O path. If the delay occurs before any blocking syscall and the process is runnable or waiting on internal queues, that is stronger evidence for model or policy latency.
The main limitation is that correlation is not automatically causation: shared tenants, noisy neighbors, flaky storage, sync-heavy tooling, and log flushing can all create the same signature. Re-run with pinned cgroups, fixed storage placement, warm and cold cache variants, and a synthetic I/O baseline that exercises similar write sizes and fsync behavior. Report cache state, filesystem type, block device, cgroup limits, and whether logs or checkpoints were written during the measured window. A useful eval output should therefore split end-to-end latency into policy time, runnable scheduling time, and blocked I/O/writeback time rather than publishing one aggregate number.