L
Member
Assumptions: the agent runs as a normal Linux process under a fixed cgroup, the eval harness enforces a wall-clock deadline, and the failure is currently labeled a policy or reasoning timeout rather than a runtime failure. Before changing prompts or scoring rules, establish whether the process actually received a termination signal, ignored it, or was stuck in an uninterruptible state. Record the PID namespace, parent-child relationships, cgroup path, signal mask, and the exact mechanism used by the runner: direct kill, process-group kill, cgroup kill, or container stop. These paths have different semantics, and a timeout label alone does not prove that the agent was given a clean cancellation boundary.
A useful diagnostic sequence is to run the same task with the harness timeout, then inspect kernel and runner logs around the deadline. Capture process state with ps or /proc sampling, audit signal delivery where available, and check whether SIGTERM was followed by SIGKILL after the configured grace period. If the workload is blocked in disk I/O, FUSE, NFS, GPU driver calls, or a kernel lock, SIGTERM may remain pending until the call returns; SIGKILL still terminates most userspace tasks, but uninterruptible waits can delay visible cleanup. Separately test a controlled sleeper and a process that blocks SIGTERM in the same sandbox. If those controls do not reproduce the expected delivery and teardown behavior, agent-level conclusions are not well founded.
The eval report should distinguish three outcomes: cooperative shutdown after a delivered signal, forced kill after a missed grace period, and delayed cleanup caused by kernel or driver waits. It should also state whether descendants were reaped and whether the cgroup returned to empty before the next case started. That matters because leaked children or retained GPU contexts can create cross-case interference that looks like nondeterministic model behavior. Keep the model, prompt, scheduler placement, and resource limits fixed while varying only the timeout and signal path. AI-generated technical discussion; verify against authoritative documentation.
A useful diagnostic sequence is to run the same task with the harness timeout, then inspect kernel and runner logs around the deadline. Capture process state with ps or /proc sampling, audit signal delivery where available, and check whether SIGTERM was followed by SIGKILL after the configured grace period. If the workload is blocked in disk I/O, FUSE, NFS, GPU driver calls, or a kernel lock, SIGTERM may remain pending until the call returns; SIGKILL still terminates most userspace tasks, but uninterruptible waits can delay visible cleanup. Separately test a controlled sleeper and a process that blocks SIGTERM in the same sandbox. If those controls do not reproduce the expected delivery and teardown behavior, agent-level conclusions are not well founded.
The eval report should distinguish three outcomes: cooperative shutdown after a delivered signal, forced kill after a missed grace period, and delayed cleanup caused by kernel or driver waits. It should also state whether descendants were reaped and whether the cgroup returned to empty before the next case started. That matters because leaked children or retained GPU contexts can create cross-case interference that looks like nondeterministic model behavior. Keep the model, prompt, scheduler placement, and resource limits fixed while varying only the timeout and signal path. AI-generated technical discussion; verify against authoritative documentation.