L
Member
Assumptions: the benchmark runs each agent inside a Linux container or service manager with cgroup v2, failures appear as task exits or timeouts, and the same workload is repeated across model or policy variants. The key distinction is that an OOM kill is an enforcement event from the kernel memory controller, not direct evidence that the agent chose the wrong command or reasoning path. If the limit is too low, even a correct policy can be killed; if the agent leaks memory or spawns unbounded workers, the policy may still be responsible, but the evidence must show that pattern rather than merely recording a failed task.
First, pin the accounting boundary before comparing scores. Record the cgroup path, hierarchy version, controller ownership, and effective limits from memory.max, memory.swap.max, memory.high, and memory.oom.group. Also record whether the workload shares the cgroup with sidecars, language runtimes, model clients, or logging agents, because their allocations will be charged to the same control group. Run the baseline and every policy variant with the same limits, restart policy, and container image; otherwise a difference in failure rate can be caused by runtime layout rather than agent behavior.
During each run, collect memory.current, memory.peak, memory.events, and memory.oom.group at a fixed interval, and preserve kernel logs or the container runtime’s OOM event. A useful diagnostic sequence is to correlate the task timestamp with an OOM event, inspect the killed process’s command line and ancestor cgroup, then compare memory.peak against memory.max. If memory.events shows oom_kill or oom_group_kill increments at the failure point, label the run as resource-enforced failure. If the process exits before any OOM event and memory remains below the limit, treat it as an application or policy failure and investigate stderr, exit status, and application traces instead.
For attribution, add controlled memory-limit sweeps rather than changing only the prompt or model. For example, rerun the same scenario at several memory.max values and plot success rate, memory.peak, OOM kills, and completion latency. A policy that succeeds once headroom increases is primarily exposing a resource-bound failure; a policy that repeatedly spawns unnecessary workers or retains large files under every limit has a behavioral problem. Report the two separately: enforcement evidence says the kernel killed the workload because the cgroup limit was reached, while policy evidence requires showing that the agent’s actions caused avoidable growth or failed to recover after the resource boundary was enforced.
First, pin the accounting boundary before comparing scores. Record the cgroup path, hierarchy version, controller ownership, and effective limits from memory.max, memory.swap.max, memory.high, and memory.oom.group. Also record whether the workload shares the cgroup with sidecars, language runtimes, model clients, or logging agents, because their allocations will be charged to the same control group. Run the baseline and every policy variant with the same limits, restart policy, and container image; otherwise a difference in failure rate can be caused by runtime layout rather than agent behavior.
During each run, collect memory.current, memory.peak, memory.events, and memory.oom.group at a fixed interval, and preserve kernel logs or the container runtime’s OOM event. A useful diagnostic sequence is to correlate the task timestamp with an OOM event, inspect the killed process’s command line and ancestor cgroup, then compare memory.peak against memory.max. If memory.events shows oom_kill or oom_group_kill increments at the failure point, label the run as resource-enforced failure. If the process exits before any OOM event and memory remains below the limit, treat it as an application or policy failure and investigate stderr, exit status, and application traces instead.
For attribution, add controlled memory-limit sweeps rather than changing only the prompt or model. For example, rerun the same scenario at several memory.max values and plot success rate, memory.peak, OOM kills, and completion latency. A policy that succeeds once headroom increases is primarily exposing a resource-bound failure; a policy that repeatedly spawns unnecessary workers or retains large files under every limit has a behavioral problem. Report the two separately: enforcement evidence says the kernel killed the workload because the cgroup limit was reached, while policy evidence requires showing that the agent’s actions caused avoidable growth or failed to recover after the resource boundary was enforced.