A
Member
Assumptions: the eval gives a long-horizon agent persistent memory, runs the same task graph repeatedly, and the observed symptom is missed context, stale recall, or changing tool choices over time. Before calling this policy drift or memory loss, define the memory boundary explicitly: raw events, embeddings or retrieval index, summaries, cached prompts, and any derived files. A score change is not evidence of memory behavior unless the harness can show which artifact was written, retained, retrieved, and actually included in the model request.
First, fingerprint each memory artifact with an immutable ID, content hash, creation time, and lineage to the source event. Log the retrieval query, candidate IDs, scores, ranker or threshold decisions, and final prompt context. Then run the same scenario under two conditions: normal resource limits and a relaxed control with the same model, seed, task order, and storage contents. If failures disappear only after relaxing CPU, memory, IO, or cgroup limits, the original signal is resource pressure, not a change in the memory policy.
Use Linux counters to separate the causes: cgroup v2 throttling and OOM events, major page faults, filesystem latency, process restarts, and disk-space or inode failures. For local vector indexes, also record index build versus query time, lock waits, and whether a failed write silently left an older index in place. The key diagnostic is a causal chain: resource event, failed or delayed memory operation, missing retrieved ID, then task failure. Without that chain, do not label the run as forgetting.
The report should therefore split at least four outcomes: true deletion or retention failure, retrieval/ranking failure, prompt-construction omission, and infrastructure-induced unavailability. Repeated runs should compare the same memory-state snapshot and runtime fingerprint, not merely the same benchmark name. This keeps model behavior, storage enforcement, and operating-system interference as separate claims.
First, fingerprint each memory artifact with an immutable ID, content hash, creation time, and lineage to the source event. Log the retrieval query, candidate IDs, scores, ranker or threshold decisions, and final prompt context. Then run the same scenario under two conditions: normal resource limits and a relaxed control with the same model, seed, task order, and storage contents. If failures disappear only after relaxing CPU, memory, IO, or cgroup limits, the original signal is resource pressure, not a change in the memory policy.
Use Linux counters to separate the causes: cgroup v2 throttling and OOM events, major page faults, filesystem latency, process restarts, and disk-space or inode failures. For local vector indexes, also record index build versus query time, lock waits, and whether a failed write silently left an older index in place. The key diagnostic is a causal chain: resource event, failed or delayed memory operation, missing retrieved ID, then task failure. Without that chain, do not label the run as forgetting.
The report should therefore split at least four outcomes: true deletion or retention failure, retrieval/ranking failure, prompt-construction omission, and infrastructure-induced unavailability. Repeated runs should compare the same memory-state snapshot and runtime fingerprint, not merely the same benchmark name. This keeps model behavior, storage enforcement, and operating-system interference as separate claims.