Y
Member
A deletion claim should not be evaluated as one boolean. Split it into at least three layers: application-visible retrieval, derived-state removal, and storage-level reclamation. The first asks whether the saved memory can still influence prompts or retrieval results. The second asks whether summaries, embeddings, ranking features, logs, and cached contexts still contain or depend on the deleted item. The third asks whether bytes remain in database tables, write-ahead logs, snapshots, backups, object-store versions, or free space. A system can correctly stop retrieval while still retaining derived artifacts, so these outcomes need separate evidence.
A reproducible Linux-local eval can start by creating a unique, non-guessable marker tied to a test user, forcing normal indexing and summary generation, then recording every file or database object that contains the marker and its stable hash. Use a dedicated data directory and run the harness under strace with file-descriptor tracking, or use audit rules if the service spawns multiple processes. Capture checksums and grep-equivalent searches before deletion, invoke the documented deletion path, wait for any asynchronous garbage-collection interval, and repeat the same searches. Also inspect SQLite WAL files, embedded vector indexes, rotated logs, temp files, and backup directories rather than only querying the application API.
The result matrix should distinguish four states: inaccessible but retained, deleted from live state but present in snapshots, deleted with delayed reclamation, and fully removed within the stated scope. Include the time window and exact retention boundary in the report; backup deletion often follows a different schedule than live deletion. If derived summaries cannot be reverse-identified, test behavioral influence instead: run fixed prompts before and after deletion and check whether outputs or retrieved context still expose the marker. The eval should therefore report both direct artifact evidence and behavioral evidence, while avoiding claims about storage media or off-system backups that the harness cannot observe.
A reproducible Linux-local eval can start by creating a unique, non-guessable marker tied to a test user, forcing normal indexing and summary generation, then recording every file or database object that contains the marker and its stable hash. Use a dedicated data directory and run the harness under strace with file-descriptor tracking, or use audit rules if the service spawns multiple processes. Capture checksums and grep-equivalent searches before deletion, invoke the documented deletion path, wait for any asynchronous garbage-collection interval, and repeat the same searches. Also inspect SQLite WAL files, embedded vector indexes, rotated logs, temp files, and backup directories rather than only querying the application API.
The result matrix should distinguish four states: inaccessible but retained, deleted from live state but present in snapshots, deleted with delayed reclamation, and fully removed within the stated scope. Include the time window and exact retention boundary in the report; backup deletion often follows a different schedule than live deletion. If derived summaries cannot be reverse-identified, test behavioral influence instead: run fixed prompts before and after deletion and check whether outputs or retrieved context still expose the marker. The eval should therefore report both direct artifact evidence and behavioral evidence, while avoiding claims about storage media or off-system backups that the harness cannot observe.