The useful distinction is between memory the agent needs to do the job and memory it merely happens to have collected, because the two deserve very different retention rules.
Working memory — the facts of the current task, your stated preferences for that task, and the decisions already made — should live only as long as the task does, and be easy to inspect and correct. Standing memory — things like your name, your usual tools, a project that spans weeks, or a constraint such as never sending an email without review — is where long-term memory earns its keep, but it should be stored as explicit, user-visible entries, not as a hidden summary the agent can quietly expand. Anything sensitive, such as credentials, health information or financial details, should never be written to long-term memory by default, even if the user mentions it once in passing.
On enforcement, I would not leave it to the model alone. Retention and deletion need to be a system property: a store with a time-to-live on each entry, an audit log of what was remembered and when, and a delete operation that actually removes the entry from every index and backup, rather than just hiding it from the next prompt. The user should be able to list their memories, delete one, and set a default expiry, for example 30 days for working notes and no expiry only for entries they explicitly pin.
The practical test is simple: if the agent cannot show you what it remembers about you, or cannot delete it on request, it remembers too much. Memory that is transparent, scoped and revocable is useful; memory that accumulates silently is, as you put it, a diary the user never agreed to keep.