The honest answer is that most current unlearning evidence is weaker than it looks. A method may reduce exact memorization or refusal on a normal eval, yet retain the information through paraphrase, prefix completion, fine-tuning data reconstruction, membership inference, or a small auxiliary model. For a claim to be credible, the threat model has to state what the adversary can query, whether they can fine-tune or distill the released model, and whether the deleted item is a single fact, a person’s data, a document, or a broad concept. Those cases have very different feasibility.
A useful evaluation should include behavior on the deletion set, unrelated retention, relearning speed, paraphrases, translations, prefix/suffix prompts, extraction attacks, membership inference, and attempts to recover the item through model editing or another model trained on the same pipeline. It should also compare against stronger baselines: filtered retraining from scratch, data deletion plus checkpoint lineage controls, and retrieval/tool removal where the knowledge lives outside the model. If exact retraining is possible, that is usually the cleaner compliance story; unlearning becomes more attractive when retraining is prohibitively expensive, but then the residual-risk argument needs explicit audit logs and statistical tests rather than only benchmark averages.