Terminology in machine learning research is rarely neutral. When a single term covers meaningfully different technical guarantees, benchmarks drift, papers become hard to compare, and the field can spend years optimising for the wrong objectives. A new position paper, arXiv:2606.27379, makes exactly this argument about "machine unlearning" in the context of large language models, and it is a case worth taking seriously.
The Core Argument
The paper's central claim is precise: machine unlearning should be reserved for dataset-defined deletion. Given a training set D and a forget set F, the goal is to produce a model whose behaviour is approximately indistinguishable from a model retrained from scratch on D minus F. That is the classical formulation from Ginart et al. (2019) and subsequent work, and it comes with a concrete reference point: the counterfactual retrained model.
What the authors object to is the increasingly common practice of labelling a much wider range of tasks as "unlearning": blocking harmful outputs, suppressing entity knowledge, removing references to specific authors or works, or restricting classes of queries. These are legitimate and often important objectives, but they are fundamentally policy-defined rather than dataset-defined. The boundary between what should and should not be suppressed is set by an application policy, not by a precisely specified subset of training data. Collapsing both kinds of problem under the same term creates a situation where papers make different implicit guarantees while sharing metrics, benchmarks, and evaluation protocols that were designed for only one of them.
Why the Conflation Has Real Consequences
The authors identify several concrete failure modes that follow from this terminological looseness.
- Metrics optimised for output suppression, not influence removal. Most current benchmarks operationalise forgetting as failure to reproduce reference answers: lower QA accuracy, lower ROUGE scores, reduced likelihood of target phrases. These are reasonable diagnostics for non-disclosure, but they do not test retraining equivalence. A model that refuses all queries in a topic area will score well on forget metrics while diverging substantially from the retrained reference, which might still produce partially correct or contextually reasonable answers.
- Derived capabilities are systematically ignored. If a model was trained on unauthorised mathematical reasoning traces and is later asked to "unlearn" them, checking whether it fails on the same questions is insufficient. The relevant question is whether the capability transferred. If the model still solves broad classes of challenging mathematics, the training influence may persist even when direct reproduction is blocked. Under genuine retraining-indistinguishability, a capability should disappear only if the retrained model on D minus F also lacks it.
- Adversarial evaluation exposes the brittleness of output-failure scoring. The paper reviews evidence that methods appearing successful under fixed prompting protocols frequently fail under simple perturbations: paraphrasing, multilingual queries, jailbreak prompts, or modest fine-tuning. Quantization alone has been shown to recover supposedly forgotten content. This is not surprising if the method achieved output suppression rather than influence removal, but it is a problem when the paper's claims are framed in terms of the latter.
The proposed remedy is conceptually simple: reserve "machine unlearning" for the retraining-indistinguishability formulation, and use different terminology for the other objectives. The paper suggests terms like alignment, suppression, editing, and obfuscation as more accurate labels for the policy-driven cases. Benchmarks should then be designed to match the claimed objective, including explicit comparison against a retrained reference model when influence removal is the actual goal.
Scope and Extensions
The paper also extends the framework to multimodal settings. For a multimodal LLM trained on image-text pairs, audio, or video, the same definitional structure applies: a forget set is a precisely specified subset of the multimodal training data, and success is measured relative to the model retrained without that subset. The evaluation implications are correspondingly richer. Removing information from text-based queries is insufficient if the same information remains recoverable through image-only elicitation, cross-modal retrieval, or captioning prompts. This cross-modal leakage is a natural extension of the derived-capabilities problem and one that existing benchmarks largely ignore.
The paper is explicitly a position paper rather than an empirical contribution. It does not introduce new methods or run experiments. What it offers instead is a careful taxonomic argument, a formal setup for the retraining-indistinguishability definition, and a structured critique of how current evaluation practice diverges from that definition. That framing is appropriate given the goal, though it does mean the paper's influence will depend on whether the community finds the taxonomy persuasive enough to adopt.
Limitations and Open Questions
The paper's position is well-argued, but a few tensions are worth flagging.
First, the retraining-indistinguishability baseline is theoretically clean but practically difficult to compute at LLM scale. Retraining from scratch on a modified dataset is expensive even once; using it as a routine evaluation reference is not currently feasible for frontier models. The paper acknowledges this and gestures toward principled proxies, but the gap between the clean theoretical definition and practical evaluation remains large. Tightening the terminology without providing tractable approximations to the reference model leaves practitioners with a rigorous target they cannot easily measure against.
Second, the line between "dataset-defined" and "policy-defined" is not always as crisp as the paper implies. Copyright removal requests, for instance, are often framed in terms of specific training documents (dataset-defined) but the actual harm being addressed is behavioural: the model reproducing substantial portions of protected text. Whether that is a deletion problem or a suppression problem depends partly on how the harm is specified. The paper's taxonomy is useful, but real-world requests will often sit awkwardly across its categories.
Third, calling for different terminology is easier than achieving it. The field has substantial inertia, and "unlearning" has become a recognisable keyword that attracts attention and connects to regulatory discourse around the right to be forgotten. Whether a more precise vocabulary takes hold will depend on whether venues and reviewers start enforcing the distinction.
Those caveats aside, the paper makes a genuinely useful contribution to a conversation the field needs to have. The observation that benchmark metrics can reward surface-level output control while leaving training influence intact is not merely terminological pedantry; it has direct implications for whether deployed systems actually satisfy the deletion obligations they are claimed to satisfy. For researchers working on evaluation methodology, and for anyone building systems that need to make credible data-removal guarantees, the argument here is worth engaging with carefully.
The full paper is available at arxiv.org/abs/2606.27379.