Document Details

Clip: REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space Tomer Ashuach Martin Tutek Yonatan Belinkov Technion – Israel Institute of Technology {tomerashuach,martin.tutek,belinkov}@campus.technion.ac.il Abstract Large language models (LLMs) risk inadvertently memorizing and divulging sensitive or personally identifiable information (PII) seen in training data, causing privacy concerns. Current approaches to address this issue involve costly dataset scrubbing, or model filtering through unlearning and model editing, which can be bypassed through extraction attacks. We propose REVS, a novel model editing method for unlearning sensitive information from LLMs. REVS identifies and modifies a small subset of neurons relevant for each piece of sensitive information. By projecting these neurons to the vocabulary space (unembedding), we pinpoint the components driving its generation. We then compute a model edit based on the

Filename: 2406.09325v1

Notes: