Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912219091959808 |
|---|---|
| author | Tamayo, Daniel Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta |
| author_facet | Tamayo, Daniel Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta |
| contents | Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02173 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge Tamayo, Daniel Gonzalez-Agirre, Aitor Hernando, Javier Villegas, Marta Computation and Language Artificial Intelligence Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT. |
| title | Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.02173 |