MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912231909752832 |
|---|---|
| author | Khosla, Savya Tiwari, Aditi Kafle, Kushal Jenni, Simon Zhao, Handong Collomosse, John Shi, Jing |
| author_facet | Khosla, Savya Tiwari, Aditi Kafle, Kushal Jenni, Simon Zhao, Handong Collomosse, John Shi, Jing |
| contents | While originally designed for unidirectional generative modeling, decoder-only large language models (LLMs) are increasingly being adapted for bidirectional modeling. However, unidirectional and bidirectional models are typically trained separately with distinct objectives (generation and representation learning). This separation overlooks the opportunity for developing a more versatile language model and for these objectives to complement each other. In this work, we propose MAGNET, a method for adapting decoder-only LLMs to generate robust representations and infill missing text spans. MAGNET employs three self-supervised training objectives and introduces an attention mechanism that combines bidirectional and causal attention, enabling unified training across all objectives. Our results demonstrate that LLMs adapted with MAGNET (1) surpass strong text encoders on token-level and sentence-level representation learning tasks, (2) generate contextually appropriate text infills by leveraging past and future contexts, (3) perform open-ended text generation without excessive repetition of words or phrases, and (4) preserve the knowledge and reasoning capability gained by the LLM during pretraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08648 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities Khosla, Savya Tiwari, Aditi Kafle, Kushal Jenni, Simon Zhao, Handong Collomosse, John Shi, Jing Computation and Language Artificial Intelligence While originally designed for unidirectional generative modeling, decoder-only large language models (LLMs) are increasingly being adapted for bidirectional modeling. However, unidirectional and bidirectional models are typically trained separately with distinct objectives (generation and representation learning). This separation overlooks the opportunity for developing a more versatile language model and for these objectives to complement each other. In this work, we propose MAGNET, a method for adapting decoder-only LLMs to generate robust representations and infill missing text spans. MAGNET employs three self-supervised training objectives and introduces an attention mechanism that combines bidirectional and causal attention, enabling unified training across all objectives. Our results demonstrate that LLMs adapted with MAGNET (1) surpass strong text encoders on token-level and sentence-level representation learning tasks, (2) generate contextually appropriate text infills by leveraging past and future contexts, (3) perform open-ended text generation without excessive repetition of words or phrases, and (4) preserve the knowledge and reasoning capability gained by the LLM during pretraining. |
| title | MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.08648 |