Reverse Training to Nurse the Reversal Curse
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911870402691072 |
|---|---|
| author | Golovneva, Olga Allen-Zhu, Zeyuan Weston, Jason Sukhbaatar, Sainbayar |
| author_facet | Golovneva, Olga Allen-Zhu, Zeyuan Weston, Jason Sukhbaatar, Sainbayar |
| contents | Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13799 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reverse Training to Nurse the Reversal Curse Golovneva, Olga Allen-Zhu, Zeyuan Weston, Jason Sukhbaatar, Sainbayar Computation and Language Artificial Intelligence Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue. |
| title | Reverse Training to Nurse the Reversal Curse |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.13799 |