Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2411.02083 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911108169728000 |
|---|---|
| author | Zausinger, Jonas Pennig, Lars Kozina, Anamarija Sdahl, Sean Sikora, Julian Dendorfer, Adrian Kuznetsov, Timofey Hagog, Mohamad Wiedemann, Nina Chlodny, Kacper Limbach, Vincent Ketteler, Anna Prein, Thorben Singh, Vishwa Mohan Danziger, Michael Morris Born, Jannis |
| author_facet | Zausinger, Jonas Pennig, Lars Kozina, Anamarija Sdahl, Sean Sikora, Julian Dendorfer, Adrian Kuznetsov, Timofey Hagog, Mohamad Wiedemann, Nina Chlodny, Kacper Limbach, Vincent Ketteler, Anna Prein, Thorben Singh, Vishwa Mohan Danziger, Michael Morris Born, Jannis |
| contents | While language models have exceptional capabilities at text generation, they lack a natural inductive bias for emitting numbers and thus struggle in tasks involving quantitative reasoning, especially arithmetic. One fundamental limitation is the nature of the cross-entropy (CE) loss, which assumes a nominal scale and thus cannot convey proximity between generated number tokens. In response, we here present a regression-like loss that operates purely on token level. Our proposed Number Token Loss (NTL) comes in two flavors and minimizes either the $L_p$ norm or the Wasserstein distance between the numerical values of the real and predicted number tokens. NTL can easily be added to any language model and extend the CE objective during training without runtime overhead. We evaluate the proposed scheme on various mathematical datasets and find that it consistently improves performance in math-related tasks. In a direct comparison on a regression task, we find that NTL can match the performance of a regression head, despite operating on token level. Finally, we scale NTL up to 3B parameter models and observe improved performance, demonstrating its potential for seamless integration into LLMs. We hope to inspire LLM developers to improve their pretraining objectives and distribute NTL as a minimalistic and lightweight PyPI package $ntloss$: https://github.com/ai4sd/number-token-loss. Development code for full paper reproduction is available separately. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02083 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models Zausinger, Jonas Pennig, Lars Kozina, Anamarija Sdahl, Sean Sikora, Julian Dendorfer, Adrian Kuznetsov, Timofey Hagog, Mohamad Wiedemann, Nina Chlodny, Kacper Limbach, Vincent Ketteler, Anna Prein, Thorben Singh, Vishwa Mohan Danziger, Michael Morris Born, Jannis Computation and Language Artificial Intelligence Computational Engineering, Finance, and Science Machine Learning While language models have exceptional capabilities at text generation, they lack a natural inductive bias for emitting numbers and thus struggle in tasks involving quantitative reasoning, especially arithmetic. One fundamental limitation is the nature of the cross-entropy (CE) loss, which assumes a nominal scale and thus cannot convey proximity between generated number tokens. In response, we here present a regression-like loss that operates purely on token level. Our proposed Number Token Loss (NTL) comes in two flavors and minimizes either the $L_p$ norm or the Wasserstein distance between the numerical values of the real and predicted number tokens. NTL can easily be added to any language model and extend the CE objective during training without runtime overhead. We evaluate the proposed scheme on various mathematical datasets and find that it consistently improves performance in math-related tasks. In a direct comparison on a regression task, we find that NTL can match the performance of a regression head, despite operating on token level. Finally, we scale NTL up to 3B parameter models and observe improved performance, demonstrating its potential for seamless integration into LLMs. We hope to inspire LLM developers to improve their pretraining objectives and distribute NTL as a minimalistic and lightweight PyPI package $ntloss$: https://github.com/ai4sd/number-token-loss. Development code for full paper reproduction is available separately. |
| title | Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Models |
| topic | Computation and Language Artificial Intelligence Computational Engineering, Finance, and Science Machine Learning |
| url | https://arxiv.org/abs/2411.02083 |