Code Pretraining Improves Entity Tracking Abilities of Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909214086004736 |
|---|---|
| author | Kim, Najoung Schuster, Sebastian Toshniwal, Shubham |
| author_facet | Kim, Najoung Schuster, Sebastian Toshniwal, Shubham |
| contents | Recent work has provided indirect evidence that pretraining language models on code improves the ability of models to track state changes of discourse entities expressed in natural language. In this work, we systematically test this claim by comparing pairs of language models on their entity tracking performance. Critically, the pairs consist of base models and models trained on top of these base models with additional code data. We extend this analysis to additionally examine the effect of math training, another highly structured data type, and alignment tuning, an important step for enhancing the usability of models. We find clear evidence that models additionally trained on large amounts of code outperform the base models. On the other hand, we find no consistent benefit of additional math training or alignment tuning across various model families. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_21068 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Code Pretraining Improves Entity Tracking Abilities of Language Models Kim, Najoung Schuster, Sebastian Toshniwal, Shubham Computation and Language Artificial Intelligence Recent work has provided indirect evidence that pretraining language models on code improves the ability of models to track state changes of discourse entities expressed in natural language. In this work, we systematically test this claim by comparing pairs of language models on their entity tracking performance. Critically, the pairs consist of base models and models trained on top of these base models with additional code data. We extend this analysis to additionally examine the effect of math training, another highly structured data type, and alignment tuning, an important step for enhancing the usability of models. We find clear evidence that models additionally trained on large amounts of code outperform the base models. On the other hand, we find no consistent benefit of additional math training or alignment tuning across various model families. |
| title | Code Pretraining Improves Entity Tracking Abilities of Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.21068 |