What is the Best Sequence Length for BABYLM?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912664236589056 |
|---|---|
| author | Salhan, Suchir Martinez, Richard Diehl Goriely, Zébulon Buttery, Paula |
| author_facet | Salhan, Suchir Martinez, Richard Diehl Goriely, Zébulon Buttery, Paula |
| contents | Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, many past submissions have defaulted to using much shorter sequence lengths. We examine the impact of sequence length on BabyLM pretraining, to answer the simple question: what sequence length should we be using when training Baby LMs? Using 100M-word training data and fixed compute budgets, we compare 125M-parameter Mamba and OPT models, finding that although longer is often better, the optimal length depends on both task and architecture. Shorter sequences are sufficient for grammatical generalization tasks whereas longer contexts benefit morphological analogical reasoning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19493 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What is the Best Sequence Length for BABYLM? Salhan, Suchir Martinez, Richard Diehl Goriely, Zébulon Buttery, Paula Computation and Language Transformer language models typically operate with a fixed-length context window, which has grown in step with large-scale pretraining datasets. In the BabyLM Challenge, however, many past submissions have defaulted to using much shorter sequence lengths. We examine the impact of sequence length on BabyLM pretraining, to answer the simple question: what sequence length should we be using when training Baby LMs? Using 100M-word training data and fixed compute budgets, we compare 125M-parameter Mamba and OPT models, finding that although longer is often better, the optimal length depends on both task and architecture. Shorter sequences are sufficient for grammatical generalization tasks whereas longer contexts benefit morphological analogical reasoning tasks. |
| title | What is the Best Sequence Length for BABYLM? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.19493 |