Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.04268 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911192604213248 |
|---|---|
| author | Algayres, Robin Saint-James, Charles-Éric Luthra, Mahi Shen, Jiayi Lin, Dongyan Benchekroun, Youssef Moritz, Rashel Pino, Juan Dupoux, Emmanuel |
| author_facet | Algayres, Robin Saint-James, Charles-Éric Luthra, Mahi Shen, Jiayi Lin, Dongyan Benchekroun, Youssef Moritz, Rashel Pino, Juan Dupoux, Emmanuel |
| contents | Children learn to speak with a low amount of data and can be taught new words on a few-shot basis, making them particularly data-efficient learners. The BabyLM challenge aims at exploring language model (LM) training in the low-data regime but uses metrics that concentrate on the head of the word distribution. Here, we introduce LongTail-Swap (LT-Swap), a benchmark that focuses on the tail of the distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. LT-Swap is a pretraining corpus-specific test set of acceptable versus unacceptable sentence pairs that isolate semantic and syntactic usage of rare words. Models are evaluated in a zero-shot fashion by computing the average log probabilities over the two members of each pair. We built two such test sets associated with the 10M words and 100M words BabyLM training sets, respectively, and evaluated 16 models from the BabyLM leaderboard. Our results not only highlight the poor performance of language models on rare words but also reveal that performance differences across LM architectures are much more pronounced in the long tail than in the head. This offers new insights into which architectures are better at handling rare word generalization. We've also made the code publicly avail |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04268 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LongTail-Swap: benchmarking language models' abilities on rare words Algayres, Robin Saint-James, Charles-Éric Luthra, Mahi Shen, Jiayi Lin, Dongyan Benchekroun, Youssef Moritz, Rashel Pino, Juan Dupoux, Emmanuel Computation and Language Artificial Intelligence Children learn to speak with a low amount of data and can be taught new words on a few-shot basis, making them particularly data-efficient learners. The BabyLM challenge aims at exploring language model (LM) training in the low-data regime but uses metrics that concentrate on the head of the word distribution. Here, we introduce LongTail-Swap (LT-Swap), a benchmark that focuses on the tail of the distribution, i.e., measures the ability of LMs to learn new words with very little exposure, like infants do. LT-Swap is a pretraining corpus-specific test set of acceptable versus unacceptable sentence pairs that isolate semantic and syntactic usage of rare words. Models are evaluated in a zero-shot fashion by computing the average log probabilities over the two members of each pair. We built two such test sets associated with the 10M words and 100M words BabyLM training sets, respectively, and evaluated 16 models from the BabyLM leaderboard. Our results not only highlight the poor performance of language models on rare words but also reveal that performance differences across LM architectures are much more pronounced in the long tail than in the head. This offers new insights into which architectures are better at handling rare word generalization. We've also made the code publicly avail |
| title | LongTail-Swap: benchmarking language models' abilities on rare words |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.04268 |