Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.22690 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908799772655616 |
|---|---|
| author | Liu, Huanyu Li, Ge Dong, Yihong Wu, Sihan Wang, Peixu Cheng, Sihao Chen, Taozhi Zhang, Kechi Zhu, Hao Liu, Tongxuan |
| author_facet | Liu, Huanyu Li, Ge Dong, Yihong Wu, Sihan Wang, Peixu Cheng, Sihao Chen, Taozhi Zhang, Kechi Zhu, Hao Liu, Tongxuan |
| contents | Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22690 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do Transformers Have the Ability for Periodicity Generalization? Liu, Huanyu Li, Ge Dong, Yihong Wu, Sihan Wang, Peixu Cheng, Sihao Chen, Taozhi Zhang, Kechi Zhu, Hao Liu, Tongxuan Machine Learning Artificial Intelligence Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research. |
| title | Do Transformers Have the Ability for Periodicity Generalization? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.22690 |