Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908613302288384 |
|---|---|
| author | Dong, Yihong Li, Ge Jiang, Xue Tao, Yongding Zhang, Kechi Zhu, Hao Liu, Huanyu Ding, Jiazheng Li, Jia Deng, Jinliang Mei, Hong |
| author_facet | Dong, Yihong Li, Ge Jiang, Xue Tao, Yongding Zhang, Kechi Zhu, Hao Liu, Huanyu Ding, Jiazheng Li, Jia Deng, Jinliang Mei, Hong |
| contents | Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer as an effective and promising architecture for advancing LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_21309 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling Dong, Yihong Li, Ge Jiang, Xue Tao, Yongding Zhang, Kechi Zhu, Hao Liu, Huanyu Ding, Jiazheng Li, Jia Deng, Jinliang Mei, Hong Computation and Language Artificial Intelligence Machine Learning Periodicity, as one of the most important basic characteristics, lays the foundation for facilitating structured knowledge acquisition and systematic cognitive processes within human learning paradigms. However, the potential flaws of periodicity modeling in Transformer affect the learning efficiency and establishment of underlying principles from data for large language models (LLMs) built upon it. In this paper, we demonstrate that integrating effective periodicity modeling can improve the learning efficiency and performance of LLMs. We introduce FANformer, which adapts Fourier Analysis Network (FAN) into attention mechanism to achieve efficient periodicity modeling, by modifying the feature projection process of attention mechanism. Extensive experimental results on language modeling show that FANformer consistently outperforms Transformer when scaling up model size and training tokens, underscoring its superior learning efficiency. Our pretrained FANformer-1B exhibits marked improvements on downstream tasks compared to open-source LLMs with similar model parameters or training tokens. Moreover, we reveal that FANformer exhibits superior ability to learn and apply rules for reasoning compared to Transformer. The results position FANformer as an effective and promising architecture for advancing LLMs. |
| title | Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.21309 |