Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.17734 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914002821447680 |
|---|---|
| author | Ikeda, Wataru Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Shibata, Keigo Suzuki, Jun |
| author_facet | Ikeda, Wataru Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Shibata, Keigo Suzuki, Jun |
| contents | This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that, while maintaining the total parameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers. Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch to examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done. Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations for multiple downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_17734 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models Ikeda, Wataru Yano, Kazuki Takahashi, Ryosuke Lee, Jaesung Shibata, Keigo Suzuki, Jun Computation and Language This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that, while maintaining the total parameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers. Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch to examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done. Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations for multiple downstream tasks. |
| title | Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.17734 |