Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.04151 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908302828371968 |
|---|---|
| author | Yano, Kazuki Ito, Takumi Suzuki, Jun |
| author_facet | Yano, Kazuki Ito, Takumi Suzuki, Jun |
| contents | Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP), which integrates parameter-efficient tuning techniques with model growth. We conduct experiments on pre-training LLMs of various sizes and demonstrate that STEP achieves up to a 53.9% reduction in maximum memory requirements compared to vanilla pre-training while maintaining equivalent performance. Furthermore, we show that the model by STEP performs comparably to vanilla pre-trained models on downstream tasks after instruction tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04151 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STEP: Staged Parameter-Efficient Pre-training for Large Language Models Yano, Kazuki Ito, Takumi Suzuki, Jun Computation and Language Pre-training large language models (LLMs) faces significant memory challenges due to the large size of model parameters. We introduce STaged parameter-Efficient Pre-training (STEP), which integrates parameter-efficient tuning techniques with model growth. We conduct experiments on pre-training LLMs of various sizes and demonstrate that STEP achieves up to a 53.9% reduction in maximum memory requirements compared to vanilla pre-training while maintaining equivalent performance. Furthermore, we show that the model by STEP performs comparably to vanilla pre-trained models on downstream tasks after instruction tuning. |
| title | STEP: Staged Parameter-Efficient Pre-training for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.04151 |