OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910440629469184 |
|---|---|
| author | Qiao, Dan Su, Yi Wang, Pinzheng Ye, Jing Xie, Wenjing Zhou, Yuechi Ding, Yuyang Tang, Zecheng Wang, Jikai Ji, Yixin Wang, Yue Guo, Pei Sun, Zechen Zhang, Zikang Li, Juntao Chao, Pingfu Chen, Wenliang Fu, Guohong Zhou, Guodong Zhu, Qiaoming Zhang, Min |
| author_facet | Qiao, Dan Su, Yi Wang, Pinzheng Ye, Jing Xie, Wenjing Zhou, Yuechi Ding, Yuyang Tang, Zecheng Wang, Jikai Ji, Yixin Wang, Yue Guo, Pei Sun, Zechen Zhang, Zikang Li, Juntao Chao, Pingfu Chen, Wenliang Fu, Guohong Zhou, Guodong Zhu, Qiaoming Zhang, Min |
| contents | Large Language Models (LLMs) have played an important role in many fields due to their powerful capabilities.However, their massive number of parameters leads to high deployment requirements and incurs significant inference costs, which impedes their practical applications. Training smaller models is an effective way to address this problem. Therefore, we introduce OpenBA-V2, a 3.4B model derived from multi-stage compression and continual pre-training from the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible training objectives, and techniques such as layer pruning, neural pruning, and vocabulary pruning to achieve a compression rate of 77.3\% with minimal performance loss. OpenBA-V2 demonstrates competitive performance compared to other open-source models of similar size, achieving results close to or on par with the 15B OpenBA model in downstream tasks such as common sense reasoning and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be compressed into smaller ones with minimal performance loss by employing advanced training objectives and data strategies, which may help deploy LLMs in resource-limited scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_05957 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning Qiao, Dan Su, Yi Wang, Pinzheng Ye, Jing Xie, Wenjing Zhou, Yuechi Ding, Yuyang Tang, Zecheng Wang, Jikai Ji, Yixin Wang, Yue Guo, Pei Sun, Zechen Zhang, Zikang Li, Juntao Chao, Pingfu Chen, Wenliang Fu, Guohong Zhou, Guodong Zhu, Qiaoming Zhang, Min Computation and Language Large Language Models (LLMs) have played an important role in many fields due to their powerful capabilities.However, their massive number of parameters leads to high deployment requirements and incurs significant inference costs, which impedes their practical applications. Training smaller models is an effective way to address this problem. Therefore, we introduce OpenBA-V2, a 3.4B model derived from multi-stage compression and continual pre-training from the original 15B OpenBA model. OpenBA-V2 utilizes more data, more flexible training objectives, and techniques such as layer pruning, neural pruning, and vocabulary pruning to achieve a compression rate of 77.3\% with minimal performance loss. OpenBA-V2 demonstrates competitive performance compared to other open-source models of similar size, achieving results close to or on par with the 15B OpenBA model in downstream tasks such as common sense reasoning and Named Entity Recognition (NER). OpenBA-V2 illustrates that LLMs can be compressed into smaller ones with minimal performance loss by employing advanced training objectives and data strategies, which may help deploy LLMs in resource-limited scenarios. |
| title | OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.05957 |