LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916150098526208 |
|---|---|
| author | Li, Jialin Nie, Qiang Fu, Weifu Lin, Yuhuan Tao, Guangpin Liu, Yong Wang, Chengjie |
| author_facet | Li, Jialin Nie, Qiang Fu, Weifu Lin, Yuhuan Tao, Guangpin Liu, Yong Wang, Chengjie |
| contents | Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS allows stacked modules to share the majority of parameters, requiring a much smaller number of unique ones per module to match or even surpass the performance of using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it to the stacked decoders of a query-based object detector, and conduct extensive experiments on the widely used MS COCO dataset. Experimental results demonstrate the effectiveness of our method, as even with a 70\% reduction in the parameters of the decoder, our method still enables the model to achieve comparable or |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04303 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking Li, Jialin Nie, Qiang Fu, Weifu Lin, Yuhuan Tao, Guangpin Liu, Yong Wang, Chengjie Computer Vision and Pattern Recognition Deep learning models, particularly those based on transformers, often employ numerous stacked structures, which possess identical architectures and perform similar functions. While effective, this stacking paradigm leads to a substantial increase in the number of parameters, posing challenges for practical applications. In today's landscape of increasingly large models, stacking depth can even reach dozens, further exacerbating this issue. To mitigate this problem, we introduce LORS (LOw-rank Residual Structure). LORS allows stacked modules to share the majority of parameters, requiring a much smaller number of unique ones per module to match or even surpass the performance of using entirely distinct ones, thereby significantly reducing parameter usage. We validate our method by applying it to the stacked decoders of a query-based object detector, and conduct extensive experiments on the widely used MS COCO dataset. Experimental results demonstrate the effectiveness of our method, as even with a 70\% reduction in the parameters of the decoder, our method still enables the model to achieve comparable or |
| title | LORS: Low-rank Residual Structure for Parameter-Efficient Network Stacking |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.04303 |