PartialFormer: Modeling Part Instead of Whole for Machine Translation

Fuente: arXiv
Saved in:
Bibliographic Details
Main Authors: Zheng, Tong, Li, Bei, Bao, Huiwen, Wang, Jiale, Shan, Weiqiao, Xiao, Tong, Zhu, Jingbo
Format: Preprint
Published: 2023
Subjects:
Online Access:
Tags: Add Tag
No Tags, Be the first to tag this record!
_version_ 1866907926919118848
author Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
author_facet Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
contents The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer's capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer.
format Preprint
id arxiv_https___arxiv_org_abs_2310_14921
institution arXiv
publishDate 2023
record_format arxiv
spellingShingle PartialFormer: Modeling Part Instead of Whole for Machine Translation
Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
Computation and Language
Artificial Intelligence
The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer's capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer.
title PartialFormer: Modeling Part Instead of Whole for Machine Translation
topic Computation and Language
Artificial Intelligence
url https://arxiv.org/abs/2310.14921