PartialFormer: Modeling Part Instead of Whole for Machine Translation

Fuente: arXiv
Guardado en:
Detalles Bibliográficos
Autores principales: Zheng, Tong, Li, Bei, Bao, Huiwen, Wang, Jiale, Shan, Weiqiao, Xiao, Tong, Zhu, Jingbo
Formato: Preprint
Publicado: 2023
Materias:
Acceso en línea:
Etiquetas: Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
_version_ 1866907926919118848
author Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
author_facet Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
contents The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer's capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer.
format Preprint
id arxiv_https___arxiv_org_abs_2310_14921
institution arXiv
publishDate 2023
record_format arxiv
spellingShingle PartialFormer: Modeling Part Instead of Whole for Machine Translation
Zheng, Tong
Li, Bei
Bao, Huiwen
Wang, Jiale
Shan, Weiqiao
Xiao, Tong
Zhu, Jingbo
Computation and Language
Artificial Intelligence
The design choices in Transformer feed-forward neural networks have resulted in significant computational and parameter overhead. In this work, we emphasize the importance of hidden dimensions in designing lightweight FFNs, a factor often overlooked in previous architectures. Guided by this principle, we introduce PartialFormer, a parameter-efficient Transformer architecture utilizing multiple smaller FFNs to reduce parameters and computation while maintaining essential hidden dimensions. These smaller FFNs are integrated into a multi-head attention mechanism for effective collaboration. We also propose a tailored head scaling strategy to enhance PartialFormer's capabilities. Furthermore, we present a residual-like attention calculation to improve depth scaling within PartialFormer. Extensive experiments on 9 translation tasks and 1 abstractive summarization task validate the effectiveness of our PartialFormer approach on machine translation and summarization tasks. Our code would be available at: https://github.com/zhengkid/PartialFormer.
title PartialFormer: Modeling Part Instead of Whole for Machine Translation
topic Computation and Language
Artificial Intelligence
url https://arxiv.org/abs/2310.14921