Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916484281794560 |
|---|---|
| author | Lu, Bo-Ru Haduong, Nikita Lin, Chien-Yu Cheng, Hao Smith, Noah A. Ostendorf, Mari |
| author_facet | Lu, Bo-Ru Haduong, Nikita Lin, Chien-Yu Cheng, Hao Smith, Noah A. Ostendorf, Mari |
| contents | Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_13112 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks Lu, Bo-Ru Haduong, Nikita Lin, Chien-Yu Cheng, Hao Smith, Noah A. Ostendorf, Mari Computation and Language Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks where multiple outputs are required for a single shared input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency by avoiding duplicate input encoding and increasing the operational intensity (ratio of numbers of arithmetic operation to memory access) of decoding process by sharing the input key-value cache. We achieve computation reduction that roughly scales with the number of subtasks, gaining up to 4.6x speed-up over state-of-the-art models for dialogue state tracking, summarization, and question-answering tasks, with comparable or better performance. |
| title | Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.13112 |