Approximately Aligned Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916995176333312 |
|---|---|
| author | Melcer, Daniel Gonugondla, Sujan Perera, Pramuditha Qian, Haifeng Chiang, Wen-Hao Wang, Yanjun Jain, Nihal Garg, Pranav Ma, Xiaofei Deoras, Anoop |
| author_facet | Melcer, Daniel Gonugondla, Sujan Perera, Pramuditha Qian, Haifeng Chiang, Wen-Hao Wang, Yanjun Jain, Nihal Garg, Pranav Ma, Xiaofei Deoras, Anoop |
| contents | It is common to reject undesired outputs of Large Language Models (LLMs); however, current methods to do so require an excessive amount of computation to re-sample after a rejection, or distort the distribution of outputs by constraining the output to highly improbable tokens. We present a method, Approximately Aligned Decoding (AprAD), to balance the distortion of the output distribution with computational efficiency, inspired by algorithms from the speculative decoding literature. AprAD allows for the generation of long sequences of text with difficult-to-satisfy constraints, while amplifying low probability outputs much less compared to existing methods. We show through a series of experiments that the task-specific performance of AprAD is comparable to methods that do not distort the output distribution, while being much more computationally efficient. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01103 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Approximately Aligned Decoding Melcer, Daniel Gonugondla, Sujan Perera, Pramuditha Qian, Haifeng Chiang, Wen-Hao Wang, Yanjun Jain, Nihal Garg, Pranav Ma, Xiaofei Deoras, Anoop Computation and Language Artificial Intelligence It is common to reject undesired outputs of Large Language Models (LLMs); however, current methods to do so require an excessive amount of computation to re-sample after a rejection, or distort the distribution of outputs by constraining the output to highly improbable tokens. We present a method, Approximately Aligned Decoding (AprAD), to balance the distortion of the output distribution with computational efficiency, inspired by algorithms from the speculative decoding literature. AprAD allows for the generation of long sequences of text with difficult-to-satisfy constraints, while amplifying low probability outputs much less compared to existing methods. We show through a series of experiments that the task-specific performance of AprAD is comparable to methods that do not distort the output distribution, while being much more computationally efficient. |
| title | Approximately Aligned Decoding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.01103 |