Optimized Speculative Sampling for GPU Hardware Accelerators
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909333457993728 |
|---|---|
| author | Wagner, Dominik Lee, Seanie Baumann, Ilja Seeberger, Philipp Riedhammer, Korbinian Bocklet, Tobias |
| author_facet | Wagner, Dominik Lee, Seanie Baumann, Ilja Seeberger, Philipp Riedhammer, Korbinian Bocklet, Tobias |
| contents | In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11016 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimized Speculative Sampling for GPU Hardware Accelerators Wagner, Dominik Lee, Seanie Baumann, Ilja Seeberger, Philipp Riedhammer, Korbinian Bocklet, Tobias Machine Learning Computation and Language In this work, we optimize speculative sampling for parallel hardware accelerators to improve sampling speed. We notice that substantial portions of the intermediate matrices necessary for speculative sampling can be computed concurrently. This allows us to distribute the workload across multiple GPU threads, enabling simultaneous operations on matrix segments within thread blocks. This results in profiling time improvements ranging from 6% to 13% relative to the baseline implementation, without compromising accuracy. To further accelerate speculative sampling, probability distributions parameterized by softmax are approximated by sigmoid. This approximation approach results in significantly greater relative improvements in profiling time, ranging from 37% to 94%, with a minor decline in accuracy. We conduct extensive experiments on both automatic speech recognition and summarization tasks to validate the effectiveness of our optimization methods. |
| title | Optimized Speculative Sampling for GPU Hardware Accelerators |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2406.11016 |