Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10251 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912273523539968 |
|---|---|
| author | Budzinskiy, Stanislav Fang, Wenyi Zeng, Longbin Petersen, Philipp |
| author_facet | Budzinskiy, Stanislav Fang, Wenyi Zeng, Longbin Petersen, Philipp |
| contents | Large language models based on transformer architectures have become integral to state-of-the-art natural language processing applications. However, their training remains computationally expensive and exhibits instabilities, some of which are expected to be caused by finite-precision computations. We provide a theoretical analysis of the impact of round-off errors within the forward pass of a transformer architecture which yields fundamental bounds for these effects. In addition, we conduct a series of numerical experiments which demonstrate the practical relevance of our bounds. Our results yield concrete guidelines for choosing hyperparameters that mitigate round-off errors, leading to more robust and stable inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10251 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Numerical Error Analysis of Large Language Models Budzinskiy, Stanislav Fang, Wenyi Zeng, Longbin Petersen, Philipp Numerical Analysis Machine Learning Large language models based on transformer architectures have become integral to state-of-the-art natural language processing applications. However, their training remains computationally expensive and exhibits instabilities, some of which are expected to be caused by finite-precision computations. We provide a theoretical analysis of the impact of round-off errors within the forward pass of a transformer architecture which yields fundamental bounds for these effects. In addition, we conduct a series of numerical experiments which demonstrate the practical relevance of our bounds. Our results yield concrete guidelines for choosing hyperparameters that mitigate round-off errors, leading to more robust and stable inference. |
| title | Numerical Error Analysis of Large Language Models |
| topic | Numerical Analysis Machine Learning |
| url | https://arxiv.org/abs/2503.10251 |