Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916907514331136 |
|---|---|
| author | Chen, Xingwu Lu, Miao Wu, Beining Zou, Difan |
| author_facet | Chen, Xingwu Lu, Miao Wu, Beining Zou, Difan |
| contents | Using more test-time computation during language model inference, such as generating more intermediate thoughts or sampling multiple candidate answers, has proven effective in significantly improving model performance. This paper takes an initial step toward bridging the gap between practical language model inference and theoretical transformer analysis by incorporating randomness and sampling. We focus on in-context linear regression with continuous/binary coefficients, where our framework simulates language model decoding through noise injection and binary coefficient sampling. Through this framework, we provide detailed analyses of widely adopted inference techniques. Supported by empirical results, our theoretical framework and analysis demonstrate the potential for offering new insights into understanding inference behaviors in real-world language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07571 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression Chen, Xingwu Lu, Miao Wu, Beining Zou, Difan Machine Learning Artificial Intelligence Using more test-time computation during language model inference, such as generating more intermediate thoughts or sampling multiple candidate answers, has proven effective in significantly improving model performance. This paper takes an initial step toward bridging the gap between practical language model inference and theoretical transformer analysis by incorporating randomness and sampling. We focus on in-context linear regression with continuous/binary coefficients, where our framework simulates language model decoding through noise injection and binary coefficient sampling. Through this framework, we provide detailed analyses of widely adopted inference techniques. Supported by empirical results, our theoretical framework and analysis demonstrate the potential for offering new insights into understanding inference behaviors in real-world language models. |
| title | Towards Theoretical Understanding of Transformer Test-Time Computing: Investigation on In-Context Linear Regression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.07571 |