Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2511.00874 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917055137054720 |
|---|---|
| author | Liu, Taowen Andronic, Marta Gündüz, Deniz Constantinides, George A. |
| author_facet | Liu, Taowen Andronic, Marta Gündüz, Deniz Constantinides, George A. |
| contents | LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_00874 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding Liu, Taowen Andronic, Marta Gündüz, Deniz Constantinides, George A. Machine Learning Numerical Analysis LLM training is resource-intensive. Quantized training improves computational and memory efficiency but introduces quantization noise, which can hinder convergence and degrade model accuracy. Stochastic Rounding (SR) has emerged as a theoretically attractive alternative to deterministic rounding, offering unbiased gradient estimates. However, its interaction with other training factors -- especially batch size -- remains under explored. In this paper, we present a theoretical and empirical study of mini-batch stochastic gradient descent (SGD) with SR, showing that increased batch sizes can compensate for reduced precision during back-propagation. Furthermore, we show that quantizing weights and activations impacts gradient variance in distinct ways. Our experiments validate these theoretical insights. |
| title | Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2511.00874 |