Energy Consumption in Parallel Neural Network Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911101225009152 |
|---|---|
| author | Huber, Philipp Li, David Muriedas, Juan Pedro Gutiérrez Hermosillo Kieckhefen, Deifilia Götz, Markus Streit, Achim Debus, Charlotte |
| author_facet | Huber, Philipp Li, David Muriedas, Juan Pedro Gutiérrez Hermosillo Kieckhefen, Deifilia Götz, Markus Streit, Achim Debus, Charlotte |
| contents | The increasing demand for computational resources of training neural networks leads to a concerning growth in energy consumption. While parallelization has enabled upscaling model and dataset sizes and accelerated training, its impact on energy consumption is often overlooked. To close this research gap, we conducted scaling experiments for data-parallel training of two models, ResNet50 and FourCastNet, and evaluated the impact of parallelization parameters, i.e., GPU count, global batch size, and local batch size, on predictive performance, training time, and energy consumption. We show that energy consumption scales approximately linearly with the consumed resources, i.e., GPU hours; however, the respective scaling factor differs substantially between distinct model trainings and hardware, and is systematically influenced by the number of samples and gradient updates per GPU hour. Our results shed light on the complex interplay of scaling up neural network training and can inform future developments towards more sustainable AI research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07706 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Energy Consumption in Parallel Neural Network Training Huber, Philipp Li, David Muriedas, Juan Pedro Gutiérrez Hermosillo Kieckhefen, Deifilia Götz, Markus Streit, Achim Debus, Charlotte Machine Learning Artificial Intelligence The increasing demand for computational resources of training neural networks leads to a concerning growth in energy consumption. While parallelization has enabled upscaling model and dataset sizes and accelerated training, its impact on energy consumption is often overlooked. To close this research gap, we conducted scaling experiments for data-parallel training of two models, ResNet50 and FourCastNet, and evaluated the impact of parallelization parameters, i.e., GPU count, global batch size, and local batch size, on predictive performance, training time, and energy consumption. We show that energy consumption scales approximately linearly with the consumed resources, i.e., GPU hours; however, the respective scaling factor differs substantially between distinct model trainings and hardware, and is systematically influenced by the number of samples and gradient updates per GPU hour. Our results shed light on the complex interplay of scaling up neural network training and can inform future developments towards more sustainable AI research. |
| title | Energy Consumption in Parallel Neural Network Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.07706 |