Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2509.01842 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914097723867136 |
|---|---|
| author | Wen, Qifu Zeng, Xi Zhou, Zihan Liu, Shuaijun Hosseinzadeh, Mehdi Su, Ningxin Rawassizadeh, Reza |
| author_facet | Wen, Qifu Zeng, Xi Zhou, Zihan Liu, Shuaijun Hosseinzadeh, Mehdi Su, Ningxin Rawassizadeh, Reza |
| contents | Early stopping monitors global validation loss and halts all parameter updates simultaneously, which is computationally costly for large transformers due to the extended time required for validation inference. We propose \textit{GradES}, a novel gradient-based early stopping approach that operates within transformer components (attention projections and Feed-Forward layer matrices). We found that different components converge at varying rates during fine-tuning for both language and vision-language models. \textit{GradES} tracks the magnitude of gradient changes in backpropagation for these matrices during training. When a projection matrix's magnitude of gradient changes fall below a convergence threshold $τ$, we exclude that projection matrix from further updates individually, eliminating costly validation passes while allowing slow converging matrices to continue learning. \textit{GradES} speeds up training time by 1.57--7.22$\times$ while simultaneously enhancing generalization through early prevention of overfitting, resulting in 1.2\% higher average accuracy in language tasks and 3.88\% on multimodal benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_01842 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stopping Wen, Qifu Zeng, Xi Zhou, Zihan Liu, Shuaijun Hosseinzadeh, Mehdi Su, Ningxin Rawassizadeh, Reza Machine Learning Artificial Intelligence 68T07 I.2; I.2.7; I.4; H.5.1 Early stopping monitors global validation loss and halts all parameter updates simultaneously, which is computationally costly for large transformers due to the extended time required for validation inference. We propose \textit{GradES}, a novel gradient-based early stopping approach that operates within transformer components (attention projections and Feed-Forward layer matrices). We found that different components converge at varying rates during fine-tuning for both language and vision-language models. \textit{GradES} tracks the magnitude of gradient changes in backpropagation for these matrices during training. When a projection matrix's magnitude of gradient changes fall below a convergence threshold $τ$, we exclude that projection matrix from further updates individually, eliminating costly validation passes while allowing slow converging matrices to continue learning. \textit{GradES} speeds up training time by 1.57--7.22$\times$ while simultaneously enhancing generalization through early prevention of overfitting, resulting in 1.2\% higher average accuracy in language tasks and 3.88\% on multimodal benchmarks. |
| title | GradES: Significantly Faster Training in Transformers with Gradient-Based Early Stopping |
| topic | Machine Learning Artificial Intelligence 68T07 I.2; I.2.7; I.4; H.5.1 |
| url | https://arxiv.org/abs/2509.01842 |