Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914570750132224 |
|---|---|
| author | Riabinin, Artem Veprikov, Andrey Bolatov, Arman Takáč, Martin Beznosikov, Aleksandr |
| author_facet | Riabinin, Artem Veprikov, Andrey Bolatov, Arman Takáč, Martin Beznosikov, Aleksandr |
| contents | We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decreases with the suboptimality gap and empirically verify that this behavior holds along optimization trajectories. Under this assumption, we establish convergence guarantees under an appropriate choice of learning rate, for which warm-up followed by decay arises naturally from the proof rather than being imposed heuristically.
Building on this theory, we develop a practical learning rate scheduler that relies only on standard hyperparameters and adapts the warm-up duration automatically at the beginning of training. We evaluate this method on large language model pretraining with LLaMA architectures and show that our adaptive warm-up selection consistently outperforms or at least matches the best manually tuned warm-up schedules across all considered setups, without additional hyperparameter search. Our source code is available at https://github.com/brain-lab-research/llm-baselines/tree/warmup |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05813 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers Riabinin, Artem Veprikov, Andrey Bolatov, Arman Takáč, Martin Beznosikov, Aleksandr Machine Learning Optimization and Control We study adaptive learning rate scheduling for norm-constrained optimizers (e.g., Muon and Lion). We introduce a generalized smoothness assumption under which local curvature decreases with the suboptimality gap and empirically verify that this behavior holds along optimization trajectories. Under this assumption, we establish convergence guarantees under an appropriate choice of learning rate, for which warm-up followed by decay arises naturally from the proof rather than being imposed heuristically. Building on this theory, we develop a practical learning rate scheduler that relies only on standard hyperparameters and adapts the warm-up duration automatically at the beginning of training. We evaluate this method on large language model pretraining with LLaMA architectures and show that our adaptive warm-up selection consistently outperforms or at least matches the best manually tuned warm-up schedules across all considered setups, without additional hyperparameter search. Our source code is available at https://github.com/brain-lab-research/llm-baselines/tree/warmup |
| title | Where Does Warm-Up Come From? Adaptive Scheduling for Norm-Constrained Optimizers |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2602.05813 |