Image de couverture de livre

The AdEMAMix Optimizer: Better, Faster, Older

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Pagliardini, Matteo, Ablin, Pierre, Grangier, David
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
par: Morwani, Depen, et autres
Publié: (2025)

Dynamic Gradient Alignment for Online Data Mixing
par: Fan, Simin, et autres
Publié: (2024)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
par: Grangier, David, et autres
Publié: (2024)

Need a Small Specialized Language Model? Plan Early!
par: Grangier, David, et autres
Publié: (2024)

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
par: Ablin, Pierre, et autres
Publié: (2025)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

Benchmarking Optimizers for Large Language Model Pretraining
par: Semenov, Andrei, et autres
Publié: (2025)

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
par: Saada, Thiziri Nait, et autres
Publié: (2025)

Scaling Laws for Optimal Data Mixtures
par: Shukor, Mustafa, et autres
Publié: (2025)

Optimal Splitting of Language Models from Mixtures to Specialized Domains
par: Seto, Skyler, et autres
Publié: (2026)

CoTFormer: A Chain-of-Thought Driven Architecture with Budget-Adaptive Computation Cost at Inference
par: Mohtashami, Amirkeivan, et autres
Publié: (2023)

DoGE: Domain Reweighting with Generalization Estimation
par: Fan, Simin, et autres
Publié: (2023)

How Smooth Is Attention?
par: Castin, Valérie, et autres
Publié: (2023)

A Primal-Dual Approach to Solving Variational Inequalities with General Constraints
par: Chavdarova, Tatjana, et autres
Publié: (2022)

DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
par: Pagliardini, Matteo, et autres
Publié: (2024)

Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints
par: Ablin, Pierre, et autres
Publié: (2023)

Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
par: Ye, Zhenzhang, et autres
Publié: (2024)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
par: Sakamoto, Keitaro, et autres
Publié: (2026)

MVICAD2: Multi-View Independent Component Analysis with Delays and Dilations
par: Heurtebise, Ambroise, et autres
Publié: (2025)

ANO : Faster is Better in Noisy Landscape
par: Kegreisz, Adrien
Publié: (2025)

FERRET: Private Deep Learning Faster And Better Than DPSGD
par: Zagardo, David
Publié: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
par: Monteiro, João, et autres
Publié: (2026)

Scaling Laws for Mixture Pretraining Under Data Constraints
par: Sedova, Anastasiia, et autres
Publié: (2026)

A framework for bilevel optimization that enables stochastic and global variance reduction algorithms
par: Dagréou, Mathieu, et autres
Publié: (2022)

A Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization
par: Dagréou, Mathieu, et autres
Publié: (2023)

Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence
par: Castin, Valérie, et autres
Publié: (2026)

Optimization without Retraction on the Random Generalized Stiefel Manifold
par: Vary, Simon, et autres
Publié: (2024)

Leveraging the true depth of LLMs
par: González, Ramón Calvo, et autres
Publié: (2025)

Multivector Neurons: Better and Faster O(n)-Equivariant Clifford Graph Neural Networks
par: Liu, Cong, et autres
Publié: (2024)

No Need to Talk: Asynchronous Mixture of Language Models
par: Filippova, Anastasiia, et autres
Publié: (2024)

Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
par: Filippova, Anastasiia, et autres
Publié: (2026)

Partial Parameter Updates for Efficient Distributed Training
par: Filippova, Anastasiia, et autres
Publié: (2025)

Compute-Optimal Quantization-Aware Training
par: Dremov, Aleksandr, et autres
Publié: (2025)

Faster Predictive Coding Networks via Better Initialization
par: Pinchetti, Luca, et autres
Publié: (2026)

Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition
par: Basharin, Artem, et autres
Publié: (2024)

Playing Markov Games Without Observing Payoffs
par: Ablin, Daniel, et autres
Publié: (2025)

FREE: Faster and Better Data-Free Meta-Learning
par: Wei, Yongxian, et autres
Publié: (2024)

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
par: Gualdoni, Eleonora, et autres
Publié: (2026)

Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation
par: Hill, Adrian, et autres
Publié: (2025)

Cheaper, Better, Faster, Stronger: Robust Text-to-SQL without Chain-of-Thought or Fine-Tuning
par: Dönder, Yusuf Denizay, et autres
Publié: (2025)