Image de couverture de livre

$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Thérien, Benjamin, Joseph, Charles-Étienne, Knyazev, Boris, Oyallon, Edouard, Rish, Irina, Belilovsky, Eugene
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Meta-learning Optimizers for Communication-Efficient Learning
par: Joseph, Charles-Étienne, et autres
Publié: (2023)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
par: Thérien, Benjamin, et autres
Publié: (2025)

Stabilizing Native Low-Rank LLM Pretraining
par: Janson, Paul, et autres
Publié: (2026)

Celo: Training Versatile Learned Optimizers on a Compute Diet
par: Moudgil, Abhinav, et autres
Publié: (2025)

Celo2: Towards Learned Optimization Free Lunch
par: Moudgil, Abhinav, et autres
Publié: (2026)

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
par: Singh, Vaibhav, et autres
Publié: (2025)

PyLO: Towards Accessible Learned Optimizers in PyTorch
par: Janson, Paul, et autres
Publié: (2025)

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
par: Legate, Gwen, et autres
Publié: (2025)

Communication Efficient LLM Pre-training with SparseLoCo
par: Sarfi, Amir, et autres
Publié: (2025)

Model Parallelism With Subnetwork Data Parallelism
par: Singh, Vaibhav, et autres
Publié: (2025)

Continual Pre-training of MoEs: How robust is your router?
par: Thérien, Benjamin, et autres
Publié: (2025)

PETRA: Parallel End-to-end Training with Reversible Architectures
par: Rivaud, Stéphane, et autres
Publié: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
par: Ibrahim, Adam, et autres
Publié: (2024)

WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
par: Fournier, Louis, et autres
Publié: (2024)

ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
par: Nabli, Adel, et autres
Publié: (2024)

Accelerating Training with Neuron Interaction and Nowcasting Networks
par: Knyazev, Boris, et autres
Publié: (2024)

Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks
par: Fournier, Louis, et autres
Publié: (2024)

Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
par: Miahi, Erfan, et autres
Publié: (2026)

Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation
par: Bakong, Killian, et autres
Publié: (2026)

FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groups
par: Nanfack, Geraldin, et autres
Publié: (2025)

DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic prediction
par: Morel, Rudy, et autres
Publié: (2025)

Test-time Generalization for Physics through Neural Operator Splitting
par: Serrano, Louis, et autres
Publié: (2026)

Efficient Refusal Ablation in LLM through Optimal Transport
par: Nanfack, Geraldin, et autres
Publié: (2026)

Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
par: Singh, Vaibhav, et autres
Publié: (2024)

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
par: Davari, MohammadReza, et autres
Publié: (2023)

On Non-Linear operators for Geometric Deep Learning
par: Sergeant-Perthuis, Grégoire, et autres
Publié: (2022)

Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
par: Lidin, Joel, et autres
Publié: (2026)

Rethinking Prompt Optimization: Reinforcement, Diversification, and Migration in Blackbox LLMs
par: Davari, MohammadReza, et autres
Publié: (2025)

When Data Falls Short: Grokking Below the Critical Threshold
par: Singh, Vaibhav, et autres
Publié: (2025)

Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
par: Abbes, Istabrak, et autres
Publié: (2025)

Knowledge Distillation for Federated Learning: a Practical Guide
par: Mora, Alessio, et autres
Publié: (2022)

Generating $π$-Functional Molecules Using STGG+ with Active Learning
par: Jolicoeur-Martineau, Alexia, et autres
Publié: (2025)

Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference
par: Riemer, Matthew, et autres
Publié: (2024)

Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
par: Hameed, Humza Wajid, et autres
Publié: (2024)

AdaFisher: Adaptive Second Order Optimization via Fisher Information
par: Gomes, Damien Martins, et autres
Publié: (2024)

DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors
par: Xie, Tianhao, et autres
Publié: (2023)

From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation
par: Nanfack, Geraldin, et autres
Publié: (2024)

Learning to Rank with Variable Result Presentation Lengths
par: Knyazev, Norman, et autres
Publié: (2025)

A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
par: Heuillet, Maxime, et autres
Publié: (2025)

Heterogeneous Low-Bandwidth Pre-Training of LLMs
par: Obeidi, Yazan, et autres
Publié: (2026)