Image de couverture de livre

Meta-learning Optimizers for Communication-Efficient Learning

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Joseph, Charles-Étienne, Thérien, Benjamin, Moudgil, Abhinav, Knyazev, Boris, Belilovsky, Eugene
Format:	Preprint
Publié:	2023
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Celo2: Towards Learned Optimization Free Lunch
par: Moudgil, Abhinav, et autres
Publié: (2026)

$μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers
par: Thérien, Benjamin, et autres
Publié: (2024)

Celo: Training Versatile Learned Optimizers on a Compute Diet
par: Moudgil, Abhinav, et autres
Publié: (2025)

PyLO: Towards Accessible Learned Optimizers in PyTorch
par: Janson, Paul, et autres
Publié: (2025)

Accelerating Training with Neuron Interaction and Nowcasting Networks
par: Knyazev, Boris, et autres
Publié: (2024)

Communication Efficient LLM Pre-training with SparseLoCo
par: Sarfi, Amir, et autres
Publié: (2025)

MuLoCo: Muon is a practical inner optimizer for DiLoCo
par: Thérien, Benjamin, et autres
Publié: (2025)

Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL
par: Miahi, Erfan, et autres
Publié: (2026)

Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training
par: Singh, Vaibhav, et autres
Publié: (2025)

Continual Pre-training of MoEs: How robust is your router?
par: Thérien, Benjamin, et autres
Publié: (2025)

FairDropout: Using Example-Tied Dropout to Enhance Generalization of Minority Groups
par: Nanfack, Geraldin, et autres
Publié: (2025)

Efficient Refusal Ablation in LLM through Optimal Transport
par: Nanfack, Geraldin, et autres
Publié: (2026)

Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
par: Singh, Vaibhav, et autres
Publié: (2024)

Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks
par: Davari, MohammadReza, et autres
Publié: (2023)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
par: Ibrahim, Adam, et autres
Publié: (2024)

Covenant-72B: Pre-Training a 72B LLM with Trustless Peers Over-the-Internet
par: Lidin, Joel, et autres
Publié: (2026)

Rethinking Prompt Optimization: Reinforcement, Diversification, and Migration in Blackbox LLMs
par: Davari, MohammadReza, et autres
Publié: (2025)

Stabilizing Native Low-Rank LLM Pretraining
par: Janson, Paul, et autres
Publié: (2026)

When Data Falls Short: Grokking Below the Critical Threshold
par: Singh, Vaibhav, et autres
Publié: (2025)

Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients
par: Legate, Gwen, et autres
Publié: (2025)

ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
par: Nabli, Adel, et autres
Publié: (2024)

WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
par: Fournier, Louis, et autres
Publié: (2024)

Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweighting
par: Hameed, Humza Wajid, et autres
Publié: (2024)

AdaFisher: Adaptive Second Order Optimization via Fisher Information
par: Gomes, Damien Martins, et autres
Publié: (2024)

DragD3D: Realistic Mesh Editing with Rigidity Control Driven by 2D Diffusion Priors
par: Xie, Tianhao, et autres
Publié: (2023)

From Feature Visualization to Visual Circuits: Effect of Adversarial Model Manipulation
par: Nanfack, Geraldin, et autres
Publié: (2024)

Learning to Rank with Variable Result Presentation Lengths
par: Knyazev, Norman, et autres
Publié: (2025)

Heterogeneous Low-Bandwidth Pre-Training of LLMs
par: Obeidi, Yazan, et autres
Publié: (2026)

LoGAH: Predicting 774-Million-Parameter Transformers using Graph HyperNetworks with 1/100 Parameters
par: Zhou, Xinyu, et autres
Publié: (2024)

Less is More: Undertraining Experts Improves Model Upcycling
par: Horoi, Stefan, et autres
Publié: (2025)

Generating $π$-Functional Molecules Using STGG+ with Active Learning
par: Jolicoeur-Martineau, Alexia, et autres
Publié: (2025)

Non-Uniform Parameter-Wise Model Merging
par: Camacho, Albert Manuel Orozco, et autres
Publié: (2024)

Model Parallelism With Subnetwork Data Parallelism
par: Singh, Vaibhav, et autres
Publié: (2025)

Incentivizing Permissionless Distributed Learning of LLMs
par: Lidin, Joel, et autres
Publié: (2025)

Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacity
par: Huang, Ningyuan, et autres
Publié: (2025)

PETRA: Parallel End-to-end Training with Reversible Architectures
par: Rivaud, Stéphane, et autres
Publié: (2024)

Attention to Mamba: A Recipe for Cross-Architecture Distillation
par: Moudgil, Abhinav, et autres
Publié: (2026)

DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone
par: Singh, Vaibhav, et autres
Publié: (2025)

Test Time Adaptation Using Adaptive Quantile Recalibration
par: Mehrbod, Paria, et autres
Publié: (2025)

Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees
par: Jolicoeur-Martineau, Alexia, et autres
Publié: (2024)