Image de couverture de livre

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Ablin, Pierre, Katharopoulos, Angelos, Seto, Skyler, Grangier, David
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
par: Grangier, David, et autres
Publié: (2024)

Optimal Splitting of Language Models from Mixtures to Specialized Domains
par: Seto, Skyler, et autres
Publié: (2026)

Need a Small Specialized Language Model? Plan Early!
par: Grangier, David, et autres
Publié: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
par: Sedova, Anastasiia, et autres
Publié: (2026)

No Need to Talk: Asynchronous Mixture of Language Models
par: Filippova, Anastasiia, et autres
Publié: (2024)

Dynamic Gradient Alignment for Online Data Mixing
par: Fan, Simin, et autres
Publié: (2024)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
par: Saada, Thiziri Nait, et autres
Publié: (2025)

Partial Parameter Updates for Efficient Distributed Training
par: Filippova, Anastasiia, et autres
Publié: (2025)

Training Bilingual LMs with Data Constraints in the Targeted Language
par: Seto, Skyler, et autres
Publié: (2024)

Compute-Optimal Quantization-Aware Training
par: Dremov, Aleksandr, et autres
Publié: (2025)

The AdEMAMix Optimizer: Better, Faster, Older
par: Pagliardini, Matteo, et autres
Publié: (2024)

Pretraining with hierarchical memories: separating long-tail and common knowledge
par: Pouransari, Hadi, et autres
Publié: (2025)

Assessing the Role of Data Quality in Training Bilingual Language Models
par: Seto, Skyler, et autres
Publié: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
par: Monteiro, João, et autres
Publié: (2026)

Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
par: Huang, Chen, et autres
Publié: (2024)

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
par: Maini, Pratyush, et autres
Publié: (2024)

Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation
par: Xie, Guofu, et autres
Publié: (2025)

Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models
par: Sundar, Anirudh, et autres
Publié: (2025)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
par: Zheng, Chen, et autres
Publié: (2025)

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
par: Lin, Yong, et autres
Publié: (2024)

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks
par: Prabhakar, Akshara, et autres
Publié: (2024)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
par: Liu, Zehua, et autres
Publié: (2025)

Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
par: Wael, Abdalrahman
Publié: (2026)

Scaling Up Temporal Domain Generalization via Temporal Experts Averaging
par: Liu, Aoming, et autres
Publié: (2025)

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
par: Jeha, Paul, et autres
Publié: (2026)

On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists
par: Fan, Dongyang, et autres
Publié: (2024)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
par: Sakamoto, Keitaro, et autres
Publié: (2026)

Efficient Stagewise Pretraining via Progressive Subnetworks
par: Panigrahi, Abhishek, et autres
Publié: (2024)

The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
par: Jin, Tian, et autres
Publié: (2025)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
par: Hui, Tingfeng, et autres
Publié: (2024)

The Geometries of Truth Are Orthogonal Across Tasks
par: Azizian, Waiss, et autres
Publié: (2025)

Table-LLM-Specialist: Language Model Specialists for Tables using Iterative Generator-Validator Fine-tuning
par: Xing, Junjie, et autres
Publié: (2024)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
par: Lv, Ang, et autres
Publié: (2025)

CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
par: Feng, Jinyuan, et autres
Publié: (2025)

Factual Consistency of Multilingual Pretrained Language Models
par: Fierro, Constanza, et autres
Publié: (2022)

Language Models Improve When Pretraining Data Matches Target Tasks
par: Mizrahi, David, et autres
Publié: (2025)

BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models
par: Chang, Aofei, et autres
Publié: (2024)

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
par: Wang, Haoxiang, et autres
Publié: (2024)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
par: Wang, Zihan, et autres
Publié: (2025)