Copertina

Optimal Splitting of Language Models from Mixtures to Specialized Domains

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Seto, Skyler, Ablin, Pierre, Filippova, Anastasiia, Ye, Jiayuan, Bethune, Louis, Katharopoulos, Angelos, Grangier, David
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
di: Ablin, Pierre, et al.
Pubblicazione: (2025)

No Need to Talk: Asynchronous Mixture of Language Models
di: Filippova, Anastasiia, et al.
Pubblicazione: (2024)

Need a Small Specialized Language Model? Plan Early!
di: Grangier, David, et al.
Pubblicazione: (2024)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
di: Grangier, David, et al.
Pubblicazione: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
di: Sedova, Anastasiia, et al.
Pubblicazione: (2026)

Partial Parameter Updates for Efficient Distributed Training
di: Filippova, Anastasiia, et al.
Pubblicazione: (2025)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
di: Bethune, Louis, et al.
Pubblicazione: (2025)

The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
di: Saada, Thiziri Nait, et al.
Pubblicazione: (2025)

Dynamic Gradient Alignment for Online Data Mixing
di: Fan, Simin, et al.
Pubblicazione: (2024)

Scaling Laws for Optimal Data Mixtures
di: Shukor, Mustafa, et al.
Pubblicazione: (2025)

Training Bilingual LMs with Data Constraints in the Targeted Language
di: Seto, Skyler, et al.
Pubblicazione: (2024)

Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
di: Jeha, Paul, et al.
Pubblicazione: (2026)

Compute-Optimal Quantization-Aware Training
di: Dremov, Aleksandr, et al.
Pubblicazione: (2025)

Scaling Categorical Flow Maps
di: Davis, Oscar, et al.
Pubblicazione: (2026)

The Geometries of Truth Are Orthogonal Across Tasks
di: Azizian, Waiss, et al.
Pubblicazione: (2025)

Assessing the Role of Data Quality in Training Bilingual Language Models
di: Seto, Skyler, et al.
Pubblicazione: (2025)

The AdEMAMix Optimizer: Better, Faster, Older
di: Pagliardini, Matteo, et al.
Pubblicazione: (2024)

Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing
di: Filippova, Anastasiia, et al.
Pubblicazione: (2026)

Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
di: Huang, Chen, et al.
Pubblicazione: (2024)

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
di: Maini, Pratyush, et al.
Pubblicazione: (2024)

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures
di: Gualdoni, Eleonora, et al.
Pubblicazione: (2026)

Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models
di: Sundar, Anirudh, et al.
Pubblicazione: (2025)

Nectar: Neural Estimation of Cached-Token Attention via Regression
di: Monteiro, João, et al.
Pubblicazione: (2026)

Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
di: Kirchhof, Michael, et al.
Pubblicazione: (2024)

TaCo: Targeted Concept Erasure Prevents Non-Linear Classifiers From Detecting Protected Attributes
di: Jourdan, Fanny, et al.
Pubblicazione: (2023)

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
di: Lin, Yong, et al.
Pubblicazione: (2024)

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
di: Qiu, Zihan, et al.
Pubblicazione: (2025)

Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts
di: Ye, Jiayuan, et al.
Pubblicazione: (2026)

Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions
di: Sedova, Anastasiia, et al.
Pubblicazione: (2026)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
di: Dong, Zican, et al.
Pubblicazione: (2025)

Pretraining with hierarchical memories: separating long-tail and common knowledge
di: Pouransari, Hadi, et al.
Pubblicazione: (2025)

Bayesian Mixture of Experts For Large Language Models
di: Dialameh, Maryam, et al.
Pubblicazione: (2025)

Mixture-of-Personas Language Models for Population Simulation
di: Bui, Ngoc, et al.
Pubblicazione: (2025)

To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
di: Sedova, Anastasiia, et al.
Pubblicazione: (2024)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
di: Li, Margaret, et al.
Pubblicazione: (2026)

Domain2Vec: Vectorizing Datasets to Find the Optimal Data Mixture without Training
di: Zhang, Mozhi, et al.
Pubblicazione: (2025)

NurseLLM: The First Specialized Language Model for Nursing
di: Khondaker, Md Tawkat Islam, et al.
Pubblicazione: (2025)

A Survey on Mixture of Experts in Large Language Models
di: Cai, Weilin, et al.
Pubblicazione: (2024)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
di: Liew, Seng Pei, et al.
Pubblicazione: (2025)