Cover Image

Sparse Upcycling: Inference Inefficient Finetuning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Doubov, Sasha, Sardana, Nikhil, Chiley, Vitaliy
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
by: Sardana, Nikhil, et al.
Published: (2023)

MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
by: Teo, Rachel S. Y., et al.
Published: (2025)

Inference-Friendly Models With MixAttention
by: Rajput, Shashank, et al.
Published: (2024)

Improving Sparse Memory Finetuning
by: Goyal, Satyam, et al.
Published: (2026)

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
by: Gupta, Prakhar, et al.
Published: (2026)

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
by: Hayou, Soufiane, et al.
Published: (2025)

The Impact of Initialization on LoRA Finetuning Dynamics
by: Hayou, Soufiane, et al.
Published: (2024)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
by: Nakamura, Taishi, et al.
Published: (2025)

When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method
by: Zhang, Biao, et al.
Published: (2024)

MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining
by: Portes, Jacob, et al.
Published: (2023)

LoRA Learns Less and Forgets Less
by: Biderman, Dan, et al.
Published: (2024)

Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
by: Sarkar, Soumajyoti, et al.
Published: (2024)

Predicting Emergent Capabilities by Finetuning
by: Snell, Charlie, et al.
Published: (2024)

Energy Considerations of Large Language Model Inference and Efficiency Optimizations
by: Fernandez, Jared, et al.
Published: (2025)

The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval
by: Chiang, Ting-Rui, et al.
Published: (2025)

Understanding Finetuning for Factual Knowledge Extraction
by: Ghosal, Gaurav, et al.
Published: (2024)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
by: Liew, Seng Pei, et al.
Published: (2025)

Training-Free Dynamic Upcycling of Expert Language Models
by: Fanì, Eros, et al.
Published: (2026)

Ensembling Finetuned Language Models for Text Classification
by: Arango, Sebastian Pineda, et al.
Published: (2024)

Transfer Learning for Finetuning Large Language Models
by: Strangmann, Tobias, et al.
Published: (2024)

FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
by: Oliaro, Gabriele, et al.
Published: (2024)

Finetuning Language Models to Emit Linguistic Expressions of Uncertainty
by: Chaudhry, Arslan, et al.
Published: (2024)

Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
by: Shi, Taiwei, et al.
Published: (2025)

Order Independence With Finetuning
by: Brown, Katrina, et al.
Published: (2025)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
by: Feng, Jiahai, et al.
Published: (2024)

Representation Noising: A Defence Mechanism Against Harmful Finetuning
by: Rosati, Domenic, et al.
Published: (2024)

KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation
by: Luo, Xindi, et al.
Published: (2024)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
by: Bethune, Louis, et al.
Published: (2025)

Towards Active Synthetic Data Generation for Finetuning Language Models
by: Kessler, Samuel, et al.
Published: (2025)

From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
by: Zhao, Eric, et al.
Published: (2025)

Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection
by: Jo, Dongwon, et al.
Published: (2026)

Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
by: Fashi, Parsa Ashrafi, et al.
Published: (2026)

Learning Dynamics of LLM Finetuning
by: Ren, Yi, et al.
Published: (2024)

ApiQ: Finetuning of 2-Bit Quantized Large Language Model
by: Liao, Baohao, et al.
Published: (2024)

Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
by: Qin, Haotong, et al.
Published: (2024)

CoBa: Convergence Balancer for Multitask Finetuning of Large Language Models
by: Gong, Zi, et al.
Published: (2024)

Internalizing World Models via Self-Play Finetuning for Agentic RL
by: Chen, Shiqi, et al.
Published: (2025)

MuonAll: Muon Variant for Efficient Finetuning of Large Language Models
by: Page, Saurabh, et al.
Published: (2025)

OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning
by: Feng, Jinyuan, et al.
Published: (2025)

Mitigating Catastrophic Forgetting in Mathematical Reasoning Finetuning through Mixed Training
by: Reynolds, John Graham
Published: (2025)