Cover Image

Sparse is Enough in Fine-tuning Pre-trained Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Song, Weixi, Li, Zuchao, Zhang, Lefei, Zhao, Hai, Du, Bo
Format:	Preprint
Published:	2023
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models
by: Zhang, Zihong, et al.
Published: (2025)

On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
by: Niu, Xueyan, et al.
Published: (2026)

Model Hemorrhage and the Robustness Limits of Large Language Models
by: Ma, Ziyang, et al.
Published: (2025)

Non-instructional Fine-tuning: Enabling Instruction-Following Capabilities in Pre-trained Language Models without Instruction-Following Data
by: Xie, Juncheng, et al.
Published: (2024)

From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
by: Ma, Xiangyu, et al.
Published: (2026)

LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
by: Chen, Yukang, et al.
Published: (2023)

Fine-tuning can Help Detect Pretraining Data from Large Language Models
by: Zhang, Hengxiang, et al.
Published: (2024)

Machine Unlearning of Pre-trained Large Language Models
by: Yao, Jin, et al.
Published: (2024)

Scaling Sparse Fine-Tuning to Large Language Models
by: Ansell, Alan, et al.
Published: (2024)

Selecting Large Language Model to Fine-tune via Rectified Scaling Law
by: Lin, Haowei, et al.
Published: (2024)

Exploring Memorization in Fine-tuned Language Models
by: Zeng, Shenglai, et al.
Published: (2023)

Fine-tuning Language Models with Generative Adversarial Reward Modelling
by: Yu, Zhang Ze, et al.
Published: (2023)

How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition
by: Dong, Guanting, et al.
Published: (2023)

InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification
by: Hu, Yujia, et al.
Published: (2024)

Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models
by: Chen, Pin-Yu, et al.
Published: (2025)

PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuning
by: Zou, Jiaru, et al.
Published: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
by: Ibrahim, Adam, et al.
Published: (2024)

Improving Low-Resource Knowledge Tracing Tasks by Supervised Pre-training and Importance Mechanism Fine-tuning
by: Zhang, Hengyuan, et al.
Published: (2024)

Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide
by: Szep, Marton, et al.
Published: (2024)

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
by: Zhang, Zihong, et al.
Published: (2026)

Information Guided Regularization for Fine-tuning Language Models
by: Sharma, Mandar, et al.
Published: (2024)

Harnessing Large Language Models: Fine-tuned BERT for Detecting Charismatic Leadership Tactics in Natural Language
by: Saeid, Yasser, et al.
Published: (2024)

SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models
by: Tang, Anke, et al.
Published: (2024)

Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning
by: Song, Haobo, et al.
Published: (2024)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
by: Samragh, Mohammad, et al.
Published: (2024)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
by: Tang, Shengkun, et al.
Published: (2026)

Efficient End-to-end Language Model Fine-tuning on Graphs
by: Xue, Rui, et al.
Published: (2023)

LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Models
by: Li, Qianxi, et al.
Published: (2023)

A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding
by: Shen, Yiqing, et al.
Published: (2024)

The Impact of Fine-tuning Large Language Models on Automated Program Repair
by: Macháček, Roman, et al.
Published: (2025)

A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
by: Shu, Dong, et al.
Published: (2025)

Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning
by: Wang, Shuhe, et al.
Published: (2024)

Fine-Grained Interpretation of Political Opinions in Large Language Models
by: Hu, Jingyu, et al.
Published: (2025)

Sequence-to-Sequence Spanish Pre-trained Language Models
by: Araujo, Vladimir, et al.
Published: (2023)

FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion
by: Tang, Anke, et al.
Published: (2024)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)

Aligning Pre-trained Models for Spoken Language Translation
by: Sedláček, Šimon, et al.
Published: (2024)

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
by: Yang, Yu, et al.
Published: (2024)

APE: Selective Fine-tuning with Acceptance Criteria for Language Model Adaptation
by: Marín, Javier
Published: (2025)

Pre-training Limited Memory Language Models with Internal and External Knowledge
by: Zhao, Linxi, et al.
Published: (2025)