Cover Image

Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ju, Yiming, Ni, Ziyi, Xing, Xingrun, Zeng, Zhixiong, Zhao, hanyu, Fan, Siqi, Zhang, Zheng
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

KLoB: a Benchmark for Assessing Knowledge Locating Methods in Language Models
by: Ju, Yiming, et al.
Published: (2023)

SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
by: Xing, Xingrun, et al.
Published: (2024)

SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation
by: Kapusuzoglu, Berkcan, et al.
Published: (2025)

Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
by: Wang, Ning, et al.
Published: (2024)

Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
by: Lai, Song, et al.
Published: (2025)

Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs
by: Zhang, Zheng, et al.
Published: (2024)

SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
by: Djuhera, Aladin, et al.
Published: (2025)

Optimal Brain Iterative Merging: Mitigating Interference in LLM Merging
by: Wang, Zhixiang, et al.
Published: (2025)

Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuning
by: Pang, Jinlong, et al.
Published: (2025)

InsideOut: Measuring and Mitigating Insider-Outsider Bias in Interview Script Generation
by: Wan, Yixin, et al.
Published: (2025)

Training-free LLM Merging for Multi-task Learning
by: Fu, Zichuan, et al.
Published: (2025)

RoSA: Enhancing Parameter-Efficient Fine-Tuning via RoPE-aware Selective Adaptation in Large Language Models
by: Pan, Dayan, et al.
Published: (2025)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
by: Hui, Tingfeng, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning With Adapters
by: Chen, Keyu, et al.
Published: (2024)

Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation
by: Ma, Xinyu, et al.
Published: (2024)

Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs
by: Ban, Hao, et al.
Published: (2025)

Selection of LLM Fine-Tuning Data based on Orthogonal Rules
by: Li, Xiaomin, et al.
Published: (2024)

CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation
by: Fawi, Muhammad
Published: (2024)

DEFT: Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection
by: Das, Devleena, et al.
Published: (2023)

Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspace
by: Zhang, Jia-Chen, et al.
Published: (2025)

Filter-then-Weight: Online Data Selection and Reweighting for LLM Fine-Tuning
by: Wang, Fangxin, et al.
Published: (2026)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
by: Yang, Yifan, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning for Medical Text Summarization: A Comparative Study of Lora, Prompt Tuning, and Full Fine-Tuning
by: Shernazarov, Ulugbek, et al.
Published: (2026)

TriAdaptLoRA: Brain-Inspired Triangular Adaptive Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
by: Liang, Yao, et al.
Published: (2025)

Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
by: Wang, Tuowei, et al.
Published: (2025)

GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning
by: Ficek, Aleksander, et al.
Published: (2024)

User-Specific Dialogue Generation with User Profile-Aware Pre-Training Model and Parameter-Efficient Fine-Tuning
by: Otsuka, Atsushi, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning for Foundation Models
by: Zhang, Dan, et al.
Published: (2025)

Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels
by: Ye, Junjie, et al.
Published: (2025)

AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories
by: Song, Yifan, et al.
Published: (2024)

DPI: Exploiting Parameter Heterogeneity for Interference-Free Fine-Tuning
by: Liu, Xiaoyu, et al.
Published: (2026)

ShadowPEFT: Shadow Network for Parameter-Efficient Fine-Tuning
by: Li, Xianming, et al.
Published: (2026)

PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization
by: Clarke, Christopher, et al.
Published: (2024)

Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
by: Liu, Deyuan, et al.
Published: (2024)

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
by: Liu, Yong, et al.
Published: (2024)

Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning
by: Guo, Chenyou, et al.
Published: (2026)

DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models
by: Deng, Wenlong, et al.
Published: (2024)

RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation
by: Nikdan, Mahdi, et al.
Published: (2024)

Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts
by: Zhang, Buze, et al.
Published: (2026)

From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning
by: Li, Yafu, et al.
Published: (2025)