Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914250015899648 |
|---|---|
| author | Ye, Hua Chen, Siyuan Zhang, Haoliang Luo, Weihao Li, Yanbin Zhang, Xuan |
| author_facet | Ye, Hua Chen, Siyuan Zhang, Haoliang Luo, Weihao Li, Yanbin Zhang, Xuan |
| contents | Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07198 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning Ye, Hua Chen, Siyuan Zhang, Haoliang Luo, Weihao Li, Yanbin Zhang, Xuan Machine Learning I.2.7; I.2.6 Large language models (LLMs) demonstrate impressive generalization abilities, yet adapting them effectively across multiple heterogeneous domains remains challenging due to inter-domain interference. To overcome this challenge, we propose a partition-based multi-stage fine-tuning framework designed to exploit inter-domain synergies while minimizing negative transfer. Our approach strategically partitions domains into subsets (stages) by balancing domain discrepancy, synergy, and model capacity constraints. We theoretically analyze the proposed framework and derive novel generalization bounds that justify our partitioning strategy. Extensive empirical evaluations on various language understanding tasks show that our method consistently outperforms state-of-the-art baselines. |
| title | Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuning |
| topic | Machine Learning I.2.7; I.2.6 |
| url | https://arxiv.org/abs/2511.07198 |