Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.03802 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911351437262848 |
|---|---|
| author | Zhou, Changhai Zhang, Shiyang Zhou, Yuhua Qiao, Qian Gao, Jun Weng, Shichao Zhang, Weizhong Jin, Cheng |
| author_facet | Zhou, Changhai Zhang, Shiyang Zhou, Yuhua Qiao, Qian Gao, Jun Weng, Shichao Zhang, Weizhong Jin, Cheng |
| contents | Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. While Quantization-Aware Fine-tuning has emerged as a viable solution, existing paradigms typically decouple quantization and adapter optimization. This separation overlooks a fundamental theoretical constraint we identify as the \textit{Fidelity-Plasticity Trade-off}: a layer's capacity to adapt to new tasks (Plasticity) is inherently constrained by the information capacity of its frozen weights (Fidelity). Aggressively quantizing semantically critical layers creates an information bottleneck that no amount of adapter rank can recover, while high precision in robust syntactic layers wastes valuable memory. To address this, we introduce \textbf{QR-Adaptor}, a unified framework that jointly optimizes per-layer quantization bit-width and LoRA rank. By formulating resource allocation as a multi-objective search aligned with the model's linguistic hierarchy, our method systematically liberates memory from redundancy-heavy layers to reinvest in capacity-critical ones. Extensive experiments demonstrate that QR-Adaptor establishes a new Pareto frontier: notably, a model fine-tuned under a strict 4-bit memory budget achieves performance rivaling 16-bit baselines, demonstrating that precise resource alignment is as critical as model size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03802 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies Zhou, Changhai Zhang, Shiyang Zhou, Yuhua Qiao, Qian Gao, Jun Weng, Shichao Zhang, Weizhong Jin, Cheng Machine Learning Artificial Intelligence Deploying and fine-tuning Large Language Models (LLMs) on resource-constrained edge devices requires navigating a strict trade-off between memory footprint and task performance. While Quantization-Aware Fine-tuning has emerged as a viable solution, existing paradigms typically decouple quantization and adapter optimization. This separation overlooks a fundamental theoretical constraint we identify as the \textit{Fidelity-Plasticity Trade-off}: a layer's capacity to adapt to new tasks (Plasticity) is inherently constrained by the information capacity of its frozen weights (Fidelity). Aggressively quantizing semantically critical layers creates an information bottleneck that no amount of adapter rank can recover, while high precision in robust syntactic layers wastes valuable memory. To address this, we introduce \textbf{QR-Adaptor}, a unified framework that jointly optimizes per-layer quantization bit-width and LoRA rank. By formulating resource allocation as a multi-objective search aligned with the model's linguistic hierarchy, our method systematically liberates memory from redundancy-heavy layers to reinvest in capacity-critical ones. Extensive experiments demonstrate that QR-Adaptor establishes a new Pareto frontier: notably, a model fine-tuned under a strict 4-bit memory budget achieves performance rivaling 16-bit baselines, demonstrating that precise resource alignment is as critical as model size. |
| title | Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.03802 |