Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.08500 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912702267392000 |
|---|---|
| author | Kapusuzoglu, Berkcan Chakraborty, Supriyo Ni, Renkun Rawls, Stephen Sahu, Sambit |
| author_facet | Kapusuzoglu, Berkcan Chakraborty, Supriyo Ni, Renkun Rawls, Stephen Sahu, Sambit |
| contents | Large language models (LLMs) adapted to financial domains often suffer from catastrophic forgetting of general reasoning capabilities essential for customer interactions and complex financial analysis. We introduce Selective Parameter Evaluation and Restoration via Model Merging (SPEAR-MM), a practical framework that preserves critical capabilities while enabling domain adaptation. Our method approximates layer-wise impact on external benchmarks through post-hoc analysis, then selectively freezes or restores transformer layers via spherical interpolation merging. Applied to LLaMA-3.1-8B for financial tasks, SPEAR-MM achieves 91.2% retention of general capabilities versus 69.7% for standard continual pretraining, while maintaining 94% of domain adaptation gains. The approach provides interpretable trade-off control and reduces computational costs by 90% crucial for resource-constrained financial institutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08500 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation Kapusuzoglu, Berkcan Chakraborty, Supriyo Ni, Renkun Rawls, Stephen Sahu, Sambit Computation and Language Artificial Intelligence Machine Learning Spectral Theory Large language models (LLMs) adapted to financial domains often suffer from catastrophic forgetting of general reasoning capabilities essential for customer interactions and complex financial analysis. We introduce Selective Parameter Evaluation and Restoration via Model Merging (SPEAR-MM), a practical framework that preserves critical capabilities while enabling domain adaptation. Our method approximates layer-wise impact on external benchmarks through post-hoc analysis, then selectively freezes or restores transformer layers via spherical interpolation merging. Applied to LLaMA-3.1-8B for financial tasks, SPEAR-MM achieves 91.2% retention of general capabilities versus 69.7% for standard continual pretraining, while maintaining 94% of domain adaptation gains. The approach provides interpretable trade-off control and reduces computational costs by 90% crucial for resource-constrained financial institutions. |
| title | SPEAR-MM: Selective Parameter Evaluation and Restoration via Model Merging for Efficient Financial LLM Adaptation |
| topic | Computation and Language Artificial Intelligence Machine Learning Spectral Theory |
| url | https://arxiv.org/abs/2511.08500 |