Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.23924 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912301775323136 |
|---|---|
| author | Ma, Ziyang Li, Zuchao Zhang, Lefei Xia, Gui-Song Du, Bo Zhang, Liangpei Tao, Dacheng |
| author_facet | Ma, Ziyang Li, Zuchao Zhang, Lefei Xia, Gui-Song Du, Bo Zhang, Liangpei Tao, Dacheng |
| contents | Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment through quantization, pruning, or decoding strategy adjustments. We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes. Through systematic analysis of various LLM frameworks, we identify key vulnerability patterns: layer expansion frequently disrupts attention mechanisms, compression techniques induce information loss cascades, and decoding adjustments amplify prediction divergences. Our investigation reveals transformer architectures exhibit inherent robustness thresholds that determine hemorrhage severity across modification types. We propose three mitigation strategies: gradient-aware pruning preserves critical weight pathways, dynamic quantization scaling maintains activation integrity, and decoding calibration aligns generation trajectories with original model distributions. This work establishes foundational metrics for evaluating model stability during adaptation, providing practical guidelines for maintaining performance while enabling efficient LLM deployment. Our findings advance understanding of neural network resilience under architectural transformations, particularly for large-scale language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23924 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Model Hemorrhage and the Robustness Limits of Large Language Models Ma, Ziyang Li, Zuchao Zhang, Lefei Xia, Gui-Song Du, Bo Zhang, Liangpei Tao, Dacheng Computation and Language Machine Learning Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment through quantization, pruning, or decoding strategy adjustments. We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes. Through systematic analysis of various LLM frameworks, we identify key vulnerability patterns: layer expansion frequently disrupts attention mechanisms, compression techniques induce information loss cascades, and decoding adjustments amplify prediction divergences. Our investigation reveals transformer architectures exhibit inherent robustness thresholds that determine hemorrhage severity across modification types. We propose three mitigation strategies: gradient-aware pruning preserves critical weight pathways, dynamic quantization scaling maintains activation integrity, and decoding calibration aligns generation trajectories with original model distributions. This work establishes foundational metrics for evaluating model stability during adaptation, providing practical guidelines for maintaining performance while enabling efficient LLM deployment. Our findings advance understanding of neural network resilience under architectural transformations, particularly for large-scale language models. |
| title | Model Hemorrhage and the Robustness Limits of Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.23924 |