Efficient Multi-modal Large Language Models via Progressive Consistency Distillation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909818262913024 |
|---|---|
| author | Wen, Zichen Wang, Shaobo Zhou, Yufa Zhang, Junyuan Zhang, Qintong Gao, Yifeng Chen, Zhaorun Wang, Bin Li, Weijia He, Conghui Zhang, Linfeng |
| author_facet | Wen, Zichen Wang, Shaobo Zhou, Yufa Zhang, Junyuan Zhang, Qintong Gao, Yifeng Chen, Zhaorun Wang, Bin Li, Weijia He, Conghui Zhang, Linfeng |
| contents | Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, as the model's parameter space struggles to quickly adapt to the substantial perturbations in the feature space induced by token compression. In this work, we propose to develop Efficient MLLMs via Progressive Consistency Distillation (EPIC), a progressive learning framework. Specifically, by decomposing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training difficulty by leveraging guidance from a teacher model and following a progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00515 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Multi-modal Large Language Models via Progressive Consistency Distillation Wen, Zichen Wang, Shaobo Zhou, Yufa Zhang, Junyuan Zhang, Qintong Gao, Yifeng Chen, Zhaorun Wang, Bin Li, Weijia He, Conghui Zhang, Linfeng Computer Vision and Pattern Recognition Visual tokens consume substantial computational resources in multi-modal large models (MLLMs), significantly compromising their efficiency. Recent works have attempted to improve efficiency by compressing visual tokens during training, either through modifications to model components or by introducing additional parameters. However, they often overlook the increased learning difficulty caused by such compression, as the model's parameter space struggles to quickly adapt to the substantial perturbations in the feature space induced by token compression. In this work, we propose to develop Efficient MLLMs via Progressive Consistency Distillation (EPIC), a progressive learning framework. Specifically, by decomposing the feature space perturbations introduced by token compression along the token-wise and layer-wise dimensions, we introduce token consistency distillation and layer consistency distillation, respectively, aiming to reduce the training difficulty by leveraging guidance from a teacher model and following a progressive learning trajectory. Extensive experiments demonstrate the superior effectiveness, robustness, and generalization capabilities of our proposed framework. |
| title | Efficient Multi-modal Large Language Models via Progressive Consistency Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.00515 |