Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911659718606848 |
|---|---|
| author | Hartman, Max Jayaraman, Vidhata Choraria, Moulik Bhimaraju, Akhil Varshney, Lav R. |
| author_facet | Hartman, Max Jayaraman, Vidhata Choraria, Moulik Bhimaraju, Akhil Varshney, Lav R. |
| contents | Vision-language models achieve incredible performance across a wide range of tasks, but their large size makes inference costly. Recent work has shown that multimodal processing contains significant redundancies, making it possible to skip certain layers with minimal performance loss. Yet current pruning techniques remain ad-hoc, relying on heuristics or hyperparameter sweeps rather than principled criteria for determining when layer skipping is beneficial. In this paper, we propose a unified framework that characterizes the redundancy conditions under which pruning can enhance efficiency without sacrificing performance. Central to our approach are experimentally verifiable and interpretable notions of redundancy that can be evaluated without requiring downstream task performance as a metric. Applying this framework, we corroborate prior findings that both early and late vision tokens are redundant across models, and we validate our conditions by showing they align with actual performance degradation. Beyond these empirical results, our framework provides a theoretically grounded understanding of redundancy in VLMs and unifies many of the ideas behind modern layer-skipping techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_25584 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models Hartman, Max Jayaraman, Vidhata Choraria, Moulik Bhimaraju, Akhil Varshney, Lav R. Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Information Theory Machine Learning Vision-language models achieve incredible performance across a wide range of tasks, but their large size makes inference costly. Recent work has shown that multimodal processing contains significant redundancies, making it possible to skip certain layers with minimal performance loss. Yet current pruning techniques remain ad-hoc, relying on heuristics or hyperparameter sweeps rather than principled criteria for determining when layer skipping is beneficial. In this paper, we propose a unified framework that characterizes the redundancy conditions under which pruning can enhance efficiency without sacrificing performance. Central to our approach are experimentally verifiable and interpretable notions of redundancy that can be evaluated without requiring downstream task performance as a metric. Applying this framework, we corroborate prior findings that both early and late vision tokens are redundant across models, and we validate our conditions by showing they align with actual performance degradation. Beyond these empirical results, our framework provides a theoretically grounded understanding of redundancy in VLMs and unifies many of the ideas behind modern layer-skipping techniques. |
| title | Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Information Theory Machine Learning |
| url | https://arxiv.org/abs/2509.25584 |