EA-ViT: Efficient Adaptation for Elastic Vision Transformer
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918104375754752 |
|---|---|
| author | Zhu, Chen Zhao, Wangbo Zhang, Huiwen Khaki, Samir Zhou, Yuhao Tang, Weidong Wang, Shuo Yuan, Zhihang Shang, Yuzhang Peng, Xiaojiang Wang, Kai Yang, Dawei |
| author_facet | Zhu, Chen Zhao, Wangbo Zhang, Huiwen Khaki, Samir Zhou, Yuhao Tang, Weidong Wang, Shuo Yuan, Zhihang Shang, Yuzhang Peng, Xiaojiang Wang, Kai Yang, Dawei |
| contents | Vision Transformers (ViTs) have emerged as a foundational model in computer vision, excelling in generalization and adaptation to downstream tasks. However, deploying ViTs to support diverse resource constraints typically requires retraining multiple, size-specific ViTs, which is both time-consuming and energy-intensive. To address this issue, we propose an efficient ViT adaptation framework that enables a single adaptation process to generate multiple models of varying sizes for deployment on platforms with various resource constraints. Our approach comprises two stages. In the first stage, we enhance a pre-trained ViT with a nested elastic architecture that enables structural flexibility across MLP expansion ratio, number of attention heads, embedding dimension, and network depth. To preserve pre-trained knowledge and ensure stable adaptation, we adopt a curriculum-based training strategy that progressively increases elasticity. In the second stage, we design a lightweight router to select submodels according to computational budgets and downstream task demands. Initialized with Pareto-optimal configurations derived via a customized NSGA-II algorithm, the router is then jointly optimized with the backbone. Extensive experiments on multiple benchmarks demonstrate the effectiveness and versatility of EA-ViT. The code is available at https://github.com/zcxcf/EA-ViT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19360 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EA-ViT: Efficient Adaptation for Elastic Vision Transformer Zhu, Chen Zhao, Wangbo Zhang, Huiwen Khaki, Samir Zhou, Yuhao Tang, Weidong Wang, Shuo Yuan, Zhihang Shang, Yuzhang Peng, Xiaojiang Wang, Kai Yang, Dawei Computer Vision and Pattern Recognition Vision Transformers (ViTs) have emerged as a foundational model in computer vision, excelling in generalization and adaptation to downstream tasks. However, deploying ViTs to support diverse resource constraints typically requires retraining multiple, size-specific ViTs, which is both time-consuming and energy-intensive. To address this issue, we propose an efficient ViT adaptation framework that enables a single adaptation process to generate multiple models of varying sizes for deployment on platforms with various resource constraints. Our approach comprises two stages. In the first stage, we enhance a pre-trained ViT with a nested elastic architecture that enables structural flexibility across MLP expansion ratio, number of attention heads, embedding dimension, and network depth. To preserve pre-trained knowledge and ensure stable adaptation, we adopt a curriculum-based training strategy that progressively increases elasticity. In the second stage, we design a lightweight router to select submodels according to computational budgets and downstream task demands. Initialized with Pareto-optimal configurations derived via a customized NSGA-II algorithm, the router is then jointly optimized with the backbone. Extensive experiments on multiple benchmarks demonstrate the effectiveness and versatility of EA-ViT. The code is available at https://github.com/zcxcf/EA-ViT. |
| title | EA-ViT: Efficient Adaptation for Elastic Vision Transformer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.19360 |