Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12119 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918530469855232 |
|---|---|
| author | Bi, Jinhe Aniri Jin, Zengjie Wang, Yifan Yan, Danqi Huang, Wenke Ma, Xiaowen Yan, Sikuan Hecker, Artur Ye, Mang Xiao, Xun Schuetze, Hinrich Tresp, Volker Ma, Yunpu |
| author_facet | Bi, Jinhe Aniri Jin, Zengjie Wang, Yifan Yan, Danqi Huang, Wenke Ma, Xiaowen Yan, Sikuan Hecker, Artur Ye, Mang Xiao, Xun Schuetze, Hinrich Tresp, Volker Ma, Yunpu |
| contents | Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a \textit{Global Semantic Drift}, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise \textbf{PRISM}, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30\% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7\% relative improvement over the baseline. The code is available for access via \href{https://github.com/bibisbar/PRISM}{this repository}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection Bi, Jinhe Aniri Jin, Zengjie Wang, Yifan Yan, Danqi Huang, Wenke Ma, Xiaowen Yan, Sikuan Hecker, Artur Ye, Mang Xiao, Xun Schuetze, Hinrich Tresp, Volker Ma, Yunpu Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, the rapid growth of these datasets introduces significant redundancy, leading to increased computational costs. Existing methods for selecting instruction data aim to prune this redundancy, but predominantly rely on computationally demanding techniques such as proxy-based inference or training-based metrics. Consequently, the substantial computational costs incurred by these selection processes often exacerbate the very efficiency bottlenecks they are intended to resolve, posing a significant challenge to the scalable and effective tuning of MLLMs. To address this challenge, we first identify a critical, yet previously overlooked, factor: the anisotropy inherent in visual feature distributions. We find that this anisotropy induces a \textit{Global Semantic Drift}, and overlooking this phenomenon is a key factor limiting the efficiency of current data selection methods. Motivated by this insight, we devise \textbf{PRISM}, the first training-free framework for efficient visual instruction selection. PRISM surgically removes the corrupting influence of global background features by modeling the intrinsic visual semantics via implicit re-centering. Empirically, PRISM reduces the end-to-end time for data selection and model tuning to just 30\% of conventional pipelines. More remarkably, it achieves this efficiency while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across eight multimodal and three language understanding benchmarks, culminating in a 101.7\% relative improvement over the baseline. The code is available for access via \href{https://github.com/bibisbar/PRISM}{this repository}. |
| title | PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.12119 |