Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.10180 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915956773617664 |
|---|---|
| author | Deng, Wenlong Zeng, Qi Zhang, Jiaming Chen, Minghui Ding, Zixin Thrampoulidis, Christos Gong, Boying Li, Xiaoxiao |
| author_facet | Deng, Wenlong Zeng, Qi Zhang, Jiaming Chen, Minghui Ding, Zixin Thrampoulidis, Christos Gong, Boying Li, Xiaoxiao |
| contents | Data valuation is essential for enhancing the transparency and accountability of large language models (LLMs) and vision-language models (VLMs). However, existing methods typically rely on gradient computations, making them computationally prohibitive for billion-parameter models and precluding batch parallelization. In this work, we introduce For-Value, a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. Leveraging the expressive power of pretrained LLMs/VLMs, we theoretically demonstrate that data valuation can be captured by the alignment between the final hidden representations and prediction errors at the last layer. In light of this insight, For-Value computes data value using a simple closed-form expression with a single forward pass, eliminating the need for costly backpropagation and enabling efficient batch calculating at scale. Extensive experiments show that For-Value matches or outperforms gradient-based baselines in detecting influential data and mislabeled data, while achieving significant efficiency improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_10180 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMs Deng, Wenlong Zeng, Qi Zhang, Jiaming Chen, Minghui Ding, Zixin Thrampoulidis, Christos Gong, Boying Li, Xiaoxiao Computation and Language Data valuation is essential for enhancing the transparency and accountability of large language models (LLMs) and vision-language models (VLMs). However, existing methods typically rely on gradient computations, making them computationally prohibitive for billion-parameter models and precluding batch parallelization. In this work, we introduce For-Value, a forward-only data valuation framework that enables efficient batch-scalable value estimation while maintaining effectiveness. Leveraging the expressive power of pretrained LLMs/VLMs, we theoretically demonstrate that data valuation can be captured by the alignment between the final hidden representations and prediction errors at the last layer. In light of this insight, For-Value computes data value using a simple closed-form expression with a single forward pass, eliminating the need for costly backpropagation and enabling efficient batch calculating at scale. Extensive experiments show that For-Value matches or outperforms gradient-based baselines in detecting influential data and mislabeled data, while achieving significant efficiency improvements. |
| title | For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.10180 |