CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910053426003968 |
|---|---|
| author | Zhuang, Xinlin Li, Yichen Liu, Xiwei Yang, Haolin Lu, Yifan Zou, Ziyun Li, Yulong Li, Huifa Chen, Dongliang Wang, Qinglei Liu, Weiyang Qian, Ying Shi, Jiangming Razzak, Imran |
| author_facet | Zhuang, Xinlin Li, Yichen Liu, Xiwei Yang, Haolin Lu, Yifan Zou, Ziyun Li, Yulong Li, Huifa Chen, Dongliang Wang, Qinglei Liu, Weiyang Qian, Ying Shi, Jiangming Razzak, Imran |
| contents | Adapting CLIP to vertical domains is typically approached by novel fine-tuning strategies or by continual pre-training (CPT) on large domain-specific datasets. Yet, data itself remains an underexplored factor in this process. We revisit this task from a data-centric perspective: Can effective data selection substitute for large-scale datasets in CPT? We introduce CHIPS (Curvature-aware Hybrid Influence in Projection Subspace), which assigns each image-text pair a utility score that integrates three complementary factors aligned with three goals: faithfulness via a curvature-aware and Newton-style alignment computed in CLIP's end-point subspace; scalability via an InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching; and retention via a selection-aware relevance weight combined with learnability to balance target adaptation against general-domain preservation. We justify this design theoretically by proving a lower-bound guarantee on the proxy's correlation with full-parameter alignment and by characterizing the bias-variance trade-offs introduced by curvature mixing and JL sketching. We evaluate CHIPS empirically across various settings: 1) CHIPS attains state-of-the-art performance among selection baselines on 17 medical benchmarks, matches full-dataset CPT with 30% of the data, and outperforms half-dataset CPT using only 10%; 2) on 31 general-domain benchmarks, CHIPS yields the least performance drop under all retention ratios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_18519 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection Zhuang, Xinlin Li, Yichen Liu, Xiwei Yang, Haolin Lu, Yifan Zou, Ziyun Li, Yulong Li, Huifa Chen, Dongliang Wang, Qinglei Liu, Weiyang Qian, Ying Shi, Jiangming Razzak, Imran Machine Learning Adapting CLIP to vertical domains is typically approached by novel fine-tuning strategies or by continual pre-training (CPT) on large domain-specific datasets. Yet, data itself remains an underexplored factor in this process. We revisit this task from a data-centric perspective: Can effective data selection substitute for large-scale datasets in CPT? We introduce CHIPS (Curvature-aware Hybrid Influence in Projection Subspace), which assigns each image-text pair a utility score that integrates three complementary factors aligned with three goals: faithfulness via a curvature-aware and Newton-style alignment computed in CLIP's end-point subspace; scalability via an InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching; and retention via a selection-aware relevance weight combined with learnability to balance target adaptation against general-domain preservation. We justify this design theoretically by proving a lower-bound guarantee on the proxy's correlation with full-parameter alignment and by characterizing the bias-variance trade-offs introduced by curvature mixing and JL sketching. We evaluate CHIPS empirically across various settings: 1) CHIPS attains state-of-the-art performance among selection baselines on 17 medical benchmarks, matches full-dataset CPT with 30% of the data, and outperforms half-dataset CPT using only 10%; 2) on 31 general-domain benchmarks, CHIPS yields the least performance drop under all retention ratios. |
| title | CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2511.18519 |