No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908577159970816 |
|---|---|
| author | Sun, Min Woo Lozano, Alejandro Tejero, Javier Gamazo Nath, Vishwesh Sun, Xiao Xiao Burgess, James Zhang, Yuhui Yuan, Kun Tibshirani, Robert Huver, Sean Yeung-Levy, Serena |
| author_facet | Sun, Min Woo Lozano, Alejandro Tejero, Javier Gamazo Nath, Vishwesh Sun, Xiao Xiao Burgess, James Zhang, Yuhui Yuan, Kun Tibshirani, Robert Huver, Sean Yeung-Levy, Serena |
| contents | Embedding vision-language models (VLMs) are typically pretrained with short text windows (<77 tokens), which forces the truncation of long-format captions. Yet, the distribution of biomedical captions from large-scale open source literature reveals that a huge portion of captions far exceed 77 tokens. To this end, we investigate the impact of pretraining on long-format biomedical captions by extending the context length of text encoders in VLMs. We find that longer context (thus, enabling additional supervision provided in long-format captions) correlates with better retrieval and classification performance. Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M image-caption pairs enriched with context-aware descriptions from full-text articles, providing longer and additional textual supervision. Using BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a text encoder supporting windows of up to 512 tokens. Our model extends context capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in Recall@1 and +2% average improvements in classification, while also converging faster than short-context. Our results demonstrate that long-context modeling is a promising direction for advancing biomedical VLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03978 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models Sun, Min Woo Lozano, Alejandro Tejero, Javier Gamazo Nath, Vishwesh Sun, Xiao Xiao Burgess, James Zhang, Yuhui Yuan, Kun Tibshirani, Robert Huver, Sean Yeung-Levy, Serena Computer Vision and Pattern Recognition Computation and Language Embedding vision-language models (VLMs) are typically pretrained with short text windows (<77 tokens), which forces the truncation of long-format captions. Yet, the distribution of biomedical captions from large-scale open source literature reveals that a huge portion of captions far exceed 77 tokens. To this end, we investigate the impact of pretraining on long-format biomedical captions by extending the context length of text encoders in VLMs. We find that longer context (thus, enabling additional supervision provided in long-format captions) correlates with better retrieval and classification performance. Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M image-caption pairs enriched with context-aware descriptions from full-text articles, providing longer and additional textual supervision. Using BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a text encoder supporting windows of up to 512 tokens. Our model extends context capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in Recall@1 and +2% average improvements in classification, while also converging faster than short-context. Our results demonstrate that long-context modeling is a promising direction for advancing biomedical VLMs. |
| title | No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2510.03978 |