Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.02566 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910069904375808 |
|---|---|
| author | Yuan, Kun Sun, Min Woo Chen, Zhen Lozano, Alejandro He, Xiangteng Li, Shi Navab, Nassir Sun, Xiaoxiao Padoy, Nicolas Yeung-Levy, Serena |
| author_facet | Yuan, Kun Sun, Min Woo Chen, Zhen Lozano, Alejandro He, Xiangteng Li, Shi Navab, Nassir Sun, Xiaoxiao Padoy, Nicolas Yeung-Levy, Serena |
| contents | There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. However, current biomedical vision-language pretraining typically compresses rich scientific figures and text into coarse figure-level pairs, discarding the fine-grained correspondences that clinicians actually rely on when zooming into local structures. To tackle this issue, we introduce Panel2Patch, a novel data pipeline that mines hierarchical structure from existing biomedical scientific literature, i.e., multi-panel, marker-heavy figures and their surrounding text, and converts them into multi-granular supervision. Given scientific figures and captions, Panel2Patch parses layouts, panels, and visual markers, then constructs hierarchical aligned vision-language pairs at the figure, panel, and patch levels, preserving local semantics instead of treating each figure as a single data sample. Built on this hierarchical corpus, we develop a granularity-aware pretraining strategy that unifies heterogeneous objectives from coarse didactic descriptions to fine region-focused phrases. By applying Panel2Patch to only a small set of the literature figures, we extract far more effective supervision than prior pipelines, enabling substantially better performance with less pretraining data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_02566 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature Yuan, Kun Sun, Min Woo Chen, Zhen Lozano, Alejandro He, Xiangteng Li, Shi Navab, Nassir Sun, Xiaoxiao Padoy, Nicolas Yeung-Levy, Serena Computer Vision and Pattern Recognition Artificial Intelligence There is a growing interest in developing strong biomedical vision-language models. A popular approach to achieve robust representations is to use web-scale scientific data. However, current biomedical vision-language pretraining typically compresses rich scientific figures and text into coarse figure-level pairs, discarding the fine-grained correspondences that clinicians actually rely on when zooming into local structures. To tackle this issue, we introduce Panel2Patch, a novel data pipeline that mines hierarchical structure from existing biomedical scientific literature, i.e., multi-panel, marker-heavy figures and their surrounding text, and converts them into multi-granular supervision. Given scientific figures and captions, Panel2Patch parses layouts, panels, and visual markers, then constructs hierarchical aligned vision-language pairs at the figure, panel, and patch levels, preserving local semantics instead of treating each figure as a single data sample. Built on this hierarchical corpus, we develop a granularity-aware pretraining strategy that unifies heterogeneous objectives from coarse didactic descriptions to fine region-focused phrases. By applying Panel2Patch to only a small set of the literature figures, we extract far more effective supervision than prior pipelines, enabling substantially better performance with less pretraining data. |
| title | From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2512.02566 |