PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915152737075200 |
|---|---|
| author | Ahmed, Faruk Yang, Lin Jaroensri, Tiam Sellergren, Andrew Matias, Yossi Hassidim, Avinatan Corrado, Greg S. Webster, Dale R. Shetty, Shravya Prabhakara, Shruthi Liu, Yun Golden, Daniel Wulczyn, Ellery Steiner, David F. |
| author_facet | Ahmed, Faruk Yang, Lin Jaroensri, Tiam Sellergren, Andrew Matias, Yossi Hassidim, Avinatan Corrado, Greg S. Webster, Dale R. Shetty, Shravya Prabhakara, Shruthi Liu, Yun Golden, Daniel Wulczyn, Ellery Steiner, David F. |
| contents | The interpretation of histopathology cases underlies many important diagnostic and treatment decisions in medicine. Notably, this process typically requires pathologists to integrate and summarize findings across multiple slides per case. Existing vision-language capabilities in computational pathology have so far been largely limited to small regions of interest, larger regions at low magnification, or single whole-slide images (WSIs). This limits interpretation of findings that span multiple high-magnification regions across multiple WSIs. By making use of Gemini 1.5 Flash, a large multimodal model (LMM) with a 1-million token context window, we demonstrate the ability to generate bottom-line diagnoses from up to 40,000 768x768 pixel image patches from multiple WSIs at 10X magnification. This is the equivalent of up to 11 hours of video at 1 fps. Expert pathologist evaluations demonstrate that the generated report text is clinically accurate and equivalent to or preferred over the original reporting for 68% (95% CI: [60%, 76%]) of multi-slide examples with up to 5 slides. While performance decreased for examples with 6 or more slides, this study demonstrates the promise of leveraging the long-context capabilities of modern LMMs for the uniquely challenging task of medical report generation where each case can contain thousands of image patches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10536 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation Ahmed, Faruk Yang, Lin Jaroensri, Tiam Sellergren, Andrew Matias, Yossi Hassidim, Avinatan Corrado, Greg S. Webster, Dale R. Shetty, Shravya Prabhakara, Shruthi Liu, Yun Golden, Daniel Wulczyn, Ellery Steiner, David F. Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning The interpretation of histopathology cases underlies many important diagnostic and treatment decisions in medicine. Notably, this process typically requires pathologists to integrate and summarize findings across multiple slides per case. Existing vision-language capabilities in computational pathology have so far been largely limited to small regions of interest, larger regions at low magnification, or single whole-slide images (WSIs). This limits interpretation of findings that span multiple high-magnification regions across multiple WSIs. By making use of Gemini 1.5 Flash, a large multimodal model (LMM) with a 1-million token context window, we demonstrate the ability to generate bottom-line diagnoses from up to 40,000 768x768 pixel image patches from multiple WSIs at 10X magnification. This is the equivalent of up to 11 hours of video at 1 fps. Expert pathologist evaluations demonstrate that the generated report text is clinically accurate and equivalent to or preferred over the original reporting for 68% (95% CI: [60%, 76%]) of multi-slide examples with up to 5 slides. While performance decreased for examples with 6 or more slides, this study demonstrates the promise of leveraging the long-context capabilities of modern LMMs for the uniquely challenging task of medical report generation where each case can contain thousands of image patches. |
| title | PolyPath: Adapting a Large Multimodal Model for Multi-slide Pathology Report Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.10536 |