Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914051738566656 |
|---|---|
| author | Deng, Boyang Peng, Songyou Genova, Kyle Wetzstein, Gordon Snavely, Noah Guibas, Leonidas Funkhouser, Thomas |
| author_facet | Deng, Boyang Peng, Songyou Genova, Kyle Wetzstein, Gordon Snavely, Noah Guibas, Leonidas Funkhouser, Thomas |
| contents | We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes. Specifically, we aim to capture frequent co-occurring changes ("trends") across a city over a certain period. Unlike previous visual analyses, our analysis answers open-ended queries (e.g., "what are the frequent types of changes in the city?") without any predetermined target subjects or training labels. These properties cast prior learning-based or unsupervised visual analysis tools unsuitable. We identify MLLMs as a novel tool for their open-ended semantic understanding capabilities. Yet, our datasets are four orders of magnitude too large for an MLLM to ingest as context. So we introduce a bottom-up procedure that decomposes the massive visual analysis problem into more tractable sub-problems. We carefully design MLLM-based solutions to each sub-problem. During experiments and ablation studies with our system, we find it significantly outperforms baselines and is able to discover interesting trends from images captured in large cities (e.g., "addition of outdoor dining,", "overpass was painted blue," etc.). See more results and interactive demos at https://boyangdeng.com/visual-chronicles. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08727 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images Deng, Boyang Peng, Songyou Genova, Kyle Wetzstein, Gordon Snavely, Noah Guibas, Leonidas Funkhouser, Thomas Computer Vision and Pattern Recognition Artificial Intelligence Computers and Society We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes. Specifically, we aim to capture frequent co-occurring changes ("trends") across a city over a certain period. Unlike previous visual analyses, our analysis answers open-ended queries (e.g., "what are the frequent types of changes in the city?") without any predetermined target subjects or training labels. These properties cast prior learning-based or unsupervised visual analysis tools unsuitable. We identify MLLMs as a novel tool for their open-ended semantic understanding capabilities. Yet, our datasets are four orders of magnitude too large for an MLLM to ingest as context. So we introduce a bottom-up procedure that decomposes the massive visual analysis problem into more tractable sub-problems. We carefully design MLLM-based solutions to each sub-problem. During experiments and ablation studies with our system, we find it significantly outperforms baselines and is able to discover interesting trends from images captured in large cities (e.g., "addition of outdoor dining,", "overpass was painted blue," etc.). See more results and interactive demos at https://boyangdeng.com/visual-chronicles. |
| title | Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2504.08727 |