PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913664358940672 |
|---|---|
| author | Ayyubi, Hammad Feng, Xuande Liu, Junzhang Lin, Xudong Wang, Zhecan Chang, Shih-Fu |
| author_facet | Ayyubi, Hammad Feng, Xuande Liu, Junzhang Lin, Xudong Wang, Zhecan Chang, Shih-Fu |
| contents | The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_14210 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction Ayyubi, Hammad Feng, Xuande Liu, Junzhang Lin, Xudong Wang, Zhecan Chang, Shih-Fu Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models. |
| title | PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2501.14210 |