Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2407.10743 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910527825903616 |
|---|---|
| author | Meijer, W. J. Kemmeren, A. C. Riemens, E. H. J. Fransman, J. E. van Bekkum, M. Burghouts, G. J. van Mil, J. D. |
| author_facet | Meijer, W. J. Kemmeren, A. C. Riemens, E. H. J. Fransman, J. E. van Bekkum, M. Burghouts, G. J. van Mil, J. D. |
| contents | This paper addresses the challenge of scaling Large Multimodal Models (LMMs) to expansive 3D environments. Solving this open problem is especially relevant for robot deployment in many first-responder scenarios, such as search-and-rescue missions that cover vast spaces. The use of LMMs in these settings is currently hampered by the strict context windows that limit the LMM's input size. We therefore introduce a novel approach that utilizes a datagraph structure, which allows the LMM to iteratively query smaller sections of a large environment. Using the datagraph in conjunction with graph traversal algorithms, we can prioritize the most relevant locations to the query, thereby improving the scalability of 3D scene language tasks. We illustrate the datagraph using 3D scenes, but these can be easily substituted by other dense modalities that represent the environment, such as pointclouds or Gaussian splats. We demonstrate the potential to use the datagraph for two 3D scene language task use cases, in a search-and-rescue mission example. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_10743 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling 3D Reasoning with LMMs to Large Robot Mission Environments Using Datagraphs Meijer, W. J. Kemmeren, A. C. Riemens, E. H. J. Fransman, J. E. van Bekkum, M. Burghouts, G. J. van Mil, J. D. Robotics Artificial Intelligence This paper addresses the challenge of scaling Large Multimodal Models (LMMs) to expansive 3D environments. Solving this open problem is especially relevant for robot deployment in many first-responder scenarios, such as search-and-rescue missions that cover vast spaces. The use of LMMs in these settings is currently hampered by the strict context windows that limit the LMM's input size. We therefore introduce a novel approach that utilizes a datagraph structure, which allows the LMM to iteratively query smaller sections of a large environment. Using the datagraph in conjunction with graph traversal algorithms, we can prioritize the most relevant locations to the query, thereby improving the scalability of 3D scene language tasks. We illustrate the datagraph using 3D scenes, but these can be easily substituted by other dense modalities that represent the environment, such as pointclouds or Gaussian splats. We demonstrate the potential to use the datagraph for two 3D scene language task use cases, in a search-and-rescue mission example. |
| title | Scaling 3D Reasoning with LMMs to Large Robot Mission Environments Using Datagraphs |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2407.10743 |