Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.15654 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914691986489344 |
|---|---|
| author | Ghaffari, Sadaf Krishnaswamy, Nikhil |
| author_facet | Ghaffari, Sadaf Krishnaswamy, Nikhil |
| contents | In this paper, we present an exploration of LLMs' abilities to problem solve with physical reasoning in situated environments. We construct a simple simulated environment and demonstrate examples of where, in a zero-shot setting, both text and multimodal LLMs display atomic world knowledge about various objects but fail to compose this knowledge in correct solutions for an object manipulation and placement task. We also use BLIP, a vision-language model trained with more sophisticated cross-modal attention, to identify cases relevant to object physical properties that that model fails to ground. Finally, we present a procedure for discovering the relevant properties of objects in the environment and propose a method to distill this knowledge back into the LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_15654 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics Ghaffari, Sadaf Krishnaswamy, Nikhil Computation and Language In this paper, we present an exploration of LLMs' abilities to problem solve with physical reasoning in situated environments. We construct a simple simulated environment and demonstrate examples of where, in a zero-shot setting, both text and multimodal LLMs display atomic world knowledge about various objects but fail to compose this knowledge in correct solutions for an object manipulation and placement task. We also use BLIP, a vision-language model trained with more sophisticated cross-modal attention, to identify cases relevant to object physical properties that that model fails to ground. Finally, we present a procedure for discovering the relevant properties of objects in the environment and propose a method to distill this knowledge back into the LLM. |
| title | Exploring Failure Cases in Multimodal Reasoning About Physical Dynamics |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.15654 |