Physical Property Understanding from Language-Embedded Feature Fields
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929303837474816 |
|---|---|
| author | Zhai, Albert J. Shen, Yuan Chen, Emily Y. Wang, Gloria X. Wang, Xinlei Wang, Sheng Guan, Kaiyu Wang, Shenlong |
| author_facet | Zhai, Albert J. Shen, Yuan Chen, Emily Y. Wang, Gloria X. Wang, Xinlei Wang, Sheng Guan, Kaiyu Wang, Shenlong |
| contents | Can computers perceive the physical properties of objects solely through vision? Research in cognitive science and vision science has shown that humans excel at identifying materials and estimating their physical properties based purely on visual appearance. In this paper, we present a novel approach for dense prediction of the physical properties of objects using a collection of images. Inspired by how humans reason about physics through vision, we leverage large language models to propose candidate materials for each object. We then construct a language-embedded point cloud and estimate the physical properties of each 3D point using a zero-shot kernel regression approach. Our method is accurate, annotation-free, and applicable to any object in the open world. Experiments demonstrate the effectiveness of the proposed approach in various physical property reasoning tasks, such as estimating the mass of common objects, as well as other properties like friction and hardness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_04242 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Physical Property Understanding from Language-Embedded Feature Fields Zhai, Albert J. Shen, Yuan Chen, Emily Y. Wang, Gloria X. Wang, Xinlei Wang, Sheng Guan, Kaiyu Wang, Shenlong Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Can computers perceive the physical properties of objects solely through vision? Research in cognitive science and vision science has shown that humans excel at identifying materials and estimating their physical properties based purely on visual appearance. In this paper, we present a novel approach for dense prediction of the physical properties of objects using a collection of images. Inspired by how humans reason about physics through vision, we leverage large language models to propose candidate materials for each object. We then construct a language-embedded point cloud and estimate the physical properties of each 3D point using a zero-shot kernel regression approach. Our method is accurate, annotation-free, and applicable to any object in the open world. Experiments demonstrate the effectiveness of the proposed approach in various physical property reasoning tasks, such as estimating the mass of common objects, as well as other properties like friction and hardness. |
| title | Physical Property Understanding from Language-Embedded Feature Fields |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.04242 |