Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918143850446848 |
|---|---|
| author | You, Yang Uy, Mikaela Angelina Han, Jiaqi Thomas, Rahul Zhang, Haotong Du, Yi Chen, Hansheng Engelmann, Francis You, Suya Guibas, Leonidas |
| author_facet | You, Yang Uy, Mikaela Angelina Han, Jiaqi Thomas, Rahul Zhang, Haotong Du, Yi Chen, Hansheng Engelmann, Francis You, Suya Guibas, Leonidas |
| contents | Reverse engineering 3D computer-aided design (CAD) models from images is an important task for many downstream applications including interactive editing, manufacturing, architecture, robotics, etc. The difficulty of the task lies in vast representational disparities between the CAD output and the image input. CAD models are precise, programmatic constructs that involve sequential operations combining discrete command structure with continuous attributes, making it challenging to learn and optimize in an end-to-end fashion. Concurrently, input images introduce inherent challenges such as photometric variability and sensor noise, complicating the reverse engineering process. In this work, we introduce a novel approach that conditionally factorizes the task into two sub-problems. First, we leverage vision-language foundation models (VLMs), a finetuned Llama3.2, to predict the global discrete base structure with semantic information. Second, we propose TrAssembler that, conditioned on the discrete structure with semantics, predicts the continuous attribute values. To support the training of our TrAssembler, we further constructed an annotated CAD dataset of common objects from ShapeNet. Putting all together, our approach and data demonstrate significant first steps towards CAD-ifying images in the wild. Code and data can be found in https://github.com/qq456cvb/Img2CAD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01437 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization You, Yang Uy, Mikaela Angelina Han, Jiaqi Thomas, Rahul Zhang, Haotong Du, Yi Chen, Hansheng Engelmann, Francis You, Suya Guibas, Leonidas Computer Vision and Pattern Recognition Graphics Reverse engineering 3D computer-aided design (CAD) models from images is an important task for many downstream applications including interactive editing, manufacturing, architecture, robotics, etc. The difficulty of the task lies in vast representational disparities between the CAD output and the image input. CAD models are precise, programmatic constructs that involve sequential operations combining discrete command structure with continuous attributes, making it challenging to learn and optimize in an end-to-end fashion. Concurrently, input images introduce inherent challenges such as photometric variability and sensor noise, complicating the reverse engineering process. In this work, we introduce a novel approach that conditionally factorizes the task into two sub-problems. First, we leverage vision-language foundation models (VLMs), a finetuned Llama3.2, to predict the global discrete base structure with semantic information. Second, we propose TrAssembler that, conditioned on the discrete structure with semantics, predicts the continuous attribute values. To support the training of our TrAssembler, we further constructed an annotated CAD dataset of common objects from ShapeNet. Putting all together, our approach and data demonstrate significant first steps towards CAD-ifying images in the wild. Code and data can be found in https://github.com/qq456cvb/Img2CAD. |
| title | Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2408.01437 |