CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916918722560000 |
|---|---|
| author | Mallis, Dimitrios Karadeniz, Ahmet Serdar Cavada, Sebastian Rukhovich, Danila Foteinopoulou, Niki Cherenkova, Kseniya Kacem, Anis Aouada, Djamila |
| author_facet | Mallis, Dimitrios Karadeniz, Ahmet Serdar Cavada, Sebastian Rukhovich, Danila Foteinopoulou, Niki Cherenkova, Kseniya Kacem, Anis Aouada, Djamila |
| contents | We propose CAD-Assistant, a general-purpose CAD agent for AI-assisted design. Our approach is based on a powerful Vision and Large Language Model (VLLM) as a planner and a tool-augmentation paradigm using CAD-specific tools. CAD-Assistant addresses multimodal user queries by generating actions that are iteratively executed on a Python interpreter equipped with the FreeCAD software, accessed via its Python API. Our framework is able to assess the impact of generated CAD commands on geometry and adapts subsequent actions based on the evolving state of the CAD design. We consider a wide range of CAD-specific tools including a sketch image parameterizer, rendering modules, a 2D cross-section generator, and other specialized routines. CAD-Assistant is evaluated on multiple CAD benchmarks, where it outperforms VLLM baselines and supervised task-specific methods. Beyond existing benchmarks, we qualitatively demonstrate the potential of tool-augmented VLLMs as general-purpose CAD solvers across diverse workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13810 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers Mallis, Dimitrios Karadeniz, Ahmet Serdar Cavada, Sebastian Rukhovich, Danila Foteinopoulou, Niki Cherenkova, Kseniya Kacem, Anis Aouada, Djamila Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics We propose CAD-Assistant, a general-purpose CAD agent for AI-assisted design. Our approach is based on a powerful Vision and Large Language Model (VLLM) as a planner and a tool-augmentation paradigm using CAD-specific tools. CAD-Assistant addresses multimodal user queries by generating actions that are iteratively executed on a Python interpreter equipped with the FreeCAD software, accessed via its Python API. Our framework is able to assess the impact of generated CAD commands on geometry and adapts subsequent actions based on the evolving state of the CAD design. We consider a wide range of CAD-specific tools including a sketch image parameterizer, rendering modules, a 2D cross-section generator, and other specialized routines. CAD-Assistant is evaluated on multiple CAD benchmarks, where it outperforms VLLM baselines and supervised task-specific methods. Beyond existing benchmarks, we qualitatively demonstrate the potential of tool-augmented VLLMs as general-purpose CAD solvers across diverse workflows. |
| title | CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Robotics |
| url | https://arxiv.org/abs/2412.13810 |