Parametric-ControlNet: Multimodal Control in Foundation Models for Precise Engineering Design Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910729795272704 |
|---|---|
| author | Zhou, Rui Zhang, Yanxia Yuan, Chenyang Permenter, Frank Arechiga, Nikos Klenk, Matt Ahmed, Faez |
| author_facet | Zhou, Rui Zhang, Yanxia Yuan, Chenyang Permenter, Frank Arechiga, Nikos Klenk, Matt Ahmed, Faez |
| contents | This paper introduces a generative model designed for multimodal control over text-to-image foundation generative AI models such as Stable Diffusion, specifically tailored for engineering design synthesis. Our model proposes parametric, image, and text control modalities to enhance design precision and diversity. Firstly, it handles both partial and complete parametric inputs using a diffusion model that acts as a design autocomplete co-pilot, coupled with a parametric encoder to process the information. Secondly, the model utilizes assembly graphs to systematically assemble input component images, which are then processed through a component encoder to capture essential visual data. Thirdly, textual descriptions are integrated via CLIP encoding, ensuring a comprehensive interpretation of design intent. These diverse inputs are synthesized through a multimodal fusion technique, creating a joint embedding that acts as the input to a module inspired by ControlNet. This integration allows the model to apply robust multimodal control to foundation models, facilitating the generation of complex and precise engineering designs. This approach broadens the capabilities of AI-driven design tools and demonstrates significant advancements in precise control based on diverse data modalities for enhanced design generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04707 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Parametric-ControlNet: Multimodal Control in Foundation Models for Precise Engineering Design Synthesis Zhou, Rui Zhang, Yanxia Yuan, Chenyang Permenter, Frank Arechiga, Nikos Klenk, Matt Ahmed, Faez Artificial Intelligence Computational Engineering, Finance, and Science Computer Vision and Pattern Recognition Human-Computer Interaction This paper introduces a generative model designed for multimodal control over text-to-image foundation generative AI models such as Stable Diffusion, specifically tailored for engineering design synthesis. Our model proposes parametric, image, and text control modalities to enhance design precision and diversity. Firstly, it handles both partial and complete parametric inputs using a diffusion model that acts as a design autocomplete co-pilot, coupled with a parametric encoder to process the information. Secondly, the model utilizes assembly graphs to systematically assemble input component images, which are then processed through a component encoder to capture essential visual data. Thirdly, textual descriptions are integrated via CLIP encoding, ensuring a comprehensive interpretation of design intent. These diverse inputs are synthesized through a multimodal fusion technique, creating a joint embedding that acts as the input to a module inspired by ControlNet. This integration allows the model to apply robust multimodal control to foundation models, facilitating the generation of complex and precise engineering designs. This approach broadens the capabilities of AI-driven design tools and demonstrates significant advancements in precise control based on diverse data modalities for enhanced design generation. |
| title | Parametric-ControlNet: Multimodal Control in Foundation Models for Precise Engineering Design Synthesis |
| topic | Artificial Intelligence Computational Engineering, Finance, and Science Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2412.04707 |