ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916399925952512 |
|---|---|
| author | Li, Peiyu Huang, Xiaobao Tian, Yijun Chawla, Nitesh V. |
| author_facet | Li, Peiyu Huang, Xiaobao Tian, Yijun Chawla, Nitesh V. |
| contents | Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12010 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation Li, Peiyu Huang, Xiaobao Tian, Yijun Chawla, Nitesh V. Computer Vision and Pattern Recognition Significant work has been conducted in the domain of food computing, yet these studies typically focus on single tasks such as t2t (instruction generation from food titles and ingredients), i2t (recipe generation from food images), or t2i (food image generation from recipes). None of these approaches integrate all modalities simultaneously. To address this gap, we introduce a novel food computing foundation model that achieves true multimodality, encompassing tasks such as t2t, t2i, i2t, it2t, and t2ti. By leveraging large language models (LLMs) and pre-trained image encoder and decoder models, our model can perform a diverse array of food computing-related tasks, including food understanding, food recognition, recipe generation, and food image generation. Compared to previous models, our foundation model demonstrates a significantly broader range of capabilities and exhibits superior performance, particularly in food image generation and recipe generation tasks. We open-sourced ChefFusion at GitHub. |
| title | ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.12010 |