Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908839120470016 |
|---|---|
| author | Liu, Guoshan Zhu, Bin Li, Yian Chen, Jingjing Ngo, Chong-Wah Jiang, Yu-Gang |
| author_facet | Liu, Guoshan Zhu, Bin Li, Yian Chen, Jingjing Ngo, Chong-Wah Jiang, Yu-Gang |
| contents | Recent advances in Multimodal Large Language Models (MLMMs) have enabled recipe generation from food images, yet outputs often contain semantically incorrect actions or ingredients despite high lexical scores (e.g., BLEU, ROUGE). To address this gap, we propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation. Our two-stage pipeline combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT): SFT builds foundational accuracy using an Action-Reasoning dataset and ingredient corpus, while RFT employs frequency-aware rewards to improve long-tail action prediction and ingredient generalization. A Semantic Confidence Scoring and Rectification (SCSR) module further filters and corrects predictions. Experiments on Recipe1M show state-of-the-art performance and markedly improved semantic fidelity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_15862 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation Liu, Guoshan Zhu, Bin Li, Yian Chen, Jingjing Ngo, Chong-Wah Jiang, Yu-Gang Computation and Language Artificial Intelligence Recent advances in Multimodal Large Language Models (MLMMs) have enabled recipe generation from food images, yet outputs often contain semantically incorrect actions or ingredients despite high lexical scores (e.g., BLEU, ROUGE). To address this gap, we propose a semantically grounded framework that predicts and validates actions and ingredients as internal context for instruction generation. Our two-stage pipeline combines supervised fine-tuning (SFT) with reinforcement fine-tuning (RFT): SFT builds foundational accuracy using an Action-Reasoning dataset and ingredient corpus, while RFT employs frequency-aware rewards to improve long-tail action prediction and ingredient generalization. A Semantic Confidence Scoring and Rectification (SCSR) module further filters and corrects predictions. Experiments on Recipe1M show state-of-the-art performance and markedly improved semantic fidelity. |
| title | Enhancing Action and Ingredient Modeling for Semantically Grounded Recipe Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.15862 |