MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918362355859456 |
|---|---|
| author | Zhang, Daoze Fu, Chenghan Nie, Zhanheng Liu, Jianyu Guan, Wanxian Gao, Yuan Song, Jun Wang, Pengjie Xu, Jian Zheng, Bo |
| author_facet | Zhang, Daoze Fu, Chenghan Nie, Zhanheng Liu, Jianyu Guan, Wanxian Gao, Yuan Song, Jun Wang, Pengjie Xu, Jian Zheng, Bo |
| contents | With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding. The data of our MBE benchmark is given in https://huggingface.co/datasets/Daoze/MM-Bench-E-Commerce. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11999 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding Zhang, Daoze Fu, Chenghan Nie, Zhanheng Liu, Jianyu Guan, Wanxian Gao, Yuan Song, Jun Wang, Pengjie Xu, Jian Zheng, Bo Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning With the rapid advancement of e-commerce, exploring general representations rather than task-specific ones has attracted increasing research attention. For product understanding, although existing discriminative dual-flow architectures drive progress in this field, they inherently struggle to model the many-to-one alignment between multiple images and texts of products. Therefore, we argue that generative Multimodal Large Language Models (MLLMs) hold significant potential for improving product representation learning. Nevertheless, achieving this goal still remains non-trivial due to several key challenges: the lack of multimodal and aspect-aware modeling modules in typical LLMs; the common presence of background noise in product images; and the absence of a standard benchmark for evaluation. To address these issues, we propose the first generative MLLM-based model named MOON for product representation learning. Our method (1) employs a guided Mixture-of-Experts (MoE) module for targeted modeling of multimodal and aspect-specific product content; (2) effectively detects core semantic regions in product images to mitigate the distraction and interference caused by background noise; and (3) introduces the specialized negative sampling strategy to increase the difficulty and diversity of negative samples. In addition, we release a large-scale multimodal benchmark MBE for various product understanding tasks. Experimentally, our model demonstrates competitive zero-shot performance on both our benchmark and the public dataset, showcasing strong generalization across various downstream tasks, including cross-modal retrieval, product classification, and attribute prediction. Furthermore, the case study and visualization illustrate the effectiveness of MOON for product understanding. The data of our MBE benchmark is given in https://huggingface.co/datasets/Daoze/MM-Bench-E-Commerce. |
| title | MOON: Generative MLLM-based Multimodal Representation Learning for E-commerce Product Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2508.11999 |