Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.12449 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912979656638464 |
|---|---|
| author | Nie, Zhanheng Fu, Chenghan Zhang, Daoze Wu, Junxian Guan, Wanxian Wang, Pengjie Xu, Jian Zheng, Bo |
| author_facet | Nie, Zhanheng Fu, Chenghan Zhang, Daoze Wu, Junxian Guan, Wanxian Wang, Pengjie Xu, Jian Zheng, Bo |
| contents | Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding. However, they still face three challenges: (i) the modality imbalance induced by modality mixed training; (ii) underutilization of the intrinsic alignment relationships among visual and textual information within a product; and (iii) limited handling of noise in e-commerce multimodal data. To address these, we propose MOON2.0, a dynamic modality-balanced MultimOdal representation learning framework for e-commerce prOduct uNderstanding. It comprises: (1) a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning to mitigate the modality imbalance; (2) a Dual-level Alignment method to better leverage semantic alignment properties inside individual products; and (3) an MLLM-based Image-text Co-augmentation strategy that integrates textual enrichment with visual expansion, coupled with Dynamic Sample Filtering to improve training data quality. We further release MBE2.0, a co-augmented Multimodal representation Benchmark for E-commerce representation learning and evaluation at https://huggingface.co/datasets/ZHNie/MBE2.0. Experiments show that MOON2.0 delivers state-of-the-art zero-shot performance on MBE2.0 and multiple public datasets. Furthermore, attention-based heatmap visualization provides qualitative evidence of improved multimodal alignment of MOON2.0. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12449 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding Nie, Zhanheng Fu, Chenghan Zhang, Daoze Wu, Junxian Guan, Wanxian Wang, Pengjie Xu, Jian Zheng, Bo Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding. However, they still face three challenges: (i) the modality imbalance induced by modality mixed training; (ii) underutilization of the intrinsic alignment relationships among visual and textual information within a product; and (iii) limited handling of noise in e-commerce multimodal data. To address these, we propose MOON2.0, a dynamic modality-balanced MultimOdal representation learning framework for e-commerce prOduct uNderstanding. It comprises: (1) a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning to mitigate the modality imbalance; (2) a Dual-level Alignment method to better leverage semantic alignment properties inside individual products; and (3) an MLLM-based Image-text Co-augmentation strategy that integrates textual enrichment with visual expansion, coupled with Dynamic Sample Filtering to improve training data quality. We further release MBE2.0, a co-augmented Multimodal representation Benchmark for E-commerce representation learning and evaluation at https://huggingface.co/datasets/ZHNie/MBE2.0. Experiments show that MOON2.0 delivers state-of-the-art zero-shot performance on MBE2.0 and multiple public datasets. Furthermore, attention-based heatmap visualization provides qualitative evidence of improved multimodal alignment of MOON2.0. |
| title | MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2511.12449 |