Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2503.17551 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918152756002816 |
|---|---|
| author | Sun, Yu Li, Yin Sun, Ruixiao Liu, Chunhui Zhou, Fangming Jin, Ze Wang, Linjie Shen, Xiang Hao, Zhuolin Xiong, Hongyu |
| author_facet | Sun, Yu Li, Yin Sun, Ruixiao Liu, Chunhui Zhou, Fangming Jin, Ze Wang, Linjie Shen, Xiang Hao, Zhuolin Xiong, Hongyu |
| contents | Transformer-based multimodal models are widely used in industrial-scale recommendation, search, and advertising systems for content understanding and relevance ranking. Enhancing labeled training data quality and cross-modal fusion significantly improves model performance, influencing key metrics such as quality view rates and ad revenue. High-quality annotations are crucial for advancing content modeling, yet traditional statistical-based active learning (AL) methods face limitations: they struggle to detect overconfident misclassifications and are less effective in distinguishing semantically similar items in deep neural networks. Additionally, audio information plays an increasing role, especially in short-video platforms, yet most pre-trained multimodal architectures primarily focus on text and images. While training from scratch across all three modalities is possible, it sacrifices the benefits of leveraging existing pre-trained visual-language (VL) and audio models. To address these challenges, we propose kNN-based Latent Space Broadening (LSB) to enhance AL efficiency and Vision-Language Modeling with Audio Enhancement (VLMAE), a mid-fusion approach integrating audio into VL models. This system deployed in production systems, leading to significant business gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17551 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion Sun, Yu Li, Yin Sun, Ruixiao Liu, Chunhui Zhou, Fangming Jin, Ze Wang, Linjie Shen, Xiang Hao, Zhuolin Xiong, Hongyu Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing Transformer-based multimodal models are widely used in industrial-scale recommendation, search, and advertising systems for content understanding and relevance ranking. Enhancing labeled training data quality and cross-modal fusion significantly improves model performance, influencing key metrics such as quality view rates and ad revenue. High-quality annotations are crucial for advancing content modeling, yet traditional statistical-based active learning (AL) methods face limitations: they struggle to detect overconfident misclassifications and are less effective in distinguishing semantically similar items in deep neural networks. Additionally, audio information plays an increasing role, especially in short-video platforms, yet most pre-trained multimodal architectures primarily focus on text and images. While training from scratch across all three modalities is possible, it sacrifices the benefits of leveraging existing pre-trained visual-language (VL) and audio models. To address these challenges, we propose kNN-based Latent Space Broadening (LSB) to enhance AL efficiency and Vision-Language Modeling with Audio Enhancement (VLMAE), a mid-fusion approach integrating audio into VL models. This system deployed in production systems, leading to significant business gains. |
| title | Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion |
| topic | Multimedia Artificial Intelligence Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2503.17551 |