A Multimodal Transformer for Live Streaming Highlight Prediction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866907981835141120 |
|---|---|
| author | Deng, Jiaxin Wang, Shiyao Shen, Dong Zhao, Liqin Yang, Fan Zhou, Guorui Meng, Gaofeng |
| author_facet | Deng, Jiaxin Wang, Shiyao Shen, Dong Zhao, Liqin Yang, Fan Zhou, Guorui Meng, Gaofeng |
| contents | Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12002 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Multimodal Transformer for Live Streaming Highlight Prediction Deng, Jiaxin Wang, Shiyao Shen, Dong Zhao, Liqin Yang, Fan Zhou, Guorui Meng, Gaofeng Multimedia Computer Vision and Pattern Recognition Recently, live streaming platforms have gained immense popularity. Traditional video highlight detection mainly focuses on visual features and utilizes both past and future content for prediction. However, live streaming requires models to infer without future frames and process complex multimodal interactions, including images, audio and text comments. To address these issues, we propose a multimodal transformer that incorporates historical look-back windows. We introduce a novel Modality Temporal Alignment Module to handle the temporal shift of cross-modal signals. Additionally, using existing datasets with limited manual annotations is insufficient for live streaming whose topics are constantly updated and changed. Therefore, we propose a novel Border-aware Pairwise Loss to learn from a large-scale dataset and utilize user implicit feedback as a weak supervision signal. Extensive experiments show our model outperforms various strong baselines on both real-world scenarios and public datasets. And we will release our dataset and code to better assess this topic. |
| title | A Multimodal Transformer for Live Streaming Highlight Prediction |
| topic | Multimedia Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.12002 |