Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917753286295552 |
|---|---|
| author | Sugihara, Tomoya Masuda, Shuntaro Xiao, Ling Yamasaki, Toshihiko |
| author_facet | Sugihara, Tomoya Masuda, Shuntaro Xiao, Ling Yamasaki, Toshihiko |
| contents | Current video summarization methods rely heavily on supervised computer vision techniques, which demands time-consuming and subjective manual annotations. To overcome these limitations, we investigated self-supervised video summarization. Inspired by the success of Large Language Models (LLMs), we explored the feasibility in transforming the video summarization task into a Natural Language Processing (NLP) task. By leveraging the advantages of LLMs in context understanding, we aim to enhance the effectiveness of self-supervised video summarization. Our method begins by generating captions for individual video frames, which are then synthesized into text summaries by LLMs. Subsequently, we measure semantic distance between the captions and the text summary. Notably, we propose a novel loss function to optimize our model according to the diversity of the video. Finally, the summarized video can be generated by selecting the frames with captions similar to the text summary. Our method achieves state-of-the-art performance on the SumMe dataset in rank correlation coefficients. In addition, our method has a novel feature of being able to achieve personalized summarization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08890 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video Sugihara, Tomoya Masuda, Shuntaro Xiao, Ling Yamasaki, Toshihiko Computer Vision and Pattern Recognition Current video summarization methods rely heavily on supervised computer vision techniques, which demands time-consuming and subjective manual annotations. To overcome these limitations, we investigated self-supervised video summarization. Inspired by the success of Large Language Models (LLMs), we explored the feasibility in transforming the video summarization task into a Natural Language Processing (NLP) task. By leveraging the advantages of LLMs in context understanding, we aim to enhance the effectiveness of self-supervised video summarization. Our method begins by generating captions for individual video frames, which are then synthesized into text summaries by LLMs. Subsequently, we measure semantic distance between the captions and the text summary. Notably, we propose a novel loss function to optimize our model according to the diversity of the video. Finally, the summarized video can be generated by selecting the frames with captions similar to the text summary. Our method achieves state-of-the-art performance on the SumMe dataset in rank correlation coefficients. In addition, our method has a novel feature of being able to achieve personalized summarization. |
| title | Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.08890 |