Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.08529 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909424817274880 |
|---|---|
| author | Nguyen, Quynh-Mai Thi Nguyen, Lan-Nhi Thi Nguyen, Cam-Van Thi |
| author_facet | Nguyen, Quynh-Mai Thi Nguyen, Lan-Nhi Thi Nguyen, Cam-Van Thi |
| contents | The objective of multimodal intent recognition (MIR) is to leverage various modalities-such as text, video, and audio-to detect user intentions, which is crucial for understanding human language and context in dialogue systems. Despite advances in this field, two main challenges persist: (1) effectively extracting and utilizing semantic information from robust textual features; (2) aligning and fusing non-verbal modalities with verbal ones effectively. This paper proposes a Text Enhancement with CommOnsense Knowledge Extractor (TECO) to address these challenges. We begin by extracting relations from both generated and retrieved knowledge to enrich the contextual information in the text modality. Subsequently, we align and integrate visual and acoustic representations with these enhanced text features to form a cohesive multimodal representation. Our experimental results show substantial improvements over existing baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08529 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TECO: Improving Multimodal Intent Recognition with Text Enhancement through Commonsense Knowledge Extraction Nguyen, Quynh-Mai Thi Nguyen, Lan-Nhi Thi Nguyen, Cam-Van Thi Computation and Language The objective of multimodal intent recognition (MIR) is to leverage various modalities-such as text, video, and audio-to detect user intentions, which is crucial for understanding human language and context in dialogue systems. Despite advances in this field, two main challenges persist: (1) effectively extracting and utilizing semantic information from robust textual features; (2) aligning and fusing non-verbal modalities with verbal ones effectively. This paper proposes a Text Enhancement with CommOnsense Knowledge Extractor (TECO) to address these challenges. We begin by extracting relations from both generated and retrieved knowledge to enrich the contextual information in the text modality. Subsequently, we align and integrate visual and acoustic representations with these enhanced text features to form a cohesive multimodal representation. Our experimental results show substantial improvements over existing baseline methods. |
| title | TECO: Improving Multimodal Intent Recognition with Text Enhancement through Commonsense Knowledge Extraction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.08529 |