Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909065715646464 |
|---|---|
| author | Nguyen, Cong-Duy Vu-Le, The-Anh Nguyen, Thong Quan, Tho Tuan, Luu Anh |
| author_facet | Nguyen, Cong-Duy Vu-Le, The-Anh Nguyen, Thong Quan, Tho Tuan, Luu Anh |
| contents | Language models have been supervised with both language-only objective and visual grounding in existing studies of visual-grounded language learning. However, due to differences in the distribution and scale of visual-grounded datasets and language corpora, the language model tends to mix up the context of the tokens that occurred in the grounded data with those that do not. As a result, during representation learning, there is a mismatch between the visual information and the contextual meaning of the sentence. To overcome this limitation, we propose GroundedBERT - a grounded language learning method that enhances the BERT representation with visually grounded information. GroundedBERT comprises two components: (i) the original BERT which captures the contextual representation of words learned from the language corpora, and (ii) a visual grounding module which captures visual information learned from visual-grounded datasets. Moreover, we employ Optimal Transport (OT), specifically its partial variant, to solve the fractional alignment problem between the two modalities. Our proposed method significantly outperforms the baseline language models on various language tasks of the GLUE and SQuAD datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_01592 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment Nguyen, Cong-Duy Vu-Le, The-Anh Nguyen, Thong Quan, Tho Tuan, Luu Anh Computation and Language Language models have been supervised with both language-only objective and visual grounding in existing studies of visual-grounded language learning. However, due to differences in the distribution and scale of visual-grounded datasets and language corpora, the language model tends to mix up the context of the tokens that occurred in the grounded data with those that do not. As a result, during representation learning, there is a mismatch between the visual information and the contextual meaning of the sentence. To overcome this limitation, we propose GroundedBERT - a grounded language learning method that enhances the BERT representation with visually grounded information. GroundedBERT comprises two components: (i) the original BERT which captures the contextual representation of words learned from the language corpora, and (ii) a visual grounding module which captures visual information learned from visual-grounded datasets. Moreover, we employ Optimal Transport (OT), specifically its partial variant, to solve the fractional alignment problem between the two modalities. Our proposed method significantly outperforms the baseline language models on various language tasks of the GLUE and SQuAD datasets. |
| title | Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2312.01592 |