CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929607781908480 |
|---|---|
| author | Zhang, Zhi Yannakoudakis, Helen Zhen, Xiantong Shutova, Ekaterina |
| author_facet | Zhang, Zhi Yannakoudakis, Helen Zhen, Xiantong Shutova, Ekaterina |
| contents | The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_09027 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension Zhang, Zhi Yannakoudakis, Helen Zhen, Xiantong Shutova, Ekaterina Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art. |
| title | CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2302.09027 |