Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911918782939136 |
|---|---|
| author | Etesam, Yasaman Yalçın, Özge Nilay Zhang, Chuxuan Lim, Angelica |
| author_facet | Etesam, Yasaman Yalçın, Özge Nilay Zhang, Chuxuan Lim, Angelica |
| contents | The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_19995 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning Etesam, Yasaman Yalçın, Özge Nilay Zhang, Chuxuan Lim, Angelica Computer Vision and Pattern Recognition The emotional theory of mind problem requires facial expressions, body pose, contextual information and implicit commonsense knowledge to reason about the person's emotion and its causes, making it currently one of the most difficult problems in affective computing. In this work, we propose multiple methods to incorporate the emotional reasoning capabilities by constructing "narrative captions" relevant to emotion perception, that includes contextual and physical signal descriptors that focuses on "Who", "What", "Where" and "How" questions related to the image and emotions of the individual. We propose two distinct ways to construct these captions using zero-shot classifiers (CLIP) and fine-tuning visual-language models (LLaVA) over human generated descriptors. We further utilize these captions to guide the reasoning of language (GPT-4) and vision-language models (LLaVa, GPT-Vision). We evaluate the use of the resulting models in an image-to-language-to-emotion task. Our experiments showed that combining the "Fast" narrative descriptors and "Slow" reasoning of language models is a promising way to achieve emotional theory of mind. |
| title | Emotional Theory of Mind: Bridging Fast Visual Processing with Slow Linguistic Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.19995 |