CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914800851746816 |
|---|---|
| author | Shin, Minjung Choi, Seongho Heo, Yu-Jung Lee, Minsu Zhang, Byoung-Tak Ryu, Jeh-Kwang |
| author_facet | Shin, Minjung Choi, Seongho Heo, Yu-Jung Lee, Minsu Zhang, Byoung-Tak Ryu, Jeh-Kwang |
| contents | We introduce CogME, a cognition-inspired, multi-dimensional evaluation metric designed for AI models focusing on story understanding. CogME is a framework grounded in human thinking strategies and story elements that involve story understanding. With a specific breakdown of the questions, this approach provides a nuanced assessment revealing not only AI models' particular strengths and weaknesses but also the characteristics of the benchmark dataset. Our case study with the DramaQA dataset demonstrates a refined analysis of the model and the benchmark dataset. We argue the need for metrics based on understanding the nature of tasks and designed to align closely with human cognitive processes. This approach provides insights beyond traditional overall scores and paves the way for more sophisticated AI development targeting higher cognitive functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2107_09847 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding Shin, Minjung Choi, Seongho Heo, Yu-Jung Lee, Minsu Zhang, Byoung-Tak Ryu, Jeh-Kwang Computer Vision and Pattern Recognition Artificial Intelligence We introduce CogME, a cognition-inspired, multi-dimensional evaluation metric designed for AI models focusing on story understanding. CogME is a framework grounded in human thinking strategies and story elements that involve story understanding. With a specific breakdown of the questions, this approach provides a nuanced assessment revealing not only AI models' particular strengths and weaknesses but also the characteristics of the benchmark dataset. Our case study with the DramaQA dataset demonstrates a refined analysis of the model and the benchmark dataset. We argue the need for metrics based on understanding the nature of tasks and designed to align closely with human cognitive processes. This approach provides insights beyond traditional overall scores and paves the way for more sophisticated AI development targeting higher cognitive functions. |
| title | CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2107.09847 |