Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915049070657536 |
|---|---|
| author | Kang, Suho Park, Jungyang Ha, Joonseo Kim, SoMin Kim, JinHyeong Park, Subeen Song, Kyungwoo |
| author_facet | Kang, Suho Park, Jungyang Ha, Joonseo Kim, SoMin Kim, JinHyeong Park, Subeen Song, Kyungwoo |
| contents | Foundation models (FMs) have achieved significant success across various tasks, leading to research on benchmarks for reasoning abilities. However, there is a lack of studies on FMs performance in exceptional scenarios, which we define as out-of-distribution (OOD) reasoning tasks. This paper is the first to address these cases, developing a novel dataset for evaluation of FMs across multiple modalities, including graphic novels, calligraphy, news articles, and lyrics. It includes tasks for instance classification, character recognition, token prediction, and text generation. The paper also proposes prompt engineering techniques like Chain-of-Thought (CoT) and CoT+Few-Shot to enhance performance. Validation of FMs using various methods revealed improvements. The code repository is accessible at: https://github.com/MLAI-Yonsei/ExceptionalBenchmark |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18001 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation Kang, Suho Park, Jungyang Ha, Joonseo Kim, SoMin Kim, JinHyeong Park, Subeen Song, Kyungwoo Artificial Intelligence Foundation models (FMs) have achieved significant success across various tasks, leading to research on benchmarks for reasoning abilities. However, there is a lack of studies on FMs performance in exceptional scenarios, which we define as out-of-distribution (OOD) reasoning tasks. This paper is the first to address these cases, developing a novel dataset for evaluation of FMs across multiple modalities, including graphic novels, calligraphy, news articles, and lyrics. It includes tasks for instance classification, character recognition, token prediction, and text generation. The paper also proposes prompt engineering techniques like Chain-of-Thought (CoT) and CoT+Few-Shot to enhance performance. Validation of FMs using various methods revealed improvements. The code repository is accessible at: https://github.com/MLAI-Yonsei/ExceptionalBenchmark |
| title | Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.18001 |