Can I understand what I create? Self-Knowledge Evaluation of Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910479663759360 |
|---|---|
| author | Tan, Zhiquan Wei, Lai Wang, Jindong Xie, Xing Huang, Weiran |
| author_facet | Tan, Zhiquan Wei, Lai Wang, Jindong Xie, Xing Huang, Weiran |
| contents | Large language models (LLMs) have achieved remarkable progress in linguistic tasks, necessitating robust evaluation frameworks to understand their capabilities and limitations. Inspired by Feynman's principle of understanding through creation, we introduce a self-knowledge evaluation framework that is easy to implement, evaluating models on their ability to comprehend and respond to self-generated questions. Our findings, based on testing multiple models across diverse tasks, reveal significant gaps in the model's self-knowledge ability. Further analysis indicates these gaps may be due to misalignment with human attention mechanisms. Additionally, fine-tuning on self-generated math task may enhance the model's math performance, highlighting the potential of the framework for efficient and insightful model evaluation and may also contribute to the improvement of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06140 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can I understand what I create? Self-Knowledge Evaluation of Large Language Models Tan, Zhiquan Wei, Lai Wang, Jindong Xie, Xing Huang, Weiran Computation and Language Machine Learning Large language models (LLMs) have achieved remarkable progress in linguistic tasks, necessitating robust evaluation frameworks to understand their capabilities and limitations. Inspired by Feynman's principle of understanding through creation, we introduce a self-knowledge evaluation framework that is easy to implement, evaluating models on their ability to comprehend and respond to self-generated questions. Our findings, based on testing multiple models across diverse tasks, reveal significant gaps in the model's self-knowledge ability. Further analysis indicates these gaps may be due to misalignment with human attention mechanisms. Additionally, fine-tuning on self-generated math task may enhance the model's math performance, highlighting the potential of the framework for efficient and insightful model evaluation and may also contribute to the improvement of LLMs. |
| title | Can I understand what I create? Self-Knowledge Evaluation of Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.06140 |