Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2307.15997 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929584205725696 |
|---|---|
| author | Wang, Ming Wu, Wenfang Gao, Chongyun Wang, Daling Feng, Shi Zhang, Yifei |
| author_facet | Wang, Ming Wu, Wenfang Gao, Chongyun Wang, Daling Feng, Shi Zhang, Yifei |
| contents | Large language models (LLMs) have received increasing attention. However, due to the complexity of its capabilities, how to rationally evaluate the capabilities of LLMs is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph and generates natural language evaluation tasks based on the task graph to evaluate the reasoning and memory abilities of LLMs respectively. Due to the very large randomness of the task construction process, it is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_15997 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | RoCar: A Relationship Network-based Evaluation Method for Large Language Models Wang, Ming Wu, Wenfang Gao, Chongyun Wang, Daling Feng, Shi Zhang, Yifei Computation and Language Artificial Intelligence Large language models (LLMs) have received increasing attention. However, due to the complexity of its capabilities, how to rationally evaluate the capabilities of LLMs is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph and generates natural language evaluation tasks based on the task graph to evaluate the reasoning and memory abilities of LLMs respectively. Due to the very large randomness of the task construction process, it is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method. |
| title | RoCar: A Relationship Network-based Evaluation Method for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2307.15997 |