Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910985470607360 |
|---|---|
| author | Galvan-Sosa, Diana Gaudeau, Gabrielle Kavumba, Pride Li, Yunmeng gu, Hongyi Yuan, Zheng Sakaguchi, Keisuke Buttery, Paula |
| author_facet | Galvan-Sosa, Diana Gaudeau, Gabrielle Kavumba, Pride Li, Yunmeng gu, Hongyi Yuan, Zheng Sakaguchi, Keisuke Buttery, Paula |
| contents | The performance and usability of Large-Language Models (LLMs) are driving their use in explanation generation tasks. However, despite their widespread adoption, LLM explanations have been found to be unreliable, making it difficult for users to distinguish good from bad explanations. To address this issue, we present Rubrik's CUBE, an education-inspired rubric and a dataset of 26k explanations, written and later quality-annotated using the rubric by both humans and six open- and closed-source LLMs. The CUBE dataset focuses on two reasoning and two language tasks, providing the necessary diversity for us to effectively test our proposed rubric. Using Rubrik, we find that explanations are influenced by both task and perceived difficulty. Low quality stems primarily from a lack of conciseness in LLM-generated explanations, rather than cohesion and word choice. The full dataset, rubric, and code are available at https://github.com/RubriksCube/rubriks_cube. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23899 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset Galvan-Sosa, Diana Gaudeau, Gabrielle Kavumba, Pride Li, Yunmeng gu, Hongyi Yuan, Zheng Sakaguchi, Keisuke Buttery, Paula Computation and Language I.2.7 The performance and usability of Large-Language Models (LLMs) are driving their use in explanation generation tasks. However, despite their widespread adoption, LLM explanations have been found to be unreliable, making it difficult for users to distinguish good from bad explanations. To address this issue, we present Rubrik's CUBE, an education-inspired rubric and a dataset of 26k explanations, written and later quality-annotated using the rubric by both humans and six open- and closed-source LLMs. The CUBE dataset focuses on two reasoning and two language tasks, providing the necessary diversity for us to effectively test our proposed rubric. Using Rubrik, we find that explanations are influenced by both task and perceived difficulty. Low quality stems primarily from a lack of conciseness in LLM-generated explanations, rather than cohesion and word choice. The full dataset, rubric, and code are available at https://github.com/RubriksCube/rubriks_cube. |
| title | Rubrik's Cube: Testing a New Rubric for Evaluating Explanations on the CUBE dataset |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2503.23899 |