Facilitating Holistic Evaluations with LLMs: Insights from Scenario-Based Experiments
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918080383287296 |
|---|---|
| author | Ishida, Toru Liu, Tongxi Wang, Hailong Cheunga, William K. |
| author_facet | Ishida, Toru Liu, Tongxi Wang, Hailong Cheunga, William K. |
| contents | Workshop courses designed to foster creativity are gaining popularity. However, even experienced faculty teams find it challenging to realize a holistic evaluation that accommodates diverse perspectives. Adequate deliberation is essential to integrate varied assessments, but faculty often lack the time for such exchanges. Deriving an average score without discussion undermines the purpose of a holistic evaluation. Therefore, this paper explores the use of a Large Language Model (LLM) as a facilitator to integrate diverse faculty assessments. Scenario-based experiments were conducted to determine if the LLM could integrate diverse evaluations and explain the underlying pedagogical theories to faculty. The results were noteworthy, showing that the LLM can effectively facilitate faculty discussions. Additionally, the LLM demonstrated the capability to create evaluation criteria by generalizing a single scenario-based experiment, leveraging its already acquired pedagogical domain knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17728 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Facilitating Holistic Evaluations with LLMs: Insights from Scenario-Based Experiments Ishida, Toru Liu, Tongxi Wang, Hailong Cheunga, William K. Computers and Society Artificial Intelligence Human-Computer Interaction Workshop courses designed to foster creativity are gaining popularity. However, even experienced faculty teams find it challenging to realize a holistic evaluation that accommodates diverse perspectives. Adequate deliberation is essential to integrate varied assessments, but faculty often lack the time for such exchanges. Deriving an average score without discussion undermines the purpose of a holistic evaluation. Therefore, this paper explores the use of a Large Language Model (LLM) as a facilitator to integrate diverse faculty assessments. Scenario-based experiments were conducted to determine if the LLM could integrate diverse evaluations and explain the underlying pedagogical theories to faculty. The results were noteworthy, showing that the LLM can effectively facilitate faculty discussions. Additionally, the LLM demonstrated the capability to create evaluation criteria by generalizing a single scenario-based experiment, leveraging its already acquired pedagogical domain knowledge. |
| title | Facilitating Holistic Evaluations with LLMs: Insights from Scenario-Based Experiments |
| topic | Computers and Society Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2405.17728 |