"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909158894206976 |
|---|---|
| author | Agarwal, Vibhor Garg, Madhav Krishan Dharmavaram, Sahiti Kumar, Dhruv |
| author_facet | Agarwal, Vibhor Garg, Madhav Krishan Dharmavaram, Sahiti Kumar, Dhruv |
| contents | This study evaluates the effectiveness of various large language models (LLMs) in performing tasks common among undergraduate computer science students. Although a number of research studies in the computing education community have explored the possibility of using LLMs for a variety of tasks, there is a lack of comprehensive research comparing different LLMs and evaluating which LLMs are most effective for different tasks. Our research systematically assesses some of the publicly available LLMs such as Google Bard, ChatGPT(3.5), GitHub Copilot Chat, and Microsoft Copilot across diverse tasks commonly encountered by undergraduate computer science students in India. These tasks include code explanation and documentation, solving class assignments, technical interview preparation, learning new concepts and frameworks, and email writing. Evaluation for these tasks was carried out by pre-final year and final year undergraduate computer science students and provides insights into the models' strengths and limitations. This study aims to guide students as well as instructors in selecting suitable LLMs for any specific task and offers valuable insights on how LLMs can be used constructively by students and instructors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01687 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | "Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students Agarwal, Vibhor Garg, Madhav Krishan Dharmavaram, Sahiti Kumar, Dhruv Computers and Society Human-Computer Interaction Machine Learning This study evaluates the effectiveness of various large language models (LLMs) in performing tasks common among undergraduate computer science students. Although a number of research studies in the computing education community have explored the possibility of using LLMs for a variety of tasks, there is a lack of comprehensive research comparing different LLMs and evaluating which LLMs are most effective for different tasks. Our research systematically assesses some of the publicly available LLMs such as Google Bard, ChatGPT(3.5), GitHub Copilot Chat, and Microsoft Copilot across diverse tasks commonly encountered by undergraduate computer science students in India. These tasks include code explanation and documentation, solving class assignments, technical interview preparation, learning new concepts and frameworks, and email writing. Evaluation for these tasks was carried out by pre-final year and final year undergraduate computer science students and provides insights into the models' strengths and limitations. This study aims to guide students as well as instructors in selecting suitable LLMs for any specific task and offers valuable insights on how LLMs can be used constructively by students and instructors. |
| title | "Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students |
| topic | Computers and Society Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2402.01687 |