QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913636245569536 |
|---|---|
| author | Prakash, Shvetank Cheng, Andrew Yik, Jason Tschand, Arya Ghosal, Radhika Uchendu, Ikechukwu Quaye, Jessica Ma, Jeffrey Grampurohit, Shreyas Giannuzzi, Sofia Balyan, Arnav Amin, Fin Pipersenia, Aadya Choudhary, Yash Nayak, Ankita Yazdanbakhsh, Amir Reddi, Vijay Janapa |
| author_facet | Prakash, Shvetank Cheng, Andrew Yik, Jason Tschand, Arya Ghosal, Radhika Uchendu, Ikechukwu Quaye, Jessica Ma, Jeffrey Grampurohit, Shreyas Giannuzzi, Sofia Balyan, Arnav Amin, Fin Pipersenia, Aadya Choudhary, Yash Nayak, Ankita Yazdanbakhsh, Amir Reddi, Vijay Janapa |
| contents | We introduce QuArch, a dataset of 1500 human-validated question-answer pairs designed to evaluate and enhance language models' understanding of computer architecture. The dataset covers areas including processor design, memory systems, and performance optimization. Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%. We observe notable struggles in memory systems, interconnection networks, and benchmarking. Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research. The dataset and leaderboard are at https://harvard-edge.github.io/QuArch/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01892 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture Prakash, Shvetank Cheng, Andrew Yik, Jason Tschand, Arya Ghosal, Radhika Uchendu, Ikechukwu Quaye, Jessica Ma, Jeffrey Grampurohit, Shreyas Giannuzzi, Sofia Balyan, Arnav Amin, Fin Pipersenia, Aadya Choudhary, Yash Nayak, Ankita Yazdanbakhsh, Amir Reddi, Vijay Janapa Hardware Architecture Artificial Intelligence Machine Learning We introduce QuArch, a dataset of 1500 human-validated question-answer pairs designed to evaluate and enhance language models' understanding of computer architecture. The dataset covers areas including processor design, memory systems, and performance optimization. Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%. We observe notable struggles in memory systems, interconnection networks, and benchmarking. Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research. The dataset and leaderboard are at https://harvard-edge.github.io/QuArch/. |
| title | QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture |
| topic | Hardware Architecture Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2501.01892 |