QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911129147539456 |
|---|---|
| author | Basit, Abdul Shao, Minghao Asif, Muhammad Haider Innan, Nouhaila Kashif, Muhammad Marchisio, Alberto Shafique, Muhammad |
| author_facet | Basit, Abdul Shao, Minghao Asif, Muhammad Haider Innan, Nouhaila Kashif, Muhammad Marchisio, Alberto Shafique, Muhammad |
| contents | Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20008 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges Basit, Abdul Shao, Minghao Asif, Muhammad Haider Innan, Nouhaila Kashif, Muhammad Marchisio, Alberto Shafique, Muhammad Artificial Intelligence Programming Languages Software Engineering 68T50, 81P68, 68T07, 68T20 I.2.7; I.2.2 Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming. |
| title | QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges |
| topic | Artificial Intelligence Programming Languages Software Engineering 68T50, 81P68, 68T07, 68T20 I.2.7; I.2.2 |
| url | https://arxiv.org/abs/2506.20008 |