QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914498721349632 |
|---|---|
| author | Slim, Ali Hamieh, Haydar Kotaich, Jawad Ghosn, Yehya Chehimi, Mahdi Mohanna, Ammar Hammoud, Hasan Abed Al Kader Ghanem, Bernard |
| author_facet | Slim, Ali Hamieh, Haydar Kotaich, Jawad Ghosn, Yehya Chehimi, Mahdi Mohanna, Ammar Hammoud, Hasan Abed Al Kader Ghanem, Bernard |
| contents | Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation.
We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08570 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation Slim, Ali Hamieh, Haydar Kotaich, Jawad Ghosn, Yehya Chehimi, Mahdi Mohanna, Ammar Hammoud, Hasan Abed Al Kader Ghanem, Bernard Machine Learning Artificial Intelligence Programming Languages Software Engineering Quantum Physics Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge. |
| title | QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation |
| topic | Machine Learning Artificial Intelligence Programming Languages Software Engineering Quantum Physics |
| url | https://arxiv.org/abs/2604.08570 |