Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913399543169024 |
|---|---|
| author | Vishwakarma, Sanjay Harkins, Francis Golecha, Siddharth Bajpe, Vishal Sharathchandra Dupuis, Nicolas Buratti, Luca Kremer, David Faro, Ismael Puri, Ruchir Cruz-Benito, Juan |
| author_facet | Vishwakarma, Sanjay Harkins, Francis Golecha, Siddharth Bajpe, Vishal Sharathchandra Dupuis, Nicolas Buratti, Luca Kremer, David Faro, Ismael Puri, Ruchir Cruz-Benito, Juan |
| contents | Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14712 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models Vishwakarma, Sanjay Harkins, Francis Golecha, Siddharth Bajpe, Vishal Sharathchandra Dupuis, Nicolas Buratti, Luca Kremer, David Faro, Ismael Puri, Ruchir Cruz-Benito, Juan Quantum Physics Artificial Intelligence Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation. |
| title | Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models |
| topic | Quantum Physics Artificial Intelligence |
| url | https://arxiv.org/abs/2406.14712 |