Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.08806 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909169059102720 |
|---|---|
| author | DeLorenzo, Matthew Gohil, Vasudev Rajendran, Jeyavijayan |
| author_facet | DeLorenzo, Matthew Gohil, Vasudev Rajendran, Jeyavijayan |
| contents | Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality.
To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_08806 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation DeLorenzo, Matthew Gohil, Vasudev Rajendran, Jeyavijayan Computation and Language Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs. |
| title | CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.08806 |