TuRTLe: A Unified Evaluation of LLMs for RTL Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909628797812736 |
|---|---|
| author | Garcia-Gasulla, Dario Kestor, Gokcen Parisi, Emanuele Albertí-Binimelis, Miquel Gutierrez, Cristian Ghorab, Razine Moundir Montenegro, Orlando Homs, Bernat Moreto, Miquel |
| author_facet | Garcia-Gasulla, Dario Kestor, Gokcen Parisi, Emanuele Albertí-Binimelis, Miquel Gutierrez, Cristian Ghorab, Razine Moundir Montenegro, Orlando Homs, Bernat Moreto, Miquel |
| contents | The rapid advancements in LLMs have driven the adoption of generative AI in various domains, including Electronic Design Automation (EDA). Unlike traditional software development, EDA presents unique challenges, as generated RTL code must not only be syntactically correct and functionally accurate but also synthesizable by hardware generators while meeting performance, power, and area constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TuRTLe, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TuRTLe integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, we benchmark a diverse set of open LLMs and analyze their strengths and weaknesses in EDA-specific tasks. Our results show that reasoning-based models, such as DeepSeek R1, consistently outperform others across multiple evaluation criteria, but at the cost of increased computational overhead and inference latency. Additionally, base models are better suited in module completion tasks, while instruct-tuned models perform better in specification-to-RTL tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_01986 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TuRTLe: A Unified Evaluation of LLMs for RTL Generation Garcia-Gasulla, Dario Kestor, Gokcen Parisi, Emanuele Albertí-Binimelis, Miquel Gutierrez, Cristian Ghorab, Razine Moundir Montenegro, Orlando Homs, Bernat Moreto, Miquel Hardware Architecture Artificial Intelligence I.2.5; J.6 The rapid advancements in LLMs have driven the adoption of generative AI in various domains, including Electronic Design Automation (EDA). Unlike traditional software development, EDA presents unique challenges, as generated RTL code must not only be syntactically correct and functionally accurate but also synthesizable by hardware generators while meeting performance, power, and area constraints. These additional requirements introduce complexities that existing code-generation benchmarks often fail to capture, limiting their effectiveness in evaluating LLMs for RTL generation. To address this gap, we propose TuRTLe, a unified evaluation framework designed to systematically assess LLMs across key RTL generation tasks. TuRTLe integrates multiple existing benchmarks and automates the evaluation process, enabling a comprehensive assessment of LLM performance in syntax correctness, functional correctness, synthesis, PPA optimization, and exact line completion. Using this framework, we benchmark a diverse set of open LLMs and analyze their strengths and weaknesses in EDA-specific tasks. Our results show that reasoning-based models, such as DeepSeek R1, consistently outperform others across multiple evaluation criteria, but at the cost of increased computational overhead and inference latency. Additionally, base models are better suited in module completion tasks, while instruct-tuned models perform better in specification-to-RTL tasks. |
| title | TuRTLe: A Unified Evaluation of LLMs for RTL Generation |
| topic | Hardware Architecture Artificial Intelligence I.2.5; J.6 |
| url | https://arxiv.org/abs/2504.01986 |