Cover Image

Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Pinckney, Nathaniel, Deng, Chenhui, Ho, Chia-Tung, Tsai, Yun-Da, Liu, Mingjie, Zhou, Wenfei, Khailany, Brucek, Ren, Haoxing
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Hardware Architecture
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation
by: Pinckney, Nathaniel, et al.
Published: (2024)

ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs
by: Deng, Chenhui, et al.
Published: (2026)

Marco: Configurable Graph-Based Task Solving and Multi-AI Agents Framework for Hardware Design
by: Ho, Chia-Tung, et al.
Published: (2025)

CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
by: Liu, Mingjie, et al.
Published: (2024)

GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation
by: Kochar, Dimple Vijay, et al.
Published: (2026)

RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models
by: Tsai, Yun-Da, et al.
Published: (2023)

ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation
by: Deng, Chenhui, et al.
Published: (2025)

VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool
by: Ho, Chia-Tung, et al.
Published: (2024)

Large Language Model (LLM) for Standard Cell Layout Design Optimization
by: Ho, Chia-Tung, et al.
Published: (2024)

PRO-V-R1: Reasoning Enhanced Programming Agent for RTL Verification
by: Zhao, Yujie, et al.
Published: (2025)

ChipAlign: Instruction Alignment in Large Language Models for Chip Design via Geodesic Interpolation
by: Deng, Chenhui, et al.
Published: (2024)

OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation
by: Liu, Shang, et al.
Published: (2025)

FVDebug: An LLM-Driven Debugging Assistant for Automated Root Cause Analysis of Formal Verification Failures
by: Bai, Yunsheng, et al.
Published: (2025)

Assessing Large Language Models in Generating RTL Design Specifications
by: Huang, Hung-Ming, et al.
Published: (2025)

ProtocolLLM: RTL Benchmark for SystemVerilog Generation of Communication Protocols
by: Sheth, Arnav, et al.
Published: (2025)

Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systems
by: Yu, Zhongzhi, et al.
Published: (2025)

Think with Self-Decoupling and Self-Verification: Automated RTL Design with Backtrack-ToT
by: Chao, Zhiteng, et al.
Published: (2025)

SpecLoop: An Agentic RTL-to-Specification Framework with Formal Verification Feedback Loop
by: Chang, Fu-Chieh, et al.
Published: (2026)

GauRast: Enhancing GPU Triangle Rasterizers to Accelerate 3D Gaussian Splatting
by: Li, Sixu, et al.
Published: (2025)

From RTL to Prompt Coding: Empowering the Next Generation of Chip Designers through LLMs
by: Krupp, Lukas, et al.
Published: (2026)

RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects
by: Allam, Ahmed, et al.
Published: (2024)

DRC-Coder: Automated DRC Checker Code Generation Using LLM Autonomous Agent
by: Chang, Chen-Chia, et al.
Published: (2024)

LAAG-RV: LLM Assisted Assertion Generation for RTL Design Verification
by: Maddala, Karthik, et al.
Published: (2024)

FVRuleLearner: Operator-Level Reasoning Tree (OP-Tree)-Based Rules Learning for Formal Verification
by: Wan, Lily Jiaxin, et al.
Published: (2026)

Annotating Slack Directly on Your Verilog: Fine-Grained RTL Timing Evaluation for Early Optimization
by: Fang, Wenji, et al.
Published: (2024)

VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation
by: Calzada, Paul E., et al.
Published: (2025)

Vectorization of Verilog Designs and its Effects on Verification and Synthesis
by: Guimarães, Maria Fernanda Oliveira, et al.
Published: (2026)

CktEvo: Repository-Level RTL Code Benchmark for Design Evolution
by: Shi, Zhengyuan, et al.
Published: (2026)

RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs
by: Jin, Pengwei, et al.
Published: (2025)

FVEval: Understanding Language Model Capabilities in Formal Verification of Digital Hardware
by: Kang, Minwoo, et al.
Published: (2024)

GSIM: Accelerating RTL Simulation for Large-Scale Designs
by: Chen, Lu, et al.
Published: (2025)

DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model
by: Liu, Yi, et al.
Published: (2025)

FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
by: Hooper, Coleman, et al.
Published: (2025)

RTL Verification for Secure Speculation Using Contract Shadow Logic
by: Tan, Qinhan, et al.
Published: (2024)

UVLLM: An Automated Universal RTL Verification Framework using LLMs
by: Hu, Yuchen, et al.
Published: (2024)

Automated Multi-Agent Workflows for RTL Design
by: Bhattaram, Amulya, et al.
Published: (2025)

Using LLMs to Facilitate Formal Verification of RTL
by: Orenes-Vera, Marcelo, et al.
Published: (2023)

Multimodal Chip Physical Design Engineer Assistant
by: Tsai, Yun-Da, et al.
Published: (2025)

SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity
by: Fan, Zichen, et al.
Published: (2025)

PipeRTL: Timing-Aware Pipeline Optimization at IR-Level for RTL Generation
by: Yin, Shuo, et al.
Published: (2026)