CVDP

CVDP, the Comprehensive Verilog Design Problems benchmark, is a framework for evaluating LLM and agent solutions on hardware verification challenges. It includes 783 problems across 13 task categories, covering RTL generation, verification, debugging, specification alignment and technical Q and A, authored by experienced hardware engineers. Problems are offered in both non-agentic and agentic formats, and the paper reports state-of-the-art models achieving no more than 34 percent pass@1 on code generation.

Evaluation runs against hidden cocotb harnesses. The open-source simulation image bundles cocotb 2.0.1, pytest, Icarus Verilog, Yosys and Verilator; a commercial dataset variant needs Cadence Xcelium and a license network. Both a direct LLM workflow and a Docker-based agentic workflow are supported, with multi-sample pass@k evaluation.

The dataset is published on HuggingFace. The README notes 20 datapoints were omitted from the initial public release for test harness issues and licensing restrictions, and reference solutions were excluded to mitigate data contamination. The Silicon Integration Initiative (Si2) is adopting CVDP into its LLM Benchmarking Coalition, which extends the problems, refines metrics and maintains a leaderboard.