VerilogEval
156 HDLBits-derived description-to-Verilog tasks in Machine and Human splits, verified by Icarus Verilog simulation with pass@k.
What it does and how it connects
VerilogEval is an evaluation harness for the VerilogEval problem-solving dataset from the paper of the same name, published at ICCAD 2023. In August 2024 the repository was revised to cover specification-to-RTL tasks in addition to the original code completion task, add in-context learning examples to prompts, and categorize common Icarus Verilog failures.
Version 2 reframes the prompts so the module interface is specified more abstractly instead of given explicitly, manages the dataset as plain text files, and adds generation and analysis scripts plus a Makefile workflow. The original v1.0 harness is preserved on the release/1.0.0 branch.
Verification is by Icarus Verilog simulation, with pass@1 reported at n=1 (temperature 0) and n=20 (temperature 0.85). The setup needs Icarus Verilog v12, Verilator and Python 3.11, and the README notes Icarus Verilog v13 is not supported.
Alternatives
Compare side by side- RTLLM 2.0
An open-source benchmark for design RTL generation from natural language with 50 designs, rather than a Verilog code generation benchmark verified by Icarus simulation.
- TuRTLe
A unified harness running VerilogEval, RTLLM, VGen, RTL-Repo and NotSoTiny under several simulators, rather than the VerilogEval benchmark alone.
- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification with hidden cocotb harnesses, rather than 156 HDLBits-derived description-to-Verilog tasks.
- RTL-Repo
1.17K single-line completion tasks testing RTL code completion across full repository context, rather than description-to-Verilog generation tasks.
- HLS-Eval
94 HLS C++ code-generation and code-edit designs with natural-language descriptions and testbenches, rather than description-to-Verilog tasks.
Sources
Not verified
- Pricing: none stated on the repo (open source benchmark).
- Real user voices: no real linked post read for this page.
- No GitHub releases exist, so latest release fields are null.
- The 156-task count and the Machine and Human splits in the one-liner: directory record and cited paper; the README read does not state the task count.
- The GitHub API classifies the repo license as NOASSERTION, while the LICENSE file text is the MIT License.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.