PyHDL-Eval
168 spec-to-RTL problems for Python-embedded HDL DSLs (PyMTL3, PyRTL, MyHDL, Migen, Amaranth) plus Verilog.
What it does and how it connects
PyHDL-Eval is an LLM evaluation framework for hardware design using Python-embedded DSLs, presented at the 6th ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD 2024) by Christopher Batten, Nathaniel Pinckney, Mingjie Liu, Haoxing Ren and Brucek Khailany of Cornell and NVIDIA.
It is a new evaluation benchmark with 168 specification-to-RTL problems covering PyMTL3, PyRTL, MyHDL, Migen and Amaranth, plus Verilog. Each problem includes a prompt, a Verilog reference, a Verilog testbench and a Python test script.
The slides report results across models including Code Gemma 7B, LLaMA3 8B and 70B, GPT-4 and GPT-4 Turbo, and show Verilog generations scoring well ahead of the Python-embedded DSLs for the models tested. The framework treats the Python DSLs as productive alternatives for RTL design and multilevel modeling, and measures whether LLMs can use them as effectively as Verilog.
Alternatives
Compare side by side- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than spec-to-RTL problems for Python-embedded HDL DSLs.
- HLS-Eval
94 HLS C++ code-generation and code-edit designs with natural-language descriptions and testbenches, rather than spec-to-RTL problems for Python-embedded HDL DSLs.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than spec-to-RTL problems for Python-embedded HDL DSLs.
- TuRTLe
A unified harness running VerilogEval, RTLLM, VGen, RTL-Repo and NotSoTiny under several simulators, rather than a Python-DSL evaluation framework.
- RTL-Repo
1.17K single-line completion tasks testing RTL code completion across full repository context, rather than spec-to-RTL problems for Python-embedded HDL DSLs.
Sources
Not verified
- License: no license is stated in the sources read.
- A public repo or dataset link: the catalog URL is the MLCAD 2024 slides PDF; no repository URL was verified.
- Pricing: none stated (research benchmark).
- Real user voices: no real linked post read for this page.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.