Benchmarking LLMs for Verilog Design Flows
Reproducible platform scoring open source language models on 50 curated Verilog tasks across combinational, sequential, FSM, and mixed designs, validating output through Verilator, Icarus simulation, waveform analysis, and formal equivalence checks.
What it does and how it connects
This is a reproducible benchmarking platform that scores open-source language models on Verilog RTL generation across 50 curated tasks: 23 combinational, 14 sequential, 8 finite state machine, and 5 mixed designs. The pipeline uses constrained prompting, post-processing, and semantic-aware iterative refinement with waveform analysis, formal equivalence verification, and AST-based repair.
Generated code is validated through Verilator compilation and Icarus Verilog simulation, and the README lists the metrics computed in every run: syntax validity, functional correctness, generation time, compile time, and simulation time. Across 12 benchmarks and 1,610 total runs over three models, Llama-3-8B, StarCoder2-7B, and TinyLlama-1.1B, the pipeline improved syntax validity from 0 percent to a 70.43 percent average and the simulation pass rate to 51.8 percent.
The README is explicit about what was not measured: synthesis quality, testbench detection rate, prompt sensitivity, hallucination index, and usability score were defined but not computed in any benchmark run. The platform and dataset are open source, and the paper states they are publicly available at the linked repository. The paper was submitted to arXiv 2026-07-23.
Alternatives
Compare side by side- VerilogEval
The 156-task English benchmark, while this platform scores 50 curated tasks with toolchain validation and iterative repair.
- RTLLM 2.0
A 50-problem Verilog benchmark, while this platform adds waveform analysis, formal equivalence, and AST-based repair across 12 benchmark runs.
- Multi-VerilogEval
A multilingual benchmark of 156 tasks in four languages, while this platform validates 50 tasks end to end through Verilator, Icarus, and formal checks.
- llm-verilog-reliability-eval
A reliability evaluation for Verilog generation, while this platform benchmarks Verilog RTL generation with toolchain validation and repair.
Sources
Not verified
- Pricing (not applicable; the platform and dataset are open source, and no price is stated)
- Real user voices (no real linked post read for this page)
- Related Error Index entries (no Verilog entries in the live error map)
- No GitHub releases exist, so latest release fields are null
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.