RTL-BenchLS
Large-scale RTL benchmark with over 10,000 formally verified Verilog designs that tests specification-to-RTL generation plus round-trip, masked-content, and repository-issue reasoning, all scored by formal equivalence checking with no manual testbenches.
What it does and how it connects
RTL-BenchLS is a benchmark for evaluating large language models on RTL reasoning and generation. It ships 10,028 formally verified Verilog designs, drawn from more than 25 open-source repositories and four augmented RTL corpora, and scores every task by formal equivalence checking, with no manual testbenches.
Beyond specification-to-RTL generation it defines three tasks: round-trip reasoning, where a design is compressed into an intermediate representation and reconstructed as equivalent RTL; masked-content reasoning, where masked logic is inferred from surrounding context; and repository-issue reasoning, where real bugs from GitHub issue reports are diagnosed and fixed. The paper reports that the best of eight evaluated models reaches only 23 percent on natural-language round-trip reasoning, 28 percent on masked-content reasoning and 12 percent on repository-issue fixing.
The repository ships the datasets as JSONL, per-model results for six models across the three tasks, a pip-installable evaluation package, and example runners for each task. Datasets under data/ and results/ are CC-BY-4.0 while the evaluation framework and example code are MIT. Formal verification hooks target Cadence Conformal, JasperGold or Synopsys Formality over local, SSH-remote or Docker setups.
Alternatives
Compare side by side- GenAI EDA Workflows Bench
A 50-task Verilog generation benchmark scored on lint, simulation and synthesis, rather than a 10,028-design benchmark scored by formal equivalence.
- LintLLM
A benchmark for LLM-based Verilog defect detection with mutation-injected faults, rather than a reasoning and generation benchmark.
- AutoBench
Grades LLM-generated Verilog testbenches, rather than grading RTL reasoning and generation itself.
- PartForge evals
An eval suite for verification workflows, rather than a Verilog design benchmark.
- Rapidflare suite
Rapidflare's internal eval practice for AI agents in electronics, rather than a Verilog design benchmark.
Sources
Not verified
- Pricing: none stated on the repo or paper page.
- Real user voices: no real linked post read for this page; the repo has no public issues.
- No GitHub releases exist, so latest release fields are null.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.