TuRTLe
Unified harness and live leaderboard running VerilogEval v2, RTLLM v2, VGen, RTL-Repo, and NotSoTiny (built on Tiny Tapeout designs) under Icarus, Verilator, Yosys, and OpenLane.
Summary rewritten from the vendor site because the catalog version did not match it.
What it does and how it connects
TuRTLe is a framework to assess LLMs across RTL generation tasks systematically, from the High Performance Artificial Intelligence group at the Barcelona Supercomputing Center. It integrates multiple existing benchmarks and automates the evaluation process: VerilogEval v2.0 (spec-to-RTL and module completion), RTLLM v1.1 and v2.0 (spec-to-RTL), VGen (module completion), RTL-Repo (single-line completion) and NotSoTiny (module completion, built on Tiny Tapeout designs).
It scores syntax correctness, functional correctness, synthesis, PPA optimization and exact line completion. The EDA tooling is Icarus Verilog and Verilator for syntax and functionality, Yosys for synthesis and equivalence, and LibreLane (OpenROAD) for PPA. Evaluation runs through a Docker image with all the EDA tooling bundled, and inference supports API-based calls to any OpenAI-compatible endpoint or local and cluster inference with vLLM and Ray.
A live leaderboard on HuggingFace Spaces shows the best open-source models per task.
Alternatives
Compare side by side- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than a unified harness across five benchmarks.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than a unified harness across five benchmarks.
- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification with hidden cocotb harnesses, rather than a unified harness across five benchmarks.
- RTL-Repo
1.17K single-line completion tasks testing RTL code completion across full repository context, rather than a unified harness across five benchmarks.
- HWE-Bench
417 real fail-to-pass bug-fix cases from six open-source hardware projects with a live leaderboard, rather than a unified harness across five generation benchmarks.
Sources
Not verified
- Pricing: none stated on the repo (open source framework).
- Real user voices: no real linked post read for this page.
- No GitHub releases exist, so latest release fields are null.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.