RTL-Repo
1.17K single-line completion tasks testing RTL code completion across full repository context.
What it does and how it connects
RTL-Repo is a benchmark for evaluating LLMs on generating Verilog code autocompletions within large, complex codebases. It assesses the model's ability to understand and remember the entire Verilog repository context and generate new code that is correct, relevant, logically consistent and adherent to coding conventions, while being aware of all components and modules in the project.
It comprises over 4,000 code samples from GitHub repositories, each containing the context of all Verilog code in the repository. The test set is 1,174 single-line completion tasks, matching the total count in the README's example evaluation output. The dataset is also available on HuggingFace Hub.
Evaluation reports edit similarity and exact match. The README leaderboard puts GPT-4 at 71.87 edit similarity and 48.5 exact match, ahead of GPT-3.5, RTLCoder and StarCoder2. The paper was accepted to IEEE LAD 2024, the first IEEE International Workshop on LLM-Aided Design.
Alternatives
Compare side by side- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than single-line completions across full repository context.
- TuRTLe
A unified harness running VerilogEval, RTLLM, VGen, RTL-Repo and NotSoTiny under several simulators, rather than the RTL-Repo benchmark alone.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than single-line completions across full repository context.
- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification, rather than single-line completions across full repository context.
- PyHDL-Eval
168 spec-to-RTL problems for Python-embedded HDL DSLs plus Verilog, rather than single-line Verilog completions across full repository context.
Sources
Not verified
- Pricing: none stated on the repo (open source benchmark).
- Real user voices: no real linked post read for this page.
- No GitHub releases exist, so latest release fields are null.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.