RealBench
60 module-level and 4 system-level tasks from real IP (AES, E203 RISC-V, SD controller), evaluated with Verilator plus JasperGold formal in Docker.
What it does and how it connects
RealBench is a benchmark for complex IP design tasks in real scenarios, with more complex tasks than module-level suites, system-level and module-level tasks, multi-modal data, and syntax, function and formal correctness verification. It draws on real IP including AES, the E203 RISC-V core and an SD controller.
Verification uses Verilator, with JasperGold formal verification available when installed, inside a Docker-based workflow. The documents are encrypted with GPG to prevent crawlers from scraping test data for direct training, and decryption is required before use. Solutions are scored with pass@k across samples per task.
The repository also provides a Docker-based workflow for evaluating LLM agents: it invokes Claude Code on each generated task, records trajectories, and evaluates the generated Verilog with the RealBench flow. The runner isolates against prompt injection by mounting only the task workdir, running as a non-root host user, disabling web search and fetch tools, and restricting outbound traffic to the configured API host.
Alternatives
Compare side by side- ChipBench
44 hierarchical Verilog modules, 89 injected-bug debug cases and 132 reference-model tasks, rather than 60 module-level and 4 system-level tasks from real IP.
- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification, rather than 60 module-level and 4 system-level tasks from real IP.
- HWE-Bench
417 real fail-to-pass bug-fix cases from six open-source hardware projects with a live leaderboard, rather than 60 module-level and 4 system-level tasks from real IP.
- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than 60 module-level and 4 system-level tasks from real IP.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than 60 module-level and 4 system-level tasks from real IP.
Sources
Not verified
- Pricing: none stated on the repo (open source benchmark).
- Real user voices: no real linked post read for this page.
- No GitHub releases exist, so latest release fields are null.
- The 60 module-level and 4 system-level task counts: directory record and paper; the README read lists the IP directories but does not state the counts.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.