ChipBench
44 hierarchical Verilog modules, 89 injected-bug debug cases, and 132 reference-model tasks; the best model scores around 30% on generation.
What it does and how it connects
ChipBench is a benchmark for AI-aided chip design that evaluates LLMs across three tasks: Verilog generation, debugging and reference-model generation. It features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases and 132 reference-model samples across Python, SystemC and CXXRTL.
The paper positions it against saturated benchmarks: existing suites report state-of-the-art pass rates above 95 percent, while ChipBench exposes substantial performance gaps. State-of-the-art Claude-4.5-opus achieves only 30.74 percent on Verilog generation and 13.33 percent on Python reference-model generation.
To help improve reference-model generation, the paper provides an automated toolbox for high-quality training-data generation. The ChipAgents research page on Build-A-Chip-Bench independently notes ChipBench scores rising from 30.7 percent at release to 90.6 percent, with headroom nearly gone, which frames ChipBench as a stepping stone toward full-flow benchmarks.
Alternatives
Compare side by side- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification, rather than generation, debugging and reference-model generation across 44 hierarchical modules.
- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than 44 hierarchical modules with debug and reference-model tasks.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than 44 hierarchical modules with debug and reference-model tasks.
- HWE-Bench
417 real fail-to-pass bug-fix cases from six open-source hardware projects with a live leaderboard, rather than 44 hierarchical modules with injected-bug debug cases.
- Phoenix-bench
511 verified Verilator instances across 114 hardware repos with Docker-pinned EDA toolchains, rather than 44 hierarchical modules with debug and reference-model tasks.
Sources
Not verified
- The GitHub API classifies the repo license as NOASSERTION, while the LICENSE file text is the MIT License (copyright 2023-2024 NVIDIA Research Projects).
- Pricing: none stated (research benchmark).
- Real user voices: no real linked post read for this page.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.