HW-Benchmarker
Industry-grade benchmark for evaluating RTL debugging capabilities.
What it does and how it connects
HW-Benchmarker is a benchmark for evaluating RTL debugging capabilities on real open-source cores, scored by simulation. Its STEAD-Bench turns a bug on a production core into a case (the failing test, its logs and waveforms, and a STEAD record naming the first incorrect signal, the time it went wrong, and the expected and actual values); the tool receives the defective tree and returns ranked candidate lines and a patch, and the harness scores both by re-running the test in a container.
Six cores are supported: scr1, ibex, rocket-chip, cva6, caliptra-rtl, and openc910, each distributed as a Docker image containing its toolchain and a warm build. LLMs, ChipAgents, Bronco AI, CRUX, and Walker are evaluated on identical cases. The current case set is cases-v3 with 47 cases, published on the releases page.
Requirements are Linux with Docker and BuildKit or Apple Silicon macOS with Docker Desktop or OrbStack, Python 3.10 or newer, about 20 GB of disk for all six cores, and an API key for each provider evaluated. Cases and gold files are not in version control; they are derived from specs and images or downloaded as pre-baked archives.
Alternatives
Compare side by side- rtlscout-bench
A benchmark suite where an LLM agent pushes a hardware design through a self-checking testbench and synthesis flow, scored on verified cost, rather than a debug benchmark scored by re-running failing tests.
- deepee
A benchmark for coding agents on embedded firmware, code repair, schematics, and PCB work, rather than for RTL debugging.
- cadclamp
A benchmark that scores AI-generated CAD on whether the part can be printed, rather than on RTL debugging.
- MARB
A benchmark that grades AI-assisted CAD assembly placement, rather than RTL debugging.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: the repo has 0 open issues, and no independent user post was found.
- Related Error Index entries: the live error map has no RTL debugging benchmark entries.
- A named individual builder beyond the repository owner handle abhimanyumagapu
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.