HWE-Bench

HWE-bench is a benchmark for evaluating coding agents on RTL and hardware bug-fixing tasks. It curates 417 real bug-fix cases from six open-source hardware projects (ibex, cva6, caliptra, rocket-chip, XiangShan and OpenTitan) across Verilog, SystemVerilog and Chisel. Each case is verified end-to-end via simulation fail-to-pass: the included test fails on the buggy commit and passes after the fix.

The evaluation runs through a Harbor-based harness with per-PR Docker images: generate task directories, run the agent, extract patches and score. A full run over all 417 cases typically takes around a day of wall-clock time with four concurrent containers. Generated tasks prohibit external access so agents cannot look up the upstream PR or patch during runs.

Results are published on an interactive leaderboard with overall and per-repository scores, file-level precision, estimated token-equivalent cost and scaffold filters. The README reports reference scores including GPT-5.5 xhigh at 79.4 percent and Claude Opus 4.7 max at 74.6 percent as of 2026-04-30. OpenTitan images are not distributed because that flow requires Synopsys VCS.