UK AISI chipbench_debug

chipbench_debug is a registration in the Inspect Evals library of a ChipBench Verilog-debugging evaluation, run under the Inspect AI framework. It is an external evaluation: the code lives in the upstream repository Plswearpants/Inspect-Eval-ChipBench at a pinned commit, and Inspect Evals lists it for discoverability.

ChipBench evaluates LLMs on AI-aided chip design across three tasks: Verilog generation (44 modules), Verilog debugging (89 cases), and reference model generation (132 cases in Python, SystemC, and CXXRTL). The debugging task injects four bug types (arithmetic, assignment, timing, state machine) into golden Verilog modules, and models receive buggy code plus a description, or additionally a VCD waveform in the one-shot variant.

Scoring uses iVerilog simulation against a golden reference, reporting pass@1, pass@5, and pass@10 over 20 samples. The eval runs from the command line with uv run inspect eval src/chipbench/chipbench.py@chipbench_debug, and the Inspect Evals project was founded in 2024 with contributions from the UK AI Security Institute.