Phoenix-bench

Phoenix-bench asks whether agentic AI systems built for software engineering transfer to realistic hardware engineering. Existing hardware LLM benchmarks isolate sub-tasks, but none jointly requires repository navigation, hierarchy-aware localization, EDA executable verification and maintenance-style patching. Phoenix-bench is a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories, each shipped with the developer patch, design-flow labels, fail-to-pass and pass-to-pass testbenches, and a Docker-pinned EDA environment so resolved-rate differences reflect agent behavior rather than toolchain availability.

The paper runs a uniform evaluation of four commercial agents and eight open-source agentic structures across four LLM backbones, plus two diagnostic interventions: file-level oracle localization and one round of testbench-log feedback. Three findings emerge. Software and hardware are fundamentally different engineering tasks: the same agent loses 37 to 58 percent from SWE-bench Verified to Phoenix-bench, because hardware bugs propagate across parallel instantiated modules through signal flow rather than along a software-style call graph.

Failures concentrate on design control-flow and finite state machine bugs, verification testbench bugs, and hard cases demanding cross-hierarchy signal-flow tracking and coordinated multi-file edits. Localization granularity matters more than localization itself: a perfect file-level oracle yields only plus 1.4 percent, while a single round of test-case feedback lifts resolved rate by 42 to 45 percent, because the test case tells the agent where the bug is and what the fix has to look like.