HSCO-Bench
End to end hardware software co-design benchmark where LLM agents pick kernels to accelerate, design and integrate heterogeneous accelerators into an SoC under resource constraints, map kernels onto them, and produce prototypes deployed on an AMD Virtex-7 FPGA, scored on valid deployment, speedup and resource use.
What it does and how it connects
HSCO-Bench is an end-to-end hardware-software co-design benchmark for accelerator-rich heterogeneous SoC generation. It is built on the ESP open-source SoC platform, and it challenges LLM agents to analyze applications to identify kernels requiring acceleration, design and integrate heterogeneous accelerators into a System-on-Chip under resource constraints, and map kernels onto the generated accelerators.
The benchmark curates 10 applications spanning machine learning and signal processing, and produces SoC prototypes deployed on the AMD Virtex-7 FPGA VC707 Evaluation Kit. The final designs are synthesized with commercial HLS tools and deployed to the FPGA for physical verification. The repo README states the benchmark requires the ESP framework, Catapult HLS and the MatchLib toolkit, plus API keys for the LLM providers it invokes.
Among the five frontier models evaluated, only two could successfully generate valid SoC prototypes, and even those designs were far from optimal. The paper reports a promising peak speedup of 16.22X, while the maximum additional resource utilization reaches only 23.67 percent. The paper also reports that Opus 4.6 achieves a 70 percent average end-to-end success rate and a geometric-mean speedup of 2.32X.
Alternatives
Compare side by side- CLOSER-Bench
A budgeted cross-stage design closure protocol pairing spec-to-RTL, RTL-to-GDS and spec-to-GDS tasks, rather than an end-to-end hardware-software co-design benchmark.
- ArchXBench
A hierarchical RTL synthesis benchmark from simple gates to full subsystems, rather than an end-to-end hardware-software co-design benchmark.
- AgenticBBO-Bench
A cross-domain benchmark for LLM agents doing black-box optimization under a fixed budget, rather than an end-to-end hardware-software co-design benchmark.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- The synthesis check re-hashes RTL and liberty but records no hash of the netlist it produced, so it cannot catch a committed gate-level netlist that was hand-edited or went stale after resynthesis. It also needs the PDK on the checking host, so signoff CI skips it.Error Index
- agent wrappers for commercial RTL-to-GDS tools shipped with two command names that were not real binaries (conformal is lec, starrc is StarXtract), a timeout left a pt_shell child holding a licence, and result markers come from documented log formats rather than real logs, with unmeasured parts labeled pending.Error Index
Sources
Not verified
- License: the GitHub API reports NOASSERTION, but the LICENSE file is the Creative Commons Attribution 4.0 International License, copyright 2026 Columbia University, System Level Design Group, so CC BY 4.0 is recorded from the file. The catalog label source-available does not match the file.
- Builder: the catalog names IBM Research and Columbia University, but the abstract page read lists only the authors, so the builder is recorded as the lead authors; the LICENSE copyright supports the Columbia affiliation.
- Pricing: none stated on the pages read.
- No independent user voice found: the sources read are the paper and the repo.
- No GitHub releases exist, so latest release fields are null.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.