HardSecBench
924 tasks over Verilog RTL and firmware C covering 76 hardware-relevant CWEs with executable tests, scoring secure code generation.
What it does and how it connects
HardSecBench is a benchmark for assessing the security awareness of LLMs under realistic specifications for hardware code generation. Existing studies mostly evaluate functional correctness and overlook security, while LLM-generated code that looks functionally sound can embed security flaws with catastrophic consequences after deployment.
The benchmark has 924 tasks spanning Verilog RTL and firmware-level C, covering 76 hardware-relevant Common Weakness Enumeration (CWE) entries. Each task includes a structured specification, a secure reference implementation and executable tests, so evaluation is grounded in execution evidence.
To automate artifact synthesis, the paper proposes a multi-agent pipeline that decouples synthesis from verification. Evaluating diverse LLMs, it finds they often satisfy functional requirements while leaving security risks, and that security results vary with prompting.
Alternatives
Compare side by side- HWE-Bench
417 real fail-to-pass bug-fix cases from six open-source hardware projects with a live leaderboard, rather than a security-awareness benchmark over RTL and firmware C.
- Phoenix-bench
511 verified Verilator instances across 114 hardware repos with Docker-pinned EDA toolchains, rather than a security-awareness benchmark over RTL and firmware C.
- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification, rather than a security-awareness benchmark over RTL and firmware C.
- ChipBench
44 hierarchical Verilog modules, 89 injected-bug debug cases and 132 reference-model tasks, rather than a security-awareness benchmark over RTL and firmware C.
- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than a security-awareness benchmark over RTL and firmware C.
Sources
Not verified
- Pricing: none stated (research benchmark).
- Real user voices: no real linked post read for this page.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.