AssertionBench
100 curated OpenCores Verilog designs with formally verified assertions, scoring correctness of LLM-generated SVA.
What it does and how it connects
AssertionBench is a benchmark to evaluate LLMs on assertion generation quantitatively. It contains 100 curated Verilog hardware designs from OpenCores with formally verified assertions for each design, generated from GoldMine and HARM assertion miners. Assertions are the collateral for simulation-based and formal verification, and their quality determines detection and diagnosis of corner-case design bugs.
The benchmark compares state-of-the-art LLMs including GPT-3.5, GPT-4o, CodeLLaMa 2 and LLaMa3-70B, using 1-shot and 5-shot learning, to assess their effectiveness in inferring functionally correct assertions. Model responses are saved in JSON format, with syntax errors corrected using GPT-3.5.
Generated assertions are formally verified using JasperGold 2022.06, following the same process used to verify the assertions mined by GoldMine and HARM. IVerilog compiles and runs the Verilog designs. The paper, by Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan and Debjit Pal, was accepted at NAACL 2025.
The catalog repo 3brahimi/assertion_data_for_llm is a fork of achieve-lab/assertion_data_for_LLM (created 2026-02-04, 0 stars); the stats reflect the fork, not the upstream repo.
Alternatives
Compare side by side- CVDP
783 engineer-authored problems across 13 categories covering RTL design and verification, including assertion creation, rather than a dedicated assertion-generation benchmark.
- VerilogEval
156 HDLBits-derived description-to-Verilog tasks verified by Icarus Verilog simulation, rather than a dedicated assertion-generation benchmark.
- HWE-Bench
417 real fail-to-pass bug-fix cases from six open-source hardware projects, rather than a dedicated assertion-generation benchmark.
- RTLLM 2.0
50 realistic natural-language-described designs scored on syntax, functionality, synthesis and PPA, rather than a dedicated assertion-generation benchmark.
- TuRTLe
A unified harness running VerilogEval, RTLLM, VGen, RTL-Repo and NotSoTiny under several simulators, rather than a dedicated assertion-generation benchmark.
Sources
Not verified
- License: no LICENSE file is in the repo and the GitHub API returns no license, so no license is stated.
- Pricing: none stated on the repo (open research data and code).
- Real user voices: no real linked post read for this page.
- No GitHub releases exist, so latest release fields are null.
- Related Error Index entries: no matching entry verified.
- The catalog repo is a fork (3brahimi/assertion_data_for_llm, created 2026-02-04, 0 stars) of the upstream achieve-lab/assertion_data_for_LLM repository.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.