specread
SpecRead: a 383-question benchmark for LLM specification comprehension in hardware design.
Summary rewritten from the vendor site because the catalog version did not match it.
What it does and how it connects
SpecRead is a 383-question benchmark that measures whether large language models comprehend hardware design specifications, independently of their ability to write RTL. Existing hardware-AI benchmarks score downstream artifacts (generated RTL, assertions, testbenches), and a model can fail those because it never understood the spec or because it understood it but cannot write correct Verilog; SpecRead separates the two, since every question is answerable from a quoted spec excerpt alone and every answer is auto-scorable.
Four task types: exact retrieval (58 questions), cross-section reasoning (58), mutated-spec contradiction detection (171), and spec-RTL consistency (96). Types 3 and 4 are the scientific focus: type-3 mutations are injected into real spec text (changed numbers, inverted rules, conflicting cross-references, deleted conditions), plus 41 distractor items that look suspicious but are fully self-consistent. All spec text and RTL come from the open-source OpenTitan project (uart, aes, aon_timer, hmac, i2c, kmac, pattgen, rv_timer, spi_device, spi_host), and 96 of 96 type-4 mutations are simulation-verified.
Headline result in the README: Ministral-3B scores 43.1 percent overall on the main set (v2.2, 165 of 383), with type scores of 55.2, 51.7, 40.4 and 35.4 percent; v2.2 scores are final after a gold-evidence audit removed 2 type-4 items and 93 gray-zone cases were double-adjudicated (Cohen's kappa 0.77). Raw model response records ship under eval/records so anyone can reproduce scoring. Code is Apache-2.0 and the dataset is CC-BY-4.0.
Alternatives
Compare side by side- veriflow-rtl-benchmark
A five-design golden-model harness that scores generated Verilog, rather than testing spec comprehension.
- llm-verilog-reliability-eval
A 30-question graded bank for Verilog generation reliability, rather than a spec-comprehension benchmark.
- llm-eval-ci-gate-rtl-verilog
A dual-tool CI gate for LLM-generated RTL, rather than a benchmark of spec comprehension.
- fail2bench
Turns agentic hardware design failures into new benchmark items, rather than testing spec comprehension on a fixed set.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: no real linked post read for this page; the repo has no public issues.
- No GitHub releases exist, so latest release fields are null.
- The catalog record says 385 questions; the README body, its task table (58 plus 58 plus 171 plus 96), and its dataset file all say 383, so 383 is used here and the one-liner was rewritten.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.