llm-verilog-reliability-eval
Reliability evaluation platform for LLM-assisted Verilog generation: graded question bank, self-checking testbenches and automated scoring. Docs are in Chinese.
What it does and how it connects
This is a reliability evaluation platform for LLM-assisted Verilog generation, published alongside the paper on reliability evaluation experiment design for LLM-generated Verilog code. The question bank holds 30 questions in three tiers (L1 combinational logic, L2 sequential logic, L3 system level, 10 each), all implemented, each with two versions of the functional specification, a golden reference model, a testbench, mutants and an equivalent-mutant list; testbench mutation scores are 100 percent. L2 and L3 questions also carry a weak testbench (mutation score 56 to 85 percent) and escape samples.
The evaluation platform runs interface checks, Verilog-2001 compilation and lint, functional simulation with a 60-second timeout, synthesis (Yosys pre-screen, Vivado as authority), nine-category error classification, a repair loop of up to 3 rounds, and SQLite storage feeding the paper's tables and figures. Simulation defaults to Icarus Verilog, with Verilator for lint; Yosys and Vivado are optional.
Results are reproducible through cached LLM responses, fixed random seeds, deterministic mutant generation and an offline golden-oracle self-test. The data folder ships the paper's full Section 4 evaluation results (DeepSeek-V4-Pro, Qwen3-Coder-Plus, GLM-5 and Qwen3-Coder-30B-A3B, 2026-10-02 to 2026-10-04, 3,300 samples) as JSONL and CSV snapshots, so the paper's tables 5 to 8 and figures 3 to 6 can be recomputed directly. The README and docs are in Chinese, and the description above is paraphrased from that text.
Alternatives
Compare side by side- veriflow-rtl-benchmark
A five-design golden-model harness for AI-written Verilog, rather than a 30-question graded bank with a repair loop.
- llm-eval-ci-gate-rtl-verilog
A dual-tool CI gate that requires static analysis and simulation to agree, rather than a graded question bank with mutation scoring.
- fail2bench
Turns agentic hardware design failures into new benchmark items, rather than running a fixed graded Verilog bank.
- specread
Tests whether models comprehend hardware specifications, rather than whether their generated Verilog works.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: no real linked post read for this page; the repo has no public issues.
- No GitHub releases exist, so latest release fields are null.
- The README and docs are in Chinese; the description above is paraphrased from that text.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.