AutoBench
Evaluation framework and dataset for automatically grading LLM-generated Verilog testbenches, using scenario checking, standardization metrics, and debug-and-reboot criteria across the full hybrid testbench generation pipeline.
What it does and how it connects
AutoBench is an evaluation framework and dataset for automatically grading LLM-generated Verilog testbenches, using a hybrid testbench structure and a self-checking system. Given only the description of the design under test, it generates comprehensive testbenches and grades them through its Eval0, Eval1, Eval2 and Eval2b framework, covering scenario checking, standardization metrics and debug-and-reboot criteria.
The dataset extends HDLBits data and ships in the repository, with experimental results linked from Google Drive. The README recommends the successor, CorrectBench, since November 2024, because it fixes bugs in AutoBench and adds self-validation and self-correction, improving the generated testbench pass ratio to 70.13 percent on GPT-4o against AutoBench's 52.18 percent and a 33.33 percent baseline.
It runs on Python 3.8 or newer with Icarus Verilog and an OpenAI or Anthropic API key, and was awarded the Best Artifact award at MLCAD 2024. The project is MIT licensed.
Alternatives
Compare side by side- RTL-BenchLS
A 10,028-design benchmark for RTL reasoning and generation, rather than a framework that grades LLM-generated testbenches.
- GenAI EDA Workflows Bench
A 50-task RTL generation platform, rather than a framework that grades LLM-generated testbenches.
- LintLLM
A linting framework for Verilog defect detection, rather than a framework that grades LLM-generated testbenches.
- PartForge evals
An eval suite for verification workflows, rather than a testbench grading framework.
- Rapidflare suite
Rapidflare's internal eval practice for AI agents in electronics, rather than a testbench grading framework.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: no real linked post read for this page; the repo's only issue is a typo fix by a co-author.
- The README recommends the successor CorrectBench since 2024-11-13, which fixes bugs in AutoBench and adds self-validation and self-correction.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.