ArchXBench
Hierarchical RTL synthesis benchmark suite running from simple gates up to full subsystems like AES cores and streaming FFT pipelines, grading LLM generated datapath heavy designs on architectural complexity and quality of results rather than single module correctness alone.
What it does and how it connects
ArchXBench (v1.5) is an LLM benchmark primarily designed for testing the capability to generate functionally correct modules written in Verilog HDL. The repo holds a hierarchical list of 71 problems in order of increasing difficulty, from simple combinational circuits to complex domain-specific hardware from cryptography, image processing and other domains: level 0 with 20 logic building blocks, level 1a, 1b and 1c with 6, 5 and 7 problems, levels 2 through 5 with 6 each, and level 6 with 9, including AES encryption and decryption and a streaming 64-point FFT.
Each problem directory holds a natural-language problem description, a design spec with the exact Verilog module signature to implement, and a testbench that drives the design under test and checks outputs. For the lower levels the testbench is run with Icarus Verilog and returns PASS or FAIL; higher levels wrap the pipeline in Makefiles that run golden model, compile, run and compare in one shot. The provided Dockerfile bundles Icarus Verilog and the Python dependencies for every problem, and the toolchain also includes Octave for reference scripts in levels 4 and 6.
Version 1.5 leaves the task set and complexity hierarchy unchanged from v1.0 and corrects and hardens the specifications and testbenches after a full audit of every benchmark across all seven levels: testbench correctness fixes, previously undisclosed benchmark data such as FIR filter coefficients now specified explicitly, golden-model and testbench consistency fixes, broken build and file references fixed, and scoring-script hardening. The repo opens with a citation for the MLCAD 2025 paper ArchXBench: A Complex Digital Systems Benchmark Suite for LLM Driven RTL Synthesis.
What engineers report
The high-pass FIR filter problem is missing reference coefficients, so the design cannot be validated against the provided golden output.
2026-01-23Read the originalThe testbench for the 16-point iterative FFT cannot load its input data because the stimuli files are missing or misnamed, and it overwrites the input buffer.
2026-01-23Read the original
Alternatives
Compare side by side- CLOSER-Bench
A budgeted cross-stage design closure protocol for hardware agents, rather than a hierarchical RTL synthesis benchmark.
- HSCO-Bench
An end-to-end hardware-software co-design benchmark deploying SoC prototypes on an FPGA, rather than a hierarchical RTL synthesis benchmark.
- AgenticBBO-Bench
A cross-domain benchmark for LLM agents doing black-box optimization under a fixed budget, rather than a hierarchical RTL synthesis benchmark.
- Phoenix-bench
511 verified Verilator instances across 114 hardware repos with Docker-pinned EDA toolchains, rather than a hierarchical RTL synthesis benchmark.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- The synthesis check re-hashes RTL and liberty but records no hash of the netlist it produced, so it cannot catch a committed gate-level netlist that was hand-edited or went stale after resynthesis. It also needs the PDK on the checking host, so signoff CI skips it.Error Index
- agent wrappers for commercial RTL-to-GDS tools shipped with two command names that were not real binaries (conformal is lec, starrc is StarXtract), a timeout left a pt_shell child holding a licence, and result markers come from documented log formats rather than real logs, with unmeasured parts labeled pending.Error Index
Sources
Not verified
- Pricing: none stated on the repo.
- One-liner note: the README frames grading as functional correctness checked by testbenches returning PASS or FAIL; the architectural complexity and quality-of-results framing in the one-liner comes from the catalog and the MLCAD paper, not the README text read.
- The last push to the repo is 2026-07-26, so the repo has not been pushed in over three months.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.