rtlscout-bench
RTLScout benchmark suite with campaign tooling and a leaderboard site generator.
What it does and how it connects
rtlscout-bench is the benchmark suite, campaign tooling, and leaderboard site generator for RTL Scout. Each model drives the RTL Scout agent for a fixed number of steps on a task's starting design, with a self-checking testbench and a full synthesis flow in the loop; the score is the best verified cost the agent reached, and entries are ranked by mean rank or by a headroom-normalized score across tasks.
The suite (suite-v2, 2026-10-12) scores four designs by area times delay on ASAP7, Verilog entries only for now. Updating the leaderboard is two commands: python -m rtlscout_bench.campaign runs whatever the matrix still needs, and python -m rtlscout_bench.publish records, shows the table diff, asks, and pushes, with the page rebuilding and deploying by itself.
Code lives in this repository and run records, best designs, summaries, and transcripts live in the rtlscout-bench-data repository as a data submodule, so every published number can be traced and rebuilt. The live leaderboard is published at plex1.github.io/rtlscout-bench, and the page can be built locally from published data with no container and no API key.
Alternatives
Compare side by side- HW-Benchmarker
A benchmark for RTL debugging on real open-source cores, scored by re-running failing tests, rather than a campaign suite that scores agents on verified synthesis cost.
- deepee
A benchmark for coding agents on embedded firmware, code repair, schematics, and PCB work, rather than for RTL synthesis cost.
- cadclamp
A benchmark that scores AI-generated CAD on whether the part can be printed, rather than on RTL design cost.
- MARB
A benchmark that grades AI-assisted CAD assembly placement, rather than RTL design cost.
Sources
Not verified
- Pricing: none stated on the repo.
- No GitHub releases exist, so latest release fields are null.
- Real user voices: the repo's only open item is a pull request by the maintainer, and no independent user post was found.
- Related Error Index entries: the live error map has no RTL Scout entries.
- A named individual builder beyond the repository owner handle plex1
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.