BenchCAD
17,900 execution-verified CadQuery programs over 106 industrial part families, scored by deterministic voxel-IoU with a live leaderboard.
What it does and how it connects
BenchCAD is a benchmark for evaluating LLMs and multimodal models on programmatic CAD. It is built on 17,900 execution-verified CadQuery programs across 106 industrial part families such as bevel gears, compression springs, twist drills and threaded adapters, drawn from 47 engineering standards including ISO, DIN, EN, ASME and IEC.
It decomposes model ability into four tasks: Vision2Code (rendered views to a CadQuery program), CodeEdit (instruction-guided edits to an existing program), Vision-QA and Code-QA (numeric questions answered from rendered views or from code). Scoring is execution-grounded and deterministic with no LLM judge: generation tasks grade by voxel IoU between the model's executed STEP solid and the ground-truth STEP, and QA tasks grade by symmetric ratio accuracy on numeric answers.
The dataset is hosted on HuggingFace and pulled on demand, with a one-command runner, pinned environment and a live leaderboard. The README states that contributed model results are re-graded from raw predictions and numbers are never self-reported.
Alternatives
Compare side by side- MUSE
Grades text-to-CAD in a three-stage funnel ending in a rubric-based VLM judge on functionality, manufacturability and assemblability, rather than scoring CadQuery programs by deterministic voxel IoU with no LLM judge.
- Parametric CAD Bench (gNucleus)
Agents author editable FreeCAD Part Design models in a multi-step loop, scored by a harmonic mean of geometry and spec consistency, rather than single-shot CadQuery generation scored by voxel IoU.
- CADBench (Doris et al.)
A unified 18,000-sample benchmark turning mesh, image or render inputs into STEP or mesh and scoring volumetric IoU, rather than an industry-standard CadQuery benchmark across 106 part families.
- Text2CAD-Bench
Tests text-to-parametric-CAD on 600 human-curated examples across four complexity levels, rather than execution-verified CadQuery programs across 106 industrial part families.
- NURBGen
Generates high-fidelity CAD models from text using NURBS surfaces, rather than scoring CadQuery programs by deterministic voxel IoU.
Sources
Not verified
- Pricing: none stated on the repo (open source benchmark; dataset is public).
- Real user voices: no real linked post read for this page; the open issues read are model-result submissions to the leaderboard, not reviews of the benchmark.
- No GitHub releases exist, so latest release fields are null.
- The GitHub API classifies the repo license as NOASSERTION; the README license section states the code is MIT and the dataset is CC BY 4.0.
- Related Error Index entries: no matching entry verified.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.