CAD-bench
Executable benchmark scoring language model CAD agents on 17 functional mechanical part and assembly tasks, checking dimensions, mating geometry, and simulated motion in Blender instead of only whether the code runs.
What it does and how it connects
CAD-bench is a native Harbor benchmark for evaluating agents that construct mechanical parts and functional assemblies in Build123D. The public dataset holds 17 self-contained tasks. Each task asks the agent to create a single /workspace/final.py file that runs on Python 3.11 with Build123D 0.10.0 and leaves the completed model in a top-level variable named part. Harbor supplies the agent adapters, sandbox lifecycle, and result storage; the repo holds the tasks and the domain verifier.
Grading runs in Harbor's separate verifier mode with networking disabled. The verifier executes the submitted program, checks the resulting topology and geometry, and applies the task-specific deterministic or Blender rigid-body evaluation, writing reward.json (with reward, overall_score, task_score, and build_success channels) and a complete grading.json record. The aggregate metric first averages overall_score within each difficulty tier, then combines the tier means with weights 1 for easy, 2 for medium, 3 for hard, and 4 for insane; missing rewards count as zero.
A separate access-controlled seven-task functional refresh (two spur-gear transfers, three three-shaft idler transfers, and two compound right-angle transfers) exists for held-out evaluation and is intentionally absent from the public repository. No Harbor registry release exists yet, so runs use the local path. The paper and metadata files in the repo are retained research records, not benchmark infrastructure.
Alternatives
Compare side by side- FrontierCAD (Normal)
A CAD Arena benchmark that scores agents recreating real parts across several CAD tools from manufacturer drawings, rather than an executable Build123d benchmark of functional parts and assemblies graded by a deterministic verifier.
- PartForge evals
A standing evaluation rerun against each model release and published as model comparisons, rather than a public task dataset with an executable verifier that anyone can run locally.
- RealCADBench
A benchmark for intent-to-program modeling in FreeCAD Python from text, drawings, and product photos, rather than Build123d tasks graded on geometry and simulated motion.
- CADGenBench
A benchmark spanning CAD generation and editing tasks scored against reference geometry, rather than functional part and assembly tasks with Blender rigid-body checks.
- PCBench
A dataset of 164 KiCad boards with routing ground truth for measuring autorouter quality, rather than a mechanical CAD benchmark.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- In a CAD generation benchmark harness, a later model reply with no code block used to throw away a script the gate had already rejected. Generation scripts can now state expected extents and volume, and a valid export that disagrees goes back for repair; separately, build123d 0.13 pulls OCCT 8 which breaks analyze_step, so the README pins 0.12.0.Error Index
- After moving to OCP 8, strict STEP volume comparison failed on an F-14 empennage sample on both versions and a direct build123d STEP import produced two invalid solids, while the project's own reader called every prototype valid.Error Index
Sources
Not verified
- Pricing: none stated on the sources read.
- Real user voices: the repo had no public issues as of 2026-11-11 (open issue count 0), so no user voice is quoted.
- Builder identity beyond the CAD-bench GitHub organization is not stated on the repo.
Last verified November 11, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.