BenchCAD 2.0
100 held out engineering cases where models build CAD parts from dimensioned drawings, assemble machines from supplied or newly modeled parts, and recover board wiring from renders, scored automatically against references with no LLM judge.
What it does and how it connects
BenchCAD 2.0 extends the original part-level benchmark to more complex tasks. Models build CAD parts from engineering drawings, assemble machines using supplied or newly modeled components, and recover circuit wiring from renders of assembled boards. The blog post, dated 2026-10-12, frames it as a benchmark that takes professional expertise in hardware engineering: real work comes as dimensioned drawings, machines assembled from dozens of parts, and existing hardware to reverse-engineer.
There are six tasks: drawing to CAD for a part and for assemblies, reverse engineering a part and an assembly from rendered views, and ECAD, recovering a wiring graph from six renders of an assembled board. Core, the 100 cases behind the reported scores, is held out and shared only under an evaluation-only agreement, to limit contamination. A separate public set of 100 open cases exists for trying the harness, and scores on it are not reportable.
Every answer is scored automatically against its reference, without an LLM judge. A part gets 0.8 for 3D overlap and 0.2 for topology, on its own bounding box; an assembly is also scored on where each part sits; a board is scored on how well its wiring graph matches the reference connections. Models run in an offline sandbox with Python's standard library, CadQuery, NumPy, SciPy and an image-cropping tool, with up to 30 rounds per case under harness v2.0.
Alternatives
Compare side by side- Bench
AI agents that run engineering design workflows end to end across existing CAD, CAE and PLM tools, rather than a held-out benchmark scoring models on 100 engineering cases.
- CADTestBench
A test-based benchmark pairing prompts with executable CADTests, rather than a held-out benchmark of drawings, assemblies and board wiring.
- RealCADBench
A benchmark for intent-to-program CAD modeling from industrial design intents, rather than a held-out benchmark of drawings, assemblies and board wiring.
- CADWorld
A 200-task benchmark for computer-use agents working through FreeCAD screenshots and GUI actions, rather than a held-out benchmark scored on geometry and wiring graphs.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- A committed STEP file quietly stopped matching its builder code: the shell referenced faces up to 529 but the data ended at 17, importing as an invalid zero-volume shell.Error Index
- Raw LLM-driven KiCad agents cannot be trusted to generate board and schematic files directly, because the file formats contain implicit relationships the model does not understand. The safe architecture is: the model proposes intent, deterministic tools execute, and a verification engine produces evidence.Error Index
Sources
Not verified
- License: no LICENSE file is in the harness repo and the README states no license, so no license is recorded; the catalog label source-available is unverified.
- Pricing: none stated on the pages read.
- No independent user voice found: the sources read are the vendor blog, the harness repo and the dataset page.
- Model scores on the blog (GPT-6 Astra 0.59, Claude Opus 5.5 0.57 at max effort) are the vendor's own reported results and are not repeated as fact in this entry.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.