Parametric CAD Bench (gNucleus)
Agents author editable FreeCAD Part-Design models from NL specs, scored by a geometry-by-spec-consistency harmonic mean on a live leaderboard.
What it does and how it connects
Parametric CAD Bench measures whether AI agents can author editable FreeCAD models from natural language. For each task the agent receives a natural-language part description and a table of key dimensions, and must produce a script that, when executed by FreeCAD, creates a saved CAD document for that part. It uses FreeCAD's Part Design workbench, so the result is a feature-based model with sketches, named operations and driven dimensions.
Scoring is a harmonic mean of two sub-scores computed by the open-source gNucleus freecad-validator: geometry similarity against a held-back reference design, and cad-spec consistency, which checks whether each key parameter in the spec exists in the saved CAD file as a named, driven dimension within tolerance. The harmonic mean is an editability gate: a model that produces perfect geometry but zero named parameters scores 0, not 0.5.
The benchmark runs a multi-step agentic loop, not a single-shot prompt: agents run a shell, write the FreeCAD script, execute it, inspect errors, fix them and retry inside a sandboxed container. The leaderboard crosses four agent frameworks with frontier models in 10 cells; GPT-5.5 via Codex leads at 0.832, and the same model on a vendor-neutral harness scores 0.744, a measured harness effect. Per-cell spend ranges from $3 to $170 across 100 trials each.
Alternatives
Compare side by side- BenchCAD
Scores execution-verified CadQuery programs by deterministic voxel IoU with no LLM judge, rather than grading editable FreeCAD models by a harmonic mean of geometry and spec consistency.
- MUSE
Grades text-to-CAD assemblies in a three-stage funnel ending in a VLM judge, rather than grading editable FreeCAD Part Design models by geometry and named-parameter consistency.
- LFORLA FreeCAD Fix
Agents repair a broken parametric FreeCAD script, scored by a deterministic geometry oracle, rather than authoring a new editable FreeCAD model from a natural-language spec.
- CADBench (Doris et al.)
A unified 18,000-sample benchmark turning mesh, image or render inputs into STEP or mesh, rather than a multi-step agentic loop authoring editable FreeCAD documents.
- Text2CAD-Bench
Tests text-to-parametric-CAD on 600 human-curated examples across four complexity levels, rather than grading editable FreeCAD documents by geometry and spec consistency.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- When an agent fans out parallel create calls through the FreeCAD MCP bridge, the shared connection races: the first object is created even though its call reports failure, and the second object is silently never created with the status still showing healthy.Error Index
- A FreeCAD MCP bridge silently ignored parameters the client materialized from JSON-Schema defaults, so box(size=20) produced a default 10mm shape. The agent believes it built the requested geometry while the model is something else entirely.Error Index
- A FreeCAD MCP bridge's autosave-before-call rewrote the active document on every tool call even with zero unsaved changes, stamping phantom diffs onto git-tracked CAD files. Agent activity pollutes version control with edits no human authored.Error Index
- A FreeCAD MCP bridge's persistent execution namespace meant that once any call bound a variable named result, every later call echoed the old value as its own output. The agent reads that output and believes its code produced a value it never computed.Error Index
- A CAD agent bridge's screenshot tool refused any image over a 50KB socket frame, so every real model screenshot was dropped, and images that did fit arrived as base64 text inside JSON instead of real image blocks. The agent's only visual feedback channel was dead for production-scale models.Error Index
Sources
Not verified
- Pricing: no price is stated on the benchmark announcement page read. The per-cell figures quoted there are benchmark run costs, not a product price.
- License terms: no license is stated on the announcement page read; commercial is the directory record's classification.
- Real user voices: no real linked post read for this page.
- A public repo for the bench itself: the scorer is the separate gNucleus-AI/freecad-validator repo and the task suite is a Harbor suite, not a git repository for the benchmark.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.