CADTestBench
First test based benchmark for text to CAD, pairing language prompts with executable CADTests that check geometry and topology of the generated model, with test suites refined against mutated models that deliberately violate one requirement so weak tests are caught.
What it does and how it connects
CADTestBench is the first test-based benchmark for text-to-CAD, built on CADTests, executable software tests that verify whether a generated CAD model satisfies the geometric and topological requirements of the input prompt. A model is correct only if it passes every test. The tests are Python predicates over the generated B-rep, using CadQuery for B-rep inspection, topology counts, bounding box dimensions, areas, volumes, center-of-mass queries, containment tests and line-to-geometry intersections.
The benchmark is derived from CADPrompt, a text-to-CAD benchmark of 200 CAD programs, each paired with an abstract prompt and a detailed prompt. In total CADTestBench comprises 5,937 tests for both prompt variants, with an average of about 15 tests per sample. Test suites are synthesized by a large language model, validated against a ground-truth reference CAD model and a set of mutants, and refined until all tests pass on the reference while each mutant is rejected by at least one test.
The paper reports that test-based evaluation is reference-free, explicit, interpretable, with failing tests returning diagnostic messages, and efficient, with no inference cost from learned evaluators. The repo also ships four paper baselines (GPT-5.2 and Claude 4.6 Sonnet, abstract and detailed splits) and a browser viewer for the preloaded results, and loads benchmark data from Hugging Face by default.
Alternatives
Compare side by side- CADWorld
A 200-task benchmark for computer-use agents working through FreeCAD screenshots and GUI actions, rather than a test-based benchmark for text-to-CAD.
- BenchCAD 2.0
100 held-out engineering cases where models build CAD parts from drawings and recover board wiring, rather than a test-based benchmark for text-to-CAD.
- RealCADBench
A benchmark for intent-to-program CAD modeling from industrial design intents, rather than a test-based benchmark for text-to-CAD.
- StepScore CAD Benchmark
A 44-metric platform scoring AI-generated CAD models against human-calibrated thresholds, rather than a test-based benchmark for text-to-CAD.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- A committed STEP file quietly stopped matching its builder code: the shell referenced faces up to 529 but the data ended at 17, importing as an invalid zero-volume shell.Error Index
- An extrude cut on a big CadQuery STEP of a brick facade with thousands of faces takes about 140 seconds in SolidCraft.Error Index
Sources
Not verified
- Pricing: none stated on the pages read.
- No independent user voice found: the sources read are the paper and the repo.
- No GitHub releases exist, so latest release fields are null.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.