PartForge evals
A standing CAD-agent evaluation of hard CAD tasks rerun against each model release, published as model comparisons on the PartForge blog.
Summary rewritten from the vendor site because the catalog version did not match it.
What it does and how it connects
PartForge evals is the standing evaluation PartForge runs on its own CAD-agent product. The latest test run, published October 20, 2026, gave each model the 27 hardest tasks, with three tries each. The tasks include parts that have to fit real hardware, parts rebuilt from a photo, and changes made over several messages, and every model used the same instructions PartForge users get, with every attempt visible on the results page.
In that run Claude Sonnet 5.5 scored 91 with a typical time of 48 seconds per task and a cost of 0.13 dollars per task, ahead of GPT-6 Sol at 90, GPT-6.1 Sol, Claude Opus 5.5, Gemini 3.8 Flash, and Claude Haiku 5.5 at 88, and Gemini 4 Argon at 87. The blog is explicit that the scores are close, all seven models within four points, and that the times are not close: Sonnet 5.5 finished a typical task in 48 seconds against 143 for Gemini 3.8 Flash, and passed build checks 96 percent of the time, tied for best.
PartForge also publishes a hand ranking of every attempt, and in that ranking Gemini 4 Argon, Gemini 3.8 Flash, and Claude Opus 5.5 all came out ahead of Sonnet 5.5, which is why the default-model change is described as a trial to watch for a couple of weeks. The blog states that PartForge will keep posting every test run as the models change. The eval is self-run by the vendor on its own product, not an independent benchmark.
Alternatives
Compare side by side- FrontierCAD (Normal)
A CAD benchmark from Normal Research's CAD Arena that scores how well AI agents recreate real parts across CAD tools, rather than a standing evaluation of hard CAD tasks.
- RealCADBench
A benchmark for intent-to-program CAD modeling from real industrial design intents, scored on FreeCAD Python output, rather than a standing evaluation of hard CAD tasks.
- PCBench
A dataset of 164 KiCad boards with routing ground truth for measuring autorouter quality, rather than a standing evaluation of hard CAD tasks.
- freerouting self-improving loop
An agent-driven experiment loop inside the freerouting repo that scores routing optimizer changes against frozen baselines, rather than a standing evaluation of hard CAD tasks.
- Easy-CAD / bendcad (jnadeau207-collab)
An agent-native CAD kernel effort built for AI agents, with certified volumes and named refusals, rather than a standing evaluation of hard CAD tasks.
Sources
Not verified
- The directory record names the builder as PartForge (Mazer); the Mazer name was not confirmed on the pages read, so the builder is recorded as PartForge.
- The directory record states 26 hard CAD tasks; the latest blog post read states 27 hardest tasks, so the count in the one-liner was rewritten and the older count is not carried.
- The veto lane and stale datasheet value example from the directory record were not confirmed on the pages read.
- Pricing: no pricing page was found on the vendor site as read.
- Real user voices (no real linked post read for this page)
- Related Error Index entries (no matching entry verified)
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.