ParamCAD-AgentBench

ParamCAD-AgentBench is a tiered benchmark for language-model agents that construct parametric CAD parts from natural-language descriptions. An agent starts from an empty CAD document, plans a feature sequence, authors complete sketch geometry, calls CAD tools, interprets structured kernel feedback, and finishes with a valid B-Rep. The benchmark asks a harder question than command-sequence prediction: can an agent build the requested 3D part under real geometry-kernel constraints, recover from failures, and complete a long dependent workflow?

The frozen v2.2 benchmark holds 2,409 models and 4,818 tasks: a Core track of 2,000 models and 4,000 tasks across four difficulty tiers (T1 Basic to T4, stratified by hidden reference macro-feature count), plus a 409-model Long-Horizon Challenge with 818 tasks. Each model has two paired tasks, abstract (shape and part relations without explicit dimensions) and detailed (major dimensions, holes, cutouts and geometric relations). The source corpus was collected from Onshape parametric modeling histories, and every accepted reference program was replayed with CadQuery and OpenCASCADE and required to produce a non-empty, solid-containing, kernel-valid B-Rep. Dev and test splits are disjoint at the model level, and test and Challenge oracles are withheld to prevent leakage of the intended construction programs.

All 2,409 rendered ground-truth models are browsable on the project page.