cadgenbench

CADGenBench measures how well AI systems produce correct 3D mechanical parts, across two tasks: generation, where an engineering drawing of a part must become a valid, geometrically correct 3D model, and editing, where an existing STEP file must be changed as requested. The benchmark is tool-agnostic: a submission is one candidate STEP or BREP file per sample, built with any tool, from build123d to Autodesk Fusion or Onshape.

The GitHub repository holds the scoring engine, metric definitions, the submission contract, and a reference baseline generator. Evaluation runs server-side on the Hugging Face leaderboard Space, scoring submissions against privately held ground truth. Validity is the gate: a candidate must be a well-formed, watertight BREP, and failure zeroes the rest; shape similarity then measures geometry distance through surface-distance F1 and volume IoU. The other two axes are interface match, mating-feature correctness via authored keep-in and keep-out sub-volumes, and topology match, Betti numbers of the tessellated boundary.

Rows publish as unvalidated, and promotion to a validated tier is a separate methodology review by the maintainer team. A sanity-check script ships with the samples so submitters can exercise the same validity gate locally before uploading.