cad-silent-failure-bench

This benchmark and harness measures silent failures in LLM CAD agents: cases where the agent emits executable, error-free CAD code that is still geometrically or semantically wrong, while claiming the task is complete. It ships 13 specification tasks in three tiers with 122 machine-checkable requirements, and a full attempt corpus of 519 full-population attempts plus fixed-protocol and ablation runs, with transcripts for the tool-use and ablation attempts.

A silent failure is a valid solid where the agent reported done and the hardened kernel oracle says not passed. In the primary population of 156 attempts per condition, single-shot agents succeeded 64.1 percent of the time with a 14.1 percent silent-failure rate, tool-use with measure-then-claim reached 77.6 percent with 3.2 percent silent failures, and planner-coder-checker reached 61.5 percent with 2.6 percent silent failures.

The suite is versioned on Zenodo, with v1.0 the version the paper reports and v1.1 adding a four-model ablation and tolerance sensitivity analysis. The benchmark code is MIT while the tasks, reference solutions and data are CC BY 4.0, and the paper is Done Is Not Correct by Carson Rodrigues, Clive Rodrigues and Aravind Reddy G.