DepthBenchCAD

DepthBenchCAD is a three-level counterfactual audit benchmark that studies how evaluation evidence should be allocated across task templates, independent model generations, and within-program counterfactual edit states under a fixed evaluation budget. It is built from the BenchCAD task corpus and released as two disjoint evaluation environments: DepthBenchCAD-A with 72 templates from eight task families, and DepthBenchCAD-B with 48 templates from six non-overlapping task families.

Every template includes a standalone CadQuery reference program, legal parameter ranges, task constraints, and 16 frozen edit states: four local, four boundary, four linked, and four semantic. The release contains 2,760 generation-level records and 44,160 state-level audit records, plus 800 doubly annotated expert-validation items. Provider-side prompts, candidate source text, token usage, raw latency logs, and decoding metadata are not included.

The audit worker executes the nominal program and all requested frozen edit states, recording build status, geometry probes, judge stage, and audit outcome. An initially invalid program contributes failures for all 16 states in the primary risk estimate. The repository ships validation and analysis scripts for materializing the task corpus and auditing new model generation manifests against the frozen states.