model-performance

model-performance measures how the crew's models and the Spark that serves them perform, so the Sponsor can make a few specific decisions. It answers four questions and nothing else: how much the Spark can serve at once and on which serving setup, what the local 27B model needs to design CAD parts reliably, whether a candidate model is better than production, and which session settings help the crew's Developer.

The CAD question is answered by a pass rate on 21 hard cases, run against a baseline and each knowledge arm (skill, references, component library). The cases come in three kinds: tight, features with tight tolerances on one part such as a living hinge, a press fit, a snap-fit hook, a dovetail, a print-in-place hinge, a gear and a thread; assembly, two to five parts that must fit, each its own STEP file; and worded, a part asked for as a person would, with the dimensions the model's to choose. The parts directory includes pi5-nvme-case, a Raspberry Pi 5 NVMe case.

Each case is a spec in parts/*.md with its checks as data, compared by cad_eval/checks.py with what cad_eval/measure.py measures in a container. Each case's reference solution passes the checks and a planted wrong one fails them. A run needs Docker, the CadQuery image, and LiteLLM, and records stay on the operator's Mac. This is an internal crew evaluation project: its issues are filed by quarters-claude, an agent account, and its results are reported per decision, not as a public leaderboard.