EDA Bench

EDA-bench is a native Harbor dataset for evaluating agents on complete KiCad project construction, framed by its authors as an open-book PCB reconstruction benchmark. The active dataset holds 35 self-contained tasks. Each task asks the agent to create a complete KiCad project under /workspace/final_project, with the instruction naming the required .kicad_pro, .kicad_sch, and .kicad_pcb files; the agent environment includes KiCad and ngspice. Harbor supplies the agent adapters, sandbox lifecycle, and result storage; the repo holds the benchmark itself.

Grading runs in Harbor's separate verifier mode with networking disabled. The verifier evaluates the submitted schematic, PCB, connectivity, external I/O behavior, physical realization, ERC/DRC results, and the task-specific functional contract, writing reward.json (reward, overall_score, task_score, build_success, and submission_exists channels), a deterministic grading.json record, and generated ERC, DRC, and ngspice evidence. The aggregate metric weights task rewards by difficulty, from 1.0 for very easy to 5.0 for extreme; missing rewards count as zero.

Each task ships with a contract directory of static and functional I/O data and a solution directory holding the Harbor oracle and sole reference project; the verifier image contains the reference project and scoring code and is never mounted into the agent environment. No Harbor registry release exists yet, so runs use the local path. The paper and the artifacts/first5_gold reports in the repo are retained research records, not benchmark infrastructure.