deepee

DeepEE is a small open benchmark for AI agents doing embedded firmware and KiCad engineering. An agent receives a prompt plus starter files, edits a fresh working directory, and submits ordinary source files, and an offline verifier checks the result. Conversation quality and screenshots do not count.

Version 1.2 has 48 tasks, 12 in each category: firmware (I2C recovery and drivers for SHT3x, BME280, MCP9808, INA219, BMP280, ADS1115, DS18B20, SCD4x, PCA9685, and MAX31855, plus Modbus CRC), code repair (scheduler rollover, ring buffers, ADC scaling, SLIP, MQTT, UTF-8, CBOR, PPP FCS, HTTP chunks, Base64, and wrap-safe tick timing), schematics (power-input repair, sensor and interface circuits, USB-C sink, RS-485, CAN, I2C level shifting, microSD, and NE555), and PCB (nRF24L01, MCP9808, and the same ten researched interface circuits as the schematic set).

Every task score is binary: all checks pass scores 1, anything fails scores 0, and the primary aggregate is Pass@1. The verifier checks required parts and pin and pad nets without comparing an entire design to golden geometry, and KiCad performs the final ERC and DRC gate. The README states this is deliberately a KiCad-native benchmark, not a tool-agnostic EDA benchmark, and that version 1.2 does not claim runtime peripheral, analog, thermal, EMI, or fabrication signoff.