EmbedBench

EmbedBench is the benchmark behind EmbedAgent, a paradigm for testing large language models in embedded system development through three simulated real-world roles: Embedded System Programmer, Architect, and Integrator. It consists of 126 manually constructed cases covering 9 electronic components across 3 hardware platforms, each with a task description, a reference solution, and automated correctness validation.

The Programmer setting gives the model a task description and a circuit schematic and asks for Arduino code. The Architect setting gives only the task description and asks the model to design the circuit and write the code. The Integrator setting gives a schematic and code for one hardware platform and asks the model to migrate the design and code to another platform.

Correctness is checked with a hardware-driven evaluation approach: the behavior of virtual hardware in the Wokwi environment is monitored in real time. The paper reports DeepSeek-R1 at only 55.6 percent pass@1 with schematic information and 50.0 percent when generating schematics itself, and cross-platform migration as the weakest area, with ESP-IDF the hardest target at 29.4 percent pass@1 for the best model. Retrieval augmentation and compiler feedback lift DeepSeek-R1 to 65.1 percent with correct schematics.