EmbedEval

EmbedEval is an LLM embedded-domain knowledge probe that measures whether models possess the implicit domain knowledge to write safe embedded C code. It covers Zephyr RTOS, ESP-IDF, STM32 HAL, FreeRTOS, Linux kernel drivers, and Yocto recipes across 267 test cases (219 public and 48 private held-out), testing interrupt safety, cache coherency, DMA alignment, power management, and real-time constraints.

Unlike benchmarks that tell the model exactly what to do, EmbedEval states what to build but not how to make it safe: a prompt may ask for a DMA transfer with a completion callback without mentioning cache-line alignment, cache flush and invalidate, or a volatile completion flag. The README reports explicit prompts pass at about 95 percent while implicit prompts pass at about 60 percent, a 35 point gap it reads as benchmarks overestimating LLM capability in embedded domains.

The repository ships a live leaderboard on Hugging Face, a methodology document, and a CLI that runs the benchmark with L0 and L3 static checks and no Docker needed. The README's leaderboard table reports Sonnet 4.6 at 68.0 percent pass@1, Sonnet 5 at 67.0 percent, and Opus 5 at 61.8 percent, with DMA, ISR concurrency, and threading the weakest categories across models.