Embedded-LLM study (MDPI)
Head-to-head of 27 models on 8 ESP32 scenarios (sensor to cloud/DB) for functional correctness and instruction-following; a study, not a reusable artifact.
What it does and how it connects
This is a study, not a reusable benchmark artifact: it systematically evaluates 27 state-of-the-art large language models across eight embedded systems scenarios of increasing complexity, from basic sensor reading to complete cloud database integration with visualization dashboards, on ESP32 microcontrollers with environmental and motion sensors.
Scenarios run from BME280 sensor reading with OLED, LCD, and web-server output, through distance measurement and cloud integrations with ThingSpeak, Thinger.io, Beebotte, Firebase, and InfluxDB, to an end-to-end ESP32 to Grafana visualization workflow. Models were tested with zero-shot prompting in direct chat mode, and scored by two expert reviewers using the Analytic Hierarchy Process across functional, instructions, output, and creativity criteria.
Top-performing models were Claude Sonnet 4.5, Claude Opus 4.1, and Gemini 2.5 Pro, with scores from 0.984 to 0.910. Performance degraded with complexity: 19 to 23 models generated compilable code for simple applications, but only 3 to 5 produced functional solutions for complex scenarios involving Grafana and cloud databases. The most frequent failure was hallucinated non-existent libraries or incorrect API usage.
Alternatives
Compare side by side- EmbedEval
EmbedEval tests implicit embedded-firmware knowledge with 267 cases, while the MDPI study evaluates 27 models on 8 ESP32 IoT scenarios by expert review.
- EmbedBench
EmbedBench tests embedded development with 126 cases across 3 hardware platforms, while the MDPI study evaluates 27 models on 8 ESP32 IoT scenarios by expert review.
- llm-firmware-benchmark
llm-firmware-benchmark is a Node harness for firmware coding tasks, while the MDPI study evaluates 27 models on 8 ESP32 IoT scenarios by expert review.
- PCEval
PCEval tests physical computing with circuits and code in simulation, while the MDPI study evaluates 27 models on 8 ESP32 IoT scenarios by expert review.
Sources
Not verified
- Builder: the article names academic authors; no builder organization was recorded for this entry
- License: the article is open access; no reusable code or dataset license was verified
- Pricing: none stated
- Real user voices (no real linked post read for this page)
- Related Error Index entries (no matching entry verified)
- A public code repository or dataset (the study reports results only)
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.