llm-firmware-benchmark
Dependency-free Node.js harness for evaluating LLMs on firmware and embedded coding tasks, with deterministic fixtures, mutation testing, sandboxed execution, and optional cross-compilation.
Summary rewritten from the vendor site because the catalog version did not match it.
What it does and how it connects
llm-firmware-benchmark is a dependency-free Node.js harness for evaluating language models on firmware and embedded coding tasks. It uses deterministic host fixtures, mutation tests, sandboxed execution, and optional cross-compilation to support reproducible validation, with general coding tasks kept as an auxiliary comparison suite.
The repository's CI badges report 136 tests, 75 C and C++ checks, and 10 sandbox fixtures. Supported provider runtimes include Claude Code, Codex, OpenCode, and OpenAI-compatible HTTP, and the harness requires Node.js 22 or newer. Task data, validation profiles, toolchains, sandbox policy, and resource limits are pinned in validation-profiles.json.
An interactive benchmark guide is published at arjunvinod.com/llm-firmware-benchmark, generated from the repository's tasks.json and validation profiles at deploy time, with a searchable catalog of every task and rubric, a grading walkthrough, and a runnable first-benchmark path.
Alternatives
Compare side by side- EmbedEval
EmbedEval tests implicit embedded-firmware knowledge with 267 cases, while llm-firmware-benchmark is a Node harness for firmware coding tasks.
- EmbedBench
EmbedBench tests embedded development with 126 cases across 3 hardware platforms, while llm-firmware-benchmark is a Node harness for firmware coding tasks.
- Embedded-LLM study (MDPI)
The MDPI study evaluates 27 models on 8 ESP32 IoT scenarios by expert review, while llm-firmware-benchmark is a Node harness for firmware coding tasks.
- PCEval
PCEval tests physical computing with circuits and code in simulation, while llm-firmware-benchmark is a Node harness for firmware coding tasks.
Sources
Not verified
- Pricing: none stated on the repo
- Real user voices (no real linked post read for this page)
- Related Error Index entries (no matching entry verified)
- A named individual builder beyond the arjun7965 repository owner
- The catalog's 96 CI tests figure (the README badges now report 136 tests)
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.