Multi-VerilogEval
First multilingual Verilog generation benchmark: 156 tasks each written in English, Japanese, Hindi, and Mongolian for 624 cases, scored on syntactic and functional correctness.
What it does and how it connects
Multi-VerilogEval is the first multilingual Verilog code generation benchmark. It is built from 156 unique underlying tasks, each written in English, Japanese, Hindi, and Mongolian, giving 624 language-specific test cases. The tasks are drawn from VerilogEval, which is derived from HDLBits, and translated through a multi-agent pipeline with human oversight.
The paper's evaluation of four representative models finds a silent failure pattern: syntactic correctness stays high at about 90 percent across languages, but functional correctness falls by up to 23.9 percent for non-English prompts in open-source and domain-specific models, while commercial models stay near parity.
To close the gap the paper proposes HE-ICL, hard-example in-context learning, a train-free method that builds few-shot demonstrations from crosslingually difficult cases. The paper reports HE-ICL closes 80 to 100 percent of the multilingual gap without any parameter updates. The paper was published 2026-05-25 in Electronics, volume 15, article 2275.
Alternatives
Compare side by side- VerilogEval
The English-only Verilog benchmark Multi-VerilogEval builds on, while Multi-VerilogEval adds Japanese, Hindi, and Mongolian prompts.
- RTLLM 2.0
A Verilog generation benchmark of 50 problems, while Multi-VerilogEval tests the same 156 tasks in four languages.
- llm-verilog-reliability-eval
A reliability evaluation for Verilog generation, while Multi-VerilogEval measures multilingual bias across four languages.
- HLS-Eval
A benchmark for LLM evaluation on high-level synthesis tasks, while Multi-VerilogEval benchmarks Verilog generation in four languages.
Sources
Not verified
- Pricing (not applicable; the benchmark is a published paper, and no price is stated)
- Real user voices (no real linked post read for this page)
- Related Error Index entries (no Verilog entries in the live error map)
- A public code or dataset repository (none is linked in the paper text read; the data availability statement says data is contained within the article)
- Repo stats (no public repo found for this paper benchmark)
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.