lumerical-benchmark
Benchmark dataset and evaluation harness for AI-generated Ansys Lumerical scripting language.
What it does and how it connects
Lumerical Benchmark is a benchmark dataset and evaluation harness for AI-generated Ansys Lumerical Scripting Language (.lsf) for photonic integrated circuit layout and FDTD simulation automation. It holds 160 tasks split strictly across device and functional categories into 80 train and 80 test tasks, with an extended set of 1,536 verified tasks. Every task has been validated against both a zero-dependency headless CAD simulator and the actual Ansys Lumerical FDTD software via the official Python automation API (lumapi).
Train categories cover waveguides and tapers, splitters and couplers, monitors and detection, and materials and stacks; test categories cover cavities and resonators, sources and illumination, solver and meshing, and analysis groups and metrics. The split keeps functional categories mutually exclusive to prevent cross-split contamination and memorization of geometric dimensions.
The harness is hardened for real-engine evaluation: Lumerical runs in a worker process so a hung script is killed after its timeout and the session restarts, a save is injected before run and runsweep so no modal Save dialog blocks the API, script variables are cleared between tasks, benchmark simulations run as a single MPI process, and failures report Lumerical's own error text instead of a generic failure message.
Alternatives
Compare side by side- veriflow-rtl-benchmark
A five-design golden-model harness for AI-written Verilog, rather than a benchmark for Lumerical scripting.
- llm-eval-ci-gate-rtl-verilog
A dual-tool CI gate for LLM-generated RTL, rather than a benchmark for photonic simulation scripting.
- specread
A 383-question benchmark for hardware spec comprehension, rather than for Lumerical scripting.
- ParamCAD-AgentBench
A tiered benchmark scoring agents that build parametric CAD parts, rather than models that write simulation scripts.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: no real linked post read for this page; the only recent activity is the owner's own pull request.
- No GitHub releases exist, so latest release fields are null.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.