fail2bench
Self-curating benchmark synthesis that turns agentic hardware design failures into new eval tasks.
What it does and how it connects
Fail2Bench is a research framework for self-curating benchmark synthesis from agentic hardware design failures. It turns every failure an LLM agent makes on hardware and systems design tasks into a versioned, reproducible benchmark item, so the resulting benchmark suite self-calibrates: it grows harder as the agent improves, and shrinks in coverage gaps the agent has already closed.
The loop is mine, learn and evaluate: run an agent (RTL repair is the documented live path, needing an Anthropic API key), mine failures into tasks, evaluate an agent version on the suite, and inspect the scoreboard and registry. An offline demo with a mock agent needs no API key, and a Streamlit dashboard ships for browsing results.
Alternatives
Compare side by side- specread
A fixed 383-question benchmark for spec comprehension, rather than a suite that grows from agent failures.
- veriflow-rtl-benchmark
A fixed five-design golden-model harness for AI-written Verilog, rather than a self-curating failure-driven suite.
- llm-eval-ci-gate-rtl-verilog
A dual-tool CI gate for LLM-generated RTL with a measured kill rate, rather than a failure-mining benchmark synthesizer.
- cad-mcp-cleanroom
Audits whether a CAD MCP command surface is deterministic and honestly read-only, rather than mining agent failures into benchmarks.
- ParamCAD-AgentBench
A fixed tiered benchmark scoring agents that build parametric CAD parts, rather than a self-curating failure-driven suite.
Sources
Not verified
- Pricing: none stated on the repo.
- Real user voices: no real linked post read for this page; the repo has no public issues.
- No GitHub releases exist, so latest release fields are null.
- Activity: last push 2026-08-29, so the project looks quiet.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.