Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

This benchmark asks whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. The authors build an MCP server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development, with 14 tools covering recurrent component-editing operations: create_component, add_signal_port, add_transactional_port, parameters, subcomponents and connections, getters, delete, save and load.

The benchmark has eight task suites: six core suites of 40 tasks each (Easy, Medium, Hard, History, Errors and Cross) and two multi-server suites of 60 tasks each (Easy-Noise and Hard-Noise). Easy, Medium and Hard isolate task length at 1, 2 and 3 to 5 expected calls; History omits entity identifiers and refers to prior tasks; Errors holds 20 heavily misspelled prompts and 20 requests where the correct response is to make no tool call; Cross combines dependency chains, cross-task references, pattern expansion and error detection in one multi-task session.

Seven open-source models are evaluated running locally through Ollama as 4-bit quantized variants, comparing ReAct single-agent and Plan-and-Act multi-agent architectures, system prompts, tool-description detail, context scope and few-shot prompting. Scoring is call-level: Expected Call Coverage measures how much of the annotated work was completed, with Extraneous Valid Call Ratio and Tool Failure Rate as complements. Strong models can achieve near-complete expected-call coverage, but reliability depends strongly on task structure and agent configuration. The full benchmark cannot be released because it encodes proprietary workflow details; representative examples are in Appendix B.