AgenticBBO-Bench
Cross domain benchmark for LLM agents doing black box optimization under a fixed evaluation budget, spanning synthetic functions, hyperparameter tuning, database tuning, chip macro placement and molecular design, scoring agents against classical numerical optimizers on solution quality per budget.
What it does and how it connects
AgenticBBO-Bench is a Python benchmark for black-box optimization with language-model agents and numerical optimizers. It accompanies the paper A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization, arXiv 2610.12183, by Ming Chen et al., 2026. It compares agents with numerical optimizers using shared initial evaluations, a host-controlled evaluation budget, isolated agent workspaces, append-only run logs, and a common scoring implementation.
The benchmark covers 77 tasks across 5 domains: BBOB with 24 tasks, hyperparameter optimization with 25, database tuning with 6, placement (BBOPlace) with 12, and molecular optimization (GuacaMol) with 10. The included main-suite contracts cover seeds 2 to 5 for each task, and frontier and diagnostic suites select smaller fixed subsets. A run fixes a task, seed, method and budget.
The README states the benchmark needs Linux, Python 3.11 and uv, and that the quick start needs no API key, Docker service or downloaded checkpoint. Task IDs and frozen reference files are retained in bbo/experiments/assets.
Alternatives
Compare side by side- CLOSER-Bench
A budgeted cross-stage design closure protocol for hardware agents, rather than a cross-domain black-box optimization benchmark.
- HSCO-Bench
An end-to-end hardware-software co-design benchmark deploying SoC prototypes on an FPGA, rather than a cross-domain black-box optimization benchmark.
- ArchXBench
A hierarchical RTL synthesis benchmark from simple gates to full subsystems, rather than a cross-domain black-box optimization benchmark.
- StepScore CAD Benchmark
A 44-metric platform scoring AI-generated CAD models against human-calibrated thresholds, rather than a cross-domain black-box optimization benchmark.
Sources
Not verified
- Pricing: none stated on the repo.
- No independent user voice found: the repo returned no public issues.
- No GitHub releases exist, so latest release fields are null.
- Fork: this repo is a fork of trxcc/agentic-bbo.
- No Error Index entry matches black-box optimization, so related errors are empty.
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.