agentic-robotics-benchmark
Benchmark of agentic robotics tasks for autonomous agents, scored in sandboxed problems.
What it does and how it connects
The Agentic Robotics Benchmark scores autonomous agents on sandboxed robotics tasks: develop a controller, plan a manipulation sequence, design a part or a mechanism, write code-as-policy, or tune a planner. Each task drops an agent into a sandboxed problem and scores whatever artifact it leaves behind, a controller process, a plan, a URDF, mesh or parameter design, generated code or a report, with a hidden grader on hidden seeds.
Tasks run on the upstream ALE engine, pinned as a submodule, in three stages (setup, agent, verify). The anchor is the reference implementation's own measured value from a real run, never a printed figure from elsewhere, and the reward saturates at a ratio sized from the metric's measured seed-to-seed spread. An untouched sandbox must score 0.0 and the oracle 1.0, or validation fails.
The repository is public and complete: every task folder is whole, solution and tests included, with the anchor hidden from the tested agent by the engine's stage isolation while it works, not from people. The website displays the tasks and takes no uploads and no submissions.
Alternatives
Compare side by side- gantry-oss
A shared ledger for people and agents developing robot hardware and software, rather than a benchmark that scores agents on sandboxed robotics tasks.
- Mechanism-AI
A constraint-driven agent for planar four-bar linkages with MuJoCo simulation, rather than a benchmark of agentic robotics tasks.
- schunk_mechatronic_gripper_mcp_server
An MCP server that drives physical SCHUNK grippers over ROS 2, rather than a benchmark that scores agents in sandboxes.
- hardware-ai
A workspace that turns a product idea into an engineering dossier for a first prototype, rather than a benchmark of robotics tasks.
Recurring errors it addresses
Error Index entries this tool is built to reduce or work around.
- Code loaded every LeRobot checkpoint in the robot's default units without looking at the checkpoint; one so101 policy trained in degrees ran as range mode and drove wrist_roll toward its limit with no warning. The fix reads the checkpoint's saved min and max and refuses to run when a joint's numbers cannot fit the assumed units.Error Index
- The tool schema a model reads gave joints no unit or range, and add_object size gave no unit; it is MuJoCo half-sizes, so an agent asking for a 3 cm cube with size=[0.03]*3 got a 6 cm one. The arm's policy list also offered humanoid and duck presets that cannot drive it. Now each joint carries rad plus min and max from the manifest.Error Index
- Docs promised that joints left out of an action hold position, but the code re-targeted them to wherever gravity had already pulled them, so shoulder_lift sagged to 0.462 rad over 200 steps; fixed to 0.029 rad.Error Index
- An LLM-driven robot turned about 370 degrees and logged it as 10, because scan matching only knows angles modulo 360 and carpet made the motors spin 3 to 6 times faster than commanded, and in the simulator a -180 degree turn really spun -2088 degrees.Error Index
- On a branched mobile manipulator, torque_vector put torque on joints that cannot feel the force, plus 1.000 on an elevator joint, and 0 where -J^T F gives plus 0.164, because the index arrays were swapped.Error Index
Sources
Not verified
- Pricing (no price stated on the repository README)
- Real user voices (recent repo activity is contributors' task pull requests)
- A named individual builder beyond the Grounded-Intelligence-Initiative organization
- A public release (no releases are published)
Last verified November 6, 2026. Listings are informational. The evidence score measures public, verifiable evidence and is not an endorsement or a compatibility claim.