CloudOpsBench: Claude Fable 5.1 tops 45 AWS tasks at 60%

CloudOpsBench arrives as enterprises weigh whether AI agents can be trusted with the day-to-day work of running cloud infrastructure. The benchmark was built by experienced DevOps engineers, and version 1.0 includes 45 tasks on AWS, with all tasks running identically in the real AWS cloud and in an emulator. That design choice matters for reproducibility: teams can compare agent behavior without provisioning live infrastructure for every test.

The results show a clear gap between the best models and full task completion. Claude Fable 5.1 solved 60 percent of tasks, while GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early; roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure. That split suggests the hard part is not retrieving what to do but deciding how far to go and when to stop before causing harm.

The project is licensed under Apache 2.0 on GitHub, and a detailed failure analysis is also available. Extensions for GCP, Azure, and OCI are planned, which would broaden the benchmark beyond AWS. For now, the published failure breakdown gives DevOps teams a concrete way to interpret agent scores when assessing whether an agent is ready for production cloud operations.