CloudOpsBench: Best AI Agent Solves 60% of 45 AWS Tasks, and Reasoning Is the Bottleneck

CloudOpsBench is a new open-source benchmark that asks a practical question: can an AI agent be trusted with cloud operations? Version 1.0 holds 45 AWS tasks. According to The Decoder, Claude Fable 5.1 solved 60 percent and GPT-6 Astra solved 53 percent.

The team, led by Ang Chen at the University of Michigan, says the main problem is not knowledge. About 80 percent of failures come from reasoning errors. The code is on GitHub under Apache 2.0.

TL;DR: scores, failure causes and limits

What kinds of tasks are in the suite?

The README says CloudOpsBench "tests whether AI agents can handle cloud operations: fix broken services, remove unused resources, and tighten access without disrupting other workloads." Each of the 45 tasks comes with a setup script, instructions for the agent, automated checks, and a reference solution.

The task list reads like a week of real infrastructure chores. Examples from the repository include:

• Remove a finished pricing experiment, including the query results it left in a shared bucket.
• Tear down a finished analytics environment by removing every resource named with its prefix.
• Remove everything belonging to a retired customer portal while keeping the billing workload intact.
• Delete a retired pilot's EBS volumes and leave other teams' volumes alone.
• Consolidate a pilot dataset into the team's shared bucket under the same keys and leave the source bucket empty.
• Disable alarm actions on a set of CloudWatch alarms ahead of maintenance.

Most of these are decommission and cleanup jobs. They look easy, which is the point. The hard part is the boundary: delete everything that carries the prefix, but nothing that merely looks similar, and do not touch a neighbor's resources.

How does the benchmark run?

According to The Decoder, all tasks run the same way in the real AWS cloud and in an emulator. The README adds that the repository provides the tasks, and you supply a harness to run setup, the agent and grading. It names an AWS emulator called Vera as one option and points to a lifecycle contract for integration.

The README also warns about real AWS use: use an isolated test account with scoped credentials and clean up after each run, because tasks can modify or delete resources and incur AWS charges. It says never to supply real AWS credentials to emulator containers.

Running the suite is therefore not a one-command job. The scripts/validate.py helper only checks task structure and does not run agents or create cloud resources.

What do the scores say?

The reported numbers are simple. Fable 5.1 reached 60 percent. GPT-6 Astra reached 53 percent. No model finished every task. The Decoder shows the benchmark team's chart of per-model results; the article text names only these two models, so we cannot give you a full table.

Treat the figures with ordinary benchmark caution:

• Small suite. With 45 tasks, one task is worth about 2.2 points. A gap of a few points is noise.
• Harness matters. Scores depend on the agent harness, prompts, tool permissions and effort settings. The article does not say which harness was used.
• Reported, not reproduced. We did not re-run any model.
• Version 1.0. The README says the 42 most recently added tasks have not yet been run end to end, and that full lifecycle execution is still a release gate. Expect fixes.

For other agent benchmarks and why leaderboards need care, see our posts on the Terminal-Bench 4.0 audit claims and on Real-SWE, a benchmark on private codebases.

Why reasoning, not knowledge, is the bottleneck

The team's finding is the most useful part. About 80 percent of failures, it says, occur because agents misjudge the scope of a task, do not check all the relevant services, or stop too early.

That matches how decommission work goes wrong in practice. A model knows how to delete a bucket. It may not know that the same project prefix also names a queue, a certificate, an image and an alarm. An agent that lists one service, finds the matching resources, deletes them and reports success has done the visible part. The task checker then looks for leftovers in other services and fails the run.

The other roughly 20 percent are wrong actions, and The Decoder notes that some have irreversible consequences. That is the failure type that matters most to operators. A missed resource costs a few dollars a month. A deleted volume that belonged to another team can cost data.

What this means if you let agents touch cloud accounts

The benchmark gives a short checklist that fits what teams already do with human operators.

1. Inventory before action. Make the agent list candidate resources across all services and show the list before it deletes anything.
2. Require a scope statement. Have the agent state what it will touch and what it will leave alone. Compare that with the real inventory.
3. Use read-only roles first. Run a discovery pass with a role that cannot delete, then grant narrow write access for the second pass.
4. Gate destructive calls. Put a human approval step in front of deletes. Our guide on when to let an agent run without a human in the loop lays out the trade-offs.
5. Add command guards. Tools such as the destructive command guard block known dangerous commands before they run.
6. Verify after. Run a second check for leftovers by prefix and tag, the way the benchmark's own graders do.

Agents that act on real infrastructure are also a security question. We covered an incident where an autonomous agent breached a real service in our post on the Hugging Face autonomous agent breach. For a runtime layer that stops AI agents before they take dangerous actions, see AgentBeam, the agent security platform from the explainx.ai team.

How to try a small version yourself

You do not need to run all 45 tasks to learn from the design.

1. Clone and validate. Clone the repository and run python3 scripts/validate.py. It checks task structure only and creates no cloud resources.
2. Read one task end to end. Open a decommission task and read the setup script, the agent instructions, the checks and the reference solution. Note how the checker looks for leftovers outside the obvious service.
3. Pick a safe environment. Use an emulator, or a throwaway AWS account with scoped credentials and a budget alarm. Never reuse a production account.
4. Run one agent on three tasks. Choose one delete task, one consolidate task and one access-tightening task. Log every call the agent makes.
5. Read the transcript for scope. Ask whether the agent listed every service before it acted, and whether it stopped when the first obvious resources were gone.
6. Repeat with a scope prompt. Add one line asking the agent to inventory all services first, and compare. If the pass rate moves, you have a cheap fix worth keeping in your own runbooks.

This is a small experiment, but it tests the exact failure mode the team reports.

How does this compare with other coding-agent results?

Coding benchmarks measure whether an agent can write a patch. CloudOpsBench measures whether it can change a live environment without side effects. Those are different skills, and a model that tops a coding leaderboard may not top this one. For model-versus-model context on the two systems named here, see our GPT-6 Astra vs Claude Fable 5.1 comparison.

The 60 percent ceiling also sets expectations. At that rate, an unsupervised agent fails roughly four tasks in ten, and some of the failures are destructive. The practical conclusion is supervised use with checks, not a ban.

What readers are asking

Is 60 percent good? For a first version of a hard benchmark, it shows headroom. For production, it means you need human review.

Does it work on Azure or GCP? Not yet. The team plans extensions.

Can I add tasks? The repository has a CONTRIBUTING.md and accepts task contributions, per its file list.

Where is the failure analysis? The Decoder says a detailed failure analysis is available from the project. Check the repository docs for the current link.

Bottom line

CloudOpsBench is a useful addition for anyone who wants to know whether an agent can run routine cloud chores. The headline numbers, 60 percent for Fable 5.1 and 53 percent for GPT-6 Astra, come from the benchmark team through one outlet. The more durable result is the diagnosis: agents lose on scope and completeness, not on API knowledge. Build your guardrails around that.

Details reflect The Decoder's November 4, 2026 report and the CloudOpsBench repository as read that day. Scores and task counts may change in later versions.

Related reading

• Terminal-Bench 4.0 audit: claimed vs verified
• Real-SWE: coding agents on private codebases
• When to let an AI agent run: human in the loop
• Destructive command guard for coding agents
• Hugging Face autonomous AI agent breach
• GPT-6 Astra vs Claude Fable 5.1
• Co-Design-Bench: GPT-6 Astra fully solves 26 percent