Meta AmbiGUI: A Benchmark for Whether GUI Agents Ask Before They Click

Meta's research team has published AmbiGUI, a benchmark and code base for one narrow but important agent behavior: knowing when to ask. The facebookresearch/AmbiGUI repository appeared on GitHub on October 23, 2026. Its README defines a GUI agent as "a model that uses a website or an app for a person: it reads the screen and the request, then clicks, types or scrolls," and the question it asks is simple. If you tell an agent to "book a table," does it ask for how many people, or does it guess?

The repository is a first commit with no stars yet, no model weights and no paper, so there are no headline results to report. What it does give you is a concrete, runnable way to measure something that computer-use demos rarely show.

TL;DR: the questions people ask first

Why asking is the hard part of computer use

Most agent benchmarks reward finishing the task. That pushes models toward acting on every instruction, even vague ones, because a wrong guess still looks like progress. In real use the cost is asymmetric: a wrong flight date, size or recipient can be expensive or irreversible, while one short question is cheap.

AmbiGUI scores both failure modes. An instruction is ambiguous "when it leaves open a choice that changes the outcome (for example which size or date), and clear otherwise." A good agent should ask on the first kind and not on the second. Asking on a clear task is a false alarm that annoys users and slows automation, so the benchmark reports ambiguous and clear tasks separately and never pools them.

That framing matters as computer-use products spread. Cursor's agents can now drive desktop apps, as we covered in Cursor Computer Use, and open stacks such as Alibaba's OpenGCUA are trying to make generalist desktop agents routine. A benchmark for restraint is a natural companion.

How the offline evaluation works

The offline mode replays recorded trajectories. At every step the model under test is shown the request, the current screenshot and the recorded history, and must either give the next action or ask one clarifying question.

Key design choices from the documentation:

• Teacher forcing. The recorded human history is used at every step, not the model's own earlier actions, so one early mistake cannot ruin the rest of the task.
• A simulated user answers. When the model asks, another language model plays the person. It knows the task's open items, the details the request left unstated, each with the value the person actually chose, and answers only what was asked.
• A cap on questioning. After ten questions in a row at one step, the task is cut off and its remaining steps count as wrong.
• An ask judge. A third model reads every question and decides which open items it targeted.
• Separate stages. Run, judge and score are separate programs, so a judge failure never costs policy GPU time again, and metrics can be recomputed from files without calling any model.

The gold file has 298 tasks, 143 ambiguous and 155 clear. With the rebuilt public data it contains 3,115 recorded steps (1,032 from Mind2Web and 2,083 from META-GUI). The 33 OSWorld rows have no recorded steps, so the offline loader skips them.

How the online evaluation works

The online mode runs a real Chromium browser on live sites. The agent sees a screenshot with numbered element marks and answers with one tool call per turn: click, type, scroll, go to a URL, message the user, or report the task complete or infeasible. A simulated user replies to messages, and a task judge decides success, asking a vision model questions about screenshots when needed.

Facts from the docs:

• 84 tasks, curated from the gold rows plus 8 annotated Mind2Web rows, with a per-task turn cap of three times the length of the recorded trajectory.
• Paper models. The paper's runs used Llama-3.2-90B-Vision-Instruct as the simulated user and Qwen3-VL-235B-A22B-Instruct as the judge, according to the docs. The policy models in the paper's runs included hosted frontier models from Anthropic, OpenAI, xAI and Google, plus an internal Meta model, and 18 open-weight policies.
• Fixed settings. Temperature 0, seed 0, a 600-second limit per rollout, and at most four images per policy request.
• Second phase. After the rollouts, an ask judge grades every message, which can be rerun under another judge without repeating the browser work.

The metrics are worth knowing by name, because they are the real contribution: Amb real asked (share of ambiguous tasks where the agent asked at least one real question), Amb on-target (share where a question targeted an open item), Clear real asked (the false-alarm rate), blocker recall, question precision and question F1. A run that asked nothing has no precision, which the report shows as "not measured" rather than zero.

A cursor reaching toward a drag-and-drop tray, illustrating the GUI agents that AmbiGUI evaluates on whether they ask before acting

How to run it

Python 3.12 on Linux is required. Install one extra per environment because the parts pin different package versions:

git clone https://github.com/facebookresearch/AmbiGUI.git ambigui && cd ambigui
python3.12 -m venv .venv && . .venv/bin/activate
pip install -e ".[offline]"   # offline evaluation
pip install -e ".[online]"    # online evaluation

The online part also needs Playwright's Chromium; the README notes that the download address built into Playwright 1.44 no longer serves files, so you set PLAYWRIGHT_DOWNLOAD_HOST=https://cdn.playwright.dev. Lock files and Dockerfiles are provided for each part.

Be ready for heavy infrastructure. Every role, meaning policy, simulated user, judge and describer, is reached through an OpenAI-compatible endpoint. To reproduce the paper's setup yourself you would serve Llama-3.2-90B-Vision and Qwen3-VL-235B with vLLM using tensor parallel 8, which is a multi-GPU job. Hosted APIs are an alternative. The docs also say live-web results differ from day to day, so compare policies only on runs made together.

You also must rebuild the benchmark: the repository is pointer-only. Run benchmark/build.py against your own copies of Mind2Web, META-GUI and OSWorld; it checks every source value against a hash and stops if a source version differs. Screenshots are not part of the rebuild.

Caveats and honest limits

• Not a reproduction of the paper. The released online file keeps two lists of open items separate, while the paper's simulated user saw one merged list. The docs say re-running with the release "is a new measurement, not a reproduction," and a correct rebuild will not match the paper's 84-task file checksum.
• Model-written labels. Gemma-4-31B wrote the open items and proposed ask windows for the gold rows; people wrote them for four online rows. Label quality therefore depends partly on a model.
• Exact re-scoring is limited. The recorded run directories are not shipped, and some open-weight tables cannot be re-scored exactly because they used an earlier credit rule.
• Non-commercial license. CC BY-NC 4.0 rules out commercial use, and the source datasets carry their own terms.
• No training code or weights. The README says an SFT row in the paper cannot be reproduced from this repository.
• Websites change. Blocks, bot walls and outages show up as env_blocked or env_unreachable endings with defined handling, but they still add noise.

What this means for what you build

If you ship an agent that acts for users, AmbiGUI gives you a template for two numbers you should track: how often it asks when it should, and how often it asks when it should not. Even without running Meta's harness, you can borrow the idea: tag test tasks as ambiguous or clear, script a simulated user that only answers what was asked, and report the groups separately.

For the field, it sits alongside other GUI-agent work we have covered, such as MAI-GYM, a 9B GUI agent RL platform and Work2Skill, which turns human GUI demonstrations into reusable skills. Those push on capability; AmbiGUI pushes on judgment.

A lightweight way to borrow the idea

You do not need Meta's harness to start. A minimal version for your own agent takes an afternoon:

1. Collect 30 to 50 real tasks from your product logs and label each as ambiguous or clear. For ambiguous ones, write down the open items and the value a typical user would pick.
2. Script a simulated user with a short prompt: reply only with the value of the item the agent asked about, and refuse anything that is a report rather than a question.
3. Score two rates separately. The share of ambiguous tasks where the agent asked something relevant, and the share of clear tasks where it asked anyway.
4. Review a sample by hand. Model judges can mislabel statements as questions, which is why AmbiGUI uses a dedicated ask judge and excludes statements.
5. Track it per release. If a new model raises task success but also raises the false-alarm rate, you have learned something a success-only benchmark would hide.

Related reading

• Cursor Computer Use: agents click and type in any desktop app
• Alibaba OpenGCUA: open generalist computer-use agents
• MAI-GYM 9B GUI agent RL platform
• Work2Skill: GUI demonstrations to agent skills
• OpenVLHarness visual agent harness
• Agent security: the AWS trust-in-depth framework

Details are accurate as of October 23, 2026, from the AmbiGUI repository README and documentation; the repository may change as the paper is published.