agent-desktop: Vercel Labs' Computer Use CLI Reads the Accessibility Tree, Not Pixels

agent-desktop is a command-line tool that gives any AI agent computer use on the desktop without looking at pixels. Published under vercel-labs/agent-desktop on GitHub (about 2.2k stars when we checked) and licensed Apache-2.0, it reads an app's real interface structure from the operating system accessibility tree, hands the agent a JSON snapshot, and lets it click, type and scroll by element reference. Version 1.0.0 landed on October 27, 2026 with Linux support, and 1.0.2 followed on October 30.

Header image: agent-desktop share card, courtesy of Vercel Labs, from the vercel-labs/agent-desktop repository.

The pitch in the README is plain: "Computer use for AI agents: read and operate desktop apps through the OS accessibility tree." Where most computer-use systems send a screenshot to a vision model and ask it to guess coordinates, agent-desktop asks the OS what is on screen and what can be done with it.

TL;DR: the questions people ask first

What is agent-desktop, and why does the accessibility tree matter?

Every major OS exposes a structured description of each window for screen readers: roles such as button, treeitem or window, names, values and the actions available. macOS calls it the Accessibility API, Windows has UI Automation, and Linux uses AT-SPI. agent-desktop treats that tree as "the source of truth for observation and semantic interaction," according to the project's own CONCEPTS.md glossary.

Computer use agent dragging a tile into a tray, illustrating agent-desktop computer use through element refs

A snapshot is the key primitive. Run agent-desktop snapshot --app Finder -i and you get one JSON envelope with a snapshot ID, a count of refs, and a tree of nodes. A node looks like "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"]. The agent then runs agent-desktop click @s8f3k2p9:e8 and snapshots again to verify the result. The README calls this the loop: observe, act on a ref, observe again, "with no pixel matching; the loop stays in your agent."

That structure buys three things a screenshot loop struggles with:

• Determinism. A button is the same ref across calls until the UI changes, so a click does not drift by a few pixels.
• Cheap observation. A text tree is far smaller to feed back into a model than a full-resolution image, and the tool offers --skeleton, --compact and --root flags to map a dense app like Slack first and then drill into one region.
• Honest failure. Refs belong to their snapshot. A bare @e8 is rejected with INVALID_ARGS, and when the UI changes an action returns STALE_REF or AMBIGUOUS_TARGET, so the agent re-snapshots instead of clicking the wrong row.

How does it work under the hood?

The project is more careful than a weekend wrapper. The README says that on macOS, CLI calls reuse a private command host per snapshot namespace and binary build, which retains native accessibility objects "so a removed row cannot silently resolve to a surviving row with the same identifier." The host exits after 15 idle minutes, and TMPDIR must be owned by the current user with mode 0700.

The concepts glossary shows how much edge-case work sits behind that. It defines a Partial Observation, a snapshot that ran out of its time budget and returns what it saw, with completeness reported on the whole result and on each cut node. It defines an Evidence Tri-State, in which every property is Known, Absent or Unknown, so a failed read can never be mistaken for a negative answer. It also describes how Chromium and Electron apps wrap content in chains of anonymous containers, and how the walker skips those wrappers so dense web apps fit the default depth budget instead of understating their reachable content "by an order of magnitude."

Actions are headless by default. The README says ref actions go through the accessibility API "without moving the pointer or taking focus," and the global --headed flag allows physical input. Four commands, key-down, key-up, mouse-down and mouse-up, refuse to run until a daemon can release a held key or button, a small but telling safety choice.

What can you do with its 60 commands?

The README lists 60 command names, 56 of them operational. They fall into groups:

Every command prints one JSON envelope with ok and either data or error. An error carries a code, a disposition and usually a suggestion, and the README says to retry "only when error.disposition.retry is safe." That contract, documented in the repo's docs/json-output.md, is what lets an agent recover without a human reading logs.

There is also an FFI library. Each release ships libagent_desktop_ffi for macOS, Linux and Windows so a program can call the tool in process rather than spawning the CLI for every command.

How do I try it in Codex, Claude Code or Cursor?

Terminal prompt with a block cursor, the starting point for installing the agent-desktop CLI

Install the CLI, then verify it:

npm install -g agent-desktop
agent-desktop version
agent-desktop permissions --guide   # macOS: grant Accessibility
agent-desktop list-apps
agent-desktop snapshot --app Finder -i

Bun, Yarn and pnpm install the same package, which includes native binaries for x64 and ARM64. The README says official Homebrew and Scoop packages are still being submitted, and that the repository no longer maintains its own tap or bucket. On Windows, no grant is needed for apps at your terminal's integrity level; on Linux you need your graphical session with at-spi2-core.

To teach an agent harness the tool, the project ships its own instructions inside the binary. Run agent-desktop skills get desktop for the loop, refs, verification and retry rules, and agent-desktop skills get platform for what differs on your OS. You can install it as a skill with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop, or paste two lines into your AGENTS.md telling the agent to read those docs before first use. The README's own demo is a clip titled "Codex.using.agent-desktop", which shows the intended pairing with a coding agent. For how skills work in general, see our agent skills guide.

How does it compare with Cursor, Cua and other computer-use tools?

The timing is not an accident. Computer use is becoming a standard coding-agent capability. Cursor added a Computer Use tool on October 23 so its agents can click through the iOS Simulator, Cua Spaces connects agents across several computers, and open-model work such as Alibaba's OpenGCUA stack and Microsoft's Mage-VL pushes vision-based agents forward on the OSWorld benchmarks.

agent-desktop is complementary, not a replacement: it is a tool any harness can call, while the models in the other rows decide what to do. The honest limit of the accessibility approach is apps that expose a poor tree, such as canvas-heavy or game-like interfaces, where a vision fallback still helps. The README's mention of an optional JEV helper, which sends screen descriptions to a configured model provider, hints at that hybrid, though the README is explicit that it is separate from the native CLI.

What are the risks and limits?

Giving an agent the whole desktop is a bigger grant than giving it a repo. The README is direct about it: "Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content. Keep them out of public logs and uploads." Anything the agent snapshots, from a password manager window to a chat client, enters the model's context and possibly your transcripts.

Practical guardrails:

• Run agents under a separate OS user, VM or spare machine for anything beyond a demo.
• Close sensitive apps before a session and avoid clipboard-get on shared sessions.
• Prefer the default headless mode, and use --headed only when an app refuses accessibility actions.
• Log commands, not snapshots, and review traces before sharing them.
• Add approval gates for destructive actions. AgentBeam, the agent security platform from the explainx.ai team, is built to stop agents before they take dangerous actions.

Platform gaps matter too. The README's support table shows Linux mouse input only on X11 with --headed, screenshots on X11 windows or via the Wayland portal, and notifications only on GNOME Shell. On macOS, shell surfaces opened through open-system-surface are marked "Planned," while Windows already has them. And macOS needs a one-time Accessibility grant; the 1.0.2 changelog notes that permissions --guide now offers the terminal that launched the tool rather than the binary itself, and that the command host restarts after a missing permission so commands succeed once you grant access.

What changed in the first week of releases?

The CHANGELOG shows a fast cadence. Version 1.0.0 on October 27 released the integrated macOS, Windows and Linux CLI and C library together, bundled native executables into npm installs "including installations with lifecycle scripts disabled," and added release verification of archives, checksums and build provenance. Version 1.0.1 the same week fixed a stale version line in the skills and automated npm and GitHub releases. Version 1.0.2 on October 30 added a one-time stderr notice when npm blocks install scripts and the Node launcher slows each call, and pointed the package at the vercel-labs repository.

Earlier listings of the project credit an individual developer's account, and the repo now sits in Vercel's labs organization, which is why you may see both names in older posts. We did not find a Vercel announcement post or a benchmark in the repository beyond a benchmarks/locator-resolution folder, so treat any speed or accuracy claims you see elsewhere as unverified until the project publishes numbers.

What this means for people who build with agents

If you run coding agents that need to test a real app, drive a native installer, or check a UI change, a structured computer-use CLI is a lower-cost, more repeatable path than screenshots. The loop is simple enough to script, the JSON contract is explicit about retries, and the FFI library means you can embed it in your own harness. The cost is that you now own a permission and privacy surface: accessibility access is among the most powerful grants on a desktop.

A reasonable first test is read-only. Install it, run list-apps and one snapshot on a harmless app, read what comes back, and only then let an agent call click. Our MCP guide explains how to wrap tools like this behind a protocol if your harness prefers MCP over shell calls. And for the broader debate about agents with OS-wide access, the Vercel KVM sandbox escape write-up is a useful reminder that where an agent runs matters as much as what it can click.

Related reading

• Cursor Computer Use: local agents in any desktop app
• Cua Spaces: cross-computer AI agents
• Alibaba OpenGCUA open computer-use stack
• Microsoft Mage-VL computer-use model
• What are agent skills? A complete guide
• What is MCP? The Model Context Protocol guide
• Vercel KVM zero-day and agent sandboxes
• Official: agent-desktop on GitHub

Version numbers, star counts and platform support reflect the repository as of October 31, 2026 and may change.