agent-desktop: connect an AI agent to desktop apps

agent-desktop is Vercel Labs’ open-source CLI and C library for reading and operating desktop applications through their accessibility trees. Use it when your agent needs to inspect a native app, select a control or enter a value. The calling agent supplies the model and decides which actions to take; the CLI supplies observations and actions.

The software is Apache-2.0 licensed. Running its native commands does not itself require an LLM subscription. Your agent client, model provider and operating machine determine the rest of the cost. Start with an app whose controls appear in a snapshot before connecting a longer automation workflow.

Check your operating system and permissions

agent-desktop uses the operating system’s accessibility interface. Check the requirements for the desktop session and the account running the commands before testing an application.

Separate agent-desktop processes still share the operating desktop and its apps. Coordinate workflows that operate on the same windows, especially when a person is also using the machine.

The platform guides distinguish supported actions from those that require headed operation. Canvas interfaces and apps with incomplete accessibility information can expose too little structure for a task. For web pages, use a browser automation tool; the project documents how to connect agent-browser to Chromium and Electron content.

Install and inspect the first app

The agent-desktop npm launcher needs Node.js 18 or later. The commands below assume its launcher is on your PATH; install the package, then inspect its version and permission status:

npm install -g agent-desktop
agent-desktop version
agent-desktop status
agent-desktop skills get desktop
agent-desktop skills get platform
agent-desktop list-apps

The package includes native binaries and platform helpers. Release archives provide a route without the JavaScript launcher; keep the executable and its matching helper together. The C library ships separately. If a package manager blocks install scripts, the fallback launcher adds startup overhead to each invocation. Follow the installation guide for that package manager rather than changing permissions across all packages.

On macOS, try a read-only observation of Finder:

agent-desktop snapshot --app Finder -i

Use an application name returned by list-apps on other systems. Read the returned JSON and choose an actual element reference before performing an action. A reference belongs to the snapshot that produced it; copying an example reference from documentation will not identify a control on your machine. When the UI changes or a reference becomes stale, take a fresh snapshot. After any action, observe again and verify the intended value or saved result.

The core agent-desktop commands fit four jobs:

For a dense app, begin with snapshot --skeleton --app APP -i --compact, then drill into the relevant region with snapshot --root REF -i --compact, replacing APP and REF with the app and current reference. This can reduce the UI text sent to a model. How much it saves depends on the tree, the task and the caller’s history handling.

Handle multiple windows and stale references

When agent-desktop returns AMBIGUOUS_TARGET because several windows match, inspect the candidate window IDs in the snapshot error’s details. Take a new snapshot scoped with --window-id and the selected ID, then choose a ref from that response. Refresh the snapshot after a STALE_REF error too. An earlier ref can point into a UI state that has since changed; replaying it adds calls without resolving that uncertainty.

The README’s trimmed Finder example shows the association between a snapshot and its elements:

{
  "snapshot_id": "s8f3k2p9",
  "tree": {
    "role": "window",
    "name": "Downloads",
    "children": [
      { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8" }
    ]
  }
}

This is an excerpt of the example’s data object, with other fields removed. The reference illustrates the format; obtain your own from the running app. If the targeted window is still ambiguous, stop that action and inspect the app/window selection instead of switching to physical input to force it through.

Default headless ref actions use accessibility APIs without moving the pointer or taking focus. Use --headed only for an action that needs physical input on a supported platform. That changes how input reaches the app; it does not give a missing control an accessibility tree. Keep a person and other automation from typing into the same desktop while a headed action runs, and verify which app received the input afterward.

Budget the model loop and desktop time

Each attempt can include an observation, a model request, an action and another observation. Model spend is the sum of fresh input, cached input and output charges across those requests, including retries. Add separately billed tools and machine time. An assumed $0.50/hour desktop reserved for 20 minutes contributes about $0.17 before model calls and other services; this is a budgeting example, not an agent-desktop tariff.

Compare cost per accepted result: a form correctly saved or a document correctly updated. A successful CLI response only establishes that the command completed; the application still needs checking. Preserve the model ID, actual billed usage, number of attempts and final artifact when comparing clients.

The native CLI has no embedded model. The optional JEV helper is separate and sends screen descriptions to its configured provider. Snapshots, clipboard content, screenshots and traces can contain private information. Decide what the calling agent may send to a provider and keep desktop artifacts out of public logs. For local model choices, see Muse Glimmer 30B; for hosted screen reasoning, compare the access and pricing in GPT-6.1 Terra.

Check app compatibility before scheduling work

Two November 2 reports show why an initial snapshot is useful but insufficient. In issue #479, evacn114 reports that agent-desktop 1.0.2 on Windows 10 with Word 16 returned document controls without the document’s paragraph text. In issue #494, kkauf reports that snapshots of Voice Memos worked on macOS 27.0.1, while actions failed with ambiguous-window errors. Both are configuration-specific user reports, open when reviewed.

Test reading, one reversible action and the resulting application state on the version you will deploy. If a document’s content is absent from the tree, use a document interface that exposes that content rather than treating an empty observation as an empty file. Repeatedly issuing the same action cannot repair missing accessibility information.

Sources

Reviewed November 3, 2026: official repository and license, platform support, installation, first-task guide, session handling, and the two linked user issue reports. The examples above are setup instructions, not a recorded automation run.

For a repository agent environment, read DeepSeek Harness. Browse the Agent guides for other workflows.