Desktop Tools

Computer Use For All

Desktop Tools gives agents eyes and hands on a Mac. Most of the work went into what an agent should believe after it acts, and then its engine was rewritten in Swift in about fifteen hours.

21 Oct 2026 5 minute read Source

Ace has a rule that all Ace development happens in Ace. That rule hits a wall as soon as the thing you're building is a desktop app. To check a native change, an agent has to look at real windows, click real buttons and work through real dialogs.

So Ace grew native desktop tools: inspect a window, click, type, drag, scroll, use menus and the clipboard, launch and quit apps. They now live in their own project, Desktop Tools, which is also a menu bar app any agent can use: Claude Code, Codex, or anything else that speaks MCP.

Clicking is the easy part. The interesting parts are who's allowed to click, and what the agent should believe afterward.

A plain library can't hold a permission

On macOS, Accessibility, Screen Recording and event synthesis are granted to an app. Not to a library, and not to whatever process happens to load one. So the work happens inside a signed app, and everything else talks to it over a Unix socket (Fig. 1):

A host that holds the grants: the Desktop Tools app, or any signed app that embeds the host library, as Ace does.
An MCP server, desktop-tools-mcp, shipped inside the app. An agent starts it, and it keeps one connection open to the host.
For apps that embed the host, a signed client CLI and a TypeScript API, so a Node or Bun process can call desktop({ op: "apps" }).

The host checks who's on the other end. It only accepts a client signed by its own team with one exact identifier, and it checks the running process, not just its pid. Then the app asks once per agent: "Allow claude-code to control this Mac?" Until you say yes, every request from that agent is refused, screenshots included. Input aimed at Desktop Tools itself is always refused. That rule came from testing the prompt, when an agent could click Allow on it for itself.

Fig. 1. Who holds the permission. macOS grants it to the app, so the work happens inside the app. Agents reach it through one connection, and each agent is asked once.

Fig. 2. Observe, act once, observe again. The dashed stamp is the one that matters.

Every action uses up its observation

The part I'm proudest of is mostly restraint. The contract (Fig. 2):

Inspect one exact window. You get accessibility text and a screenshot, and it never steals focus.
The observation comes with a snapshot of that window and the controls it saw.
Act once against the snapshot. Every action that sends input uses up its observation, even one whose result is uncertain. Want to do something else? Inspect again.

Most input goes to the target window in the background, and the physical pointer doesn't move. The few tools that need the real pointer or keyboard, like hover or a Command-click, say so. If the window moved, the app restarted or the control vanished, the action is refused before anything is sent. Every action reports one of three outcomes:

completed: the input reached that exact window. Some tools also read the change back. Otherwise it's still not proof the app did what you wanted, so look.
refused: nothing was sent.
unknown: input may have landed, and nobody can say.

Unknown is a real answer, and the system is built so nobody pretends otherwise. A model's instinct is to retry. On a desktop, retrying a click you can't confirm is how you send the same message twice. What an agent can do instead is say what it expects, such as a window appearing. If that change shows up, an unknown becomes completed.

Fig. 3. Fifteen patches, then sixty-four, then none. The changes that mattered were carried into the rewrite as ports.

Fig. 4. Where old setups land. Every path ends at the server inside the app; the one that can't get there is told where it is.

Sixty-four patches

Underneath the app, until October 20, was Peekaboo 4.8.0 plus patches applied in a fixed order (Fig. 3). Each one came out of something that broke while an agent was using it. Clicking a window that had moved used to come back as "maybe dispatched," the scariest possible answer. Now it's a clean refusal.

At the start of October there were fifteen. By October 19 there were 64, more than 15,000 lines of patch files, and we had never moved to a newer Peekaboo. It was a fork in all but name, so that night it became one, with every change applied in a copy inside the repo.

The same evening I asked how insane it would be to rewrite the whole thing from the ground up, without Peekaboo, maybe in Rust. I don't like TypeScript, and we had to have Swift anyway for the system APIs and the signed app, so it might as well be one language. Rust wouldn't make it noticeably faster. The time went to two places: the Node server started a fresh signed client for every call, and every Accessibility read is a round trip into the target app. "If rust isn't worth it, we skip," I said. "I can see it with how deeply we will be interfacing with macos in general." So: one Swift codebase, no Peekaboo, no Node. I filed the plan and closed it as not planned. An hour later I picked it back up: "start chewing through a pure swift spike and see how it goes."

Same answers, faster

The rewrite, called native2, didn't get to change what a model sees. The 48 tools' schemas and instructions were frozen, and each tool was ported to give the same result, field for field: 583 recorded calls, then live runs against a small window that logs every mouse event. The Peekaboo code that mattered came along as ports, like clicking a background window without moving the pointer.

My bar was "as good or better than the original." The agent doing the port didn't merge the first time it thought it was done, because the Mac had been locked all night and nothing had run on a live screen. The two stacks could run side by side, so native2 went in next to the old one first.

desktop_windows went from 115 ms to 6 ms at the median, and inspecting a Ghostty window from 551 ms to 42. The app's binaries shrank from about 84 MB to about 6 MB. The few changes callers see are deliberate: launch and open stay in the background unless asked, so an agent opening an app doesn't take my keyboard, and a key refused because focus left the window now says refused, not unknown, since nothing was sent.

The hard part was everyone already set up for the old engine (Fig. 4). The MCP server now lives in the app, so agents don't need Node. An npx setup at 0.17.0 or later hands its connection to that server. One pinned to an older release reaches the new client, which answers CLIENT_OUTDATED and names the server to set up instead.

0.17.0 went out on October 20, about fifteen hours after the spike started, and 0.17.1 followed that afternoon. Ace still runs the old stack. Deleting Peekaboo waits until it moves.

The desktop tools are careful about the one thing they own. They never act on a window they haven't just looked at, and they never claim more than they know.