# Computer use

> A desktop in the workspace that agents drive with screenshots, clicks and keystrokes: Claude's computer toolset, agent tools, the CLI and MCP.

## What computer use gives an agent

A workspace with computer use on has a Linux desktop, 1280x800 by default, that your agent operates like a person:
it takes screenshots, moves the pointer, clicks, types and presses keys. Use it for software that has no API: web apps
behind a login, desktop applications, anything an agent should see before it acts.

- **On demand.** Switch computer use on for a workspace or for every workspace of a template. The platform starts the
  desktop on the first computer call, in about 0.35 s, and keeps it running with the workspace.
- **Fast.** A click and the screenshot after it take about 20 ms in the workspace's region (p50, 1280x800 PNG). A
  batch of actions runs in one request.
- **Part of the workspace.** Commands, files and the desktop share one machine: `exec` of
  `chromium https://example.com &` opens the browser on the desktop. Suspend and resume keep its windows.

| Client | Version | Switch | Actions |
| --- | --- | --- | --- |
| TypeScript SDK | `@shardflux/sdk` **(0.15.0+)** | `open({ computerUse })`, `workspace.setComputerUse()` | `workspace.computer.act()`, `computerToolset()` |
| Python SDK | `shardflux` **(0.11.0+)** | `open(computer_use=)`, `ws.set_computer_use()` | `ws.computer.act()`, `computer_toolset()` |
| CLI | `@shardflux/cli` **(0.10.0+)** | `shard ws computer enable`, `shard ws open --computer-use on` | `shard ws computer click\|type\|key\|...` |
| Agent tools and MCP server | `@shardflux/sdk` **(0.15.0+)**, `@shardflux/mcp` **(0.9.0+)** | `workspace_set_computer_use` (MCP) | `computer`, `computer_batch` |
| HTTP | `/v1` | `PUT /v1/workspaces/{id}/computer-use` | `POST <cell_endpoint>/v1/workspaces/{id}/computer/actions` |

The desktop comes with the `python-node-browser` template, version 15 and later, and every template built on it.

## Switch it on

```ts
import { Shardflux } from '@shardflux/sdk';

const cloud = new Shardflux();
const ws = await cloud.workspaces.open({ key: 'agent/42', template: 'python-node-browser', computerUse: true });
```

- `computerUse: true` (Python `computer_use=True`) switches it on for a new or existing workspace; `false` switches it
  off; `null` (`None`) follows the template. `workspace.setComputerUse()` changes it later.
- `cloud.templates.setComputerUse(slug, true)` switches it on for every workspace of one of your templates; a
  workspace's own setting wins.
- In the dashboard, the workspace's **Overview** has a **Computer use** row with the switch.
- `workspace.computerUse` reads `{ enabled, workspace, template, available }`.

While computer use is on, the workspace's tool tokens carry the `computer` tool. An API key needs the `computer` tool
permission, which keys created with every tool have.

## With Claude's computer toolset

`computerToolset()` answers Claude's computer toolset: all of a model turn's computer calls run as one batch, and each
answer comes back as the `tool_result` Claude expects.

```ts
import Anthropic from '@anthropic-ai/sdk';
import { computerToolset } from '@shardflux/sdk';

const anthropic = new Anthropic();
const computer = computerToolset(ws);
const messages = [{ role: 'user', content: 'Open example.com in Chromium and tell me the page title.' }];

for (;;) {
  const msg = await anthropic.messages.create({ model: 'claude-opus-5-5', max_tokens: 4096, tools: [computer.definition], messages });
  messages.push({ role: 'assistant', content: msg.content });
  if (msg.stop_reason !== 'tool_use') break;
  messages.push({ role: 'user', content: await computer.run(msg.content) });
}
```

```python
from anthropic import Anthropic
from shardflux import computer_toolset

client = Anthropic()
computer = computer_toolset(ws)
messages = [{"role": "user", "content": "Open example.com in Chromium and tell me the page title."}]

while True:
    msg = client.messages.create(
        model="claude-opus-5-5", max_tokens=4096, tools=[computer.definition], messages=messages
    )
    messages.append({"role": "assistant", "content": msg.content})
    if msg.stop_reason != "tool_use":
        break
    messages.append({"role": "user", "content": computer.run(msg.content)})
```

Every `tool_result` echoes `toolset_name: "computer"`. A turn that does not end with a screenshot gets one after its
last action, on the last result.

## Agent tools for any model

`workspaceTools()` (Python `workspace_tools()`) includes two computer tools while the workspace's computer use is on:

| Tool | Takes | Answers |
| --- | --- | --- |
| `computer` | One `action` with its parameters | `ok`, `output`, `error`, `cursor`, `display`, and `screenshot` (a screenshot after an input action, the image of `screenshot` and `zoom`) |
| `computer_batch` | `actions`: 1-50 actions | `ok`, `results` (per action: `ok`, `skipped`, `output`, `error`, `took_ms`, a zoom's `image`), `cursor`, `display`, `screenshot` |

Both take `screenshot: false` to skip the screenshot, `settle_ms` (the wait before it when the screen may change,
default 250) and `format: "jpeg"` with `quality` for smaller images. Images are base64 in `data_base64` with their
`mime_type`, `width` and `height`.

## Drive it from code

```ts
const shot = await ws.computer.screenshot(); // { format: 'png', width: 1280, height: 800, data }
const r = await ws.computer.act(
  [
    { action: 'left_click', coordinate: [640, 400] },
    { action: 'type', text: 'hello' },
    { action: 'key', text: 'Return' },
  ],
  { screenshot: true },
);
// r.results: one per action; r.screenshot after the last one; r.cursor; r.display
```

- `act(actions, { screenshot, settleMs, format, quality })` runs the actions in order. The first failure stops the
  batch; the actions after it come back `skipped`.
- `status()`, `start({ width, height })` (640x480 to 2560x1600) and `stop()` manage the desktop directly.

## From the command line

```sh
shard ws open acme/demo --template python-node-browser --computer-use on
shard ws exec-start acme/demo -- chromium https://example.com
shard ws computer screenshot acme/demo --out desktop.png
shard ws computer click acme/demo 640 400
shard ws computer type acme/demo "hello" --screenshot after.png
shard ws computer key acme/demo ctrl+s
```

| Command | Action |
| --- | --- |
| `click <ws> [x y] [--button left\|right\|middle] [--count 2\|3] [--mods keys]` | A click, double or triple click at a point or at the pointer |
| `type <ws> <text\|->` | Types text (Unicode; `-` reads standard input) |
| `key <ws> <keys> [--repeat n]` | Keys and combinations: `Return`, `ctrl+s`, `alt+Tab`, several separated by spaces |
| `scroll <ws> up\|down\|left\|right [x y] [--amount n]` | Wheel clicks (default 3) |
| `move <ws> <x> <y>`, `drag <ws> <x1> <y1> <x2> <y2>` | Moves the pointer; drags with the left button |
| `mouse-down <ws>`, `mouse-up <ws>`, `hold-key <ws> <keys> <seconds>` | Holds the left button or keys |
| `zoom <ws> <x0> <y0> <x1> <y1> [--out F]` | Saves an enlarged view of a region |
| `cursor <ws>`, `wait <ws> <seconds>` | Prints the pointer position; waits |
| `act <ws> <json\|->` | Any of these in one batch: a JSON array of actions |
| `status`, `start [--width --height]`, `stop` | The desktop itself |

Every input command takes `--screenshot F` (a screenshot after it), `--settle MS` and `--format jpeg --quality N`.

## From MCP

The Shardflux MCP server lists, for API keys with the `computer` permission:

- `computer` and `computer_batch`, with their images as MCP image content;
- `workspace_computer_desktop`: the desktop's status, a start at a chosen screen size, or a stop;
- `workspace_set_computer_use` (`state`: `on`, `off` or `inherit`) for every key.

## Actions

| Action | Parameters |
| --- | --- |
| `screenshot` | |
| `zoom` | `region`: `[x0, y0, x1, y1]`, answered with that region enlarged |
| `left_click`, `right_click`, `middle_click`, `double_click`, `triple_click` | `coordinate` (default: at the pointer), `text`: modifiers held, e.g. `shift` |
| `left_click_drag` | `start_coordinate`, `coordinate`, `text` (modifiers) |
| `mouse_move` | `coordinate` |
| `left_mouse_down`, `left_mouse_up` | |
| `cursor_position` | Answered with `X=…, Y=…` |
| `scroll` | `scroll_direction` (`up`, `down`, `left`, `right`), `scroll_amount`, `coordinate`, `text` (modifiers) |
| `type` | `text` |
| `key` | `text`: keys such as `Return`, `ctrl+s`, `alt+Tab`, several separated by spaces; `repeat` |
| `hold_key` | `text`, `duration` (seconds) |
| `wait` | `duration` (seconds) |

Coordinates are screen pixels from the top-left of a screenshot. Key names follow X keysyms (`Return`, `Escape`,
`Page_Down`, `F5`), and the common aliases (`enter`, `esc`, `cmd`, `ArrowUp`) work too.

## Your own desktop

A template built on `python-node-browser` 15 or later has the desktop. Its session (window manager and panel) is the
script `/etc/shardflux/desktop/session`: replace it in your template to start another window manager or your
application.