Computer use

What computer use gives an agent

A workspace with computer use on has a Linux desktop, 1280x800 by default, that your agent operates like a person: it takes screenshots, moves the pointer, clicks, types and presses keys. Use it for software that has no API: web apps behind a login, desktop applications, anything an agent should see before it acts.

• On demand. Switch computer use on for a workspace or for every workspace of a template. The platform starts the desktop on the first computer call, in about 0.35 s, and keeps it running with the workspace.
• Fast. A click and the screenshot after it take about 20 ms in the workspace's region (p50, 1280x800 PNG). A batch of actions runs in one request.
• Part of the workspace. Commands, files and the desktop share one machine: exec of chromium https://example.com & opens the browser on the desktop. Suspend and resume keep its windows.

The desktop comes with the python-node-browser template, version 15 and later, and every template built on it.

Switch it on

import { Shardflux } from '@shardflux/sdk';

const cloud = new Shardflux();
const ws = await cloud.workspaces.open({ key: 'agent/42', template: 'python-node-browser', computerUse: true });

• computerUse: true (Python computer_use=True) switches it on for a new or existing workspace; false switches it off; null (None) follows the template. workspace.setComputerUse() changes it later.
• cloud.templates.setComputerUse(slug, true) switches it on for every workspace of one of your templates; a workspace's own setting wins.
• In the dashboard, the workspace's Overview has a Computer use row with the switch.
• workspace.computerUse reads { enabled, workspace, template, available }.

While computer use is on, the workspace's tool tokens carry the computer tool. An API key needs the computer tool permission, which keys created with every tool have.

With Claude's computer toolset

computerToolset() answers Claude's computer toolset: all of a model turn's computer calls run as one batch, and each answer comes back as the tool_result Claude expects.

import Anthropic from '@anthropic-ai/sdk';
import { computerToolset } from '@shardflux/sdk';

const anthropic = new Anthropic();
const computer = computerToolset(ws);
const messages = [{ role: 'user', content: 'Open example.com in Chromium and tell me the page title.' }];

for (;;) {
  const msg = await anthropic.messages.create({ model: 'claude-opus-5-5', max_tokens: 4096, tools: [computer.definition], messages });
  messages.push({ role: 'assistant', content: msg.content });
  if (msg.stop_reason !== 'tool_use') break;
  messages.push({ role: 'user', content: await computer.run(msg.content) });
}

from anthropic import Anthropic
from shardflux import computer_toolset

client = Anthropic()
computer = computer_toolset(ws)
messages = [{"role": "user", "content": "Open example.com in Chromium and tell me the page title."}]

while True:
    msg = client.messages.create(
        model="claude-opus-5-5", max_tokens=4096, tools=[computer.definition], messages=messages
    )
    messages.append({"role": "assistant", "content": msg.content})
    if msg.stop_reason != "tool_use":
        break
    messages.append({"role": "user", "content": computer.run(msg.content)})

Every tool_result echoes toolset_name: "computer". A turn that does not end with a screenshot gets one after its last action, on the last result.

Agent tools for any model

workspaceTools() (Python workspace_tools()) includes two computer tools while the workspace's computer use is on:

Both take screenshot: false to skip the screenshot, settle_ms (the wait before it when the screen may change, default 250) and format: "jpeg" with quality for smaller images. Images are base64 in data_base64 with their mime_type, width and height.

Drive it from code

const shot = await ws.computer.screenshot(); // { format: 'png', width: 1280, height: 800, data }
const r = await ws.computer.act(
  [
    { action: 'left_click', coordinate: [640, 400] },
    { action: 'type', text: 'hello' },
    { action: 'key', text: 'Return' },
  ],
  { screenshot: true },
);
// r.results: one per action; r.screenshot after the last one; r.cursor; r.display

• act(actions, { screenshot, settleMs, format, quality }) runs the actions in order. The first failure stops the batch; the actions after it come back skipped.
• status(), start({ width, height }) (640x480 to 2560x1600) and stop() manage the desktop directly.

From the command line

shard ws open acme/demo --template python-node-browser --computer-use on
shard ws exec-start acme/demo -- chromium https://example.com
shard ws computer screenshot acme/demo --out desktop.png
shard ws computer click acme/demo 640 400
shard ws computer type acme/demo "hello" --screenshot after.png
shard ws computer key acme/demo ctrl+s

Every input command takes --screenshot F (a screenshot after it), --settle MS and --format jpeg --quality N.

From MCP

The Shardflux MCP server lists, for API keys with the computer permission:

• computer and computer_batch, with their images as MCP image content;
• workspace_computer_desktop: the desktop's status, a start at a chosen screen size, or a stop;
• workspace_set_computer_use (state: on, off or inherit) for every key.

Actions

Coordinates are screen pixels from the top-left of a screenshot. Key names follow X keysyms (Return, Escape, Page_Down, F5), and the common aliases (enter, esc, cmd, ArrowUp) work too.

Your own desktop

A template built on python-node-browser 15 or later has the desktop. Its session (window manager and panel) is the script /etc/shardflux/desktop/session: replace it in your template to start another window manager or your application.