computer

Drive a full desktop on the iii bus. Where the browser worker gives an agent a Chromium tab, computer gives it a whole screen: it hands the model a screenshot, clicks and scrolls at the coordinates it reads off that image, and types wherever the desktop's focus is. The harness discovers computer::* as functions automatically, so a model sees the screen and acts on it with no glue, and the console gets a live viewport of whatever the desktop is doing.

The worker does the one thing no other worker can: capture the screen and move the cursor. Three drivers sit behind one surface, picked per session:

• native (no endpoint, no image): drive the local machine this worker runs on, capture and input injection built in.
• sandbox (an image): boot a fresh desktop inside an iii-sandbox microVM and drive it through iii primitives alone (sandbox::exec / sandbox::fs), with no socket into the guest. A fixed virtual display means 1:1 coordinates, no HiDPI or multi-monitor ambiguity, and nothing to grant on the host.
• remote (an endpoint): drive an already-running desktop through the executor inside it, over a WebSocket. computer connects; that desktop boots out of band.

Everything else composes with workers that already exist: run commands and touch files with the shell worker, persist with state, schedule with cron.

Install

iii trigger compose::add worker=computer

To drive a throwaway desktop instead of your own machine, add the sandbox worker too — it boots the microVM the image driver runs in:

iii trigger compose::add worker=iii-sandbox

macOS, native driver only. Capture and input are permission-gated and macOS degrades both silently, so the worker checks and fails loud. Grant the app that runs the worker Screen Recording (without it every screenshot is the wallpaper with all windows stripped) and Accessibility (without it clicks and typing are dropped) in System Settings > Privacy & Security, then restart it. The sandbox driver needs neither.

Linux, native driver. On an X11 session (Xorg or Xvfb) a native session captures and acts by coordinate like macOS/Windows; coordinates are physical root pixels. Run a window manager: on a bare X server a click gives no keyboard focus, so toolkits that wait for focus (egui, Bevy, Slint) ignore typed keys. On a Wayland desktop (GNOME and KDE default to it) the session is semantic-only: sessions::start reports coordinates: { supported: false }, screen is 0x0, screenshots and coordinate actions are refused with unsupported_session, and you read and act by ref. Ref actions need the AT-SPI bus; when accessibility is off, semantic.reason carries a warning with the command that turns it on for the login session (gdbus call … IsEnabled ''). The worker never runs it itself: it also persists toolkit-accessibility=true in dconf. Chromium/Electron apps also need --force-renderer-accessibility.

Quickstart

Start a session, look at the screen, click what you see:

# drive this machine: a bare start picks the native driver when neither
# sandbox_image nor default_endpoint is configured
iii trigger computer::sessions::start --json '{}'
# -> { "session_id": "c1", "endpoint": "native", "os": "macos", "screen": { "width": 1512, "height": 945 }, "semantic": { "supported": true } }

# see the screen: an image block the vision model renders inline
iii trigger computer::screenshot --json '{"session_id":"c1"}'

# click a pixel read off that screenshot (top-left origin)
iii trigger computer::act --json '{"session_id":"c1","action":"click","x":640,"y":360}'

iii trigger computer::sessions::stop --json '{"session_id":"c1"}'

Swap the first call for {"image":"desktop"} to boot a sandboxed Linux desktop instead (see Sandbox desktop image), or {"endpoint":"ws://host:8000"} to drive a remote one. Everything after that call is identical: the session id is the only handle.

The same from a sibling worker:

use iii_sdk::protocol::TriggerRequest;
use iii_sdk::{register_worker, InitOptions};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let started = iii
        .trigger(TriggerRequest {
            function_id: "computer::sessions::start".into(),
            payload: json!({}),
            action: None,
            timeout_ms: Some(30_000),
        })
        .await?;
    let session_id = started["session_id"].as_str().unwrap();

    iii.trigger(TriggerRequest {
        function_id: "computer::act".into(),
        payload: json!({ "session_id": session_id, "action": "click", "x": 640, "y": 360 }),
        action: None,
        timeout_ms: Some(10_000),
    })
    .await?;

    Ok(())
}

The rest of the surface: computer::act — pointer actions addressed by coordinate (click, right_click, double_click, move, drag, scroll) and keyboard actions that land wherever the desktop's focus is (type with text, press / hotkey with keys, e.g. ["cmd","c"]) — computer::observe (a screenshot and, with include_a11y, an app's accessibility tree), computer::find, computer::displays, computer::sessions::list and computer::sessions::stop.

For commands and files, mind which machine you mean: on a native session the desktop IS this host, so the shell worker is the right tool. A sandboxed desktop is reached with sandbox::exec / sandbox::fs, and a remote one through its own guest executor — shell would run on the host instead, which is a different machine entirely.

On a native session the worker also acts by element: computer::observe { include_a11y: true, include_image: false } or computer::find { selector: "button[name='Save']" } return lines carrying [ref=…], and computer::act { action: "invoke" | "set_value" | "toggle" | "select", ref } acts on that element through the OS accessibility API — the worker moves no mouse, sends no keys and sets no focus. The effect is not verified; confirm with another find. Refs live until an observe with include_a11y or a find with matches replaces them, any coordinate or keyboard action (type, press, hotkey, console input included) runs on any native session (they share the desktop), a timed-out, cancelled or crashed accessibility call resets the session's executor, or the session stops or the worker restarts. macOS needs Accessibility and Screen Recording for the process running the worker. On Windows a password field's value is hidden only when its name looks like a password (or the app itself masks it): xa11y does not expose UIA's IsPassword. On Linux a password field is recognised by its AT-SPI role, whatever its name. Remote and sandbox sessions stay coordinate-only.

computer::run { app_name, goal, inputs } drives one app toward a goal in a single call: each step, the judge worker picks the next element and operation from the tree, the worker picks the verb and acts by ref, and the run returns a status, its steps and the tree with live refs. Text comes only from inputs, keyed by field label. Without a judge worker it returns judge_unavailable and the tree.

Console

The worker ships its own console page (page computer): a session rail, a live viewport fed by the live screencast, and click / type / scroll forwarding straight into the desktop. It is injected into any running console at registration time — nothing to install, nothing to rebuild. Every computer::* call in chat and traces renders through the same asset.

Configuration

Stored in the configuration worker under the computer key; every field is editable live from the console. The screencast rate applies to running sessions; everything a driver is built with — endpoint, OS label, timeouts, capture limits, and the sandbox display and network settings — is read at sessions::start, so a change reaches sessions started after it.

computer:
  default_endpoint: ''      # guest-executor endpoint for a remote desktop; empty = drive the local machine
  os: linux                 # guest OS label recorded on sessions
  max_sessions: 2           # concurrent desktop connections
  idle_stop_ms: 300000      # stop sessions idle this long; 0 disables
  screencast_fps: 15        # live-view frame rate cap
  max_screenshot_dimension: 1280 # downscale screenshots/frames to this longest edge (native)
  screenshot_quality: 70    # JPEG quality 1-100 (native)
  command_timeout_ms: 120000 # timeout for each driver action (fixed at connect)
  connect_timeout_ms: 15000  # driver connect timeout at session start
  sandbox_image: ''          # iii-sandbox image for a sandbox session; empty = name it per call
  sandbox_width: 1280        # sandbox virtual display width
  sandbox_height: 800        # sandbox virtual display height
  sandbox_network: true      # give the sandboxed desktop network access; false keeps it offline
  sandbox_idle_timeout_secs: 86400 # idle_timeout_secs for sandbox::create; kept high, the worker owns teardown
  screen_capture_preflight: true # macOS: ask for Screen Recording at native start, fail loud if denied

Sandbox desktop image

The sandbox driver needs a desktop inside the guest (Xvfb, xdotool, imagemagick, openbox). A prebaked image lives in images/desktop: build it, push it, register it as a custom_images entry on the iii-sandbox worker, then start a session with { "image": "desktop" }. iii-sandbox boots on macOS Apple Silicon (libkrun) and Linux (/dev/kvm).

Custom trigger types

Sibling workers (and the console page) subscribe to session activity instead of polling. Both bindings accept an optional { "session_id": "..." } filter.

Trigger type	Fires when	Payload to subscribers
computer::session-started	A session connected and is ready	{ session_id, endpoint, os, screen, timestamp }
computer::session-stopped	A session ended	{ session_id, reason: "stopped" | "idle", timestamp }

Live screen

computer::screencast::start / stop, computer::frame and the computer::frame-changed trigger type drive the live viewport. The worker captures at screencast_fps and keeps only the newest frame per session in memory (no history). After each frame it fires computer::frame-changed with a small, image-free notification; the viewer reads the frame with computer::frame (notify, then fetch), so the console and any number of watchers follow the desktop without polling, and a slow watcher skips frames instead of queueing them.

Trigger type	Fires when	Payload to subscribers
computer::frame-changed	A new frame was stored (change: "updated"), or the frame was dropped because the screencast or session stopped (change: "cleared")	{ session_id, epoch, frame_seq, change, width, height, mime?, bytes?, timestamp }

The binding config { "session_id": "..." } is required (a viewer follows one desktop). Order frames by (epoch, frame_seq); epoch changes when the worker restarts or restores the session. Bind first, then read computer::frame once for the current frame; read it again after each notification (pass since_frame to skip an unchanged image) and after a reconnect. These are internal console plumbing rather than agent surface, and stay out of agent function lists.