> ## Documentation Index
> Fetch the complete documentation index at: https://iii.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Changelog
> Product updates and announcements for iii.
## Register triggers from anywhere: `engine::register_trigger` + invocation metadata
New `engine::register_trigger` / `engine::unregister_trigger` builtins register a trigger whose fire is delivered to a `function_id`, with arbitrary `metadata` recovered at fire time. The call returns the trigger id, unregister is idempotent, and session `allowed_trigger_types` RBAC is honored.
Per-invocation metadata is now a first-class argument through the whole invocation path — delivered to the target function as a separate argument, never injected into the payload. All four SDKs read it idiomatically: Rust two-arg closures (`|data, meta: Option|`) and `TriggerRequest.metadata(...)`, Node `(data, metadata?)` and `trigger({ ..., metadata })`, Python `def handler(data, metadata=None)` (forwarded only if the handler declares it), Go `iii.MetadataFromContext(ctx)`.
## Triggers survive reconnects and provider reloads
Two lifecycle fixes. When a trigger-type provider re-registers (e.g. after a reload), the engine now replaces the stale registrator and replays existing bindings to it — previously new bindings landed but never fired. And `engine::register_trigger` bindings are durable engine-side state, removed only by explicit `engine::unregister_trigger` — a CLI's short-lived connection or a proxy restart no longer reaps them. Worker-owned `RegisterTrigger` bindings keep connection ownership and disconnect GC.
## `engine::functions::list` results are relevance-ranked
Search results are ordered by relevance instead of alphabetically: id-anchored matches first (a `::` segment equals the needle, or the id starts with it), id substrings second, description-only matches last. The same functions are returned — only the order changes. Listing without a search term stays alphabetical.
## `engine::functions::info` accepts batches
`engine::functions::info` now takes a `function_ids: [...]` batch (up to 32, deduplicated in request order) alongside the historical single `function_id`, eliminating one round-trip per function after `functions::list`. Batch calls return `{ functions: [...] }` with a per-id `FunctionDetail` or `{ function_id, error: "forbidden" | "not_found" }` — one bad id never fails the batch. Single-id calls keep the flat wire shape and error codes unchanged.
## Live pending spans: traces show work in flight
Trace views previously only showed spans after they finished, so long-running invocations looked idle. With the new `live_spans` setting (default on for `exporter: memory`, opt-in for `both`), the engine mirrors spans into the in-memory store at start as `pending` snapshots; the final span replaces the snapshot in place, and OTLP still exports only finished spans. `engine::traces::list` / `group_by` understand pending spans (duration-so-far) and gain `trace_ids` batch lookup, `exclude_attributes` filtering, `label_attribute` for group headings, and merged `trace_tags` from `iii.tag.*` attributes. There is also a global feed that includes every span seen in each processing window.
## Distributed traces: inbound `tracestate` honored, remote-parent traces visible
The engine previously seeded only `traceparent` on inbound HTTP and dropped the companion `tracestate` header; it is now honored at the HTTP boundary and exposed on spans through the traces API. Traces entering iii from a remote caller were also invisible — `traces::list` and `traces::tree` only treated spans with no parent as roots, so a trace whose server span pointed at a remote (never-stored) parent was hidden. Spans with dangling remote parents are now treated as roots. The `baggage` header is redacted in header logs.
## RabbitMQ priority queues
The RabbitMQ queue provider supports priority queues on both function and subscriber queues. Configure `max_priority` (declares the queue with `x-max-priority`) and `priority_field` (an integer field in the message `data` that stamps each message's priority — no enqueue API or SDK change) under `queue_configs.`; subscribers use `maxPriority` in their `queue_config`. Main and retry queues both carry `x-max-priority`, so priority survives retries and DLQ redrive. Note that `x-max-priority` is fixed at queue creation. Subscriber consumers also gain per-consumer `basic_qos` prefetch for broker-side backpressure.
## `iii t` — a shorthand for `trigger`
`iii t my::function key=value` is equivalent to `iii trigger my::function key=value`, with all arguments and dynamic help unchanged.
## One `iii.worker.yaml` for the registry and the engine
Manifests published from the workers repo carry `iii`, `deploy`, and `manifest` top-level keys — required by the registry's release CI. The engine's validator treated them as unknown keys and hard-failed `iii worker add`, so a single manifest could not satisfy both. The validator now accepts the three publish metadata keys and ignores them at add/start time; a non-string value still gets the friendly dotted-path error.
## Bundle workers: pick any rootfs, install dependencies at first boot
Bundle manifests may now set `runtime.base_image` to **any OCI image reference** (e.g. `oven/bun:1`), not just the engine-preset images — the rootfs is publisher-chosen, in the same trust context as the `scripts.start` command the engine already executes. A ref with characters that don't belong in an OCI reference is rejected at **install time** instead of silently falling back to the default image at start.
Bundles may also declare `scripts.install` — it runs once inside the sandbox VM (guarded like the rest of the boot script), which python bundles need for pip/browser bootstrap that can't be vendored per-arch into the archive. `scripts.setup` stays rejected: OS provisioning belongs in the base image.
## Compound `scripts.start` commands run correctly
A `scripts.start` with shell metacharacters (`&&`, pipes, env prefixes) is now exec'd via a quoted `sh -c`, so compound start commands behave the same as they do in a terminal instead of being split naively on whitespace.
## Configuration: env placeholders in numeric fields, and a friendlier config directory
A lone `${VAR:default}` placeholder in a numeric field — e.g. `port: ${HTTP_PORT:3111}` — is now env-expanded and coerced to the right scalar type, so schema validation passes instead of rejecting it as a string. The default config store moved from `./data/configuration` to `./iii-config` (one-time auto-migration on first boot; explicit `directory:` overrides are untouched), and direct edits to `./iii-config/*.yaml` hot-reload through the same rebind path as `configuration::set`. The HTTP worker also gracefully shuts down the old listener when its host or port rebinds.
## Docker projects start cleanly on Linux
`iii project init --docker && docker compose up` could crash-loop on Linux: the distroless engine image runs as non-root UID 65532 with no writable data directory. The image now pre-creates `/app/data` and `/data` owned by that UID, and the generated compose template mounts an `iii_data` named volume at `/app/data`, so configuration data persists across restarts. Generated compose files also load `.env` via `env_file`, so `${VAR}` placeholders in worker configs resolve from the project's `.env`. The stale port `9464` exposure is dropped.
## `exporter: both` no longer strips the service name from OTLP spans
With `exporter: both`, the tee exporter didn't forward `set_resource` to the wrapped OTLP exporter, so every engine span shipped with an empty resource — Tempo showed root spans as `` and span-metrics lost their `service` label. Engine spans now carry the configured `service_name` and version in OTLP payloads. Pure `exporter: otlp` was never affected.
## Node SDK: `ChannelWriter` reconnects after a dropped WebSocket
After a connection drop, sends went to a dead socket — throwing or silently dropping data — because the writer's ready flag was never reset on `close`/`error`. Sends now check the live socket state, requeue on failure, and the writer transparently reconnects and flushes the queue when the new socket opens.
## Worker configuration moves to `./config` and edits apply in real time
Every worker's runtime settings live in the **configuration worker** — one schema-validated YAML file per worker — and 0.21 makes that layer visible, relocatable, and truly live. The new [Using iii / Configuration](/next/using-iii/configuration) page documents the full lifecycle: a worker's `config:` block in `config.yaml` is a first-boot seed; after that, its settings are managed in real time from the file on disk, the console, or `configuration::set`.
**The default store moves from `./data/configuration/` to `./config/`.** When running on the default location, entries still in `./data/configuration` are migrated across once on boot (logged at INFO); a file that already exists at the new location is skipped with a WARNING and the legacy copy is left in place to reconcile. An explicit `directory:` override disables the migration entirely. The folder is set on the configuration worker's own block:
```yaml config.yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
workers:
- name: configuration
config:
adapter:
name: fs
config:
directory: ./config
```
**Editing a persisted file on disk now hot-applies.** Direct edits to `./config/.yaml` are picked up by the fs adapter's watcher (the watch path is canonicalized, fixing macOS FSEvents on relative paths) and fan out through the same validate/apply path as `configuration::set` — a hand edit rebinds a port the same way an SDK call does. Edits are validated against the registered schema after `${VAR}` expansion: an invalid edit is rejected with a warning and the previous good value stays in effect. The worker's own writes are deduped so a save can't trigger a reload loop.
**`${VAR}` placeholders work in numeric fields.** A field that consists of a single placeholder — e.g. `port: ${HTTP_PORT:3111}` — is env-expanded and YAML-coerced to the schema's scalar type (int, bool, float), so it validates as the integer `3111` instead of being rejected as a string. Validation always runs against the expanded value; the template is stored verbatim and re-expanded on every read. A placeholder with no env value and no default keeps the entry on its last valid value instead of failing the load.
**Host/port rebinds free the old port for real.** When a configuration change rebinds the `iii-http` listener, the old server is now shut down gracefully — closing its open connections, including idle keep-alive connections that would otherwise keep serving the old address — with a hard-abort backstop after 10 seconds so a stuck connection can't pin the old port. The engine REST API server rebinds the same way.
## SDK: a major, breaking reorganization of the public surface
The Node, Python, and Rust SDKs are reorganized for a consistent, well-factored public surface. Names are aligned across the three languages, the shared types move into a new package, and the rest of the root surface is grouped into named submodules. This release is breaking: many symbols move out of the SDK root.
Upgrading an existing app? Follow the step-by-step
[Upgrading from 0.19.x to 0.20.x](/next/upgrading/from-0-19-x) guide.
### Shared types extracted into `@iii-dev/helpers`
The HTTP, queue, stream, and worker-connection-manager types move out of the root SDK into a new separately published package, **`@iii-dev/helpers`** (npm), **`iii-helpers`** (crate and PyPI, import `iii_helpers`), organized into four submodules. The root exports are removed; import these types from the helpers package.
| Submodule | Node | Python | Rust |
| ------------------------- | -------------------------------------------- | --------------------------------------- | ---------------------------------------- |
| http | `@iii-dev/helpers/http` | `iii_helpers.http` | `iii_helpers::http` |
| queue | `@iii-dev/helpers/queue` | `iii_helpers.queue` | `iii_helpers::queue` |
| stream | `@iii-dev/helpers/stream` | `iii_helpers.stream` | `iii_helpers::stream` |
| worker-connection-manager | `@iii-dev/helpers/worker-connection-manager` | `iii_helpers.worker_connection_manager` | `iii_helpers::worker_connection_manager` |
`IStream`, the streaming `StreamRequest` and `StreamResponse`, `MiddlewareFunctionInput`, and `TriggerActionEnqueue` stay in the root SDK. See [the helpers entry](./0-20-0/extract-helpers-lib) for the full symbol list and migration.
### Cross-language parity: renames and removals
* Main client handle aligned to `IIIClient` (Node was `ISdk`, Rust was `III`).
* Telemetry config aligned to `TelemetryOptions` (Rust was `WorkerTelemetryMeta`).
* Buffered HTTP request and response renamed `ApiRequest` and `ApiResponse` to `HttpRequest` and `HttpResponse`; streaming keeps `StreamRequest` and `StreamResponse`. Rust `StreamCallRequest` and `StreamJoinLeaveCallRequest` become `StreamChangeEvent` and `StreamJoinLeaveEvent`.
* Stream helper type names aligned across the three SDKs: the change-event detail is `StreamChangeEventDetail` everywhere (Rust was `StreamEventDetail`, Node was an inline object), the result types are `StreamSetResult` / `StreamUpdateResult` / `StreamDeleteResult` (Rust had bare `SetResult` / `UpdateResult`, Node and Rust had a bare `DeleteResult`), and `MergePath` is now a named export in all three.
* Removed: `UpdateBuilder`, the Rust `Value` re-export, `IIIForbiddenError` and `IIITimeoutError` (Python), the `TriggerActionType` alias, the Rust `FieldPath` path helper, and the root `Logger` and OpenTelemetry re-exports (use `@iii-dev/observability`).
See [the parity entry](./0-20-0/cross-language-parity) for the full table.
### Submodule grouping
The remaining root surface is grouped into submodules (`iii-sdk/` / `iii.` / `iii_sdk::`):
* **errors** (`InvocationError`, `Error`), **channel**, **trigger**, **runtime**: the first slice. The `III` prefix is dropped from the invocation error types. These keep their old root paths working as deprecated aliases. See [the first slice](./0-20-0/organize-sdk-imports).
* **engine** (`EngineFunctions`, `EngineTriggers`, `RemoteFunctionHandler`), **protocol** (the message and register-input types), **internal** (`InternalHttpRequest`), and Python **utils** (the format helpers): these move out of the root with the root export removed. `IIIConnectionState` (runtime) and `TriggerActionVoid` (Python trigger) likewise leave the root. See [the submodule entry](./0-20-0/submodule-clean-break).
### Compatibility
This release is not backward compatible. The first errors/channel/trigger/runtime slice keeps deprecated root aliases, but everything extracted into `@iii-dev/helpers`, the renamed types, and the `engine`/`protocol`/`internal`/`utils` groups are removed from the root SDK. Update imports to the submodule and helpers paths.
### Observability moved into `@iii-dev/helpers`
The `Logger`, `initOtel`, `withSpan`, `executeTracedRequest`, and the rest of the public observability API move from the standalone `iii-observability` packages into a new `observability` submodule of the helpers package. The OpenTelemetry dependencies are now bundled in helpers (net-neutral for SDK users). The standalone `@iii-dev/observability` / `iii-observability` packages become deprecated shims that re-export from helpers; they continue to work but emit a deprecation signal. See [the observability entry](./0-20-0/observability-into-helpers) for the full path table and migration.
| Submodule | Node | Python | Rust |
| ------------- | -------------------------------- | --------------------------- | ---------------------------- |
| observability | `@iii-dev/helpers/observability` | `iii_helpers.observability` | `iii_helpers::observability` |
## Saving a file in a local overlay worker restarts cleanly the first time
Editing and saving a file in a local-path worker on the overlay copy-in path could fail on the **first** save with `iii: ERROR /mnt/host-src … is not a virtiofs mountpoint`, then work on the next save. The host source watcher tried an in-VM **fast restart** first — but that re-runs the boot script, and in copy-in mode the script **detaches** `/mnt/host-src` once it has copied your project into the VM. The re-run hit the now-absent share and aborted; the watcher then fell back to a full VM restart, which is why the second save worked.
The fast in-VM restart is now skipped for overlay copy-in workers, so they take a full VM restart — the only path that re-copies your edited source. (Even when it didn't error, a fast restart would have run **stale** source, since the project is copied at boot rather than live-mounted.) Legacy live-mount workers keep the fast restart; bundle workers don't auto-restart on source edits at all. Net: one save, one clean restart, running your latest changes.
## `config.yaml` worker blocks are stripped once their value is persisted
Every worker's runtime settings already live in the configuration worker — the `config.yaml` block was **seed-only**, read once on first boot and ignored afterwards. But the block still sat in `config.yaml`, looking authoritative while being dead. Now, once a worker's value is persisted in the configuration store, its `config:` block is **removed from `config.yaml`** on the next boot: the `- name:` entry is kept and the block is replaced with a one-line breadcrumb pointing at where the value now lives (`./data/configuration/.yaml` with the default `fs` adapter) and how to change it (`iii config set ` or `configuration::set`). This covers built-in workers (stripped on the first boot after they seed) and external workers like `shell` that register over the bus (stripped on the first boot after their value is in the store).
The rewrite is line-based and byte-preserving on purpose — it deletes only the contiguous block and inserts one comment, so comments, key order, and `${VAR:default}` placeholders in **other** blocks are left exactly as written (a `serde_yaml` round-trip would have mangled all three). The file is written atomically, and the strip is best-effort: a read/write failure is logged and never fails boot, and a second boot with the block already gone is a no-op. The strip runs before the file watcher starts, so it never trips a config reload.
Relatedly, the persisted store files are now **value-only**: the configuration worker writes just the setting value to `.yaml` and no longer dumps the JSON schema alongside it, so the on-disk files are smaller and hand-editable.
## Overlay workers boot on every architecture
The shared read-only overlay base introduced in 0.19.5 was built as **squashfs**, but the libkrunfw guest kernel on **x86\_64** ships without `CONFIG_SQUASHFS`. Mounting the lower returned `ENODEV`, the overlay never assembled, the VM never reached ready, and the worker timed out. It only surfaced on x86\_64 — aarch64 (Apple Silicon) dev kernels carry squashfs, so local boots passed while Linux/x86\_64 hosts and CI runners failed.
The overlay lower is now built as **erofs** — the one read-only filesystem the libkrunfw kernel mounts on **every** architecture (erofs + lz4 are present on both x86\_64 and aarch64). The image is **uncompressed**, so the guest mounts it with no decompressor and pays no read-time decompression cost. The committed libkrunfw firmware is refreshed (kernel 6.12.68, ABI soname unchanged) to a build that carries erofs on every arch. Building the base is a pure-Rust, std-only erofs writer with no external dependency, and it streams through an on-disk spool to stay within bounded memory.
Trade-off: an uncompressed erofs base cache is **larger than the old squashfs+gzip** (\~3.3×, observed 763 MB vs 230 MB for the node base). A worker's stale `.sqfs` base from a 0.19.5 prerelease is garbage-collected the next time the base is extracted.
The overlay opt-out is unchanged — `III_ROOTFS_MODE=legacy` (env) or `rootfs.mode: legacy` in `config.yaml` falls back to the per-worker-clone boot.
## Workers share one read-only rootfs instead of cloning it per worker
A managed worker used to get a **full private copy** of its base-image rootfs under `~/.iii/managed//`. With several workers on a box that duplication dominated disk use. Workers now boot from a **shared read-only base** — built once per base image, host-side — layered with a **per-worker writable ext4 upper** via overlayfs; `iii-init` assembles the overlay and `pivot_root`s into it. The per-worker managed directory drops from a full clone to a small trampoline plus a sparse upper file — in a node-worker test, **731 MiB → \~9 MiB on disk** — and the base image is stored once and reused across every worker built on it. The model is image-independent: the upper ships as a prebuilt golden ext4, so even a minimal or distroless base with no filesystem tools boots, and nothing inside the image is used to assemble the overlay.
Overlay is **on by default**, but activates only when `iii-worker` is built with an embedded, version-matched `iii-init` — so default development builds keep the exact legacy clone behavior. Opt out explicitly with `III_ROOTFS_MODE=legacy` (env) or a top-level `rootfs.mode: legacy` in `config.yaml`; both fall back to the per-worker-clone boot with no overlay disks attached. The env var wins over `config.yaml` when both are set. On a worker's first overlay boot, the orphaned dep cache from the old layout is reclaimed.
## Local workers no longer leave empty dependency folders in your project
A local-path worker mounted your project directory writable into the VM, so the dep and build directories created during install (`node_modules`, `.venv`, `dist`, …) leaked back onto the host as empty folders. Under overlay, a local worker now mounts the host project **read-only** and copies the source into a VM-local workspace on the upper (dep and build dirs excluded at any depth), so the host project is only ever read — those phantom folders no longer appear. Bundle and legacy workers keep the live mount.
## Overlay local workers boot reliably and pack large base images
Two reliability fixes for the new overlay boot path:
* **The host source share mounts reliably.** In copy-in mode the host project is shared into the VM as a virtio-fs device that `iii-init` mounts right after the root pivot. virtio-fs device registration is asynchronous, so a mount issued too early could transiently fail — and because that share is boot-critical (the worker aborts when it isn't mounted), it surfaced as an **intermittent** boot failure (`/mnt/host-src … is not a virtiofs mountpoint`). `iii-init` now retries each share mount for \~2s to ride out the device-probe race; a genuinely failed mount logs the cause instead of being silently swallowed.
* **Large base images pack within bounded memory and file descriptors.** Building the shared read-only base streams each file's bytes through a single on-disk spool instead of holding the whole image in memory or keeping one open descriptor per file, so even a large base (the node image carries \~15k files) packs cleanly.
## `iii-pubsub` runtime settings live in the configuration worker
The pub/sub worker now registers its config schema under the `iii-pubsub` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — no restart.
The pub/sub `adapter` is a **full hot-swap tier** (unlike `iii-state`'s restart-tier storage adapter): a runtime edit rebuilds the backend — `local` (in-process broadcast) or `redis` (cross-instance delivery) — re-subscribes the live subscriptions onto the new backend before swapping it in so there is no delivery gap, then tears down the previous backend, aborting its per-topic tasks rather than leaking them. A value that fails to build the backend is gated and keeps the previous one running.
The `adapter` field advertises a **concrete per-adapter schema** — a discriminated union keyed on `name` over the built-in `local` and `redis` adapters. Each branch is closed: `redis` carries a typed `redis_url`, and `local` (which takes no config) is a closed empty object, so `configuration::set` rejects an unknown adapter name *and* a junk config key on either branch, while a schema-driven UI renders per-adapter fields instead of a free-form object. Deserialization stays lenient, so a hand-edited persisted file is still tolerated at boot.
As with the other workers, the config.yaml block is **seed-only** once a value is persisted; change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-pubsub.yaml` with the default `fs` adapter). `${VAR:default}` placeholders expand on read.
## `iii-stream` runtime settings live in the configuration worker
The WebSocket stream worker now registers its config schema under the `iii-stream` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — no restart. The schema carries per-field descriptions, rejects unknown keys at set time, and expands `${VAR:default}` placeholders on read. As with the other workers, the config.yaml block is **seed-only** once a value is persisted — change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-stream.yaml` with the default `fs` adapter), and a runtime edit survives engine restarts.
Each field applies on its own tier. `auth_function` applies to new connections immediately — no rebind. A `host`/`port` change **rebinds the listener**: the new address is bound and the server respawned before the old listener is torn down, a failed bind keeps the previous server, and live connections on the old address are dropped (clients reconnect). An `adapter` change **hot-swaps the pub/sub backend**: the new backend is built, swapped in, and its event pump restarted; the swap is gated (a value that fails to build keeps the previous backend), and existing connections stay bound to the previous backend until they close, so prefer a quiet moment to repoint the adapter in a multi-instance deployment.
The `adapter` field advertises a **concrete per-adapter schema** — a discriminated union keyed on `name` over the built-in `kv`, `redis`, and `bridge` adapters, each with its own typed `config` (`kv`: `store_method` / `file_path` / `save_interval_ms` / `channel_size`; `redis`: `redis_url`; `bridge`: `bridge_url`). `configuration::set` validates adapter settings against it and rejects an unknown adapter name or stray config key, and a schema-driven UI renders per-adapter fields instead of a free-form object. Deserialization stays lenient, so a hand-edited persisted file is still tolerated at boot.
## `iii-cron` runtime settings live in the configuration worker
The cron worker now registers its config schema under the `iii-cron` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — no restart. The config.yaml block is **seed-only** once a value is persisted; change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-cron.yaml` with the default `fs` adapter), and a runtime edit survives engine restarts. `${VAR:default}` placeholders expand on read, and values are validated against the schema (which carries a per-field description and rejects unknown keys) at set time.
Cron's only setting is the distributed-lock `adapter`, and a change is a **full hot-swap**: the new lock backend is built first (gated — a value that fails to build keeps the previous backend, config, and jobs), the old backend is shut down, then every live cron job is re-registered onto the new one (best-effort per job), and the `(config, scheduler)` pair is swapped in atomically. The brief swap window is a scheduling gap on this instance — a job whose fire time lands in it is skipped, not double-run — while across a multi-instance fleet the old and new lock backends cannot coordinate mid-migration, so prefer a quiet moment to repoint the adapter.
The `adapter` field advertises a **concrete per-adapter schema** — a discriminated union keyed on `name` over the built-in `kv` and `redis` lock backends, each with its own typed `config` (`kv`: `store_method` / `file_path` / `save_interval_ms` / `lock_ttl_ms` / `lock_index`; `redis`: `redis_url` only, since the lock TTL and key prefix are fixed by the adapter). `configuration::set` validates adapter settings against it and rejects an unknown adapter name or a stray config key (e.g. a `redis` `lock_ttl_ms`), and a schema-driven UI renders per-adapter fields instead of a free-form object. Deserialization stays lenient, so a hand-edited persisted file is still tolerated at boot.
## `iii-observability` is fully configurable through the configuration worker
The engine's observability worker registers its complete configuration surface — trace/metric/log exporters, sampling (ratio, per-operation rules, rate limits), in-memory store limits and retention, alert rules, span-collapse rules, and the engine log level/format — under the `iii-observability` configuration entry. The config.yaml block seeds the entry on first boot and is ignored afterwards; runtime edits survive restarts.
Changes hot-apply per field tier: ingest gates, `logs_console_output`, and `logs_sampling_ratio` are read live; store limits retune on the next insert/sweep; the sampler, alert rules (with cooldown/firing continuity for surviving rules), span-collapse cache, and engine log level rebuild and swap in place; the OTLP logs exporter and log-retention tasks restart with new settings. Restart-tier fields (trace exporter wiring, resource identity, log format) are warned about and apply at the next engine start — the persisted entry is read at boot, before logging init, so they genuinely take effect.
The schema rejects invalid values at `configuration::set` time (unknown fields, sampling ratios outside `0..=1`, zero counts, log batch/flush sizes outside the exporter's accepted range), and out-of-range values that predate a schema tightening are clamped on read.
`engine::alerts::list` now also returns the configured `rules` alongside the live alert states, so a caller sees the active thresholds and their firing status in one call.
This work also fixed three latent dead-config bugs: `metrics_max_count`/`metrics_retention_seconds` and `logs_max_count` were silently shadowed by defaults claimed before the worker booted, and advanced `sampling` rules never reached the engine tracer because the sampler was built before the global config existed. All three now apply — and the first two are live-tunable.
## `iii-queue` runtime settings live in the configuration worker
The queue worker now registers its config schema under the `iii-queue` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — no restart. Per-queue settings live in `queue_configs`, keyed by queue name (with the always-present built-in `default` queue): `max_retries`, `concurrency`, `type` (`standard` or `fifo`), `message_group_field`, `backoff_ms`, and `poll_interval_ms`. A runtime edit diffs `queue_configs` and restarts only the consumers that changed — added queues start a consumer, removed queues stop theirs, untouched queues keep running. The schema carries per-field descriptions and rejects invalid values at set time, including `concurrency: 0` (which would otherwise panic the consumer's bounded channel) and a `fifo` queue with no `message_group_field`; the same guards re-run on the config.yaml seed, which bypasses the schema.
The `adapter` is a **full hot-swap tier** (like `iii-pubsub`, unlike `iii-state`'s restart-tier storage adapter): a runtime edit re-instantiates the transport and restarts every consumer on it, with no engine restart. The new transport is built first and the swap is gated — a value that fails to build keeps the previous transport, config, and consumers running. Because the new transport starts empty, messages still queued on the old one do not migrate, and in-flight deliveries finish acking against the old transport before it is dropped — so prefer a quiet moment to repoint the adapter.
The `adapter` field advertises a **concrete per-adapter schema** — a discriminated union keyed on `name` over the built-in `builtin` (in-process, the default), `redis`, and `bridge` adapters, each with its own typed `config` (`builtin`: `store_method` (`in_memory` / `file_based`) / `file_path` / `save_interval_ms`; `redis`: `redis_url`; `bridge`: `bridge_url`). The `rabbitmq` adapter and its config (`amqp_url` / `max_attempts` / `prefetch_count` / `queue_mode`) join the union only when the `rabbitmq` feature is compiled in, so `configuration::set` never accepts an adapter name the engine cannot build. The set is closed: `configuration::set` rejects an unknown adapter name or a stray config key, and a schema-driven UI renders per-adapter fields instead of a free-form object. Deserialization stays lenient, so a hand-edited persisted file is still tolerated at boot.
As with the other workers, the config.yaml block is **seed-only** once a value is persisted; change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-queue.yaml` with the default `fs` adapter). `${VAR:default}` placeholders expand on read.
## `iii.worker.yaml` is validated before anything is installed
Local `worker add` (CLI and `worker::add` trigger) now validates the manifest **before any artifact is written**. A typed schema is the single source of truth, and it is strict where it matters:
* **Unknown keys are rejected** — collected and sorted, so a manifest with several typos surfaces them all in one run:
```
unknown key(s) in ./my-worker/iii.worker.yaml: [runtimee, scrpits].
Supported fields are: name, description, runtime.base_image,
scripts.(setup|install|start), env, dependencies, resources.(cpus|memory).
```
* **Deprecated keys warn but still work** (`runtime.kind` / `package_manager` / `entry` / `language`, top-level `config` / `language` / `entry`) — honored for now, scheduled for removal in a future version, with migration hints per key. Silence with `III_NO_DEPRECATION_WARN=1`.
* **Shape errors read in plain English** — `runtime: node` reports "`runtime` must be a mapping", not serde's "expected struct RuntimeSection". A manifest missing `name` says so precisely instead of the false "No project manifest detected".
* The same strict validation re-runs at `start`, with a 64 KiB size cap that defuses hostile (billion-laughs) or accidental multi-GB manifests before they are slurped into host memory.
**Breaking:** a local manifest with unknown keys that previously installed silently now fails `worker::add`. Fix the typo or remove the key — `worker::validate` (below) tells you exactly which.
## Author → check → add: `worker::validate` and the manifest contract
Two new ops close the authoring loop for LLMs and automation:
* **`worker::validate`** dry-runs a manifest — inline string or host path — and returns `{ valid, name, errors, unknown_keys, deprecated_keys, warnings }` without touching anything. `valid: true` means `worker::add` would accept it.
* **`worker::schema { function_id: "iii.worker.yaml" }`** serves the manifest's JSON Schema (closed-world: every field described, unknown keys rejected) plus a ready-to-write hello-world bundle using the correct `iii-sdk` package — self-tested against our own validator so the example can never drift from the rules.
## `worker::status`: one worker, full picture
`worker::status { name }` reports a single worker's config entry, sandbox state, process liveness, and recent logs in one call, with actionable hints — a local worker without logs yet reports that it is still provisioning rather than implying failure. Like every `worker::*` op it carries schemars-generated request/response schemas via `engine::functions::info`.
## Local-path workers: stop re-adding after code edits
Local workers run their project directory live and get a host-side source watcher: editing a source file auto-restarts the worker, and editing a dependency manifest (`package.json`, `Cargo.toml`, …) forces a full restart. Re-running `worker::add` after a code change is pure waste — it re-registers triggers and re-runs install scripts. The `worker::add` / `worker::update` descriptions now say this to LLM callers directly: re-add with `force: true` only when `iii.worker.yaml` itself changes, and `worker::update` never touches local-path workers (it re-resolves registry workers pinned in `iii.lock`).
Also fixed: a forced local re-add returned the raw path (`/tmp/hello-world`) as the worker name instead of the manifest name (`hello-world`) — no other `worker::*` op accepts the path form, so the response was a dead end.
## `worker add --force` reruns install when dependencies change
`--force` is the documented way to rebuild a local worker after its `iii.worker.yaml` or lock file changes. It stops the worker and clears artifacts, but a leftover binary or OCI image under `~/.iii` could trip an `if freed == 0` guard that left the managed directory in place — and with it the `.iii-prepared` marker and the `/var/iii/deps` caches. The next boot saw the marker, skipped `setup`/`install`, and reused stale dependencies, so a freshly added package surfaced as a `ModuleNotFoundError` at runtime.
The managed directory is now always wiped on `--force` (it is a distinct path from the image cache, so there was never a double-count to guard against), and the `.iii-prepared` marker is removed explicitly as a backstop even when the directory wipe partially fails. A changed lock file reinstalls.
## W120 tells you when the lock holder is the daemon itself
When the worker lock is held by the `iii-worker-ops` daemon — the process serving the `worker::*` API — the W120 `LockBusy` error now says so and tells the caller **not** to kill that pid. From a real harness session: an LLM saw "lock held by pid N", killed N, and took down the entire worker management API it was using.
## `worker::logs` output is terminal-sanitized
Logs fetched over the bus are stripped of ANSI/OSC escapes and spinner residue — raw escape bytes are token noise for LLM consumers and can rewrite the reader's terminal. Pass `raw: true` to opt out.
## `iii-state` runtime settings live in the configuration worker
The state worker now registers its config schema under the `iii-state` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — no restart. Two new runtime knobs apply live: `triggers_enabled` globally pauses/resumes state change-trigger fan-out, and `max_value_bytes` rejects oversized `state::set` writes with `VALUE_TOO_LARGE`. `save_interval_ms` retunes the file-backed `kv` save cadence by respawning the save loop. The schema carries per-field descriptions and rejects invalid values (e.g. `max_value_bytes: 0`, `save_interval_ms: 10`) at set time.
The storage `adapter` is **restart-tier**: a change is logged and takes effect at the next engine start, where a boot-read of the persisted `iii-state` entry drives adapter construction — so a runtime-edited adapter survives restarts. As with the other workers, the config.yaml block is **seed-only** once a value is persisted; change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-state.yaml` with the default `fs` adapter). `${VAR:default}` placeholders expand on read.
The `adapter` field advertises a **concrete per-adapter schema** — a discriminated union keyed on `name` over the built-in `kv`, `redis`, and `bridge` adapters, each with its own typed `config` (`kv`: `store_method` / `file_path` / `save_interval_ms`; `redis`: `redis_url`; `bridge`: `bridge_url`). `configuration::set` validates adapter settings against it and rejects an unknown adapter name or stray config key, and a schema-driven UI renders per-adapter fields instead of a free-form object. Deserialization stays lenient, so a hand-edited persisted file is still tolerated at boot.
## OTLP observability exports work with TLS collectors
OTLP trace and metric exporters now honor OpenTelemetry protocol environment variables and automatically enable TLS for HTTPS gRPC collector endpoints. HTTP/protobuf endpoints are normalized to the correct `/v1/traces` or `/v1/metrics` signal path even when the configured endpoint already includes another OTLP signal path.
Log export also applies headers from `OTEL_EXPORTER_OTLP_LOGS_HEADERS` or the global `OTEL_EXPORTER_OTLP_HEADERS`, including percent-decoded header values, so authenticated collectors receive the configured credentials.
## Hardening
* The VM UDP relay now writes a real pseudo-header checksum on response packets (strict guest stacks drop zero-checksum UDP), bounds payloads to the 16-bit IP length limit instead of silently truncating, and no longer injects empty frames into the guest's rx ring.
* Shell frame readers (proto, client, relay) zero-initialize their buffers instead of using uninitialized memory before the socket read.
## `iii-http` runtime settings live in the configuration worker
The HTTP worker now registers its config schema under the `iii-http` configuration entry, seeds it from the config.yaml block on first boot, and hot-applies changes via `configuration::set` — CORS, timeout, concurrency limit, and global middleware swap without dropping the listener; a `host`/`port` change binds the new address before tearing down the old one, and a failed bind keeps the previous server running. `${VAR:default}` placeholders expand in string fields. The schema carries per-field descriptions and rejects invalid values (including `concurrency_request_limit: 0`) at set time.
The config.yaml block is **seed-only**: once a value is persisted, later edits to that block are ignored — change settings via `configuration::set` or by editing the persisted file (`./data/configuration/iii-http.yaml` with the default `fs` adapter).
**Breaking — HTTP error envelope:** errors generated by the `iii-http` server itself (handler invocation failure, middleware failure/timeout, unmet route condition, route-miss 404s, and response-stream build failures) now return `{"error": {"code", "message", "error_id"?}}` instead of flat strings or plain-text bodies. `code` is machine-readable (e.g. `MIDDLEWARE_TIMEOUT`, `CONDITION_NOT_MET`, `INTERNAL_ERROR`, `NOT_FOUND`); `error_id` correlates 5xx responses with server logs. Bodies returned by your own handlers and middleware pass through unchanged. Clients parsing the old flat string must read `error.message` instead.
## Nothing the engine started survives it — zombie-worker leak fixed
An abnormal engine death (`kill -9`, OOM, crash, dev hard-restart) used to leave the engine's entire spawn tree running forever: orphaned `worker-manager-daemon` / `sandbox-daemon` processes reconnect-looped indefinitely (one incident accumulated 19), an orphaned sandbox-daemon pinned live multi-GB libkrun VMs, and managed-worker VMs, `__watch-source` sidecars, and binary workers all survived a `killall -9 iii`.
Engine death is now detected three ways — a lifeline pipe the kernel closes on any death including `SIGKILL`, an `III_ENGINE_PID` handshake that flows down the whole spawn tree, and a hardened reparent fallback — and triggers a cascade that tears down daemons, VMs, watchers, and workers. The worker-ops daemon runs a session reaper that stops every `config.yaml` worker host-side without needing the engine. Self-exits leave a one-line breadcrumb in `~/.iii/logs/.log`. Residual: binary workers leak only if the engine **and** the worker-ops daemon are `SIGKILL`ed in the same instant.
## Daemon no longer panics when the engine dies with broken stdio
When the engine died abnormally, its stdout/stderr pipes became broken, and the SDK connection thread's reconnect logging hit the tracing layer's `eprintln!` fallback — which panicked (`failed printing to stderr: Broken pipe`) and killed the daemon before its engine-gone reaper could run. The tracing writer now swallows write errors (`ResilientStdout`) so logging can never panic the process; the durable exit-log redirect stays for forensics.
## Live trace feed is tree-correct and lower-latency
Spans whose real parent was an engine-internal wrapper rendered as detached "phantom roots" in the live detail view. The detail stream now rebuilds each touched trace through the same prune+collapse pipeline as the REST tree, emitting corrected `parent_span_id` and the full ancestor chain (upsert-by-`span_id`, so frames stay self-contained). The Node SDK's `BatchSpanProcessor` flush delay drops from OpenTelemetry's default 5000ms to 100ms (configurable via `spansFlushIntervalMs`), cutting the dominant source of console lag; console trace polling drops from 3s to 1s.
## Go SDK: connection-scoped replies no longer leak across reconnects
The Go SDK drained connection-scoped replies (pong, `InvocationResult`, `TriggerRegistrationResult`) from a single shared outbound channel. If a reply was buffered when the socket dropped, the next connection's `writeLoop` could send that stale reply on the wrong socket. Replies now route through a per-connection channel created on connect and detached on teardown; a reply enqueued with no live connection is dropped.
## Conflicting HTTP routes are rejected instead of crashing the worker
Registering two HTTP routes with identical structure but different path-parameter names — e.g. `GET users/:id` and `GET users/:userId` — used to panic axum's matcher and take down the entire `iii-http` worker thread. `register_router` now detects the structural conflict up front and rejects the second registration with a descriptive error:
```
Route 'GET users/:userId' conflicts with already-registered route 'GET users/:id':
routes with identical structure but different path-parameter names are not supported
```
The first route keeps serving; only the conflicting registration fails, and the worker stays up. Route conflict tests were added across the Go, Node, Python, and Rust SDKs.
## HTTP trigger unregister is owner-aware
When two workers registered the same `method` + `path` — during a rolling deploy or a reconnect — a departing worker's route cleanup could delete the route the new worker had just taken over, dropping the endpoint to a 404. `unregister` now checks ownership (`trigger_id` + `worker_id`) and skips the removal when the route already belongs to a different owner, so the live worker's route keeps resolving. Removal by the actual owner is unchanged.
## Install script retries transient download failures
`install.sh` now wraps every GitHub API call and binary download (`iii`, `iii-init`, `iii-worker`) in a retry (`--retry 5`, `--retry-delay 2`, `--connect-timeout 10`). Transient 5xx responses and connection timeouts are retried instead of failing the install on the first hiccup. Only widely-supported curl flags are used, so older curl builds keep working.
## `worker::*` management API is self-describing — and `kind: "local"` now works over the trigger
Every `worker::*` op (`add`/`remove`/`update`/`start`/`stop`/`list`/`clear`/`schema`) now publishes its request JSON Schema, a description, and `default_timeout_ms` / `idempotent` metadata through `engine::functions::info` and `worker::schema`, so an LLM or automation caller can discover the full contract without out-of-band docs. Workers can also report a one-line `description` (Node, Go, Rust, and Python SDKs) that surfaces in `engine::workers::list` / `engine::workers::info`.
**Breaking — error codes on the wire:**
* Malformed `worker::*` payloads now return **W105** (`BadRequest`) instead of **W101** (`InvalidSource`). The envelope's `details.hint` names the `worker::schema` call that returns the request schema. **W101** and **W102** are now reserved (documented but never emitted) — consumers matching `W101` for malformed payloads should match `W105`.
* `worker::*` op failures now surface the **W-code** as the transport `ErrorBody.code` (previously the generic `"invocation_failed"`, with the W-code only inside the message envelope). Consumers that matched `code == "invocation_failed"` to detect worker-op failures should match the W-code instead.
**Breaking — `worker::add { kind: "local" }` over the trigger:** the identical request that previously returned **W102** (rejection) now **succeeds**. The `path` resolves on the engine/daemon host and the install runs the manifest's `setup`/`install`/`start` scripts there. Because the engine does not authenticate worker identity, treat a daemon reachable by untrusted workers as a host-level code-execution surface — prefer registry names or OCI references for distributed workers, and lock down the daemon when exposing it.
## `engine::triggers::info` now exposes `response_schema`
Trigger types can declare the schema a bound handler must **return** when the trigger fires. `engine::triggers::info` surfaces it as a new optional `response_schema` field alongside the existing `configuration_schema` (how to configure the trigger) and `request_schema` (what the handler receives) — the full trigger contract is now discoverable from a single call:
```json theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
{
"id": "http",
"configuration_schema": { "…": "route fields — path, method, middleware" },
"request_schema": { "…": "envelope your handler receives" },
"response_schema": {
"properties": {
"status_code": { "…": "HTTP status to send; defaults to 200 when omitted" },
"headers": { "…": "{ \"Header-Name\": \"value\" } map or [\"Header-Name: value\"] strings" },
"body": { "…": "serialized as JSON, text, or bytes per your Content-Type" }
}
},
"instance_count": 1
}
```
The `http` trigger type is the first to declare a return contract: its `response_schema` is the response envelope the `iii-http` worker reads from a handler's return value — `status_code` / `headers` / `body`, every field optional. Previously, "what should my HTTP handler return" wasn't discoverable from the trigger itself: you had to inspect an already-bound handler via `engine::functions::info`, or guess field names (`status` vs `status_code` — it's `status_code`).
Trigger types that place no constraint on the handler's return omit the field entirely, so existing consumers of `engine::triggers::info` are unaffected. In-process (Rust) trigger types can declare their own contract with the new `TriggerType::with_call_response_format::()` builder.
## SDK: inbound `unregistertrigger` for custom trigger types
When a trigger instance is removed — via `trigger.unregister()` or because the subscribing worker disconnects — the engine notifies the worker that owns the trigger type so it can run `unregisterTrigger` and tear down listeners, routes, or subscriptions.
Node, Browser, Python, and Rust SDKs already handled inbound `registertrigger`; they now handle inbound `unregistertrigger` the same way. Custom trigger type providers (`registerTriggerType`) receive the binding `id` (and can look up stored config from their own registry keyed by that id).
**Built-in trigger types** (`http`, `cron`, `state`, `subscribe`, `durable:subscriber`, stream, and others) are unchanged: the engine calls each in-process worker’s `unregister_trigger` directly and never sends a WebSocket message to an SDK worker.
### What this fixes
* Unregistering a trigger bound to a **custom** trigger type now invokes the provider’s `unregisterTrigger` callback instead of leaving stale bindings server-side.
* When the **provider worker reconnects**, the engine re-sends `registertrigger` for existing bindings (unchanged); cleanup on consumer disconnect now correctly pairs with `unregisterTrigger` on the provider.
## SDK surface trimming — deprecated and unused exports removed
**Breaking (import-time only).** A cleanup pass across all three SDKs removed re-exports and aliases that were back-compat shims, orphaned types, or thin wrappers over upstream crates. None change runtime behavior — each is a mechanical import swap.
### Observability re-exports dropped (Node + Python)
The `Logger` and OTel re-exports that `iii-sdk` kept for back-compat when the observability surface moved to `iii-observability` in 0.16.0 are now removed. Import from the observability package directly:
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Node — before
import { Logger } from 'iii-sdk'
// after
import { Logger } from '@iii-dev/observability'
```
```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# Python — before
from iii import Logger, init_otel, with_span, OtelConfig
# after
from iii_observability import Logger, init_otel, with_span, OtelConfig
```
The full set removed from the Python `iii` package: `Logger`, `init_otel`, `shutdown_otel`, `flush_otel`, `with_span`, `execute_traced_request`, `OtelConfig`, `ReconnectionConfig`, `BaggageSpanProcessor`, `current_span_id` / `current_trace_id`, `current_span_is_recording`, `record_span_event`, `set_current_span_attribute` / `set_current_span_error`, the baggage and traceparent inject/extract helpers, `redact` / `redact_and_truncate` / `resolve_max_bytes_from_env`, `DEFAULT_ALLOWLIST`, and `REDACTED_PLACEHOLDER`. All live in `iii_observability`.
### Rust SDK: crate-root re-exports and dead types removed
| Removed from `iii_sdk` | Replacement |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| `Value` (re-export of `serde_json::Value`) | depend on `serde_json` and use `serde_json::Value` |
| `UpdateBuilder` | build a `Vec` with `UpdateOp::set` / `increment` / `decrement` / `append` / `remove` / `merge` |
| `FieldPath` | `UpdateOp` path fields now take `impl Into` — pass `String` / `&str` directly |
| `MergePath` (crate root) | still available at `iii_sdk::types::MergePath` |
| `TriggerTypeInfo` | none — it was orphaned and never wired to anything |
### Node SDK: `TriggerActionType` alias removed
The `TriggerActionType` type alias is gone — use `TriggerAction` directly. The `TriggerAction.Enqueue()` / `TriggerAction.Void()` runtime helpers are unchanged.
### Python SDK: `IIIForbiddenError` / `IIITimeoutError` removed
Both exception subclasses are deleted. All rejections — including timeouts and RBAC denials — now raise `IIIInvocationError`; branch on its `.code` (`"FORBIDDEN"`, `"TIMEOUT"`) instead of catching distinct types.
```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# before
try:
result = iii.trigger(...)
except IIITimeoutError:
...
# after
try:
result = iii.trigger(...)
except IIIInvocationError as e:
if e.code == "TIMEOUT":
...
```
## Channel and stream helpers moved to a `helpers` submodule
**Breaking.** `createChannel` / `createStream` (and the channel utility types) are no longer instance methods or crate-root exports — they moved to a dedicated `helpers` submodule across all three SDKs. This keeps the core `iii` client surface focused on registration and invocation, and groups the channel/stream plumbing in one importable place.
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Node — before
const ch = iii.createChannel(bufferSize)
iii.createStream(name, stream)
// after
import { createChannel, createStream } from 'iii-sdk/helpers'
const ch = createChannel(iii, bufferSize)
createStream(iii, name, stream)
```
```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# Python — before
ch = iii.create_channel()
# after
from iii.helpers import create_channel, create_channel_async, create_stream
ch = create_channel(iii)
```
```rust theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Rust — before
let ch = iii.create_channel(buffer_size);
// after
let ch = iii_sdk::helpers::create_channel(&iii, buffer_size);
```
The same submodule now also carries the channel utilities — `ChannelDirection`, `ChannelItem`, `extractChannelRefs` / `extract_channel_refs`, and `isChannelRef` / `is_channel_ref` — which were previously top-level exports. `ChannelReader`, `ChannelWriter`, and `StreamChannelRef` stay at the package root.
## `iii-worker` warns when `scripts.install` is omitted
A worker manifest with no `scripts.install` now emits a warning at load time instead of silently skipping the install step, so a missing setup phase is visible during local runs and CI rather than surfacing later as a runtime failure.
## Observability: `getTracer` / `getMeter` / `SpanKind` dropped from the public Node API
**Breaking.** `@iii-dev/observability` no longer exports `getTracer`, `getMeter`, or `SpanKind` from its main entry point. `getTracer` / `getMeter` moved to a first-party-only `@iii-dev/observability/internal` subpath; they were never intended for application code. External consumers should:
* instrument with `withSpan` / `initOtel`, and
* import `SpanKind` from `@opentelemetry/api` directly.
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// before
import { getTracer, SpanKind } from '@iii-dev/observability'
// after
import { SpanKind } from '@opentelemetry/api'
// (getTracer is internal — instrument via withSpan)
```
## Stored logs are stripped of ANSI escape codes
Log lines captured by the observability pipeline now have terminal color/formatting escape sequences removed before storage, so persisted logs render as clean text in the dashboard and downstream consumers instead of leaking raw `\x1b[...m` codes.
## Single `register_function` entry point in the Rust SDK
**Breaking.** The Rust SDK's function registration is collapsed into a single entry point that mirrors Node and Python:
```rust theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
iii.register_function("greet", RegisterFunction::new(greet));
iii.register_function(
"http::fetch",
RegisterFunction::new_async(fetch).description("Fetches a URL"),
);
iii.register_function(
"ext::lambda",
RegisterFunction::http(http_config),
);
```
`RegisterFunction` carries the handler plus all optional metadata. There are three constructors — `new`, `new_async`, `http` — and `Value` is accepted by `new` / `new_async`, so no separate `untyped` constructor is needed. `register_function_with`, the tuple form, `untyped`, `IntoFunctionRegistration`, `IntoFunctionHandler`, `RegisterFunctionOptions`, `iii_fn`, `iii_async_fn`, `IIIFn`, and `IIIAsyncFn` are removed.
Handler error type is fixed to `IIIError`. `IIIError` now implements `From` / `From<&str>` so existing `Result` handlers can migrate by updating the return type and relying on `?`-propagation.
See [the migration entry](./0-16-0/align-rust-register-function-with-signature) for the full before/after diff, builder methods, and step-by-step migration.
## `Logger` and OpenTelemetry primitives moved to `iii-observability`
The `Logger`, `OtelConfig`, `ReconnectionConfig` (OTel variant), and the full OTel surface (`init_otel` / `shutdown_otel` / `flush_otel` / `with_span` / `execute_traced_request`, baggage and traceparent helpers, `current_span_id` / `current_trace_id`, span ops, payload redaction, `BaggageSpanProcessor`) now ship from a new shared package in every supported language:
| Language | Package | Import |
| -------- | ------------------------------- | ------------------------------------------------------------------------------------------- |
| Node | `@iii-dev/observability` (npm) | `import { Logger, initOtel, withSpan, executeTracedRequest } from '@iii-dev/observability'` |
| Python | `iii-observability` (PyPI) | `from iii_observability import Logger, init_otel, with_span, execute_traced_request` |
| Rust | `iii-observability` (crates.io) | `use iii_observability::{Logger, init_otel, with_span, execute_traced_request};` |
This isolates telemetry concerns from the SDK transport so workers that don't need OTel pull a smaller dependency set, and so the surface stays consistent across languages.
Two helpers that previously only existed in the Rust SDK are now available in Node and Python as well:
* `flush_otel` / `flushOtel` — force-flushes every provider without tearing OTel down. Use it before short-lived process exits where you still need pending spans, metrics, and logs delivered.
* `execute_traced_request` / `executeTracedRequest` — wraps an outgoing HTTP call (httpx in Python, `fetch` in Node) in an OTel `CLIENT` span. Injects W3C traceparent, records HTTP semantic-convention attributes, sets `ERROR` status on `>= 400` responses, and records exceptions on network errors.
### Migration
Python and Rust continue to re-export the moved symbols from the SDK package for back-compat. Node removes the `iii-sdk/telemetry` subpath entry point — the named exports from `iii-sdk` itself stay, so `import { Logger } from 'iii-sdk'` keeps working. Direct imports from the new packages are preferred:
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before (Node)
import { Logger, initOtel, withSpan } from 'iii-sdk'
// After (Node)
import { Logger, initOtel, withSpan } from '@iii-dev/observability'
```
```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# Before (Python)
from iii import Logger
from iii.telemetry import init_otel, with_span
# After (Python)
from iii_observability import Logger, init_otel, with_span
```
```rust theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before (Rust)
use iii_sdk::{Logger, OtelConfig, init_otel, with_span, execute_traced_request};
// After (Rust)
use iii_observability::{Logger, OtelConfig, init_otel, with_span, execute_traced_request};
```
The new packages publish in lock-step with the rest of the monorepo on the same `iii/v*` release tag, so versions stay aligned with `iii-sdk`.
## `register_service` removed from all SDKs
**Breaking.** `register_service` / `registerService`, along with the `RegisterServiceInput` and `RegisterServiceMessage` types, are removed from the Node, Browser, Python, and Rust SDKs, and the engine no longer handles the message. Services were an organizational-only grouping that never affected invocation or routing, so there is no replacement — drop all `register_service` calls.
## Unused telemetry accessors removed
**Breaking.** Alongside the observability move, low-level telemetry accessors that were exported but unused are gone:
* **Node** (`iii-sdk`): `getTracer`, `getMeter`, `SpanStatusCode` — import `SpanStatusCode` from `@opentelemetry/api`; tracer and meter are internal.
* **Python** (`iii`): `get_tracer`, `get_meter`, `is_initialized` are now private (`_get_tracer`, `_get_meter`, `_is_initialized`) — use the `opentelemetry` API directly.
* **Rust** (`iii_sdk`): the `get_tracer`, `get_meter`, `is_initialized`, `SpanKind`, and `SpanStatus` re-exports — obtain meters via `opentelemetry::global::meter(...)` and import `SpanKind` from `opentelemetry::trace`.
For custom metrics, use the OpenTelemetry global meter directly rather than the SDK's `getMeter` / `get_meter`.
## `sandbox::run` — one call from zero to result
A new meta-function composes `sandbox::create` + `sandbox::fs::write` + `sandbox::exec` + `sandbox::stop` into a single call. The classic four-step "create → write → exec → stop" dance drops to one. The sandbox is auto-stopped on both success and failure unless you pass `keep_sandbox: true`.
```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# before (4 calls)
SB=$(iii trigger sandbox::create image=python | jq -r .sandbox_id)
iii trigger sandbox::fs::write sandbox_id="$SB" path=/workspace/run.py content='print(2+2)'
iii trigger sandbox::exec sandbox_id="$SB" cmd=python3 args='["/workspace/run.py"]'
iii trigger sandbox::stop sandbox_id="$SB"
# after (1 call)
iii trigger sandbox::run --json '{"image":"python","code":"print(2+2)"}'
```
## `sandbox::catalog::list`
A new function returns the daemon's image catalog — bundled presets plus operator-registered `custom_images` entries from `iii.config.yaml`. Closes the "what images are available on this host?" discovery loop without operator hand-off.
## `sandbox::exec` and `sandbox::create` accept more input shapes
`sandbox::exec.cmd` now accepts three shapes:
* `cmd` + `args` (classic POSIX)
* `argv` array
* shell-line `cmd` (shlex-split when `args` / `argv` are empty)
`sandbox::exec.env` and `sandbox::create.env` accept **either** a `Vec<"K=V">` list **or** a `{ K: V }` map. Env-var names are pinned to `[A-Za-z_][A-Za-z0-9_]*`; digit-leading or `/`/`-`/`=` names are rejected as `S001`.
## `sandbox::fs::read` returns inline bodies for small text
Additive: a new optional `body` field on the `sandbox::fs::read` response carries the file contents as a UTF-8 string for text files under 1 MiB that decode cleanly. The existing `content: StreamChannelRef` field is still always populated and still delivers the same bytes, so peers that statically type `content` as a stream ref keep working unchanged. New callers can short-circuit the channel subscription whenever `body` is present:
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
const { content, body } = await trigger({ function_id: 'sandbox::fs::read', payload: { sandbox_id, path } })
const text = body ?? await readChannel(content) // prefer inline body, fall back to stream
```
Cost: small text is buffered into the channel as well as the inline body so legacy subscribers still receive it. Bounded at 1 MiB per call.
## Structured `sandbox::*` errors with resubmittable `fix` payloads
Every `sandbox::*` function now returns a structured envelope on failure:
```json theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
{
"code": "S211",
"type": "FsParentNotFound",
"message": "parent directory /workspace/a/b does not exist",
"docs_url": "https://github.com/iii-hq/iii/.../README.md#S211",
"retryable": false,
"fix": { "parents": true },
"fix_note": "merge `fix` into the original request and resubmit: `parents: true` auto-creates missing intermediate directories"
}
```
* `docs_url` anchors directly at the in-repo `S`-code subsection. **Breaking:** the base URL flipped from `https://iii.dev/docs/errors/sandbox/Sxxx` to `https://github.com/iii-hq/iii/blob/main/crates/iii-worker/src/sandbox_daemon/README.md#Sxxx` while the canonical `iii.dev` error pages are still pending. Bookmarks and scrapers built on the old URL need to follow the new anchors.
* `fix` is a non-null JSON payload the agent can merge into the original request and resubmit verbatim when recovery is unambiguous (parent-missing writes, `sandbox::run` sub-step failures, etc.).
* `fix_note` describes how to use the fix or — when `fix` is `null` — explains why no auto-recovery exists.
* `sandbox::run` sub-step failures surface the inner `S`-code transparently and name the failing step in `fix.context`, plus `fix.sandbox_id` when `keep_sandbox: true`.
* FS error `message` strings now carry a kind prefix (e.g. `"file not found: {path}"` instead of bare `{path}`). The authoritative `code` / `type` fields are unchanged; only callers that grep the message text are affected.
## `sandbox::exec` default timeout raised to 5 minutes
**Breaking.** The default `timeout_ms` for `sandbox::exec` moves from 30 s to 300 s. Sized for cold `npm install` / `pip install` / `cargo build`. Previously the 30 s default fired as an opaque engine-gate denial before the daemon could return a structured `timed_out: true` response. Callers that relied on the 30 s fast-fail to bound runaway commands should now set `timeout_ms` explicitly.
## Handler-boundary tracing on every `sandbox::*` handler
Every `sandbox::*` handler emits a `tracing::info!` event on both success and error with a stable field set: `function_id`, `sandbox_id`, `success`, `error_code`, `error_type`, `retryable`, `duration_ms`. Operators can dashboard sandbox usage without grepping unstructured logs.
## Telemetry re-exports removed from public SDK surface
**Breaking.** Convenience re-exports of OpenTelemetry accessors were dropped from the Rust, Node, Python, and browser SDKs. Underlying behavior is unchanged — only the public surface is smaller. Users who need a tracer or meter directly should depend on the OpenTelemetry library for their language.
Removed symbols by language:
| Symbol | Rust (`iii::*`) | Node (`iii-sdk/telemetry`) | Python (`iii.telemetry` / `iii.logger`) | Browser |
| ------------------------------- | --------------------------------------------------- | -------------------------- | --------------------------------------- | ------------------------- |
| `get_tracer` / `getTracer` | dropped (still at `iii::telemetry::get_tracer`) | dropped | renamed `_get_tracer` | already absent (asserted) |
| `get_meter` / `getMeter` | dropped (still at `iii::telemetry::get_meter`) | dropped | renamed `_get_meter` | already absent (asserted) |
| `is_initialized` | dropped (still at `iii::telemetry::is_initialized`) | n/a | renamed `_is_initialized` | already absent (asserted) |
| `SpanKind` | dropped (use `opentelemetry::trace::SpanKind`) | n/a | n/a | already absent (asserted) |
| `SpanStatus` / `SpanStatusCode` | dropped (use `opentelemetry::trace::Status`) | dropped | n/a | already absent (asserted) |
### Migration
* For custom spans, prefer `withSpan` / `with_span` / `run_in_span`. These preserve trace context.
* To obtain a tracer or meter directly, depend on `@opentelemetry/api` (Node) or the `opentelemetry` crate / Python package and call its accessors. Rust users can also keep using `iii::telemetry::get_tracer` / `iii::telemetry::get_meter`.
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before (Node)
import { getTracer, getMeter, SpanStatusCode } from 'iii-sdk/telemetry'
// After (Node)
import { trace, metrics, SpanStatusCode } from '@opentelemetry/api'
const tracer = trace.getTracer('my-service')
const meter = metrics.getMeter('my-service')
```
```rust theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before (Rust)
use iii::{get_tracer, get_meter, SpanKind, SpanStatus};
// After (Rust)
use opentelemetry::global;
use opentelemetry::trace::{SpanKind, Status};
let meter = global::meter("my-service");
```
```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# Before (Python)
from iii.telemetry import get_tracer, get_meter, is_initialized
# After (Python)
from opentelemetry import trace, metrics
tracer = trace.get_tracer("my-service")
meter = metrics.get_meter("my-service")
```
## `iii sandbox` subcommand removed
**Breaking.** The `iii sandbox` CLI subcommand is gone. Every sandbox operation now goes through `iii trigger`:
```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# before
iii sandbox create python --idle-timeout 300
iii sandbox exec "$SB" -- python3 -c 'print(2+2)'
iii sandbox stop "$SB"
# after
SB=$(iii trigger sandbox::create image=python idle_timeout_secs=300 | jq -r .sandbox_id)
iii trigger sandbox::exec sandbox_id="$SB" cmd=python3 args='["-c","print(2+2)"]'
iii trigger sandbox::stop sandbox_id="$SB"
```
Each call also accepts a single `--json ''` payload (e.g. `iii trigger sandbox::exec --json '{"sandbox_id":"…","cmd":"python3","args":["-c","print(2+2)"]}'`), equivalent to the kv form shown above.
`iii trigger` is request/response only, so the streaming flows the old subcommand offered (`exec` stdout/stderr stream, `upload`, `download`) are no longer available from the terminal. Use the SDK from worker code for those: `sandbox::exec` and `sandbox::fs::write` / `sandbox::fs::read` still expose the streaming channel.
## `iii trigger` reshape
**Breaking.** `iii trigger` no longer accepts `--function-id` and `--payload`. The new form takes the function path as a positional argument and accepts payload fields as `key=value` tokens, an `--json ''` flag, or both:
```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# kv form
iii trigger orders::process amount=149.99 currency=USD
# JSON form
iii trigger orders::process --json '{"amount": 149.99, "currency": "USD"}'
# Combined: --json is the base, kv overrides individual keys
iii trigger orders::process --json '{"amount": 100}' amount=149.99
```
See [Triggers](../using-iii/triggers) for the full reference.
## `iii update --list-targets`
`iii update` now exposes a `--list-targets` flag that prints every target accepted by `iii update ` (e.g. `self`, `console`, `worker`). Passing an unknown target now points users at this flag instead of failing silently. Rollback is not supported; reinstall a prior version manually with `curl -fsSL https://iii.dev/install.sh | sh -s -- --version `.
## Migrating from Motia
**Breaking.** The Motia framework is deprecated in favor of using `iii-sdk` directly. Moving to the SDK unlocks multi-worker orchestration, browser connectivity via `iii-browser-sdk` with RBAC, and a direct understanding of iii's three primitives — Workers, Functions, and Triggers. Your existing Motia project becomes one worker in a larger iii deployment instead of a standalone monolith.
[Node / TypeScript migration guide →](./0-11-0/migrating-from-motia-js) · [Python migration guide →](./0-11-0/migrating-from-motia-py)
## SDK discovery wrappers removed
**Breaking.** The convenience discovery wrappers were removed from the Node, browser, Rust, and Python SDKs:
* `listFunctions` / `list_functions` / `list_functions_async`
* `listWorkers` / `list_workers` / `list_workers_async`
* `listTriggers` / `list_triggers` / `list_triggers_async`
* `listTriggerTypes` / `list_trigger_types` / `list_trigger_types_async`
* `onFunctionsAvailable` / `on_functions_available`
Discovery now goes through the core primitives directly: call `trigger()` against the built-in engine functions and register `engine::functions-available` like any other trigger type. This keeps the SDK surfaces aligned with the engine's "use the primitives directly" design.
## Worker RBAC
The **iii-worker-manager** now supports role-based access control. Configure auth functions that validate WebSocket upgrade requests, attach per-session allow/deny lists for functions, control trigger registration, and auto-prefix function IDs for namespace isolation. An optional middleware function lets you intercept every invocation for audit logging, rate limiting, or payload enrichment.
[Read the Worker RBAC guide →](/0-11-0/how-to/worker-rbac)
## Trigger format, validation, and metadata
Trigger types now accept **`trigger_request_format`** and **`call_request_format`** fields (JSON Schema) so the engine can validate trigger configs and call payloads at registration time. Triggers also support an arbitrary **`metadata`** field for tagging and filtering.
[Define request/response formats →](/0-11-0/how-to/define-request-response-formats) · [Trigger architecture →](/0-11-0/architecture/trigger-types)
## Browser SDK
Your browser is now a first-class iii worker. The new `iii-browser-sdk` package connects to the engine over a single WebSocket and exposes the same core primitives as the Node SDK — `registerFunction`, `trigger`, `registerTrigger`, and `createChannel` all work identically. Build real-time dashboards, collaborative apps, and bi-directional frontends without REST endpoints or polling.
[Use iii in the browser →](/0-11-0/how-to/use-iii-in-the-browser)
## Sandbox and Container Workers
Workers can now run as **container workers** or **sandbox workers**. Container workers are OCI images managed through the `iii worker` CLI — add an image, configure it in `config.yaml`, and the engine pulls, extracts, and runs it in an isolated sandbox. For local development, `iii worker add ./my-project` registers a local directory as a first-class managed worker that runs inside a lightweight microVM with auto-detected runtimes, dependency caching, and full lifecycle support (`start`, `stop`, `list`, `remove`) — no Dockerfiles needed. Requires macOS Apple Silicon or Linux with KVM.
[Managing Container Workers →](/0-11-0/how-to/managing-container-workers) · [Developing Sandbox Workers →](/0-11-0/how-to/developing-sandbox-workers)
## `iii worker exec`
A new `iii worker exec -- ` command runs arbitrary commands inside a running worker's microVM — think `docker exec` for iii workers. stdin/stdout/stderr flow through, exit codes pass back, Ctrl-C delivers SIGINT (twice for SIGKILL). TTY mode auto-detects when both stdin and stdout are terminals, so `iii worker exec my-worker -- sh` in a terminal gives you a real interactive shell with line editing and job control. Pass `--timeout 30s` to bound runaway commands (exit 124 matches coreutils).
[Exec into a running worker →](/0-11-0/how-to/managing-container-workers#5-exec-into-a-running-worker)
## Reproducible worker installs
Registry-managed workers can now be pinned in `iii.lock`. `iii worker add` writes the resolved worker graph when the registry provides one, binary workers can record artifacts for multiple platform targets, `iii worker verify` checks that `config.yaml` is represented in the lockfile, and `iii worker update [worker]` refreshes locked pins intentionally.
[Reproduce Worker Installs →](/0-11-0/how-to/reproduce-worker-installs)
## Topic-based fan-out queues
**Breaking.** The topic-based queue API has been renamed. The trigger type changes from `queue` to `durable:subscriber`, and the publish function changes from `enqueue` to `iii::durable::publish`:
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before
registerTrigger({ type: 'queue', function_id: 'my::handler', config: { topic: 'order.created' } })
trigger({ function_id: 'enqueue', payload: { topic: 'order.created', data } })
// After
registerTrigger({ type: 'durable:subscriber', function_id: 'my::handler', config: { topic: 'order.created' } })
trigger({ function_id: 'iii::durable::publish', payload: { topic: 'order.created', data } })
```
Messages now fan out to every subscriber, with each function processing its copy independently and retrying on its own schedule. If a function has multiple replicas, they compete on a shared per-function queue. An optional `condition_function_id` lets you filter messages server-side before they reach the handler.
[Use topic-based queues →](/0-11-0/how-to/use-topic-queues)
## Node SDK: `registerFunction` signature change
**Breaking.** The `registerFunction` API now takes the function ID as a plain string instead of an options object:
```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// Before
registerFunction({ id: 'function-id' }, handler)
// After
registerFunction('function-id', handler, {})
```
The options object (metadata, request/response formats) moves to an optional third argument.
## Everything is a worker
**Breaking.** We simplified iii down to three primitives: **Workers**, **Functions**, and **Triggers**. Modules were always workers in disguise -- they connect to the engine, register functions, and react to triggers just like SDK workers do. Now the naming reflects that.
* **Config YAML** — `modules:` top-level key renamed to `workers:`, `class:` field renamed to `name:` with short identifiers.
* **Rust API** — `Module` trait → `Worker`, `register_module!` → `register_worker!`, `EngineBuilder::add_module()` → `add_worker()`.
* **Adapter IDs** — changed from long Rust-style paths to short names: `kv`, `redis`, `builtin`, `rabbitmq`, `local`, `bridge`.
[Read the full story and migration guide →](./0-11-0/everything-is-a-worker)