> ## Documentation Index
> Fetch the complete documentation index at: https://iii.dev/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Traces, metrics, and logs for a iii system through the iii-observability worker.

Every call in a iii system passes through the engine: function invocations, trigger firings, and
channel messages all flow across it. Because every worker communicates through the engine, it can
trace work as it moves from one worker to the next, giving you observability across the whole system
without each worker instrumenting itself. The automatically supplied `iii-observability` worker
turns that information into OpenTelemetry traces, metrics, and logs. Do not declare it in either
`config.yaml` or `worker-compose.yaml`.

> [!NOTE]
>
> This page is a quick tour. For the complete configuration and the full list of query functions,
> see the [iii-observability worker docs](https://workers.iii.dev/workers/iii-observability).

## OpenTelemetry support

`iii-observability` is OpenTelemetry-based: it produces distributed traces across worker hops,
metrics, and structured logs, and it can export them to any OTel-compatible backend or keep them
in-memory for local development. Sampling, retention, and exporter targets are configured on the
worker; see the worker docs for the full set of options.

## Logging

Worker code emits structured logs through the `Logger` from the observability SDK, which routes them
into the same OpenTelemetry pipeline (so log lines correlate with the trace they happened in) rather
than to raw stdout.

> [tab] Node / TypeScript
>
> ```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
> import { Logger } from "@iii-dev/helpers/observability";
>
> const logger = new Logger();
>
> // each level takes a message and optional structured data
> logger.debug("cache lookup", { key });
> logger.info("processing link", { slug });
> logger.warn("retrying upstream", { attempt });
> logger.error("failed to persist", { slug, err });
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
> from iii_helpers.observability import Logger
>
> logger = Logger()
>
> # each level takes a message and optional structured data
> logger.debug("cache lookup", {"key": key})
> logger.info("processing link", {"slug": slug})
> logger.warn("retrying upstream", {"attempt": attempt})
> logger.error("failed to persist", {"slug": slug, "err": str(err)})
> ```

> [tab] Rust
>
> ```rust theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
> use iii_helpers::observability::Logger;
> use serde_json::json;
>
> let logger = Logger::new();
>
> // each level takes a message and optional structured data
> logger.debug("cache lookup", Some(json!({ "key": key })));
> logger.info("processing link", Some(json!({ "slug": slug })));
> logger.warn("retrying upstream", Some(json!({ "attempt": attempt })));
> logger.error("failed to persist", Some(json!({ "slug": slug, "err": err.to_string() })));
> ```

### Where the lines go

`Logger` does not write to your worker's own output. It sends each line to the engine, which:

* prints it in the engine's output, labeled with the worker's OpenTelemetry service name (set it
  with `OTEL_SERVICE_NAME`), unless the observability worker sets `logs_console_output: false`;
* stores it: `iii trigger engine::logs::list` returns it, and the [console](https://iii.dev/docs/0-25-0/using-iii/console.md)
  shows it next to the trace it was logged in.

Under `iii compose --up`, the engine's output goes to `engine.log` in the compose state directory,
`.iii/compose/<namespace>/` by default (see [Compose logs](https://iii.dev/docs/0-25-0/using-iii/compose.md#compose-logs)).

Before the worker's OpenTelemetry export starts, or when it is off (`OTEL_ENABLED=false`), `Logger`
falls back to the language's own logging, and the line stays in the worker's process:

| SDK | Fallback | Shown without setup |
| - | - | - |
| Node | `console.*` | yes |
| Python | `logging`, logger `iii.logger` | warnings and errors only, on stderr |
| Rust | the `tracing` crate | no: install a subscriber, for example `tracing_subscriber::fmt().init()` |

For lines in your worker's own terminal, print them yourself (`console.log`, `print`, `println!`).

You can also query recorded logs and emit one-off log lines through the observability worker's
triggers, for example:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# emit a log line from the CLI
iii trigger engine::log::info --json '{"message":"hello from the CLI"}'

# find it in the stored logs
iii trigger engine::logs::list | grep -C 10 hello
```

## Traces

A trace captures the spans of a call as it moves across workers, the headline view for debugging a
request end to end. Traces come from worker function invocations: a worker records a span each time
it handles a call. Built-in `engine::*` functions are not traced by default (set
`III_OTEL_TRACE_BUILTINS=true` to include them), so call a worker function first to produce a trace.
The [sandbox](https://iii.dev/docs/0-25-0/creating-workers/sandboxes.md) worker's `sandbox::run` is an easy one to start with:

```yaml worker-compose.yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
engine:
  workers:
    iii-sandbox: {}
```

Start this file with `iii compose --up` before calling the sandbox function. `iii-sandbox` is an
engine-owned exception, so it belongs under the top-level `engine.workers`, not under `containers`.

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# run something to produce a trace
iii trigger sandbox::run image=python lang=python code='print(1)'

# count the recorded traces (workers export spans a moment after the call,
# so re-run this until it reports a non-zero count)
iii trigger engine::traces::list | jq '.total'

# grab the most recent trace id and print its span tree
TID=$(iii trigger engine::traces::list | jq -r '.traces[-1].trace_id')
iii trigger engine::traces::tree trace_id=$TID
```

> [!NOTE]
>
> `engine::traces::tree` needs a real `trace_id` and traces export on a short delay. If
> `engine::traces::list` still reports `0` or the call above to get a `trace_id` resolves to `null`
> and fails then wait a few seconds and try again.

## React to logs and spans

The observability worker hosts two event sources you can [observe](https://iii.dev/docs/0-25-0/using-iii/observers.md):
`observability::on-log`, one event per log record, and `observability::on-span`, the ids of the
traces that recorded spans in each window of about 300 ms. They replace the `log` and `trace`
trigger types, which keep working.

```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
// invoke alerts::log for every ERROR log record
worker.observe(
  {
    function_id: "observability::on-log",
    deliver: "input",
    filters: { "input.severity_text": { $eq: "ERROR" } },
  },
  "alerts::log",
);
```

Each event runs its targets in a trace of its own, and log records and spans in that trace are not
events again. A target that logs while it handles an event does not trigger itself: the SDK loggers
record the trace the handler runs in, and so does `engine::log::*` when the call passes no
`trace_id`.

Activity outside that trace is bounded by depth. A span the engine records for a call made inside a
chain of observers carries an `iii.observe_depth` attribute (set when the depth is above 0). These
events take that depth, so a target that logs or makes calls on every event stops at
`observers.max_observe_depth`.

Observers have their own metrics (`iii.observer.activations.total`, `iii.observer.dropped.total`,
`iii.observers`, and more), and calls a worker makes to its own functions are counted in
`iii.invocations.total` with `dispatch="local"`. See
[Engine-collected metrics](https://iii.dev/docs/0-25-0/reference/engine-protocol.md#engine-collected-metrics).

## Health and clearing

Check engine health, or clear stored observability data while developing:

```bash theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}}
# overall health status
iii trigger engine::health::check | jq -r .status

# clear stored logs
iii trigger engine::logs::clear

# the logs are now empty
iii trigger engine::logs::list

# clear stored traces
iii trigger engine::traces::clear

# the trace count is now zero
iii trigger engine::traces::list | jq '.total'
```

> [!NOTE]
>
> Logs populate quickly. It is possible to call `engine::logs::clear` immediately followed by
> `engine::logs::list` and still see new logs that were generated in between the two calls.

## Seeing it in the console

The [console](https://iii.dev/docs/0-25-0/using-iii/console.md) renders this observability data visually, traces, logs, and
metrics for a running system, so you usually do not query the functions above by hand during
development.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.