iii-observability
v0.26.0-rc.7OpenTelemetry-based traces, metrics, logs, alerts, and sampling.
Install
iii trigger compose::add worker=iii-observability@0.26.0-rc.7Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
skill doc
iii-observability
The iii-observability worker provides OpenTelemetry-backed observability for the iii engine:
distributed tracing, structured logs, metrics with rollups, alert rules, sampling configuration, and
baggage propagation. Every surface is a callable engine::* function, plus two engine-hosted
sources: observability::on-log, called once per ingested log record, and observability::on-span,
called once per coalesced window of span activity. Functions span nine sub-namespaces — emit, query
stored telemetry, inspect configuration, and propagate context.
The worker is on by default (enabled: true). When disabled, the emit and read functions still
register but become no-ops and the sources are never called. Core config:
service_name/service_version/service_namespace (OTel resource attributes), exporter
(memory | otlp | both), endpoint (OTLP collector URL, default http://localhost:4317),
sampling_ratio (0.0–1.0), memory_max_spans, and per-pillar toggles/limits
(metrics_enabled, logs_enabled, retention and count caps). Most fields accept OTEL_* env
overrides. The in-memory query functions (logs, traces) require the memory or both exporter.
Traces and metrics use OTLP/gRPC by default; https:// endpoints enable TLS and http:// endpoints
use cleartext. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf to use OTLP/HTTP protobuf for traces
and metrics, or use OTEL_EXPORTER_OTLP_TRACES_PROTOCOL / OTEL_EXPORTER_OTLP_METRICS_PROTOCOL for
signal-specific overrides. The HTTP/protobuf path is appended automatically (/v1/traces,
/v1/metrics) when the configured endpoint is a base collector URL. Logs export over OTLP/HTTP to
/v1/logs.
Use OTEL_EXPORTER_OTLP_HEADERS for collector headers and the signal-specific
OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_METRICS_HEADERS, or
OTEL_EXPORTER_OTLP_LOGS_HEADERS when needed. The logs exporter reads
OTEL_EXPORTER_OTLP_LOGS_HEADERS first and falls back to OTEL_EXPORTER_OTLP_HEADERS. Keep
credentials in environment variables or a secret manager, not in config files.
When to Use
- Emit structured logs or read back stored logs/spans/metrics from inside a function instead of shelling out to a collector.
- React in real time to log entries (page on errors, archive everything) without polling.
- Inspect engine health, active sampling rules, or alert state operationally.
- Propagate OpenTelemetry baggage across calls for cross-cutting context.
Boundaries
engine::baggage::setdoes not propagate back to the caller — baggage propagation happens at the SDK/invocation level via headers.- The in-memory query functions return nothing useful unless the
memory(orboth) exporter is configured; withotlp-only the data lives in your collector. - When
logs_enabledis off the log pipeline is dormant andobservability::on-logis never called; the ingestlevelsets the minimum severity stored. - This worker observes telemetry — it is not a general event bus (
pubsub) or durable queue (queue); those are standalone Compose workers.
Functions
engine::log::info,engine::log::warn,engine::log::error,engine::log::debug,engine::log::trace— emit a log entry at the named severity; same input shape, only the level differs.engine::logs::list— read stored OTel logs, filtered by time, trace correlation, or severity.engine::logs::clear— wipe the in-memory log store.engine::traces::list— list compact trace summaries with aggregate status/counts and optional attribute projection.engine::traces::spans— list full span records for detail/timeline consumers.engine::traces::tree— walk a single trace as a parent/child tree.engine::traces::group_by— aggregate stored spans by an attribute value (counts, duration, errors per group).engine::traces::clear— wipe stored spans.engine::metrics::list— list metrics with aggregated stats and optional time bucketing.engine::rollups::list— list metric rollup aggregations across 1-minute, 5-minute, and 1-hour windows.engine::baggage::get,engine::baggage::get_all— read one or all OpenTelemetry baggage keys.engine::baggage::set— set a baggage key locally (does not propagate to the caller).engine::sampling::rules— list the active sampling rules.engine::health::check— return engine health (status, per-component breakdown, version).engine::alerts::list— inspect configured alert rules and current state.engine::alerts::evaluate— manually run an alert evaluation pass.
Reacting to logs and spans
Observe observability::on-log when a function should run every time a log entry lands in the
engine's OTel log pipeline — emitted via engine::log::*, ingested via OTLP, or recorded by any
worker using structured logging. Each activation carries the same OTel-shaped record, so filters and
handlers can route by severity, attribute, or trace correlation.
Reach for it when:
- A specific severity (typically
error) should page a human, post to Slack, or open a ticket. - You want real-time fan-out of log entries to a downstream sink (archive, analytics, transformer)
without polling
engine::logs::list.
Use engine::logs::list instead when you need to query stored entries on demand rather than react
to each as it arrives.
How to observe
- Register a handler:
iii.registerFunction('monitoring::on-error', handler). - Observe the source:
iii.observe(
{
function_id: "observability::on-log",
filters: { "input.severity_number": { $gte: 17 } }, // ERROR and above; omit for every record
},
"monitoring::on-error",
);observability::on-log is called only when the logs pipeline is enabled. The target's return value
is ignored; activations run asynchronously after each entry is stored.
Observe observability::on-span for a "traces changed" tick: { trace_ids }, the distinct traces
with span activity in a ~300ms window. Re-read them with engine::traces::*.
For the OTel log record shape, call engine::functions::info { function_id: "observability::on-log" }:
its request_format is the record.