Skip to content
Sign in

iii-observability

v0.26.0-rc.1

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

Engine module
Baked into the iii engine; no separate install required.

Install

bash
iii trigger compose::add worker=iii-observability@0.26.0-rc.1

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

skill doc

index.md

iii-observability

The iii-observability worker provides OpenTelemetry-backed observability for the iii engine: distributed tracing, structured logs, metrics with rollups, alert rules, sampling configuration, and baggage propagation. Every surface is a callable engine::* function, plus two engine-hosted sources: observability::on-log, called once per ingested log record, and observability::on-span, called once per coalesced window of span activity. Functions span nine sub-namespaces — emit, query stored telemetry, inspect configuration, and propagate context.

The worker is on by default (enabled: true). When disabled, the emit and read functions still register but become no-ops and the sources are never called. Core config: service_name/service_version/service_namespace (OTel resource attributes), exporter (memory | otlp | both), endpoint (OTLP collector URL, default http://localhost:4317), sampling_ratio (0.0–1.0), memory_max_spans, and per-pillar toggles/limits (metrics_enabled, logs_enabled, retention and count caps). Most fields accept OTEL_* env overrides. The in-memory query functions (logs, traces) require the memory or both exporter.

Traces and metrics use OTLP/gRPC by default; https:// endpoints enable TLS and http:// endpoints use cleartext. Set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf to use OTLP/HTTP protobuf for traces and metrics, or use OTEL_EXPORTER_OTLP_TRACES_PROTOCOL / OTEL_EXPORTER_OTLP_METRICS_PROTOCOL for signal-specific overrides. The HTTP/protobuf path is appended automatically (/v1/traces, /v1/metrics) when the configured endpoint is a base collector URL. Logs export over OTLP/HTTP to /v1/logs.

Use OTEL_EXPORTER_OTLP_HEADERS for collector headers and the signal-specific OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_METRICS_HEADERS, or OTEL_EXPORTER_OTLP_LOGS_HEADERS when needed. The logs exporter reads OTEL_EXPORTER_OTLP_LOGS_HEADERS first and falls back to OTEL_EXPORTER_OTLP_HEADERS. Keep credentials in environment variables or a secret manager, not in config files.

When to Use

  • Emit structured logs or read back stored logs/spans/metrics from inside a function instead of shelling out to a collector.
  • React in real time to log entries (page on errors, archive everything) without polling.
  • Inspect engine health, active sampling rules, or alert state operationally.
  • Propagate OpenTelemetry baggage across calls for cross-cutting context.

Boundaries

  • engine::baggage::set does not propagate back to the caller — baggage propagation happens at the SDK/invocation level via headers.
  • The in-memory query functions return nothing useful unless the memory (or both) exporter is configured; with otlp-only the data lives in your collector.
  • When logs_enabled is off the log pipeline is dormant and observability::on-log is never called; the ingest level sets the minimum severity stored.
  • This worker observes telemetry — it is not a general event bus (pubsub) or durable queue (queue); those are standalone Compose workers.

Functions

  • engine::log::info, engine::log::warn, engine::log::error, engine::log::debug, engine::log::trace — emit a log entry at the named severity; same input shape, only the level differs.
  • engine::logs::list — read stored OTel logs, filtered by time, trace correlation, or severity.
  • engine::logs::clear — wipe the in-memory log store.
  • engine::traces::list — list compact trace summaries with aggregate status/counts and optional attribute projection.
  • engine::traces::spans — list full span records for detail/timeline consumers.
  • engine::traces::tree — walk a single trace as a parent/child tree.
  • engine::traces::group_by — aggregate stored spans by an attribute value (counts, duration, errors per group).
  • engine::traces::clear — wipe stored spans.
  • engine::metrics::list — list metrics with aggregated stats and optional time bucketing.
  • engine::rollups::list — list metric rollup aggregations across 1-minute, 5-minute, and 1-hour windows.
  • engine::baggage::get, engine::baggage::get_all — read one or all OpenTelemetry baggage keys.
  • engine::baggage::set — set a baggage key locally (does not propagate to the caller).
  • engine::sampling::rules — list the active sampling rules.
  • engine::health::check — return engine health (status, per-component breakdown, version).
  • engine::alerts::list — inspect configured alert rules and current state.
  • engine::alerts::evaluate — manually run an alert evaluation pass.

Reacting to logs and spans

Observe observability::on-log when a function should run every time a log entry lands in the engine's OTel log pipeline — emitted via engine::log::*, ingested via OTLP, or recorded by any worker using structured logging. Each activation carries the same OTel-shaped record, so filters and handlers can route by severity, attribute, or trace correlation.

Reach for it when:

  • A specific severity (typically error) should page a human, post to Slack, or open a ticket.
  • You want real-time fan-out of log entries to a downstream sink (archive, analytics, transformer) without polling engine::logs::list.

Use engine::logs::list instead when you need to query stored entries on demand rather than react to each as it arrives.

How to observe

  1. Register a handler: iii.registerFunction('monitoring::on-error', handler).
  2. Observe the source:
iii.observe(
  {
    function_id: "observability::on-log",
    filters: { "input.severity_number": { $gte: 17 } }, // ERROR and above; omit for every record
  },
  "monitoring::on-error",
);

observability::on-log is called only when the logs pipeline is enabled. The target's return value is ignored; activations run asynchronously after each entry is stored.

Observe observability::on-span for a "traces changed" tick: { trace_ids }, the distinct traces with span activity in a ~300ms window. Re-read them with engine::traces::*.

For the OTel log record shape, call engine::functions::info { function_id: "observability::on-log" }: its request_format is the record.

Search the registry

Find workers, kits and executables, or jump to a page.