OpenTelemetry-based traces, metrics, logs, alerts, and sampling.
iii trigger compose::add worker=iii-observabilityRun in your project with the iii engine running. Exact versions are immutable and digest-pinned.
Force an immediate alert-rule evaluation against current metrics and return any triggered_alerts, bypassing the periodic tick. Returns evaluated:false when the alert manager is not initialized; produces nothing without configured rules.
Alerts triggered by this evaluation pass (present when the manager is initialized).
List configured alert rules with their current evaluated states and a firing_count. Returns empty when no alert rules are configured or the alert manager is not initialized.
Current evaluated alert states (serialized).
Configured alert rules (present when the alert manager is initialized).
Action to take when alert triggers (Log, Webhook { url }, Function { path })
The webhook URL to send the alert to
The function path to invoke
Minimum interval between alert triggers in seconds (default: 60)
Whether the alert is enabled (default: true)
Metric name to monitor (e.g., "iii.invocations.error")
Name of the alert (for identification)
Comparison operator (>, >=, <, <=, ==, !=)
Threshold value for the alert
Time window in seconds to evaluate the metric (default: 60)
Read one baggage entry by key from the current OTEL context, returning { value } (null if unset). Diagnostic only: reads ambient process context, not per-invocation baggage.
The baggage key to retrieve
Read all baggage entries from the current OTEL context as a { baggage } map. Diagnostic only: reflects ambient process context, not per-invocation baggage.
Set a baggage key/value on a fresh OTEL context for verification only; the new context is not propagated to the caller or global state. Use SDK-level headers for real propagation.
The baggage key to set
The baggage value to set
Report observability subsystem health: per-component status (otel, metrics, logs, spans) marked healthy with counts or disabled, plus engine version. Always succeeds regardless of configuration.
Each component is `{ status: "healthy"|"disabled", details: <object|null> }`.
Record a DEBUG-level OTEL log (severity 5) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.
Additional structured data/attributes
The log message
Service name (defaults to function name if not provided)
Optional span ID for correlation
Optional trace ID for correlation
Record an ERROR-level OTEL log (severity 17) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.
Additional structured data/attributes
The log message
Service name (defaults to function name if not provided)
Optional span ID for correlation
Optional trace ID for correlation
Record an INFO-level OTEL log (severity 9) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.
Additional structured data/attributes
The log message
Service name (defaults to function name if not provided)
Optional span ID for correlation
Optional trace ID for correlation
Record a TRACE-level OTEL log (severity 1) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.
Additional structured data/attributes
The log message
Service name (defaults to function name if not provided)
Optional span ID for correlation
Optional trace ID for correlation
Record a WARN-level OTEL log (severity 13) with optional trace/span correlation and structured data. No-op when logs_enabled is false or dropped by logs_sampling_ratio.
Additional structured data/attributes
The log message
Service name (defaults to function name if not provided)
Optional span ID for correlation
Optional trace ID for correlation
Drop every stored OTEL log, returning { success: true }. Succeeds as a no-op when log storage was never initialized (logs_enabled false).
List stored OTEL logs filtered by trace/span id, severity, and time range, with pagination and a total count. Returns an empty result when logs_enabled is false (no log storage).
End time in Unix timestamp milliseconds
Maximum number of logs to return
Pagination offset (default: 0)
Minimum severity number (1-24, higher = more severe)
Filter by severity text (e.g., "ERROR", "WARN", "INFO")
Filter by span ID
Start time in Unix timestamp milliseconds
Filter by trace ID
Stored OTEL log records (serialized).
Echo of the applied filters (present only when storage exists).
Total matching logs before pagination.
Return engine invocation/worker counters and span-derived latency percentiles, plus stored SDK metrics filtered by name/time and optionally aggregated by interval. engine_metrics is always present; sdk_metrics is empty when no metric storage exists.
Aggregate interval in seconds
End time in Unix timestamp milliseconds
Filter by metric name
Start time in Unix timestamp milliseconds
Time-bucketed aggregates (present only when an aggregate_interval was requested and produced data).
Per-function invocation counts.
Echo of the applied query filters (present only when a filter was supplied).
Stored SDK metric points (serialized).
Return pre-aggregated metric rollups and histograms for a level (0=1m, 1=5m, 2=1h) over a time range (default last hour), optionally by metric name. Falls back to on-the-fly aggregation over metric storage when no rollup storage exists.
End time in Unix timestamp milliseconds
Rollup level index (0 = 1 min, 1 = 5 min, 2 = 1 hour)
Filter by metric name
Start time in Unix timestamp milliseconds
Pre-aggregated histogram rollups (serialized).
Pre-aggregated metric rollups (serialized).
"on_the_fly" when computed live because no rollup storage exists.
Report the active trace sampling config (default ratio, per-operation/service rules, parent_based) and the logs sampling_ratio, read from live config. Defaults to ratio 1.0 with no rules when sampling is unconfigured.
Drop every span from the in-memory trace store, returning { success: true }. Requires exporter memory or both, else fails memory_exporter_not_enabled.
Aggregate stored spans by one attribute into groups (trace_ids, span_count, duration, error_count), newest-first, capped at limit (default 100); skips spans lacking the attribute and engine-internal spans unless include_internal. Requires exporter memory or both.
Span attribute key to group by. Spans without this attribute are skipped.
Include engine-internal spans. Defaults to false, matching `traces::list`.
Attribute whose value becomes each group's human-readable `label` (e.g. group by `iii.session.id` with label `iii.session.name`). The newest group member carrying the attribute wins, so renames surface.
Max groups returned after sorting by `first_seen_ms` descending. Default 100.
Earliest end_time (ms since epoch) to include.
Display label resolved from `label_attribute`, when requested and present.
List compact trace summaries with aggregate status/counts, filtering, pagination, sort, and optional attribute_projection. Child spans are searched and aggregated in the engine; full span payloads are available from engine::traces::spans. Requires exporter memory or both, else fails memory_exporter_not_enabled.
Attribute keys to project onto each trace summary. Only these arbitrary attributes are returned; full span attributes remain on `engine::traces::spans` and `engine::traces::tree`.
Filter by span attributes (array of [key, value] pairs, AND logic, exact match)
End time in unix timestamp milliseconds (include spans overlapping before this)
Exclude listed rows whose own attributes match ANY [key, value] pair (OR logic, exact match). Applied to the root/row span itself — hiding a function hides traces rooted at it, not traces that merely call it.
`engine::traces::spans` only: `false` returns each span without its `events` and `links`, where invocation payloads ride, for views that draw spans without opening one. Defaults to true.
Include internal engine traces (engine.* functions). Defaults to false.
Pagination limit (default: 100)
Maximum span duration in milliseconds (sub-ms precision)
Minimum span duration in milliseconds (sub-ms precision)
Filter by span name (case-insensitive substring match)
Pagination offset (default: 0)
Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.
Filter by service name (case-insensitive substring match)
Sort field: "start_time" | "duration" (alias "duration_ms") | "service_name" | "name" (default: "start_time"). Unknown values fall back to "start_time".
Sort order: "asc" | "desc" (default: "asc")
Start time in unix timestamp milliseconds (include spans overlapping after this)
Filter by status. For `engine::traces::list`, accepts `error`, `pending`, `ok`, or `unset` (case-insensitive); other values match no traces. For `engine::traces::spans`, this remains a case-insensitive substring match on the raw span status.
Filter by specific trace ID
Filter to a specific set of trace IDs (ignored when `trace_id` is set; used by grouped views to expand one group's members in a single call).
Indicates whether the result includes complete durable history.
Total matching traces before pagination.
One compact aggregate per matching trace.
Only keys requested through `attribute_projection` are present.
List full stored spans, including attributes, events and links, with filtering and pagination. Use engine::traces::list for compact trace summaries. Requires exporter memory or both, else fails memory_exporter_not_enabled.
Attribute keys to project onto each trace summary. Only these arbitrary attributes are returned; full span attributes remain on `engine::traces::spans` and `engine::traces::tree`.
Filter by span attributes (array of [key, value] pairs, AND logic, exact match)
End time in unix timestamp milliseconds (include spans overlapping before this)
Exclude listed rows whose own attributes match ANY [key, value] pair (OR logic, exact match). Applied to the root/row span itself — hiding a function hides traces rooted at it, not traces that merely call it.
`engine::traces::spans` only: `false` returns each span without its `events` and `links`, where invocation payloads ride, for views that draw spans without opening one. Defaults to true.
Include internal engine traces (engine.* functions). Defaults to false.
Pagination limit (default: 100)
Maximum span duration in milliseconds (sub-ms precision)
Minimum span duration in milliseconds (sub-ms precision)
Filter by span name (case-insensitive substring match)
Pagination offset (default: 0)
Search across all spans in each trace, not just root spans. When true and a `name` filter is set, traces are matched if ANY span in the trace matches the name filter. Defaults to false.
Filter by service name (case-insensitive substring match)
Sort field: "start_time" | "duration" (alias "duration_ms") | "service_name" | "name" (default: "start_time"). Unknown values fall back to "start_time".
Sort order: "asc" | "desc" (default: "asc")
Start time in unix timestamp milliseconds (include spans overlapping after this)
Filter by status. For `engine::traces::list`, accepts `error`, `pending`, `ok`, or `unset` (case-insensitive); other values match no traces. For `engine::traces::spans`, this remains a case-insensitive substring match on the raw span status.
Filter by specific trace ID
Filter to a specific set of trace IDs (ignored when `trace_id` is set; used by grouped views to expand one group's members in a single call).
Full stored span records, including attributes, events and links.
Total matching spans before pagination.
Build the nested span tree for one trace_id as { roots }, pruning no-op trigger wrappers and collapsing configured pass-through spans. Requires exporter memory or both, else fails memory_exporter_not_enabled.
Trace ID to build the tree for
Root span-tree nodes (each a serialized, flattened span with nested children).
Internal: re-apply the iii-observability configuration when the authoritative configuration entry changes.
Serve the built-in iii-observability Console configuration UI assets.
Find workers, kits and executables, or jump to a page.