iii
Modules

Observability

OpenTelemetry-based traces, metrics, logs, alerts, and sampling.

Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions.

modules::observability::OtelModule

Sample Configuration

- class: modules::observability::OtelModule
  config:
    enabled: true
    service_name: my-service
    service_version: 1.0.0
    exporter: memory
    metrics_enabled: true
    logs_enabled: true
    memory_max_spans: 1000
    sampling_ratio: 1.0
    alerts:
      - name: high-error-rate
        metric: iii.invocations.error
        threshold: 10
        operator: ">"
        window_seconds: 60
        action:
          type: log

Configuration

enabled
boolean

Whether OpenTelemetry tracing export is enabled. Defaults to false. Can also be set via OTEL_ENABLED environment variable.

service_name
string

Service name reported in traces and metrics. Defaults to "iii". Can also be set via OTEL_SERVICE_NAME.

service_version
string

Service version reported in traces (service.version OTEL attribute). Can also be set via SERVICE_VERSION.

service_namespace
string

Service namespace (service.namespace OTEL attribute). Can also be set via SERVICE_NAMESPACE.

exporter
string

Trace exporter type. Options:

  • memory — store traces in memory, queryable via engine::traces::list
  • otlp — export to an OTLP collector via gRPC
  • both — export via OTLP and keep in memory (enables log triggers alongside OTLP export)

Defaults to otlp. Can also be set via OTEL_EXPORTER_TYPE.

endpoint
string

OTLP collector endpoint. Used when exporter is otlp or both. Defaults to "http://localhost:4317". Can also be set via OTEL_EXPORTER_OTLP_ENDPOINT.

sampling_ratio
number

Global trace sampling ratio from 0.0 (sample nothing) to 1.0 (sample everything). Defaults to 1.0. Can also be set via OTEL_TRACES_SAMPLER_ARG.

sampling
SamplingConfig

Advanced per-operation and per-service sampling rules.

memory_max_spans
number

Maximum number of spans to keep in memory when using memory or both exporter. Defaults to 1000. Can also be set via OTEL_MEMORY_MAX_SPANS.

metrics_enabled
boolean

Whether metrics collection is enabled. Defaults to false. Can also be set via OTEL_METRICS_ENABLED.

metrics_exporter
string

Metrics exporter type: memory (queryable via API) or otlp. Defaults to memory. Can also be set via OTEL_METRICS_EXPORTER.

metrics_retention_seconds
number

How long to retain metrics in memory in seconds. Defaults to 3600 (1 hour). Can also be set via OTEL_METRICS_RETENTION_SECONDS.

metrics_max_count
number

Maximum number of metric data points to keep in memory. Defaults to 10000. Can also be set via OTEL_METRICS_MAX_COUNT.

logs_enabled
boolean

Whether structured log storage is enabled. When not set, log storage is always initialized by the module.

logs_exporter
string

Logs exporter type: memory, otlp, or both. Defaults to memory. Can also be set via OTEL_LOGS_EXPORTER.

logs_max_count
number

Maximum number of log entries to keep in memory. Defaults to 1000.

logs_retention_seconds
number

How long to retain logs in memory in seconds. Defaults to 3600 (1 hour).

logs_sampling_ratio
number

Fraction of logs to retain (0.0 to 1.0). Defaults to 1.0 (keep all).

logs_console_output
boolean

Whether to print ingested logs to the console via tracing. Defaults to true.

level
string

Minimum log level for the engine itself. Options: trace, debug, info, warn, error. Defaults to info.

format
string

Log output format: default (human-readable) or json (structured JSON). Defaults to default.

alerts
AlertRule[]

List of alert rules evaluated against metrics.

Functions

Logging

engine::log::info
function

Log an informational message.

engine::log::warn
function

Log a warning message. Same parameters as engine::log::info.

engine::log::error
function

Log an error message. Same parameters as engine::log::info.

engine::log::debug
function

Log a debug message. Same parameters as engine::log::info.

engine::log::trace
function

Log a trace-level message. Same parameters as engine::log::info.

Logs API

engine::logs::list
function

Query stored log entries.

engine::logs::clear
function

Clear all stored log entries from memory.

Traces API

engine::traces::list
function

List stored trace spans.

engine::traces::tree
function

Retrieve a trace as a hierarchical span tree.

engine::traces::clear
function

Clear all stored trace spans from memory.

Metrics API

engine::metrics::list
function

List collected metrics with aggregated statistics.

engine::rollups::list
function

List metric rollup aggregations (1-minute, 5-minute, 1-hour windows).

Baggage API

engine::baggage::get
function

Get a baggage value from the current trace context.

engine::baggage::set
function

Set a baggage value in the current trace context.

engine::baggage::get_all
function

Get all baggage key-value pairs from the current trace context.

Sampling API

engine::sampling::rules
function

List all active sampling rules and their current configuration.

Health API

engine::health::check
function

Check engine health status.

Alerts API

engine::alerts::list
function

List all configured alert rules and their current state.

engine::alerts::evaluate
function

Manually trigger evaluation of all alert rules against current metrics.

Trigger Type

This module adds a new Trigger Type: log.

Register a function to react to log entries as they are produced.

Log Entry Payload

timestamp_unix_nano
number

Timestamp of the log entry in Unix nanoseconds.

observed_timestamp_unix_nano
number

Observed timestamp in Unix nanoseconds.

severity_number
number

Numeric severity level (1–24).

severity_text
string

Severity text (e.g., "INFO", "WARN", "ERROR").

body
string

The log message content.

attributes
object

Structured attributes attached to the log entry.

trace_id
string

Distributed tracing ID for correlating this log entry across services.

span_id
string

Span ID for correlation within a trace.

resource
object

Resource attributes associated with the log entry.

service_name
string

Name of the service that produced the log entry.

instrumentation_scope_name
string

Name of the instrumentation scope.

instrumentation_scope_version
string

Version of the instrumentation scope.

Sample Code

const fn = iii.registerFunction(
  { id: 'monitoring::onError' },
  async (logEntry) => {
    await sendAlert({
      message: logEntry.body,
      severity: logEntry.severity_text,
      traceId: logEntry.trace_id,
    })
    return {}
  },
)

iii.registerTrigger({
  type: 'log',
  function_id: fn.id,
  config: { level: 'error' },
})

Cron

Schedule functions with cron expressions.

Stream

Build durable streams for real-time data subscriptions.

On this page

Sample Configuration
Configuration
Functions
Logging
Logs API
Traces API
Metrics API
Baggage API
Sampling API
Health API
Alerts API
Trigger Type
Log Entry Payload
Sample Code