> ## Documentation Index > Fetch the complete documentation index at: https://iii.dev/docs/llms.txt > Use this file to discover all available pages before exploring further. # Observability > OpenTelemetry-based traces, metrics, logs, alerts, and sampling. Full OpenTelemetry observability for III Engine: distributed tracing, structured logs, performance metrics, alert rules, and trace sampling — all queryable via built-in functions. ``` iii-observability ``` ## Sample Configuration ```yaml theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}} - name: iii-observability config: enabled: true service_name: my-service service_version: 1.0.0 exporter: memory metrics_enabled: true logs_enabled: true memory_max_spans: 1000 sampling_ratio: 1.0 alerts: - name: high-error-rate metric: iii.invocations.error threshold: 10 operator: ">" window_seconds: 60 action: type: log ``` ## Configuration Whether OpenTelemetry tracing export is enabled. Defaults to `false`. Can also be set via `OTEL_ENABLED` environment variable. Service name reported in traces and metrics. Defaults to `"iii"`. Can also be set via `OTEL_SERVICE_NAME`. Service version reported in traces (`service.version` OTEL attribute). Can also be set via `SERVICE_VERSION`. Service namespace (`service.namespace` OTEL attribute). Can also be set via `SERVICE_NAMESPACE`. Trace exporter type. Options: * `memory` — store traces in memory, queryable via `engine::traces::list` * `otlp` — export to an OTLP collector via gRPC * `both` — export via OTLP and keep in memory (enables log triggers alongside OTLP export) Defaults to `otlp`. Can also be set via `OTEL_EXPORTER_TYPE`. OTLP collector endpoint. Used when `exporter` is `otlp` or `both`. Defaults to `"http://localhost:4317"`. Can also be set via `OTEL_EXPORTER_OTLP_ENDPOINT`. Global trace sampling ratio from `0.0` (sample nothing) to `1.0` (sample everything). Defaults to `1.0`. Can also be set via `OTEL_TRACES_SAMPLER_ARG`. Advanced per-operation and per-service sampling rules. Default sampling rate for operations not matching any rule. Ordered list of sampling rules evaluated per span. Operation name pattern (supports wildcards like `"api.*"`). Service name pattern to match. Sampling rate for this rule (`0.0` to `1.0`). If `true`, inherit the sampling decision from the parent span. Maximum number of traces to sample per second. Maximum number of spans to keep in memory when using `memory` or `both` exporter. Defaults to `1000`. Can also be set via `OTEL_MEMORY_MAX_SPANS`. Whether metrics collection is enabled. Defaults to `false`. Can also be set via `OTEL_METRICS_ENABLED`. Metrics exporter type: `memory` (queryable via API) or `otlp`. Defaults to `memory`. Can also be set via `OTEL_METRICS_EXPORTER`. How long to retain metrics in memory in seconds. Defaults to `3600` (1 hour). Can also be set via `OTEL_METRICS_RETENTION_SECONDS`. Maximum number of metric data points to keep in memory. Defaults to `10000`. Can also be set via `OTEL_METRICS_MAX_COUNT`. Whether structured log storage is enabled. When not set, log storage is always initialized by the worker. Logs exporter type: `memory`, `otlp`, or `both`. Defaults to `memory`. Can also be set via `OTEL_LOGS_EXPORTER`. Maximum number of log entries to keep in memory. Defaults to `1000`. How long to retain logs in memory in seconds. Defaults to `3600` (1 hour). Fraction of logs to retain (`0.0` to `1.0`). Defaults to `1.0` (keep all). Whether to print ingested logs to the console via tracing. Defaults to `true`. Minimum log level for the engine itself. Options: `trace`, `debug`, `info`, `warn`, `error`. Defaults to `info`. Log output format: `default` (human-readable) or `json` (structured JSON). Defaults to `default`. List of alert rules evaluated against metrics. Unique name for the alert rule. Metric name to monitor (e.g., `iii.invocations.error`). Threshold value to compare against. Comparison operator: `>`, `>=`, `<`, `<=`, `==`, `!=`. Defaults to `>`. Time window in seconds over which to evaluate the metric. Defaults to `60`. Minimum interval between alert fires in seconds. Defaults to `60`. Whether the alert rule is active. Defaults to `true`. Action to take when the alert fires. `{ "type": "log" }` — Log the alert (default) `{ "type": "webhook", "url": "https://..." }` — Send a webhook notification `{ "type": "function", "path": "my::alert::handler" }` — Invoke a registered function ## Functions ### Logging Log an informational message. The message to log. Optional structured fields to attach to the log entry. Optional trace ID for correlation. Optional span ID for correlation. Service name. Defaults to the function name if not provided. Log a warning message. Same parameters as `engine::log::info`. Log an error message. Same parameters as `engine::log::info`. Log a debug message. Same parameters as `engine::log::info`. Log a trace-level message. Same parameters as `engine::log::info`. ### Logs API Query stored log entries. Start time in Unix timestamp milliseconds. End time in Unix timestamp milliseconds. Filter by trace ID. Filter by span ID. Minimum severity number (1–24, higher = more severe). Filter by severity text (e.g., `"ERROR"`, `"WARN"`, `"INFO"`). Pagination offset. Defaults to `0`. Maximum number of entries to return. Array of log entries. Total number of matching log entries before pagination. Echo of all input query parameters used for the request. Omitted when log storage has not yet been initialized. Response timestamp in Unix milliseconds. Clear all stored log entries from memory. ### Traces API List stored trace spans. Filter by specific trace ID. Filter by service name (case-insensitive substring match). Filter by span name (case-insensitive substring match). Filter by status (case-insensitive substring match). Minimum span duration in milliseconds. Maximum span duration in milliseconds. Start time in Unix timestamp milliseconds. End time in Unix timestamp milliseconds. Sort field: `"duration"`, `"start_time"`, or `"name"`. Defaults to `"start_time"`. Sort order: `"asc"` or `"desc"`. Defaults to `"asc"`. Filter by span attributes. Array of `[key, value]` pairs (AND logic, exact match). Include internal engine traces (`engine.*` functions). Defaults to `false`. Pagination offset. Defaults to `0`. Pagination limit. Defaults to `100`. Array of span objects. Total number of matching spans before pagination. Applied pagination offset. Applied pagination limit. Retrieve a trace as a hierarchical span tree. The trace ID to retrieve. Array of root spans, each with nested child spans in a `children` field. Clear all stored trace spans from memory. ### Metrics API List collected metrics with aggregated statistics. Start time in Unix timestamp milliseconds. End time in Unix timestamp milliseconds. Filter by metric name. Aggregate interval in seconds. Built-in engine counters: `invocations` (total, success, error, deferred, by\_function), `workers` (spawns, deaths, active), and `performance` (avg\_duration\_ms, p50\_duration\_ms, p95\_duration\_ms, p99\_duration\_ms, min\_duration\_ms, max\_duration\_ms). Raw SDK metric data points collected from storage. Time-bucketed aggregations, present only when `aggregate_interval` is provided alongside a time range. Response timestamp in Unix milliseconds. Echo of the input query parameters, present when any time filter or interval was provided. List metric rollup aggregations (1-minute, 5-minute, 1-hour windows). Start time in Unix timestamp milliseconds. End time in Unix timestamp milliseconds. Rollup level index: `0` = 1 minute, `1` = 5 minutes, `2` = 1 hour. Filter by metric name. Array of rollup objects with time-bucketed aggregations. Array of histogram rollup objects for distribution metrics. The rollup level applied: `0` = 1 minute, `1` = 5 minutes, `2` = 1 hour. Echo of the input query parameters (`start_time`, `end_time`, `metric_name`). Response timestamp in Unix milliseconds. ### Baggage API Get a baggage value from the current trace context. Baggage key to retrieve. Set a baggage value in the current trace context. Baggage key. Baggage value. Get all baggage key-value pairs from the current trace context. ### Sampling API List all active sampling rules and their current configuration. ### Health API Check engine health status. Health status (e.g., `"healthy"`). Per-component health with `otel`, `metrics`, `logs`, and `spans` sub-statuses. Current time in Unix timestamp milliseconds. Engine version. ### Alerts API List all configured alert rules and their current state. Manually trigger evaluation of all alert rules against current metrics. ## Trigger Type This worker adds a new Trigger Type: `log`. Register a function to react to log entries as they are produced. The log level to subscribe to: `info`, `warn`, `error`, `debug`, or `trace`. When omitted, the trigger fires for all levels. ### Log Entry Payload Timestamp of the log entry in Unix nanoseconds. Observed timestamp in Unix nanoseconds. Numeric severity level (1–24). Severity text (e.g., `"INFO"`, `"WARN"`, `"ERROR"`). The log message content. Structured attributes attached to the log entry. Distributed tracing ID for correlating this log entry across services. Span ID for correlation within a trace. Resource attributes associated with the log entry. Name of the service that produced the log entry. Name of the instrumentation scope. Version of the instrumentation scope. ### Sample Code ```typescript theme={"theme":{"light":"catppuccin-latte","dark":"dark-plus"}} const fn = iii.registerFunction( 'monitoring::onError', async (logEntry) => { await sendAlert({ message: logEntry.body, severity: logEntry.severity_text, traceId: logEntry.trace_id, }) return {} }, ) iii.registerTrigger({ type: 'log', function_id: fn.id, config: { level: 'error' }, }) ```