Queue

A module for asynchronous job processing. It supports two modes: topic-based queues (register a consumer per topic, emit events) and named queues (enqueue function calls via TriggerAction.Enqueue, no trigger registration).

modules::queue::QueueModule

Queue modes

Topic-based queues

Register a consumer for a topic and emit events to it.

1. Register a consumer with registerTrigger({ type: 'queue', function_id: 'my::handler', config: { topic: 'order.created' } }). This subscribes the handler to that topic.
2. Emit events by calling trigger({ function_id: 'enqueue', payload: { topic: 'order.created', data: payload } }) or trigger({ function_id: 'enqueue', payload: { topic, data }, action: TriggerAction.Void() }) for fire-and-forget. The enqueue function routes the payload to all subscribers of that topic.
3. Action on the trigger: the handler receives the data as its input. Optional queue_config on the trigger controls per-subscriber retries and concurrency.

The producer knows the topic name; consumers register to receive it. Queue settings can live at the trigger registration site.

Named queues

Define queues in iii-config.yaml, then enqueue function calls directly. No trigger registration.

1. Define queues in queue_configs (see Configuration).
2. Enqueue a function call with trigger({ function_id: 'orders::process', payload, action: TriggerAction.Enqueue({ queue: 'payment' }) }). The engine routes the job to the named queue and invokes the function when a worker consumes it.
3. Action on the trigger: the target function receives payload as its input. Retries, concurrency, and FIFO are configured in iii-config.yaml.

The producer targets the function and queue explicitly. Queue configuration is centralized.

When to use which

Both modes are valid. Named queues offer config-driven retries, concurrency, and FIFO. See Queue System for design rationale.

Sample Configuration

- class: modules::queue::QueueModule
  config:
    queue_configs:
      default:
        max_retries: 5
        concurrency: 5
        type: standard
      payment:
        max_retries: 10
        concurrency: 2
        type: fifo
        message_group_field: transaction_id
    adapter:
      class: modules::queue::BuiltinQueueAdapter
      config:
        store_method: file_based
        file_path: ./data/queue_store

Configuration

Queue Configuration

Each entry in queue_configs defines an independent named queue with its own retry, concurrency, and ordering settings.

Adapters

modules::queue::BuiltinQueueAdapter

Built-in in-process queue. No external dependencies. Suitable for single-instance deployments — messages are not shared across engine instances.

class: modules::queue::BuiltinQueueAdapter
config:
  store_method: file_based   # in_memory | file_based
  file_path: ./data/queue_store  # required when store_method is file_based

modules::queue::RedisAdapter

Uses Redis as the queue backend. Enables message distribution across multiple engine instances. Does not support retries or dead-letter queues.

class: modules::queue::RedisAdapter
config:
  redis_url: ${REDIS_URL:redis://localhost:6379}

modules::queue::RabbitMQAdapter

Uses RabbitMQ as the queue backend. Supports durable delivery, retries, and dead-letter queues across multiple engine instances.

The engine owns consumer loops, retry acknowledgement, and backoff logic — RabbitMQ is used as a transport only. Retry uses explicit ack + republish to a retry exchange with an x-attempt header, keeping compatibility with both classic and quorum queues.

class: modules::queue::RabbitMQAdapter
config:
  amqp_url: ${RABBITMQ_URL:amqp://localhost:5672}

Adapter Comparison

Queue Flow