queue
v0.22.0Durable function queues - hosts `queue::on-message`, which topic subscribers observe, and the queue/DLQ service functions.
Install
iii trigger compose::add worker=queue@0.22.0Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
skill doc
queue
The queue worker provides durable topic delivery for iii. A consumer
subscribes by observing queue::on-message with the topic as args, publishers
call iii::durable::publish, and the worker persists messages, retries failed
deliveries, moves exhausted jobs to DLQ, and exposes redrive/discard
inspection functions.
Install it with iii call compose::add worker=queue. The legacy built-in queue
service must be removed from the engine config before this worker starts
because both serve the same queues.
When to Use
- A topic event must be processed reliably by a subscribed function.
- A failed consumer should retry automatically before the message lands in DLQ.
- You need to inspect, redrive, or discard dead-lettered messages.
- Local or single-instance durable queues are enough, using in-memory storage for development or file-backed storage for restart survival.
Boundaries
- Not fire-and-forget broadcast. Use
pubsubwhen missed events are acceptable. - Three transports ship today:
builtin(in-process, full DLQ/retry/fifo),redis(pub/sub only, no DLQ), andrabbitmq(full: retry/DLQ/priority/fifo). Thebridgeadapter is not ported — it was engine-internal; the engine'sQueueEnqueuercut (MOT-3829) serves that path. - Engine
CallAction.Enqueue/ named function queues require the separateQueueEnqueuerengine cut. Where it is missing, useiii::durable::publishandqueue::on-messagesubscribers. queue::on-messageis private: only this worker calls it. Observe it; a call from another worker is refused (FORBIDDEN).- File-backed mode survives worker restarts. In-memory mode loses pending jobs
on restart or transport hot-swap. An unreachable
redis/rabbitmqtarget at boot fails the boot (no fallback tobuiltin). fifomode is strictly serial — there is no grouped-fifo partitioning bygroup_idlike the engine'sGroupedFifoWorker.- This worker carries work items, not key/value or stream state. Use
stateorstreamfor those.
Functions
iii::durable::publish: publish a JSON payload to a queue/topic. Input accepts{ "queue": "...", "data": ... }and also acceptstopicas an alias.iii::queue::redrive: move all DLQ messages for a queue back to the main queue. Input acceptsqueueortopic.iii::queue::redrive_message: move one DLQ message back bymessage_id.iii::queue::discard_message: purge one DLQ message bymessage_id.engine::queue::list_topics: list known queue topics.engine::queue::topic_stats: returndepth,consumer_count, anddlq_depthfor one topic.engine::queue::dlq_topics: list DLQ topics with message counts.engine::queue::dlq_messages: browse DLQ messages withoffsetandlimit.queue::on-message: one call per message for each subscriber of a topic. Observe it with args{ topic, max_retries?, backoff_ms?, queue_config?, group? }; the target receives the published data and its result acks the message.
Subscribing
Observe queue::on-message when a function should consume every message
published to a topic with retries and DLQ handling:
iii.observe(
{
function_id: 'queue::on-message',
args: { topic: 'orders.created', max_retries: 3, backoff_ms: 1000 },
},
'orders::process',
{ id: 'orders-process' },
)Args:
topic(required): topic to consume.queueis accepted as an alias.max_retries(default3): failed attempts before moving to DLQ.backoff_ms(default1000): base retry delay. Retries use the same exponential curve as the builtin queue:backoff_ms * 2^(attempts - 1).queue_config(optional): per-subscriber tuning (type: fifo,concurrency,maxPriorityon RabbitMQ).group(optional): observers naming the same group share one queue, so the group handles each message once. A member whose worker disconnects leaves the group, and the group's messages go to the members that stay.
The target receives the published data. Returning acks the message; throwing
retries it with backoff and then moves it to the DLQ. The worker calls the
target through its observer, so the engine's observers.activation_timeout_ms
(30 s by default) bounds each delivery: a target that has not answered by then
fails it with TIMEOUT, and the message is retried while that call still runs.
Raise the setting in the engine's config.yaml for targets that run longer. Filters read the message
as input, and a message an observer's filters reject is acked without
calling its target. Every subscriber of a topic receives every message, except
that a group shares one. Args without a topic, or with an unknown field, are
refused with INVALID_ARGS.
Each subscriber's durable queue is keyed by its observer id: pin the id
option to keep the queue and its backlog across restarts of the observing
worker. The restarted worker, connecting under the same worker name in the
same namespace, observes under the same id and replaces the observer it left
behind; any other worker that observes under that id is refused with
OBSERVER_ID_CONFLICT until the engine's observers.observer_grace_ms ends.
When the observer is removed (dispose(), or its worker
disconnected), the consumer stops at once and the queue keeps receiving the
topic for subscriber_retention_ms (30 s by default, like the engine's
observers.observer_grace_ms): a worker that restarts under the same name
within the engine's grace period re-observes its id; when it is back within
the retention as well, it receives its unacked backlog and everything
published while it was away. After the retention the queue keeps only what
it holds, for a later observer with the same id. Keep
subscriber_retention_ms at least the engine's observers.observer_grace_ms.
Configuration
Use the configuration worker entry named queue. The default is in-memory:
adapter:
name: builtinFor restart survival, use file-backed storage:
adapter:
name: builtin
config:
store_method: file_based
file_path: data/queue
save_interval_ms: 5000For pub/sub-only delivery against a shared Redis (no DLQ, no durability):
adapter:
name: redis
config:
redis_url: redis://localhost:6379For full retry/DLQ/priority/fifo delivery against RabbitMQ:
adapter:
name: rabbitmq
config:
amqp_url: amqp://localhost:5672
max_attempts: 3
prefetch_count: 10
queue_mode: standard
priority_field: priorityChanging the adapter config hot-swaps the transport and restarts every consumer — the new adapter is built first, so a bad config leaves the previous one serving. File-backed jobs survive by construction; in-memory jobs are lost, matching the builtin adapter's in-process durability profile.