Skip to content
Sign in

workflow

v0.3.4

Deterministic, crash-resumable multi-agent workflow orchestrator over harness turns.

license: Apache-2.0Used in 1 kitautomationdurablemulti-agentorchestrationworkflow

Install

bash
iii trigger compose::add worker=workflow

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

workflow worker

A deterministic DAG orchestrator running over the iii harness. Accepts workflow::start / workflow::tick / workflow::stop calls; persists run state in state; fans out nodes as child harness sessions with deterministic ids so duplicate deliveries are idempotent.

The always-on harness observers (harness::on-turn-completed → wake, harness::hook::on-pre-call → stamp-reply, harness::hook::on-post-turn → validate-result) are created once at startup. An observer created before the harness is up stays pending and becomes active when the harness registers its function, and again after a harness restart. workflow::start is registered at boot, but answers "not ready, retry" until the stamp-reply observer is active, or until 10 s passed while the harness is absent, so an agent's call cannot run unstamped while the worker boots. If the harness refuses the stamp-reply hook, at boot or when it starts or restarts later, workflow::start refuses every call with the reason until a worker restart after the fix. The last — harness::hook::on-pre-generate → inject-guidance — follows the inject_guidance knob in the workflow configuration entry (ON by default): turning it off shrinks prompts (the harness's # Granted functions catalog still advertises the workflow::* ids) and the worker observes or disposes it hot, no restart.


Getting the result back (don't poll)

workflow::start returns a run_id immediately. There are a few ways to learn the outcome; prefer push over polling workflow::status in a loop.

  1. Observe the run (for an AGENT caller). For a call from an agent turn, workflow::start returns {run_id, next}. next is the harness::observe call that wakes your session when the run ends. Make it exactly as given, then end your turn:

    // workflow::start returns
    {
      "run_id": "r_…",
      "next": {
        "call": "harness::observe",
        "arguments": {
          "function_id": "workflow::on-run-completed",
          "args": { "run_id": "r_…" },
          "label": "workflow r_…",
          "lifecycle": { "expires_in_ms": 82800000 }
        },
        "then": "END YOUR TURN after this call; …"
      }
    }

    The outcome arrives as [notification] workflow r_…: {…}, at once if the run already ended: the worker replays a finished run to an observer of it. The wake expires an hour before the run is collected (24 h after it ended), so an expired wake can observe again and still get the replay; a collected run is refused with UNKNOWN_RUN. A notification cut at the harness's 8,000-character limit is incomplete: call workflow::status once for the run. next is present only when the pre-call hook stamped your session; you pass no session id. A session whose policy narrows options.functions must allow harness::observe and workflow::on-run-completed.

    reply_to was removed: a start that passes it is refused with this recipe.

  2. Completion callback (push), for a WORKER caller. Pass notify to workflow::start. When the run reaches a terminal state the worker calls your function once with {run_id, status, result, result_error, failed_nodes?} (failed_nodes is omitted when no node failed; see Events):

    workflow::start {
      "nodes": { /* ... */ },
      "input": { /* ... */ },
      "notify": { "function_id": "myworker::wf-done" }  // optional "queue", defaults to "workflow-notify"
    }

    Delivery is durable (enqueued) and at-least-once — make the handler idempotent and dedup on run_id. This is the push path; no polling. The worker defines its workflow-notify queue at boot, apart from the workflow tick queue, so a slow callback cannot delay ticks; a custom queue must already be defined with queue::define.

  3. Observe every run. Every terminal run also calls workflow::on-run-completed. Observe it without args to see all runs, e.g. for audit (see Events); to wait for one run, pass args { run_id } instead.

    iii.observe(
      { function_id: 'workflow::on-run-completed', filters: { 'input.status': { $eq: 'failed' } }, capture: ['output'] },
      'audit::workflow-failed',
    )

workflow::status remains for one-off inspection, but a live tracker should observe the run (agents: the next call) or pass notify (workers) rather than run a status loop.


The request

workflow::start takes one level of nodes:

{"input": {"topic": "durable workflows"},
 "nodes": {
   "research": {"task": "List the key findings on the topic.",
                "schema": {"type": "object", "properties": {"findings": {"type": "array", "items": {"type": "string"}}},
                           "required": ["findings"]}},
   "report":   {"reads": "research", "task": "Write a one-page report from these findings."}}}

The worker refuses an unknown key at either level, and lists every shape problem of a request in one refusal. A key whose value is null counts as omitted. The removed shape (definition, agent, input.from, depends_on, fanout.over, output.from, the node: prefix) is refused with the recipe for each moved key.

Top-level key Default Meaning
nodes required One sub-agent session per node, keyed by node id. An id has no #, . or /.
input null Any JSON. Every node without reads or fanout receives it.
output every unread node A node id returns that result as is; an array of ids returns an object keyed by id. An unread node is one no other node names in reads or fanout: one returns its result as is, several an object keyed by id. A loop node or a loop body node returns its last pass; see Loops.
max_concurrency no cap Live node sessions at once (running, or reserved for a send). At least 1.
default_functions everything except workflow::*, configuration::*, approval::*, harness::hook::* and harness::send Dispatch policy of every node without functions.
idempotency_key, notify none See Getting the result back.
Node key Default Meaning
task required The node's opening message. Its input follows as JSON, fenced as untrusted data.
reads the run input A node id delivers that result as is (a text node as a JSON string, a fanout node as the array of its copies' results); an array of ids delivers an object keyed by id. The node runs after them.
fanout none One copy per element of an array; see Fanout.
schema text node JSON Schema, with a top-level type, of the JSON value the node delivers. Required on a node a fanout path points at and on a loop node.
model the caller's model A model id from router::models::list; an unregistered id is refused at start.
provider the routed provider Only with model or profile.
profile none Agent profile id (lowercase letters, digits, - and _, at most 64 characters) sent as the harness options.agent. It excludes system_prompt, and a model it pins wins over model.
system_prompt none Appended to the child's built-in identity prompt.
functions default_functions The node session's dispatch policy: an allow-list array or { "allow", "deny" }. List a denied id to give it back; a node that starts a sub-run lists both workflow::start and workflow::on-run-completed.
filesystem_root the caller's folder Absolute folder the node's session runs in. It must lie inside the caller's folder with symlinks resolved, checked once at start on the worker's host; a run without a caller folder cannot set it. Fanout copies and loop passes share it.
timeout_ms default_pending_timeout_ms (30 min) Wall clock an attempt may run, from its send. At least 1000.
retries max_node_retries (1) Refires after a timeout, each in a new @r<n> session. At most 10. A failed turn is not refired; see read_only.
read_only false true states that the node has no side effects, so a turn that fails on a transient provider failure is continued once in its own session; see Transient provider failures. The worker does not check it.
on_error "fail" "continue": a failed or cancelled node counts as finished with a null result, so its readers run and the run can complete. The error stays in node_errors.
label none Session title (fanout copies get #<i>, loop passes #it<k>) and node_details.label. 1 to 80 characters.
phase none Group counted in workflow::status phases. 1 to 80 characters.
loop none Repeat this node and its body while a field of its result holds; see Loops.

The model. A node that omits model and provider runs on the caller's model, read once at start from the caller session's harness::status context and written into the stored definition, so retries, loop passes and fanout copies stay on it. It also takes the caller's provider, unless it has a profile. With no caller session, or an unreadable model, a node without a profile is refused; a profile node then runs on the profile's pinned model. With output omitted, every unread node is required: a failed side node fails the run unless it sets on_error: "continue".

Fanout

A fanout node runs one copy per element of an array.

fanout The array Copies start
"plan.items" The items array of plan's JSON result, read once. When plan and every node in reads are settled.
["security","cost"] The array as written. When every node in reads is settled; without reads, at once.
The node has A copy receives
fanout only its element
fanout and reads: "draft" {"input": <draft's result>, "fanout_item": <its element>}
fanout and reads: ["draft","spec"] {"input": {"draft": …, "spec": …}, "fanout_item": <its element>}

input is what a plain node with the same reads receives. A copy never receives the run input. The node's result, for its readers and output, is the array of its copies' results in element order (the order written, for a written array); a failed copy of an on_error: "continue" node reads as null. A fanout cannot iterate another fanout's result: read it with a join node that returns one array.

Several critics, each with an angle, over one draft:

{"input": {"feature": "..."},
 "nodes": {
   "draft":  {"task": "Write a one-page design note for the feature."},
   "critic": {"fanout": ["security", "cost", "operability"], "reads": "draft",
              "task": "Criticize the draft in `input` from the angle in `fanout_item`. List concrete problems."},
   "final":  {"reads": ["draft", "critic"], "task": "Revise the draft. Fix every valid problem in `critic`."}}}

Three critic copies start when draft is done; final receives {"draft": "<the draft>", "critic": ["…", "…", "…"]} and is the run's result.

JSON nodes

A JSON node (one with schema) runs under the harness's text contract, so its session has no submit_result tool. Its opening message ends with the schema and asks the node to deliver its result by calling workflow-result::submit with {"result": <value>}; the value is checked against the schema, and an invalid one is refused with the cause so the node calls again. workflow::validate-result observes the harness's post-turn hook for node sessions: a turn that submitted passes, and so does a final message that is itself a valid JSON value (one fenced code block is tolerated); anything else is denied with the cause and the schema, so the harness re-prompts the turn, at most max_validation_retries times. A turn that ends with a wake armed (harness::status reads expects_wake) passes whatever it says, so a node can park on a status line and answer on the woken turn. A node that submitted settles when its turn ends, even with a wake armed. The worker then stops its session tree: armed wakes are disposed, and background shell jobs and child sessions are killed. A node that must wait for a wake submits on the woken turn. A node that must leave a background job running ends with no wake armed. When the node settles, reconcile checks its result again and stores the parsed value. A result that does not parse or match (a turn the hook did not check, for example while this worker restarted) gets the same correction sent into the node's session, at most twice per attempt; then the node fails. The failed node's error (node_errors, the run's result_error) names the cause without the schema. workflow::start rejects a schema that does not compile.

Provider-native structured output is not used for a node, even on a model that supports it: the schema is checked after the answer, not enforced while the model writes it. The node session's harness result (harness::status result, harness::on-turn-completed, the result any other post-turn validator on that session receives) is the answer text; the parsed value is only in workflow::status and workflow::node-result.

There is no separate finish signal: with no wake armed, a final message that parses and matches the schema settles the node. A node that may park needs a schema with required fields a status line cannot satisfy. Under {"type": "object"} alone, or any schema with no required, a status object such as {"status": "waiting on build"} sent after a refused harness::observe becomes the node's result.

Transient provider failures

The harness resumes a turn whose provider fails temporarily (an outage, a rate limit, a cut stream) up to max_transient_resumes times, at once, and then fails the turn with failure.retryable: true on harness::status. A node with read_only: true whose turn fails that way is not failed: it stays running, and at least 60 s later the worker sends one continue message into the same session, so the transcript and the provider's prompt cache carry over and the work already done is not paid again. The continue asks the node to resume where it stopped, without repeating calls that already returned, and to deliver its result.

  • Once per node uid, over all its attempts: each fanout copy and loop pass has its own, and a timeout refire does not give a new one. A second transient failure fails the node, and so does any failure of a node without read_only.
  • The continue runs in what is left of timeout_ms, which still counts from the attempt's send, and needs at least 5 minutes of it when sent: it reloads the whole transcript, so a continue the timeout stops at once would be paid for nothing. With less than 7 minutes left at the failure (the wait, one default sweep period and those 5 minutes), the node fails at once instead. A pass that reaches the due continue with less than 5 minutes left (a slower sweep_expression, say) fails the node with the transient error instead of sending it.
  • Before sending, the worker reads the session again: a session that moved on (a person typed into it, say) is settled as usual instead. With a caller session, every pass reads the caller while a node waits: a stopped caller turn stops the run, and an unreadable caller holds the continue for the next pass.
  • The transient error never reaches node_errors unless the continue fails too; with on_error: "continue" only that final failure reads as null.
  • An older harness, which reports no failure on harness::status, makes every failed turn fail the node.

Loops

A loop on a node (the loop node) runs it and its body nodes again while a field of its JSON result equals a string, as a do-while:

"work":   {"task": "Implement the ticket; fix every finding in loop_previous.", "profile": "deepwork-impl",
           "on_error": "continue"},
"verify": {"task": "Review the work.", "profile": "deepwork-adversary", "reads": "work", "on_error": "continue",
           "schema": {"type": "object",
             "properties": {"verdict": {"type": "string", "enum": ["pass", "reject"]},
                            "findings": {"type": "array", "items": {"type": "string"}}},
             "required": ["verdict", "findings"]},
           "loop": {"body": ["work"], "while": {"field": "verdict", "equals": "reject"}, "max_repeats": 2}},
"report": {"task": "Report the outcome.", "reads": "verify"}
  • while.field is a dotted path into the loop node's result; a missing field, a non-string value or a null result ends the loop. max_repeats (1 to 9) counts passes after the first, so 2 means at most 3 passes. The loop node must have a schema, and when its schema declares the field (and an enum for it), the start checks field and equals against it. The start refuses a field whose declared type is not string, and a field the schema does not list in required: each segment of the path must be in the required of its enclosing object, even with additionalProperties, unless that level uses anyOf, oneOf, allOf or $ref. A result without the field ends the loop after one pass.
  • A new pass starts only when every node of the loop is done in the last pass. A failed node ends the loop, even with on_error: "continue"; to retry a pass that timed out, set the body node's retries.
  • Pass k of a node runs as uid <id>#it<k> (the first pass keeps <id>), in its own session, with the node's timeout_ms and retries. workflow::node-result with <id>#it<k> returns pass k (<id>#it0 the first pass); the bare id of a loop node or body node returns its last pass, like reads and output. It returns null for a node that is not done.
  • Every node of the loop receives {"input": <its input>, "loop_previous": <the loop node's result of the previous pass>}, and a body node also "own_previous": <its own result of the previous pass>. Both are null on the first pass. Inside the loop, reads reads the same pass.
  • Outside the loop a node can read only the loop node: readers wait for its last pass and read that result. output may name the loop node and any body node: each returns its last pass, all from the same pass, and the run waits for the loop to end even when output names only a body node. A body node whose last pass failed under on_error: "continue" returns null (its error is in failed_nodes), never an earlier pass; read one with workflow::node-result and <id>#it<k>. Without on_error: "continue", a failed pass of any node of a loop the output needs fails the run.
  • Put the loop node in output, or read it from a later node. After the run, a result whose while.field still equals equals means the loop gave up: the repeats ran out, or a node of the last pass failed (see failed_nodes). Any other value means the condition stopped holding. The pass count is the number of <id>#it<k> uids in nodes, plus one.
  • Say in the task of each node of the loop how to use loop_previous (for example: fix every finding in it). A body node whose result is the work itself (a text or a JSON document, nothing left on disk) starts every pass from scratch unless its task says to build on own_previous (for example: keep own_previous and change only what the findings ask for).
  • Every body node must feed the loop node, a body node must not read its loop node, a node belongs to at most one loop, and no node of a loop may declare fanout or be iterated by one.
  • Under max_concurrency, free slots go to ready uids in node-id order, so an earlier node's pass can take a slot before a later node's first pass.

workflow::status

workflow::status { run_id, details? } returns run_id, status, nodes (uid → state), node_errors, node_results, result, result_error, created_at, updated_at and phases.

  • nodes also lists the nodes that have not started as pending. An unexpanded fanout appears once under its node id; a fanout over an empty array lists nothing. Each loop pass is listed under its own <id>#it<k> uid, in nodes, node_errors, node_results and node_details.
  • phases counts node uids per phase: {"code": {"total": 2, "done": 0, "failed": 0, "cancelled": 0, "running": 1}}. failed includes the cancelled ones. Nodes without a phase are left out, and the map is omitted when empty. Each loop pass counts once, so total grows with every repeat.

With details: true it adds node_details for at most 100 uids (running first, then pending, failed, cancelled, done) and details_truncated: true when it cut the list:

"node_details": {"kan-12": {
  "state": "running", "label": "KAN-12 · x", "phase": "code", "model": "m",
  "profile": "deepwork-lead", "session_id": "wf_r_ab_kan-12@r0", "attempt": 0,
  "started_at": 1, "finished_at": 2, "elapsed_ms": 1,
  "harness_status": "completed", "expects_wake": true, "last_generate_at": 3,
  "usage": {"complete": true, "sessions": 3, "turns": 12, "function_calls": 40,
            "function_call_errors": 0, "input_tokens": 1, "output_tokens": 1,
            "cache_read_tokens": 1, "cache_write_tokens": 1,
            "reasoning_tokens": null, "cost_usd": 0.42}}}

usage comes from harness::metrics over the node session's tree, and model and last_generate_at from its context snapshots. harness_status and expects_wake come from harness::status, for running nodes only. A running node waiting for its transient retry also has retry_at, the epoch ms its continue is due; until then its harness_status reads failed. A node whose send never reached the harness gets no live fields. These live reads run without the run lock, 16 at a time, with a 5 s timeout each. A failed read leaves its fields out, and nothing live is stored. model of a node with a profile is shown only once observed, because the profile may pin another model. Callers compute idle time from last_generate_at.

elapsed_ms is finished_at minus started_at, or the time since started_at while the node runs. It is the clock timeout_ms is checked against, so harness queue wait, parked time and the 2 s settle poll are included. It covers the current attempt: a timeout refire starts it again, and attempt counts those; each earlier attempt ran the full timeout_ms.

Limits

  • Parked nodes and stop. A node whose session waits on an armed wake stays running until a later turn ends it, unless it already submitted its result. When this worker stops a node session (timeout, refire, workflow::stop, a run ending with nodes in flight, a submitted node settling with a wake armed), it disposes every observer in the session's tree after the stop, so a parked session ends cancelled instead of waking later. That stop also kills the session tree's background shell jobs and child sessions; for a submitted node the session reads cancelled in the harness while the node is done. This needs harness v1.9.4 or newer: an older harness ignores quiet_deadlines, so disposing a parked node's armed wake sends it a [notification] wake removed message and starts a new turn right away. A wake that delivers between the stop and the disposal starts one more turn, which a second stop cancels; wakes that turn arms before the second stop stay armed, so the session can still wake once more. If harness::stop fails, the observers stay and the session can still wake; nothing retries the stop or the disposal for a refired, timed-out or settled submitted node; only workflow::stop gets a second pass when the run finalizes. A node stopped from outside while it holds armed wakes settles only when its wakes retire or at its timeout_ms, unless it had submitted. A submitted one settles at once from its turn's status: done if the turn had completed, cancelled if the stop cancelled a running turn.
  • A finished node settles on a second poll. A node whose harness::status reads finished is polled again 2 s later and settles from that reply, so a parked node does not settle on its parking turn. Each tick or sweep pass that settles a node takes about 2 s longer, holding the run lock and one of the 10 tick-queue slots that all runs share. A wake whose turn starts more than 2 s after its claim, or whose dispatch fails after the claim, can still settle the parking turn. A message that reaches a finished node session within the 2 s makes the node settle on that later turn.
  • The timeout counts queue wait. timeout_ms runs from the send, so it includes harness queue wait and parked time. The sweep checks it once a minute.
  • retries defaults to 1. A node that times out is refired once unless it sets retries: 0, and a refire repeats the node's side effects.
  • A transient retry waits 1 to 2 minutes. The continue goes out with the first tick or sweep pass at or after retry_at, and the default sweep runs once a minute. A stop of an idle caller does not cancel a node waiting for its continue; workflow::stop does. A stop of a busy caller is seen only while it is the caller's current turn: one followed by a new caller turn before the next pass (up to one sweep period) is missed. read_only is a promise the worker does not check: a node with side effects that sets it gets continued too, though the continue asks it not to repeat calls that already returned.
  • No ordering-only dependency. A node runs after the nodes it names in reads or fanout, and receives their results, so in "do X, then validate" over shared state the validator reads X's result, which may be text.
  • Loops are bounded and quiet. Each pass is a new session; continuity is the work it leaves behind plus loop_previous and, for a body node, own_previous. A loop ends without an error when its repeats run out, when the condition reads a missing field, or when a pass would push the run past 50,000 materialized nodes. Put the loop node in output, or read it from a later node: a last result whose while.field still equals equals means the loop gave up (the repeats ran out, or a node of the last pass failed; see failed_nodes), any other value means the condition stopped holding, and the pass count is the number of <id>#it<k> uids in nodes, plus one. Say in the task of each node of the loop how to use loop_previous (for example: fix every finding in it), and in a body node that answers with the work itself, to build on own_previous. No node of a loop can fan out. Outside its loop a node reads only the loop node; output may also name a body node, which returns its last pass.
  • The run result is capped at 1 MiB. An output whose combined value is larger fails the run, and result_error says so. The node results stay readable with workflow::node-result, so name fewer nodes in output when their results are large.
  • Usage covers the current attempt. A refired node reports the usage of its newest @r<n> session tree only.
  • Stopping an idle caller. Stopping an idle caller cancels the running nodes, but consumers of on_error: continue nodes still fire. Use workflow::stop to end the run.
  • finished_at after an upgrade. Nodes that ended before the upgrade get the time of the run's next write as finished_at; a run that had already ended has none.
  • reply_to runs after an upgrade. A run started with reply_to before the upgrade gets no delivery: its record still loads, but nothing posts its outcome and nothing wakes its caller. To recover, prompt the caller to observe the run with args { run_id } (a run that already ended is replayed at once; one still running reports when it ends) or to read workflow::status, within 24 h of the run's end.
  • Forgotten observe. An agent that never makes the next call is not woken; the outcome stays in workflow::status for 24 h. The worker logs a warning and counts workflow.runs.ended_unobserved for a run with a stamped caller, no notify and no observer of that run (an every-run observer does not count). A run that ends before its caller's observe lands counts too, so the counter is an upper bound on stalled callers. Do not subtract workflow.runs.replayed: it counts every replay (a second observer, or one re-listed after a restart). To find the runs really stalled, match the warning's run_id against the replayed the run's end log lines.

Events

Function Args Filter on Event
workflow::on-run-completed { run_id? }: one run, replayed if it already ended; omitted: every run input.run_id, input.status (no replay) RunCompletedEvent { run_id, status: completed | failed | cancelled, result, result_error, failed_nodes? }

The function is private (only this worker calls it; anyone may observe it) and returns the event it is called with. The worker calls it once per terminal run, at least once (a crash mid-finalize re-fires it), and only while an observer wants that run; dedup on run_id. failed_nodes lists the node '<uid>': <error> of each node a completed run tolerated failing (on_error: continue), and is omitted when there are none.

An observer with args { run_id } gets only that run. Adding it for a run that already ended replays the event to it at once, through the observer: a call target receives the bare event, not the Observed envelope of a live delivery, so a worker with its own target must accept both (or use notify). An observer accepted during the run's final tick can get both the live event and the replay. Adding one for an unknown or collected run is refused with UNKNOWN_RUN, and unknown args with INVALID_ARGS. filters get no replay: to wait for one run, use args, never filters.

Observers

Observed function Args / filters Target Target receives
cron::on-tick args { expression: <sweep_expression> } workflow::sweep the tick (capture: [output], read as payload.output)
harness::on-turn-completed filters { "input.session_id": { "$prefix": "wf_" } } workflow::wake TurnCompletedEvent of a node session (capture: [output], read as payload.output)
harness::hook::on-pre-call args { functions: ["workflow::start"], on_error: "fail_closed", timeout_ms: 30000 } workflow::stamp-reply the pre-call hook input; returns the hook decision. A hook error or timeout refuses the agent's workflow::start
harness::hook::on-post-turn args { sessions: ["wf_*"], on_error: "fail_open", timeout_ms: 30000, retry_prompt: <answer-format nudge> } workflow::validate-result the post-turn hook input; returns deny with a reason for a JSON node's answer that is not a JSON value matching its schema, unless a wake is armed. An error or timeout passes the turn, and reconcile still checks the result
harness::hook::on-pre-generate args { on_error: "fail_open", inject_prompt: <workflow guidance> }, only while inject_guidance is on workflow::inject-guidance not called: the harness joins inject_prompt into the system prompt
configuration::on-updated filters on input.id and input.event_type (through iii-config-client) workflow::on-config-change the configuration event (capture: [output], read as payload.output)

A sweep_expression change observes the new expression first, then disposes the old observer; if the cron worker refuses the new expression (INVALID_CRON_EXPRESSION), the old observer and schedule stay.


Scaling & concurrency

Single-writer-per-run (the correctness model)

Every path that reads, mutates, and writes a run record — workflow::tick, reconcile, the pending-timeout sweep, and workflow::stop — holds the per-run WorkflowLocks guard before touching state. This serialises writes within one process and closes the read-modify-write race.

Idempotency for duplicate event deliveries is provided by deterministic ids: the child session id (wf_<run_id>_<node_uid>@r<attempt>) and the workflow_node_result/<run_id>/<node_uid> key are stable, so a re-delivered tick that re-fires an already-running node is a no-op at the harness layer. A tick persists each node's attempt session before sending it; a later tick that finds the session reserved asks harness::status before sending again. A workflow::start idempotency_key is checked and written under one fixed lock ("idem"), and the sweep deletes the key with its run under that lock.

Node sessions carry the caller as metadata.parent_session_id, so they sit in the caller's session tree: stopping the caller's session stops every running node. When the caller's turn reads cancelled, the run ends cancelled, even with on_error: continue nodes; an idle caller is not detected (see Limits). workflow::stop stops one run's node sessions without touching the caller.

Node sessions run in the caller's folder: the pre-call hook stamps the caller turn's metadata.fs_scope.root as caller_fs_root (replacing any value the agent sent), and every node send carries it as options.metadata.fs_scope.root, unless the node sets filesystem_root. Without one, nodes get the harness default folder. A non-agent caller (CLI, worker) is not hooked, so its caller_fs_root is used as sent; an empty one is refused. A node granted workflow::start and workflow::on-run-completed (its next observe; workflow::* is denied by default) passes its own folder to a nested run and is woken when it ends. The hook runs only on calls made from an agent turn, so the harness refuses workflow::start as an observer binding target and fp refuses it as a pipe step; both need the harness and fp releases that carry those refusals.

One instance per namespace

A function id has one host per namespace: the engine rejects a second registration of the same id with FUNCTION_NAMESPACE_CONFLICT. The worker registers every workflow::* function, workflow::on-run-completed included, so the first instance in a namespace hosts them all and the per-run WorkflowLocks live in that one process. A second instance in the same namespace gets every registration rejected while its observers (the sweep and the harness hooks) still deliver to the first instance's functions, so each sweep and hook runs twice. Run one workflow worker per namespace.

Why not state::compare-and-set

state::set is an unconditional overwrite. state::compare-and-set exists, but the worker does not adopt it for run records or idempotency keys:

  • it compares the whole stored value, so every run-record write would have to carry the full previous record;
  • an absent key compares equal to a stored null;
  • it has no size guard;
  • on Redis it retries a scope-wide WATCH up to 8 times, so concurrent writers to different runs in the same scope contend with each other.

Since the engine allows one instance per namespace (above), the in-process locks give every run one writer, and WorkflowRunRecord carries no version field.

Search the registry

Find workers, kits and executables, or jump to a page.