# workflow

> Deterministic, crash-resumable multi-agent workflow orchestrator over harness turns.

| field | value |
|-------|-------|
| version | 0.3.4 |
| type | binary |
| license | Apache-2.0 |
| repo | https://github.com/iii-hq/workers |
| supported_targets | x86_64-apple-darwin, aarch64-apple-darwin, x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu, x86_64-unknown-linux-musl, armv7-unknown-linux-gnueabihf |
| author | iii |

## installation

```sh
iii trigger compose::add worker=workflow@0.3.4
```

## dependencies

- `configuration` @ `latest`
- `cron` @ `latest`
- `harness` @ `latest`
- `queue` @ `latest`
- `state` @ `latest`

## readme

# workflow worker

A deterministic DAG orchestrator running over the iii harness. Accepts
`workflow::start` / `workflow::tick` / `workflow::stop` calls; persists run
state in state; fans out nodes as child harness sessions with deterministic
ids so duplicate deliveries are idempotent.

The always-on harness observers (`harness::on-turn-completed` → wake,
`harness::hook::on-pre-call` → stamp-reply, `harness::hook::on-post-turn` →
validate-result) are created once at startup. An
observer created before the harness is up stays `pending` and becomes active
when the harness registers its function, and again after a harness restart.
`workflow::start` is registered at boot, but answers "not ready, retry" until
the stamp-reply observer is active, or until 10 s passed while the harness is
absent, so an agent's call cannot run unstamped while the worker boots. If the
harness refuses the stamp-reply hook, at boot or when it starts or restarts
later, `workflow::start` refuses every call with the reason until a worker
restart after the fix.
The last — `harness::hook::on-pre-generate` → inject-guidance — follows the
`inject_guidance` knob in the `workflow` configuration entry (ON by default):
turning it off shrinks prompts (the harness's `# Granted functions` catalog
still advertises the `workflow::*` ids) and the worker observes or disposes it
hot, no restart.

---

## Getting the result back (don't poll)

`workflow::start` returns a `run_id` immediately. There are a few ways to learn
the outcome; prefer push over polling `workflow::status` in a loop.

0. **Observe the run (for an AGENT caller).** For a call from an agent turn,
   `workflow::start` returns `{run_id, next}`. `next` is the
   `harness::observe` call that wakes your session when the run ends. Make it
   exactly as given, then end your turn:

   ```jsonc
   // workflow::start returns
   {
     "run_id": "r_…",
     "next": {
       "call": "harness::observe",
       "arguments": {
         "function_id": "workflow::on-run-completed",
         "args": { "run_id": "r_…" },
         "label": "workflow r_…",
         "lifecycle": { "expires_in_ms": 82800000 }
       },
       "then": "END YOUR TURN after this call; …"
     }
   }
   ```

   The outcome arrives as `[notification] workflow r_…: {…}`, at once if the
   run already ended: the worker replays a finished run to an observer of it.
   The wake expires an hour before the run is collected (24 h after it ended),
   so an expired wake can observe again and still get the replay; a collected
   run is refused with `UNKNOWN_RUN`. A notification cut at the harness's
   8,000-character limit is incomplete: call `workflow::status` once for the
   run. `next` is present only when the pre-call hook stamped your session; you
   pass no session id. A session whose policy narrows `options.functions` must
   allow `harness::observe` and `workflow::on-run-completed`.

   `reply_to` was removed: a start that passes it is refused with this recipe.

1. **Completion callback (push), for a WORKER caller.** Pass `notify` to
   `workflow::start`. When the run reaches a terminal state the worker calls
   your function once with `{run_id, status, result, result_error, failed_nodes?}`
   (`failed_nodes` is omitted when no node failed; see [Events](#events)):

   ```jsonc
   workflow::start {
     "nodes": { /* ... */ },
     "input": { /* ... */ },
     "notify": { "function_id": "myworker::wf-done" }  // optional "queue", defaults to "workflow-notify"
   }
   ```

   Delivery is durable (enqueued) and **at-least-once** — make the handler
   idempotent and dedup on `run_id`. This is the push path; no polling. The
   worker defines its `workflow-notify` queue at boot, apart from the `workflow`
   tick queue, so a slow callback cannot delay ticks; a custom `queue` must
   already be defined with `queue::define`.

2. **Observe every run.** Every terminal run also calls
   `workflow::on-run-completed`. Observe it without args to see *all* runs,
   e.g. for audit (see [Events](#events)); to wait for one run, pass args
   `{ run_id }` instead.

   ```ts
   iii.observe(
     { function_id: 'workflow::on-run-completed', filters: { 'input.status': { $eq: 'failed' } }, capture: ['output'] },
     'audit::workflow-failed',
   )
   ```

`workflow::status` remains for one-off inspection, but a live tracker should
observe the run (agents: the `next` call) or pass `notify` (workers) rather than
run a status loop.

---

## The request

`workflow::start` takes one level of nodes:

```jsonc
{"input": {"topic": "durable workflows"},
 "nodes": {
   "research": {"task": "List the key findings on the topic.",
                "schema": {"type": "object", "properties": {"findings": {"type": "array", "items": {"type": "string"}}},
                           "required": ["findings"]}},
   "report":   {"reads": "research", "task": "Write a one-page report from these findings."}}}
```

The worker refuses an unknown key at either level, and lists every shape
problem of a request in one refusal. A key whose value is `null` counts as
omitted. The removed shape (`definition`, `agent`, `input.from`, `depends_on`,
`fanout.over`, `output.from`, the `node:` prefix) is refused with the recipe
for each moved key.

| Top-level key | Default | Meaning |
|---|---|---|
| `nodes` | required | One sub-agent session per node, keyed by node id. An id has no `#`, `.` or `/`. |
| `input` | `null` | Any JSON. Every node without `reads` or `fanout` receives it. |
| `output` | every unread node | A node id returns that result as is; an array of ids returns an object keyed by id. An unread node is one no other node names in `reads` or `fanout`: one returns its result as is, several an object keyed by id. A loop node or a loop body node returns its last pass; see [Loops](#loops). |
| `max_concurrency` | no cap | Live node sessions at once (running, or reserved for a send). At least 1. |
| `default_functions` | everything except `workflow::*`, `configuration::*`, `approval::*`, `harness::hook::*` and `harness::send` | Dispatch policy of every node without `functions`. |
| `idempotency_key`, `notify` | none | See [Getting the result back](#getting-the-result-back-dont-poll). |

| Node key | Default | Meaning |
|---|---|---|
| `task` | required | The node's opening message. Its input follows as JSON, fenced as untrusted data. |
| `reads` | the run `input` | A node id delivers that result as is (a text node as a JSON string, a fanout node as the array of its copies' results); an array of ids delivers an object keyed by id. The node runs after them. |
| `fanout` | none | One copy per element of an array; see [Fanout](#fanout). |
| `schema` | text node | JSON Schema, with a top-level `type`, of the JSON value the node delivers. Required on a node a `fanout` path points at and on a loop node. |
| `model` | the caller's model | A model id from `router::models::list`; an unregistered id is refused at start. |
| `provider` | the routed provider | Only with `model` or `profile`. |
| `profile` | none | Agent profile id (lowercase letters, digits, `-` and `_`, at most 64 characters) sent as the harness `options.agent`. It excludes `system_prompt`, and a model it pins wins over `model`. |
| `system_prompt` | none | Appended to the child's built-in identity prompt. |
| `functions` | `default_functions` | The node session's dispatch policy: an allow-list array or `{ "allow", "deny" }`. List a denied id to give it back; a node that starts a sub-run lists both `workflow::start` and `workflow::on-run-completed`. |
| `filesystem_root` | the caller's folder | Absolute folder the node's session runs in. It must lie inside the caller's folder with symlinks resolved, checked once at start on the worker's host; a run without a caller folder cannot set it. Fanout copies and loop passes share it. |
| `timeout_ms` | `default_pending_timeout_ms` (30 min) | Wall clock an attempt may run, from its send. At least 1000. |
| `retries` | `max_node_retries` (1) | Refires after a timeout, each in a new `@r<n>` session. At most 10. A failed turn is not refired; see `read_only`. |
| `read_only` | `false` | `true` states that the node has no side effects, so a turn that fails on a transient provider failure is continued once in its own session; see [Transient provider failures](#transient-provider-failures). The worker does not check it. |
| `on_error` | `"fail"` | `"continue"`: a failed or cancelled node counts as finished with a `null` result, so its readers run and the run can complete. The error stays in `node_errors`. |
| `label` | none | Session title (fanout copies get ` #<i>`, loop passes ` #it<k>`) and `node_details.label`. 1 to 80 characters. |
| `phase` | none | Group counted in `workflow::status` `phases`. 1 to 80 characters. |
| `loop` | none | Repeat this node and its `body` while a field of its result holds; see [Loops](#loops). |

**The model.** A node that omits `model` and `provider` runs on the caller's
model, read once at start from the caller session's `harness::status`
context and written into the stored definition, so retries, loop passes and
fanout copies stay on it. It also takes the caller's provider, unless it has a
`profile`. With no caller session, or an unreadable model, a node without a
`profile` is refused; a `profile` node then runs on the profile's pinned
model. With `output` omitted, every unread node is required: a failed side
node fails the run unless it sets `on_error: "continue"`.

### Fanout

A fanout node runs one copy per element of an array.

| `fanout` | The array | Copies start |
|---|---|---|
| `"plan.items"` | The `items` array of `plan`'s JSON result, read once. | When `plan` and every node in `reads` are settled. |
| `["security","cost"]` | The array as written. | When every node in `reads` is settled; without `reads`, at once. |

| The node has | A copy receives |
|---|---|
| `fanout` only | its element |
| `fanout` and `reads: "draft"` | `{"input": <draft's result>, "fanout_item": <its element>}` |
| `fanout` and `reads: ["draft","spec"]` | `{"input": {"draft": …, "spec": …}, "fanout_item": <its element>}` |

`input` is what a plain node with the same `reads` receives. A copy never
receives the run `input`. The node's result, for its readers and `output`, is
the array of its copies' results in element order (the order written, for a
written array); a failed copy of an `on_error: "continue"` node reads as
`null`. A fanout cannot iterate another fanout's result: read it with a join
node that returns one array.

Several critics, each with an angle, over one draft:

```jsonc
{"input": {"feature": "..."},
 "nodes": {
   "draft":  {"task": "Write a one-page design note for the feature."},
   "critic": {"fanout": ["security", "cost", "operability"], "reads": "draft",
              "task": "Criticize the draft in `input` from the angle in `fanout_item`. List concrete problems."},
   "final":  {"reads": ["draft", "critic"], "task": "Revise the draft. Fix every valid problem in `critic`."}}}
```

Three `critic` copies start when `draft` is done; `final` receives
`{"draft": "<the draft>", "critic": ["…", "…", "…"]}` and is the run's result.

### JSON nodes

A JSON node (one with `schema`) runs under the harness's text contract, so its
session has no `submit_result` tool. Its opening message ends with the schema
and asks the node to deliver its result by calling `workflow-result::submit`
with `{"result": <value>}`; the value is checked against the schema, and an
invalid one is refused with the cause so the node calls again.
`workflow::validate-result` observes the harness's post-turn hook for node
sessions: a turn that submitted passes, and so does a final message that is
itself a valid JSON value (one fenced code block is tolerated); anything else
is denied with the cause and the schema, so the harness re-prompts the turn,
at most `max_validation_retries` times. A turn that ends with a wake armed
(`harness::status` reads `expects_wake`) passes whatever it says, so a node can
park on a status line and answer on the woken turn. A node that submitted
settles when its turn ends, even with a wake armed. The worker then stops its
session tree: armed wakes are disposed, and background shell jobs and child
sessions are killed. A node that must wait for a wake submits on the woken
turn. A node that must leave a background job running ends with no wake armed.
When the node settles, reconcile checks its result again and stores the parsed
value. A result that does not parse or match (a turn the hook did not check,
for example while this worker restarted) gets the same correction sent into the
node's session, at most twice per attempt; then the node fails. The failed
node's error (`node_errors`, the run's `result_error`) names the cause without
the schema. `workflow::start` rejects a schema that does not compile.

Provider-native structured output is not used for a node, even on a model that
supports it: the schema is checked after the answer, not enforced while the
model writes it. The node session's harness result (`harness::status` `result`,
`harness::on-turn-completed`, the `result` any other post-turn validator on
that session receives) is the answer text; the parsed value is only in
`workflow::status` and `workflow::node-result`.

There is no separate finish signal: with no wake armed, a final message that
parses and matches the schema settles the node. A node that may park needs a
schema with `required` fields a status line cannot satisfy. Under
`{"type": "object"}` alone, or any schema with no `required`, a status object
such as `{"status": "waiting on build"}` sent after a refused
`harness::observe` becomes the node's result.

### Transient provider failures

The harness resumes a turn whose provider fails temporarily (an outage, a rate
limit, a cut stream) up to `max_transient_resumes` times, at once, and then
fails the turn with `failure.retryable: true` on `harness::status`. A node with
`read_only: true` whose turn fails that way is not failed: it stays running,
and at least 60 s later the worker sends one continue message into the same
session, so the transcript and the provider's prompt cache carry over and the
work already done is not paid again. The continue asks the node to resume where it stopped,
without repeating calls that already returned, and to deliver its result.

- Once per node uid, over all its attempts: each fanout copy and loop pass has
  its own, and a timeout refire does not give a new one. A second transient
  failure fails the node, and so does any failure of a node without
  `read_only`.
- The continue runs in what is left of `timeout_ms`, which still counts from
  the attempt's send, and needs at least 5 minutes of it when sent: it reloads
  the whole transcript, so a continue the timeout stops at once would be paid
  for nothing. With less than 7 minutes left at the failure (the wait, one
  default sweep period and those 5 minutes), the node fails at once instead.
  A pass that reaches the due continue with less than 5 minutes left (a slower
  `sweep_expression`, say) fails the node with the transient error instead of
  sending it.
- Before sending, the worker reads the session again: a session that moved on
  (a person typed into it, say) is settled as usual instead. With a caller
  session, every pass reads the caller while a node waits: a stopped caller
  turn stops the run, and an unreadable caller holds the continue for the next
  pass.
- The transient error never reaches `node_errors` unless the continue fails
  too; with `on_error: "continue"` only that final failure reads as `null`.
- An older harness, which reports no `failure` on `harness::status`, makes
  every failed turn fail the node.

## Loops

A `loop` on a node (the loop node) runs it and its `body` nodes again while a
field of its JSON result equals a string, as a do-while:

```jsonc
"work":   {"task": "Implement the ticket; fix every finding in loop_previous.", "profile": "deepwork-impl",
           "on_error": "continue"},
"verify": {"task": "Review the work.", "profile": "deepwork-adversary", "reads": "work", "on_error": "continue",
           "schema": {"type": "object",
             "properties": {"verdict": {"type": "string", "enum": ["pass", "reject"]},
                            "findings": {"type": "array", "items": {"type": "string"}}},
             "required": ["verdict", "findings"]},
           "loop": {"body": ["work"], "while": {"field": "verdict", "equals": "reject"}, "max_repeats": 2}},
"report": {"task": "Report the outcome.", "reads": "verify"}
```

- `while.field` is a dotted path into the loop node's result; a missing field,
  a non-string value or a null result ends the loop. `max_repeats` (1 to 9)
  counts passes after the first, so `2` means at most 3 passes. The loop node
  must have a `schema`, and when its schema declares the field (and an `enum`
  for it), the start checks `field` and `equals` against it. The start refuses
  a field whose declared `type` is not `string`, and a field the schema does
  not list in `required`: each segment of the path must be in the `required`
  of its enclosing object, even with `additionalProperties`, unless that level
  uses `anyOf`, `oneOf`, `allOf` or `$ref`. A result without the field ends the
  loop after one pass.
- A new pass starts only when every node of the loop is `done` in the last
  pass. A failed node ends the loop, even with `on_error: "continue"`; to
  retry a pass that timed out, set the body node's `retries`.
- Pass `k` of a node runs as uid `<id>#it<k>` (the first pass keeps `<id>`), in
  its own session, with the node's `timeout_ms` and `retries`.
  `workflow::node-result` with `<id>#it<k>` returns pass `k` (`<id>#it0` the
  first pass); the bare id of a loop node or body node returns its last pass,
  like `reads` and `output`. It returns null for a node that is not `done`.
- Every node of the loop receives `{"input": <its input>, "loop_previous":
  <the loop node's result of the previous pass>}`, and a body node also
  `"own_previous": <its own result of the previous pass>`. Both are null on
  the first pass. Inside the loop, `reads` reads the same pass.
- Outside the loop a node can read only the loop node: readers wait for its
  last pass and read that result. `output` may name the loop node and any
  body node: each returns its last pass, all from the same pass, and the run
  waits for the loop to end even when `output` names only a body node. A body
  node whose last pass failed under `on_error: "continue"` returns `null` (its
  error is in `failed_nodes`), never an earlier pass; read one with
  `workflow::node-result` and `<id>#it<k>`. Without `on_error: "continue"`, a
  failed pass of any node of a loop the output needs fails the run.
- Put the loop node in `output`, or read it from a later node. After the run,
  a result whose `while.field` still equals `equals` means the loop gave up:
  the repeats ran out, or a node of the last pass failed (see `failed_nodes`).
  Any other value means the condition stopped holding. The pass count is the
  number of `<id>#it<k>` uids in `nodes`, plus one.
- Say in the `task` of each node of the loop how to use `loop_previous` (for
  example: fix every finding in it). A body node whose result is the work
  itself (a text or a JSON document, nothing left on disk) starts every pass
  from scratch unless its `task` says to build on `own_previous` (for example:
  keep own_previous and change only what the findings ask for).
- Every body node must feed the loop node, a body node must not read its loop
  node, a node belongs to at most one loop, and no node of a loop may declare
  `fanout` or be iterated by one.
- Under `max_concurrency`, free slots go to ready uids in node-id order, so an
  earlier node's pass can take a slot before a later node's first pass.

## `workflow::status`

`workflow::status { run_id, details? }` returns `run_id`, `status`, `nodes`
(uid → state), `node_errors`, `node_results`, `result`, `result_error`,
`created_at`, `updated_at` and `phases`.

- `nodes` also lists the nodes that have not started as `pending`. An
  unexpanded fanout appears once under its node id; a fanout over an empty
  array lists nothing. Each loop pass is listed under its own `<id>#it<k>` uid,
  in `nodes`, `node_errors`, `node_results` and `node_details`.
- `phases` counts node uids per `phase`: `{"code": {"total": 2, "done": 0,
  "failed": 0, "cancelled": 0, "running": 1}}`. `failed` includes the
  `cancelled` ones. Nodes without a
  phase are left out, and the map is omitted when empty. Each loop pass counts
  once, so `total` grows with every repeat.

With `details: true` it adds `node_details` for at most 100 uids (running
first, then pending, failed, cancelled, done) and `details_truncated: true`
when it cut the list:

```jsonc
"node_details": {"kan-12": {
  "state": "running", "label": "KAN-12 · x", "phase": "code", "model": "m",
  "profile": "deepwork-lead", "session_id": "wf_r_ab_kan-12@r0", "attempt": 0,
  "started_at": 1, "finished_at": 2, "elapsed_ms": 1,
  "harness_status": "completed", "expects_wake": true, "last_generate_at": 3,
  "usage": {"complete": true, "sessions": 3, "turns": 12, "function_calls": 40,
            "function_call_errors": 0, "input_tokens": 1, "output_tokens": 1,
            "cache_read_tokens": 1, "cache_write_tokens": 1,
            "reasoning_tokens": null, "cost_usd": 0.42}}}
```

`usage` comes from `harness::metrics` over the node session's tree, and `model`
and `last_generate_at` from its context snapshots. `harness_status` and
`expects_wake` come from `harness::status`, for running nodes only. A running
node waiting for its [transient retry](#transient-provider-failures) also has
`retry_at`, the epoch ms its continue is due; until then its `harness_status`
reads `failed`. A node whose
send never reached the harness gets no live fields. These live
reads run without the run lock, 16 at a time, with a 5 s timeout each. A failed
read leaves its fields out, and nothing live is stored. `model` of a node with
a `profile` is shown only once observed, because the profile may pin another
model. Callers compute idle time from `last_generate_at`.

`elapsed_ms` is `finished_at` minus `started_at`, or the time since
`started_at` while the node runs. It is the clock `timeout_ms` is checked
against, so harness queue wait, parked time and the 2 s settle poll are
included. It covers the current attempt: a timeout refire starts it again, and
`attempt` counts those; each earlier attempt ran the full `timeout_ms`.

## Limits

- **Parked nodes and stop.** A node whose session waits on an armed wake stays
  running until a later turn ends it, unless it already submitted its result.
  When this worker stops a node session (timeout, refire, `workflow::stop`, a
  run ending with nodes in flight, a submitted node settling with a wake armed),
  it disposes every observer in the session's tree after the stop, so a parked
  session ends cancelled instead of waking later. That stop also kills the
  session tree's background shell jobs and child sessions; for a submitted node
  the session reads cancelled in the harness while the node is done. This needs
  harness v1.9.4 or newer: an older harness ignores `quiet_deadlines`, so
  disposing a parked node's armed wake sends it a `[notification] wake removed`
  message and starts a new turn right away. A wake that delivers between the
  stop and the disposal starts one more turn, which a second stop cancels; wakes
  that turn arms before the second stop stay armed, so the session can still
  wake once more. If `harness::stop` fails, the observers stay and the session
  can still wake; nothing retries the stop or the disposal for a refired,
  timed-out or settled submitted node; only `workflow::stop` gets a second pass
  when the run finalizes. A node stopped from outside while it holds armed wakes
  settles only when its wakes retire or at its `timeout_ms`, unless it had
  submitted. A submitted one settles at once from its turn's status: done if the
  turn had completed, cancelled if the stop cancelled a running turn.
- **A finished node settles on a second poll.** A node whose `harness::status`
  reads finished is polled again 2 s later and settles from that reply, so a
  parked node does not settle on its parking turn. Each tick or sweep pass that
  settles a node takes about 2 s longer, holding the run lock and one of the 10
  tick-queue slots that all runs share. A wake whose turn starts more than 2 s
  after its claim, or whose dispatch fails after the claim, can still settle the
  parking turn. A message that reaches a finished node session within the 2 s
  makes the node settle on that later turn.
- **The timeout counts queue wait.** `timeout_ms` runs from the send, so it
  includes harness queue wait and parked time. The sweep checks it once a
  minute.
- **`retries` defaults to 1.** A node that times out is refired once unless it
  sets `retries: 0`, and a refire repeats the node's side effects.
- **A transient retry waits 1 to 2 minutes.** The continue goes out with the
  first tick or sweep pass at or after `retry_at`, and the default sweep runs
  once a minute. A stop of an idle caller does not cancel a node waiting for its
  continue; `workflow::stop` does. A stop of a busy caller is seen only while
  it is the caller's current turn: one followed by a new caller turn before the
  next pass (up to one sweep period) is missed. `read_only` is a promise the worker does not
  check: a node with side effects that sets it gets continued too, though the
  continue asks it not to repeat calls that already returned.
- **No ordering-only dependency.** A node runs after the nodes it names in
  `reads` or `fanout`, and receives their results, so in "do X, then
  validate" over shared state the validator reads X's result, which may be
  text.
- **Loops are bounded and quiet.** Each pass is a new session; continuity is
  the work it leaves behind plus `loop_previous` and, for a body node,
  `own_previous`. A loop ends without an error when its repeats run out, when
  the condition reads a missing field, or when a pass would push the run past
  50,000 materialized nodes. Put the loop node in `output`, or read it from a
  later node: a last result whose `while.field` still equals `equals` means
  the loop gave up (the repeats ran out, or a node of the last pass failed;
  see `failed_nodes`), any other value means the condition stopped holding,
  and the pass count is the number of `<id>#it<k>` uids in `nodes`, plus one.
  Say in the `task` of each node of the loop how to use `loop_previous` (for
  example: fix every finding in it), and in a body node that answers with the
  work itself, to build on `own_previous`. No node of a loop can fan out.
  Outside its loop a node reads only the loop node; `output` may also name a
  body node, which returns its last pass.
- **The run result is capped at 1 MiB.** An `output` whose combined value is
  larger fails the run, and `result_error` says so. The node results stay
  readable with `workflow::node-result`, so name fewer nodes in `output` when
  their results are large.
- **Usage covers the current attempt.** A refired node reports the usage of
  its newest `@r<n>` session tree only.
- **Stopping an idle caller.** Stopping an idle caller cancels the running
  nodes, but consumers of `on_error: continue` nodes still fire. Use
  `workflow::stop` to end the run.
- **`finished_at` after an upgrade.** Nodes that ended before the upgrade get
  the time of the run's next write as `finished_at`; a run that had already
  ended has none.
- **`reply_to` runs after an upgrade.** A run started with `reply_to` before
  the upgrade gets no delivery: its record still loads, but nothing posts its
  outcome and nothing wakes its caller. To recover, prompt the caller to
  observe the run with args `{ run_id }` (a run that already ended is replayed
  at once; one still running reports when it ends) or to read
  `workflow::status`, within 24 h of the run's end.
- **Forgotten observe.** An agent that never makes the `next` call is not
  woken; the outcome stays in `workflow::status` for 24 h. The worker logs a
  warning and counts `workflow.runs.ended_unobserved` for a run with a stamped
  caller, no `notify` and no observer of that run (an every-run observer does
  not count). A run that ends before its
  caller's observe lands counts too, so the counter is an upper bound on
  stalled callers. Do not subtract `workflow.runs.replayed`: it counts every
  replay (a second observer, or one re-listed after a restart). To find the
  runs really stalled, match the warning's `run_id` against the `replayed the
  run's end` log lines.

---

## Events

| Function | Args | Filter on | Event |
|---|---|---|---|
| `workflow::on-run-completed` | `{ run_id? }`: one run, replayed if it already ended; omitted: every run | `input.run_id`, `input.status` (no replay) | `RunCompletedEvent { run_id, status: completed \| failed \| cancelled, result, result_error, failed_nodes? }` |

The function is private (only this worker calls it; anyone may observe it) and
returns the event it is called with. The worker calls it once per terminal run,
at least once (a crash mid-finalize re-fires it), and only while an observer
wants that run; dedup on `run_id`. `failed_nodes` lists the
`node '<uid>': <error>` of each node a completed run tolerated failing
(`on_error: continue`), and is omitted when there are none.

An observer with args `{ run_id }` gets only that run. Adding it for a run that
already ended replays the event to it at once, through the observer: a call
target receives the bare event, not the `Observed` envelope of a live delivery,
so a worker with its own target must accept both (or use `notify`). An observer
accepted during the run's final tick can get both the live event and the
replay. Adding one for an unknown or collected run is refused with
`UNKNOWN_RUN`, and unknown args with `INVALID_ARGS`. `filters` get no replay:
to wait for one run, use args, never filters.

## Observers

| Observed function | Args / filters | Target | Target receives |
|---|---|---|---|
| `cron::on-tick` | args `{ expression: <sweep_expression> }` | `workflow::sweep` | the tick (`capture: [output]`, read as `payload.output`) |
| `harness::on-turn-completed` | filters `{ "input.session_id": { "$prefix": "wf_" } }` | `workflow::wake` | `TurnCompletedEvent` of a node session (`capture: [output]`, read as `payload.output`) |
| `harness::hook::on-pre-call` | args `{ functions: ["workflow::start"], on_error: "fail_closed", timeout_ms: 30000 }` | `workflow::stamp-reply` | the pre-call hook input; returns the hook decision. A hook error or timeout refuses the agent's `workflow::start` |
| `harness::hook::on-post-turn` | args `{ sessions: ["wf_*"], on_error: "fail_open", timeout_ms: 30000, retry_prompt: <answer-format nudge> }` | `workflow::validate-result` | the post-turn hook input; returns `deny` with a reason for a JSON node's answer that is not a JSON value matching its schema, unless a wake is armed. An error or timeout passes the turn, and reconcile still checks the result |
| `harness::hook::on-pre-generate` | args `{ on_error: "fail_open", inject_prompt: <workflow guidance> }`, only while `inject_guidance` is on | `workflow::inject-guidance` | not called: the harness joins `inject_prompt` into the system prompt |
| `configuration::on-updated` | filters on `input.id` and `input.event_type` (through `iii-config-client`) | `workflow::on-config-change` | the configuration event (`capture: [output]`, read as `payload.output`) |

A `sweep_expression` change observes the new expression first, then disposes
the old observer; if the cron worker refuses the new expression
(`INVALID_CRON_EXPRESSION`), the old observer and schedule stay.

---

## Scaling & concurrency

### Single-writer-per-run (the correctness model)

Every path that reads, mutates, and writes a run record — `workflow::tick`,
`reconcile`, the pending-timeout sweep, and `workflow::stop` — holds the
per-run `WorkflowLocks` guard before touching state.  This serialises writes
within one process and closes the read-modify-write race.

Idempotency for duplicate event deliveries is provided by deterministic ids:
the child session id (`wf_<run_id>_<node_uid>@r<attempt>`) and the
`workflow_node_result/<run_id>/<node_uid>` key are stable, so a re-delivered
tick that re-fires an already-running node is a no-op at the harness layer.
A tick persists each node's attempt session before sending it; a later tick
that finds the session reserved asks `harness::status` before sending again.
A `workflow::start` `idempotency_key` is checked and written under one fixed
lock (`"idem"`), and the sweep deletes the key with its run under that lock.

Node sessions carry the caller as `metadata.parent_session_id`, so they sit in
the caller's session tree: stopping the caller's session stops every running
node. When the caller's turn reads `cancelled`, the run ends `cancelled`, even
with `on_error: continue` nodes; an idle caller is not detected (see Limits).
`workflow::stop` stops one run's node sessions without touching the caller.

Node sessions run in the caller's folder: the pre-call hook stamps the caller
turn's `metadata.fs_scope.root` as `caller_fs_root` (replacing any value the
agent sent), and every node send carries it as
`options.metadata.fs_scope.root`, unless the node sets `filesystem_root`.
Without one, nodes get the harness default folder. A non-agent caller (CLI, worker) is not hooked, so its
`caller_fs_root` is used as sent; an empty one is refused. A node granted
`workflow::start` and `workflow::on-run-completed` (its `next` observe;
`workflow::*` is denied by default) passes its own folder to a nested run and
is woken when it ends. The hook runs only on
calls made from an agent turn, so the harness refuses `workflow::start` as an
observer binding target and `fp` refuses it as a pipe step; both need the
harness and `fp` releases that carry those refusals.

### One instance per namespace

A function id has one host per namespace: the engine rejects a second
registration of the same id with `FUNCTION_NAMESPACE_CONFLICT`. The worker
registers every `workflow::*` function, `workflow::on-run-completed` included,
so the first instance in a namespace hosts them all and the
per-run `WorkflowLocks` live in that one process. A second instance in the same
namespace gets every registration rejected while its observers (the sweep and
the harness hooks) still deliver to the first instance's functions, so each
sweep and hook runs twice. Run one workflow worker per namespace.

### Why not `state::compare-and-set`

`state::set` is an unconditional overwrite. `state::compare-and-set` exists,
but the worker does not adopt it for run records or idempotency keys:

- it compares the whole stored value, so every run-record write would have to
  carry the full previous record;
- an absent key compares equal to a stored `null`;
- it has no size guard;
- on Redis it retries a scope-wide `WATCH` up to 8 times, so concurrent writers
  to different runs in the same scope contend with each other.

Since the engine allows one instance per namespace (above), the in-process
locks give every run one writer, and `WorkflowRunRecord` carries no `version`
field.

## api reference

```json
{
  "functions": [
    {
      "description": "Deliver this workflow node's result: call it once with {\"result\": <the JSON value your node's output schema requires>}. An error names what to fix: call it again. Once it is accepted, end your turn with one short line, not the JSON. The node is finished once the turn ends: submit last. Only a workflow node's own session can call it.",
      "metadata": {},
      "name": "workflow-result::submit",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "caller_session_id": {
            "default": null,
            "description": "Stamped from the calling session by the workflow's pre-call hook; a supplied value is overwritten.",
            "type": [
              "string",
              "null"
            ]
          },
          "result": {
            "description": "The node's result: the JSON value its output schema requires."
          }
        },
        "required": [
          "result"
        ],
        "title": "SubmitRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "accepted": {
            "type": "boolean"
          },
          "node": {
            "description": "The node uid the result was stored for.",
            "type": "string"
          }
        },
        "required": [
          "accepted",
          "node"
        ],
        "title": "SubmitResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal pre_generate hook: appends workflow orchestration guidance to the agent system prompt. The target of the worker's harness::hook::on-pre-generate observer; not called directly.",
      "metadata": {},
      "name": "workflow::inject-guidance",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "GenerateContext": {
            "properties": {
              "system_prompt": {
                "default": "",
                "description": "The system prompt assembled so far (base + any prior hook's mutation).",
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "description": "The `pre_generate` hook envelope slice we read; the generation context is nested under `generate`.",
        "properties": {
          "generate": {
            "$ref": "#/definitions/GenerateContext"
          }
        },
        "title": "PreGenerateEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "PreGenerateMutations": {
            "description": "The harness applies `system_prompt` only when the key is present (`HookRunner`'s `parse_mutations`), so `None` serializes to an empty object: the safe no-op that preserves the harness's assembled prompt.",
            "properties": {
              "system_prompt": {
                "description": "Full replacement system prompt (base + appended guidance). The harness overwrites, it does not merge.",
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          }
        },
        "description": "Hook envelope returned to the harness: the mutations to apply to the generation.",
        "properties": {
          "mutations": {
            "$ref": "#/definitions/PreGenerateMutations"
          }
        },
        "required": [
          "mutations"
        ],
        "title": "PreGenerateResponse",
        "type": "object"
      }
    },
    {
      "description": "Fetch the stored JSON result of one node: `node_uid` is the node id, '{node_id}#{i}' for a fanout item, or '{node_id}#it{k}' for loop pass k ('#it0' is the first pass; the bare id of a node in a loop is its last pass; workflow::status lists the done passes under `node_results`). Returns {result: null} when that node is not done.",
      "metadata": {},
      "name": "workflow::node-result",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "node_uid": {
            "description": "The node id, `\"{node_id}#{i}\"` for a fanout item, or `\"{node_id}#it{k}\"` for loop pass k (`#it0` is the first pass); the bare id of a loop node or body node returns its last pass. `uid` is accepted as an alias.",
            "type": "string"
          },
          "run_id": {
            "description": "The `run_id` returned by `workflow::start`.",
            "type": "string"
          }
        },
        "required": [
          "node_uid",
          "run_id"
        ],
        "title": "NodeResultRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "result": {
            "description": "The node's stored JSON result, or null if the node has not completed (or produced no stored result). Always serialized — even when absent it is an explicit `{ \"result\": null }`, so callers never special-case a missing key."
          }
        },
        "title": "NodeResultResponse",
        "type": "object"
      }
    },
    {
      "description": "A workflow run reached a terminal state (completed, failed or cancelled); the event is {run_id, status, result, result_error, failed_nodes}. To wait for one run, observe it with args {run_id}: a run that already ended is replayed at once, and an unknown or collected run is refused with UNKNOWN_RUN. Without args it reports every run; `filters` on input.run_id or input.status narrow that but get no replay, so never wait for one run with filters. Delivery is at-least-once: dedup on run_id. A replay reaches a call target as the bare event, not the observed envelope. A notification ending in `…(truncated)`, or with an array ending in an `…N more entries omitted` string, is incomplete: call workflow::status once for the run. If the wake expires unfired, observe again the same way. Called by the workflow worker only.",
      "metadata": {},
      "name": "workflow::on-run-completed",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "RunStatus": {
            "enum": [
              "running",
              "awaiting_nodes",
              "completed",
              "failed",
              "cancelled"
            ],
            "type": "string"
          }
        },
        "description": "A run's terminal outcome: the `workflow::on-run-completed` event and the `notify` callback payload.",
        "properties": {
          "failed_nodes": {
            "description": "`node '<uid>': <error>` per node that failed in a completed run (a tolerated `on_error: continue` failure); omitted when none.",
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "result": {
            "default": null,
            "description": "The run's result; null unless it completed."
          },
          "result_error": {
            "default": null,
            "description": "Why the run failed; null otherwise.",
            "type": [
              "string",
              "null"
            ]
          },
          "run_id": {
            "type": "string"
          },
          "status": {
            "$ref": "#/definitions/RunStatus"
          }
        },
        "required": [
          "run_id",
          "status"
        ],
        "title": "RunCompletedEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "RunStatus": {
            "enum": [
              "running",
              "awaiting_nodes",
              "completed",
              "failed",
              "cancelled"
            ],
            "type": "string"
          }
        },
        "description": "A run's terminal outcome: the `workflow::on-run-completed` event and the `notify` callback payload.",
        "properties": {
          "failed_nodes": {
            "description": "`node '<uid>': <error>` per node that failed in a completed run (a tolerated `on_error: continue` failure); omitted when none.",
            "items": {
              "type": "string"
            },
            "type": "array"
          },
          "result": {
            "default": null,
            "description": "The run's result; null unless it completed."
          },
          "result_error": {
            "default": null,
            "description": "Why the run failed; null otherwise.",
            "type": [
              "string",
              "null"
            ]
          },
          "run_id": {
            "type": "string"
          },
          "status": {
            "$ref": "#/definitions/RunStatus"
          }
        },
        "required": [
          "run_id",
          "status"
        ],
        "title": "RunCompletedEvent",
        "type": "object"
      }
    },
    {
      "description": "Internal pre_call hook on workflow::start: stamps the caller's session and folder into the arguments. Always continues; not called directly.",
      "metadata": {},
      "name": "workflow::stamp-reply",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "StampCall": {
            "properties": {
              "arguments": {
                "default": null
              },
              "id": {
                "default": null,
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          }
        },
        "description": "The slice of the `pre_call` hook envelope we need (lenient: ignores the other envelope fields the harness sends).",
        "properties": {
          "call": {
            "anyOf": [
              {
                "$ref": "#/definitions/StampCall"
              },
              {
                "type": "null"
              }
            ]
          },
          "metadata": {
            "default": null,
            "description": "Caller turn's `options.metadata`; its `fs_scope.root` is the caller's folder, stamped as `caller_fs_root`."
          },
          "session_id": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          },
          "turn_id": {
            "default": null,
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "StampReplyEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "StampReplyMutations": {
            "properties": {
              "arguments": {
                "description": "The call's arguments, rewritten with the caller's stamped session and folder (any supplied `caller_fs_root` removed)."
              }
            },
            "required": [
              "arguments"
            ],
            "type": "object"
          }
        },
        "description": "Hook outcome returned to the harness: always `continue`, with rewritten `workflow::start` arguments when it stamps the caller's session or folder, or removes a supplied `caller_fs_root`. The harness's `parse_output` treats a missing/`\"continue\"` decision and absent mutations as \"continue unchanged\", so the no-op case is the same wire the old `null` reply produced — but the response schema stays typed (the registry publish gate refuses AnyValue response schemas).",
        "properties": {
          "decision": {
            "description": "Always `\"continue\"` — this hook never denies or holds a call.",
            "type": "string"
          },
          "mutations": {
            "anyOf": [
              {
                "$ref": "#/definitions/StampReplyMutations"
              },
              {
                "type": "null"
              }
            ]
          }
        },
        "required": [
          "decision"
        ],
        "title": "StampReplyResponse",
        "type": "object"
      }
    },
    {
      "description": "Run a multi-agent workflow and return {run_id, next} immediately. Declarative DAG: fan-out, barrier/join, durable, crash-resumable. Shape: {\"nodes\":{\"a\":{\"task\":\"…\"},\"b\":{\"task\":\"…\",\"reads\":\"a\"}}}; a node omits `model` to run on your model. To get the result make the `harness::observe` call that `next` gives and END YOUR TURN — it arrives as a `[notification]` when the run ends; workers pass `notify`. Never poll workflow::status in a loop.",
      "metadata": {},
      "name": "workflow::start",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "Fanout": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "items": true,
                "type": "array"
              }
            ],
            "description": "`\"<id>.<path to an array>\"`, or the array itself."
          },
          "LoopCondition": {
            "additionalProperties": false,
            "description": "`field` of the loop node's result equals `equals`.",
            "properties": {
              "equals": {
                "description": "The loop repeats while the field is this string; a missing field, a non-string value or a null result does not hold.",
                "type": "string"
              },
              "field": {
                "description": "Dotted path into the loop node's JSON result, e.g. `\"verdict\"`. The loop node's `schema` must list each segment in `required`.",
                "type": "string"
              }
            },
            "required": [
              "equals",
              "field"
            ],
            "type": "object"
          },
          "LoopSpec": {
            "additionalProperties": false,
            "description": "A bounded do-while loop on the loop node. Pass `k` of a node runs as uid `<id>#it<k>` (the first pass keeps `<id>`); every node of the loop gets `{\"input\": <its input>, \"loop_previous\": <the loop node's previous result>}`, and a body node also `\"own_previous\": <its own previous result>`, both null on the first pass: the same form as a fanout copy's `{\"input\": …, \"fanout_item\": …}`.",
            "properties": {
              "body": {
                "default": [],
                "description": "Other node ids that repeat with the loop node; the loop node must read each of them, directly or through other body nodes. No loop node may declare `fanout`.",
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "max_repeats": {
                "description": "Passes after the first, 1 to 9: `2` means at most 3 passes. The loop also ends quietly at the run's materialized-node cap.",
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "while": {
                "allOf": [
                  {
                    "$ref": "#/definitions/LoopCondition"
                  }
                ],
                "description": "The loop repeats while this condition holds on the loop node's JSON result."
              }
            },
            "required": [
              "max_repeats",
              "while"
            ],
            "type": "object"
          },
          "Node": {
            "description": "One sub-agent session. Its opening message is `task`, followed by its input as JSON, fenced as untrusted data.",
            "properties": {
              "fanout": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/Fanout"
                  },
                  {
                    "type": "null"
                  }
                ],
                "description": "One copy of this node runs per element of an array: `\"<id>.<path>\"` names an array in that node's JSON result (that node needs `schema`), or write the array itself. Without `reads` a copy receives its element; with `reads` it receives `{\"input\": <what reads delivers>, \"fanout_item\": <its element>}`. The node's result is the array of its copies' results, in element order. Not allowed on a node of a loop."
              },
              "filesystem_root": {
                "description": "Absolute folder the node's session runs in, in place of the caller's folder. It must lie inside the caller's folder (symlinks resolved), which is checked once, at start; a run without a caller folder cannot set it. Fanout copies and loop passes share it.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "functions": {
                "description": "Child dispatch policy: an allow-list array, a single string, or `{ \"allow\": [...], \"deny\": [...] }`; omit to inherit `default_functions`."
              },
              "label": {
                "description": "Display name, at most 80 characters: the node session's title (a fanout copy gets ` #<i>`) and the node's entry in `workflow::status` details.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "loop": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/LoopSpec"
                  },
                  {
                    "type": "null"
                  }
                ],
                "description": "Bounded do-while loop carried by this node (the loop node): this node and its `body` run again, as new sessions, while `while` holds on this node's result. Outside the loop a node can read only this node, and readers get its last pass; `output` may also name a body node, which returns its last pass (`null` when that pass failed). Put the loop node in `output`, or read it from a later node. After the run, a result whose `while.field` still equals `equals` means the loop gave up: the repeats ran out, or a node of the last pass failed (see `failed_nodes`). Any other value means the condition stopped holding. The pass count is the number of `<id>#it<k>` uids in `nodes`, plus one. Say in the `task` of each node of the loop how to use `loop_previous` (for example: fix every finding in it), and in a body node whose result is the work itself, to build on `own_previous`, its own result of the previous pass."
              },
              "model": {
                "description": "Omit to run this node on your own model. Otherwise a model id registered on this engine (discover them with `router::models::list`); an unregistered id is refused at start.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "on_error": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/OnError"
                  },
                  {
                    "type": "null"
                  }
                ],
                "description": "`\"fail\"` (default): a failed or cancelled node fails the run when the output needs it. `\"continue\"`: it counts as finished with a `null` result, so readers still run and the run can complete; the error stays in `node_errors`. Retries run first; a run abort still cancels everything."
              },
              "phase": {
                "description": "Group name, at most 80 characters; `workflow::status` counts the nodes of each phase (total, done, failed, running).",
                "type": [
                  "string",
                  "null"
                ]
              },
              "profile": {
                "description": "Agent profile id (`directory::agents::*`: lowercase letters, digits, `-` and `_`, at most 64 characters), sent as the harness `options.agent`. The profile supplies the system prompt, so `system_prompt` must be omitted, and a model it pins (also through `extends`) wins over `model`. The profile grants no permissions: `functions` still decides them.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "provider": {
                "description": "Provider override; defaults to the routed provider for the model. Only with `model` or `profile`.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "read_only": {
                "description": "`true`: the node has no side effects, so a turn that fails on a transient provider failure is continued once, in the same session, a minute later, within what is left of `timeout_ms`. The worker neither checks nor enforces it (`functions` still decides what the node may call). Default `false`: such a turn fails the node.",
                "type": [
                  "boolean",
                  "null"
                ]
              },
              "reads": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/NodeRefs"
                  },
                  {
                    "type": "null"
                  }
                ],
                "description": "The node(s) whose result this node receives; it runs after them. One id delivers that result as is (a text node arrives as a JSON string, a fanout node as the array of its copies' results); an array of ids delivers an object keyed by id. Omit it to receive the run `input`. On a `fanout` node the value arrives under `input`."
              },
              "retries": {
                "description": "Refires after a timeout, each in a new session; a failed turn is not refired (see `read_only`). At most 10; the default refires once, and a refire repeats the node's side effects. `0` fails the node at its first timeout.",
                "format": "uint32",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "schema": {
                "additionalProperties": true,
                "description": "JSON Schema, with a top-level `type`, of the JSON value this node delivers; declare every field read downstream. Omit it for a text node.",
                "type": [
                  "object",
                  "null"
                ]
              },
              "system_prompt": {
                "description": "System prompt appended to the child's built-in identity prompt.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "task": {
                "description": "What this node does: the text of its opening message, where the instructions go.",
                "type": "string"
              },
              "timeout_ms": {
                "description": "Milliseconds an attempt may run, wall clock from its send (harness queue wait and parked time included; checked once a minute). At least 1000; defaults to the worker's `default_pending_timeout_ms`. Fanout copies use the node's value.",
                "format": "uint64",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              }
            },
            "required": [
              "task"
            ],
            "type": "object"
          },
          "NodeRefs": {
            "anyOf": [
              {
                "type": "string"
              },
              {
                "items": {
                  "type": "string"
                },
                "type": "array"
              }
            ],
            "description": "A node id, or an array of node ids."
          },
          "NotifySpec": {
            "additionalProperties": false,
            "description": "Completion callback: `function_id` is called once on terminal state with `{run_id, status, result, result_error, failed_nodes}`; at-least-once, dedup on `run_id`.",
            "properties": {
              "function_id": {
                "description": "Function id to call on terminal state, e.g. `\"myworker::wf-done\"`.",
                "type": "string"
              },
              "queue": {
                "description": "Queue for durable delivery; defaults to the worker's `\"workflow-notify\"` queue.",
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "required": [
              "function_id"
            ],
            "type": "object"
          },
          "OnError": {
            "description": "What a node's failure does to the rest of the run.",
            "oneOf": [
              {
                "description": "A failed or cancelled node fails the run when the output needs it.",
                "enum": [
                  "fail"
                ],
                "type": "string"
              },
              {
                "description": "A failed or cancelled node counts as finished with a `null` result: its consumers still fire, and a fan-in keeps the `null` at its position.",
                "enum": [
                  "continue"
                ],
                "type": "string"
              }
            ]
          }
        },
        "description": "Run a DAG of sub-agent sessions, durable and crash-resumable. Example: `{\"nodes\":{\"a\":{\"task\":\"<what a does>\"},\"b\":{\"task\":\"<what b does>\",\"reads\":\"a\"}},\"input\":<any JSON>}`.",
        "properties": {
          "caller_fs_root": {
            "description": "Caller's folder. On a call from an agent turn the `workflow::stamp-reply` hook replaces it with the turn's `metadata.fs_scope.root`; the harness refuses `workflow::start` as a binding target and `fp` as a pipe step, since both skip that hook. Every node session runs in it. Omit it to use the harness default folder.",
            "type": [
              "string",
              "null"
            ]
          },
          "caller_session_id": {
            "description": "Orchestrator session, stamped by the `workflow::stamp-reply` hook from the caller's turn; never trusted from the agent.",
            "type": [
              "string",
              "null"
            ]
          },
          "default_functions": {
            "description": "Dispatch policy inherited by every node that omits `functions` (same shape); when unset those nodes get everything except `workflow::*`, `configuration::*`, `approval::*`, `harness::hook::*` and `harness::send`."
          },
          "idempotency_key": {
            "description": "Dedupe key; a repeated key returns the original `run_id` instead of launching a duplicate run.",
            "type": [
              "string",
              "null"
            ]
          },
          "input": {
            "default": null,
            "description": "Any JSON value. Every node without `reads` or `fanout` receives it."
          },
          "max_concurrency": {
            "description": "At most this many node sessions live at once (running, or reserved for a send); further ready nodes wait for a slot. At least 1; unset means no cap.",
            "format": "uint32",
            "minimum": 0,
            "type": [
              "integer",
              "null"
            ]
          },
          "nodes": {
            "additionalProperties": {
              "$ref": "#/definitions/Node"
            },
            "description": "One sub-agent session per node, keyed by node id; an id must not contain `#`, `.` or `/`.",
            "type": "object"
          },
          "notify": {
            "anyOf": [
              {
                "$ref": "#/definitions/NotifySpec"
              },
              {
                "type": "null"
              }
            ],
            "description": "Completion callback called once when the run reaches a terminal state, instead of polling `workflow::status`."
          },
          "output": {
            "anyOf": [
              {
                "$ref": "#/definitions/NodeRefs"
              },
              {
                "type": "null"
              }
            ],
            "description": "What the run returns: a node id returns that node's result as is, an array of ids an object keyed by id. A node of a loop, the loop node or a body node, returns its last pass. Omit it to return every node no other node reads (one such node: its result as is)."
          }
        },
        "required": [
          "nodes"
        ],
        "title": "StartRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "next": {
            "description": "For an agent caller: the `harness::observe` call that delivers the run's outcome to this session. Make it exactly as given, then end the turn."
          },
          "run_id": {
            "type": "string"
          }
        },
        "required": [
          "run_id"
        ],
        "title": "StartResponse",
        "type": "object"
      }
    },
    {
      "description": "Return a snapshot of a workflow run (status, per-node state with unstarted nodes as pending, node_errors, node_results, phases, result, result_error), or null if not found. `details: true` adds node_details: label, phase, model, session, timings, live usage and harness status per node (at most 100 nodes). Each call costs a turn: do not poll in a loop; observe `workflow::on-run-completed` with args `{run_id}`, or pass `notify` to workflow::start, to be pushed the outcome.",
      "metadata": {},
      "name": "workflow::status",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "Read a single snapshot of a workflow run's status.",
        "properties": {
          "details": {
            "default": false,
            "description": "Also return `node_details` (at most 100 nodes: running first, then pending, failed, cancelled, done). Reads each node's usage and harness status live.",
            "type": "boolean"
          },
          "run_id": {
            "description": "The `run_id` returned by `workflow::start`.",
            "type": "string"
          }
        },
        "required": [
          "run_id"
        ],
        "title": "StatusRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "anyOf": [
          {
            "$ref": "#/definitions/StatusResponse"
          },
          {
            "type": "null"
          }
        ],
        "definitions": {
          "NodeDetail": {
            "description": "One node in `node_details`. Definition fields come from the run's definition; `harness_status`, `expects_wake`, `last_generate_at`, `usage` and an observed `model` are read live and omitted when that read fails.",
            "properties": {
              "attempt": {
                "description": "0 for the first attempt; each timeout refire adds one.",
                "format": "uint32",
                "minimum": 0,
                "type": [
                  "integer",
                  "null"
                ]
              },
              "elapsed_ms": {
                "description": "Ms the current attempt has run: `finished_at`, or now while it runs, minus `started_at`.",
                "format": "int64",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "expects_wake": {
                "description": "The session waits on an armed wake; running nodes only.",
                "type": [
                  "boolean",
                  "null"
                ]
              },
              "finished_at": {
                "description": "Epoch ms the node ended.",
                "format": "int64",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "harness_status": {
                "description": "`harness::status` turn status; running nodes only.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "label": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "last_generate_at": {
                "description": "Epoch ms of the last model request anywhere in the node's session tree.",
                "format": "int64",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "model": {
                "description": "The model the node session last ran; for a node without a profile, the definition's model until one is observed.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "phase": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "profile": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "retry_at": {
                "description": "Epoch ms the continue of the node's transient provider failure is due; while set, `harness_status` reads the failed turn.",
                "format": "int64",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "session_id": {
                "description": "The current attempt's session.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "started_at": {
                "description": "Epoch ms of the current attempt's send.",
                "format": "int64",
                "type": [
                  "integer",
                  "null"
                ]
              },
              "state": {
                "$ref": "#/definitions/NodeState"
              },
              "usage": {
                "description": "`harness::metrics` totals of the current attempt's session tree, plus `complete`."
              }
            },
            "required": [
              "state"
            ],
            "type": "object"
          },
          "NodeState": {
            "enum": [
              "pending",
              "running",
              "done",
              "failed",
              "cancelled"
            ],
            "type": "string"
          },
          "PhaseCounts": {
            "properties": {
              "cancelled": {
                "description": "The cancelled ones among `failed`.",
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "done": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "failed": {
                "description": "Failed and cancelled nodes.",
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "running": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              },
              "total": {
                "format": "uint32",
                "minimum": 0,
                "type": "integer"
              }
            },
            "required": [
              "cancelled",
              "done",
              "failed",
              "running",
              "total"
            ],
            "type": "object"
          },
          "RunStatus": {
            "enum": [
              "running",
              "awaiting_nodes",
              "completed",
              "failed",
              "cancelled"
            ],
            "type": "string"
          },
          "StatusResponse": {
            "properties": {
              "created_at": {
                "format": "int64",
                "type": "integer"
              },
              "details_truncated": {
                "description": "`node_details` was cut at 100 nodes.",
                "type": "boolean"
              },
              "node_details": {
                "additionalProperties": {
                  "$ref": "#/definitions/NodeDetail"
                },
                "description": "Only with `details: true`: per-node detail keyed by node uid.",
                "type": "object"
              },
              "node_errors": {
                "additionalProperties": {
                  "type": "string"
                },
                "description": "Errors for nodes that failed, keyed by node uid — e.g. \"no provider registered for model …\". Present so a caller can diagnose a `failed` run without digging into stored state. Omitted when no node carries an error.",
                "type": "object"
              },
              "node_results": {
                "additionalProperties": {
                  "type": "string"
                },
                "description": "Nodes that have a stored result: node uid → result_ref (its key in state). Fetch the value with `workflow::node-result { run_id, node_uid }`. Lets a caller recover partial work from a run that failed partway (the run-level `result` is only set on a Completed run). Omitted when empty.",
                "type": "object"
              },
              "nodes": {
                "additionalProperties": {
                  "$ref": "#/definitions/NodeState"
                },
                "description": "Per-node state keyed by node uid. Definition nodes that have not started are `pending`; an unexpanded fanout is listed once under its node id.",
                "type": "object"
              },
              "phases": {
                "additionalProperties": {
                  "$ref": "#/definitions/PhaseCounts"
                },
                "description": "Node uid counts per definition `phase` (`failed` includes `cancelled`); nodes without a phase are left out. Omitted when empty.",
                "type": "object"
              },
              "result": true,
              "result_error": {
                "description": "Run-level failure summary (set when `status == failed`).",
                "type": [
                  "string",
                  "null"
                ]
              },
              "run_id": {
                "type": "string"
              },
              "status": {
                "$ref": "#/definitions/RunStatus"
              },
              "updated_at": {
                "format": "int64",
                "type": "integer"
              }
            },
            "required": [
              "created_at",
              "nodes",
              "run_id",
              "status",
              "updated_at"
            ],
            "type": "object"
          }
        },
        "title": "Nullable_StatusResponse"
      }
    },
    {
      "description": "Cancel a workflow run, cascading harness::stop to each live node session.",
      "metadata": {},
      "name": "workflow::stop",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "run_id": {
            "type": "string"
          }
        },
        "required": [
          "run_id"
        ],
        "title": "StopRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "stopping": {
            "type": "boolean"
          }
        },
        "required": [
          "stopping"
        ],
        "title": "StopResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal cron sweep: reconcile AwaitingNodes runs and time out nodes past their deadline. Not called directly.",
      "metadata": {},
      "name": "workflow::sweep",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "The `cron::on-tick` event, delivered as output. Informational only: the sweep scans every run record whatever the tick says.",
        "properties": {
          "expression": {
            "default": null,
            "description": "The expression that fired.",
            "type": [
              "string",
              "null"
            ]
          },
          "scheduled_time": {
            "default": null,
            "description": "RFC 3339, UTC.",
            "type": [
              "string",
              "null"
            ]
          }
        },
        "title": "SweepEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "ok": {
            "type": "boolean"
          },
          "swept": {
            "description": "Number of runs touched (reconciled and/or timed out) this sweep.",
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          }
        },
        "required": [
          "ok",
          "swept"
        ],
        "title": "SweepResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal durable workflow step.",
      "metadata": {},
      "name": "workflow::tick",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "run_id": {
            "type": "string"
          },
          "step": {
            "default": 0,
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "wake": {
            "default": false,
            "description": "Sent by `workflow::wake`: runs even below the step floor.",
            "type": "boolean"
          }
        },
        "required": [
          "run_id"
        ],
        "title": "TickRequest",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "skipped": {
            "type": "boolean"
          }
        },
        "required": [
          "skipped"
        ],
        "title": "TickResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal post_turn hook on workflow node sessions: a JSON node's turn that delivered no result with workflow-result::submit (and whose final message is not a valid result either) is denied, so the harness re-prompts it; a turn that ends with a wake armed passes. Not called directly.",
      "metadata": {},
      "name": "workflow::validate-result",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "The slice of the post_turn hook input this hook reads.",
        "properties": {
          "result": {
            "default": null,
            "description": "The turn's result: the final message's text under a text contract."
          },
          "session_id": {
            "default": "",
            "type": "string"
          }
        },
        "title": "PostTurnEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "description": "The hook decision returned to the harness.",
        "properties": {
          "decision": {
            "description": "`continue`, or `deny` to re-prompt the turn.",
            "type": "string"
          },
          "reason": {
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "decision"
        ],
        "title": "ValidateResultResponse",
        "type": "object"
      }
    },
    {
      "description": "Internal: on a harness::on-turn-completed event, wake the owning run with a tick. Best-effort latency optimization; the cron sweep is the durable backstop. Not called directly.",
      "metadata": {},
      "name": "workflow::wake",
      "request_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "definitions": {
          "Caller": {
            "description": "Who made an invocation. `worker_id` / `worker_name` are stamped by the engine from the calling connection; `function_id`, `invocation_id` and `parent` are asserted by the calling SDK.",
            "properties": {
              "function_id": {
                "description": "The handler the call was made from; absent for a top-level call.",
                "type": [
                  "string",
                  "null"
                ]
              },
              "invocation_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "parent": {
                "anyOf": [
                  {
                    "$ref": "#/definitions/ParentCaller"
                  },
                  {
                    "type": "null"
                  }
                ],
                "description": "The caller of that handler, one level only."
              },
              "worker_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "worker_name": {
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          },
          "InvocationStatus": {
            "enum": [
              "success",
              "error"
            ],
            "type": "string"
          },
          "ObservedError": {
            "description": "What a call failed with: [`Observed::error`], [`Call::error`](crate::Call::error).",
            "properties": {
              "code": {
                "type": "string"
              },
              "message": {
                "type": "string"
              }
            },
            "required": [
              "code",
              "message"
            ],
            "type": "object"
          },
          "ParentCaller": {
            "description": "[`Caller::parent`]: a caller without a parent of its own.",
            "properties": {
              "function_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "invocation_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "worker_id": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "worker_name": {
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "type": "object"
          },
          "WakeEvent": {
            "description": "The `harness::on-turn-completed` event (delivered as output); only `session_id` is read.",
            "properties": {
              "session_id": {
                "default": "",
                "type": "string"
              }
            },
            "type": "object"
          }
        },
        "description": "What an observer's target receives when iii delivers: the observed call, per the observer's `capture`.\n\n```rust,no_run # use iii_sdk::{Error, IIIClient, Observed, RegisterFunction}; # use serde_json::Value; # fn demo(iii: &IIIClient) { iii.register_function( \"alerts::payment-failed\", RegisterFunction::new(|event: Observed<Value>| { println!(\"{} failed: {:?}\", event.function_id, event.error); Ok::<_, Error>(()) }), ); # } ```",
        "properties": {
          "activation_id": {
            "type": "string"
          },
          "caller": {
            "anyOf": [
              {
                "$ref": "#/definitions/Caller"
              },
              {
                "type": "null"
              }
            ]
          },
          "duration_ms": {
            "format": "double",
            "type": "number"
          },
          "error": {
            "anyOf": [
              {
                "$ref": "#/definitions/ObservedError"
              },
              {
                "type": "null"
              }
            ]
          },
          "function_id": {
            "type": "string"
          },
          "input": true,
          "invocation_id": {
            "type": "string"
          },
          "metadata": true,
          "namespace": {
            "type": [
              "string",
              "null"
            ]
          },
          "observer_id": {
            "type": "string"
          },
          "output": {
            "anyOf": [
              {
                "$ref": "#/definitions/WakeEvent"
              },
              {
                "type": "null"
              }
            ]
          },
          "started_at_ms": {
            "format": "uint64",
            "minimum": 0,
            "type": "integer"
          },
          "status": {
            "$ref": "#/definitions/InvocationStatus"
          }
        },
        "required": [
          "activation_id",
          "duration_ms",
          "function_id",
          "invocation_id",
          "observer_id",
          "started_at_ms",
          "status"
        ],
        "title": "Observed_for_AnyValue_and_WakeEvent",
        "type": "object"
      },
      "response_schema": {
        "$schema": "http://json-schema.org/draft-07/schema#",
        "properties": {
          "woke": {
            "type": "boolean"
          }
        },
        "required": [
          "woke"
        ],
        "title": "WakeResponse",
        "type": "object"
      }
    }
  ]
}
```
