Skip to content
Sign in

llm-router

v1.5.5

One front door + provider protocol in front of every LLM provider.

license: Apache-2.0Used in 1 kitllmmodelsprovidersrouter

Install

bash
iii trigger compose::add worker=llm-router

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

llm-router

One front door for every LLM provider. The router owns routing, the provider registry, credential resolution, the model catalog, streaming relay, retries, and a single failure contract — consumers call one chat surface and never talk to a provider directly.

llm-router is a standalone iii worker. Providers plug in as separate workers at runtime through a self-registration protocol (iii call compose::add worker=provider-<x>); the router never compiles against a provider, and removing a provider worker removes the provider.

Install

iii call compose::add worker=llm-router

Quickstart

A consumer streams a turn by creating an iii channel, handing the router the channel's write endpoint, and reading frames from the read endpoint while router::chat runs. Any SDK works; Node shown:

import { createChannel } from 'iii-sdk/helpers';

const { reader, writerRef } = await createChannel(iii);
reader.onMessage((frame) => {
  const event = JSON.parse(frame); // AssistantMessageEvent
  if (event.type === 'text_delta') process.stdout.write(event.delta);
});

const res = await iii.call({
  function_id: 'router::chat',
  payload: {
    writer_ref: writerRef, // direction "write"
    model: 'claude-sonnet-4',
    messages: [{ role: 'user', content: [{ type: 'text', text: 'Hello' }], timestamp: Date.now() }],
  },
  timeoutMs: 620_000, // outer timeout ≥ the router's 600s stream budget
});
// res: { ok, provider, model, stop_reason, usage }

The streaming contract: every stream ends with exactly one terminal frame (done or error). When the router has to kill a stream itself (idle timeout, provider crash), it synthesizes the terminal frame and attaches the partial content, so consumers never hang on a half-open stream.

Both router::chat and router::complete accept an optional retry_max: the maximum number of retries after the first provider attempt. Omit it to keep the router's configured retry behavior, or pass 0 when the caller must authorize and budget each attempt itself. A caller can only reduce the configured limit: the effective ceiling is the smaller of the two, fixed for the turn. This bounds router dispatches; it does not change a provider's own retry policy or make missing usage a billable cost of zero. If a caller may need to abort, it must send its own unique request_id with the original request: an omitted id is generated internally and is not returned in the response. Caller timeouts still require explicit router::abort with that same caller-supplied request_id.

Functions

Consumer surface

Function Purpose
router::chat Stream a turn into the caller's channel; returns the turn summary.
router::complete Non-streaming convenience over the same pipeline; returns the final message.
router::abort Cancel an in-flight turn by request_id.
router::route Read-only routing preview: {model, provider?} → {provider, candidates}, same rules and error codes as router::chat. Pin the result as the explicit provider on the chat call when you need the provider before streaming.
router::models::list List catalog models, filterable by provider / capability / modality (chat by default, stt, tts, any).
router::models::get Fetch one model record (null when unknown).
router::models::supports Check one capability flag for one model.
router::embed Batch text embeddings through the first embed-capable provider in the registry, then embed-only workers ({model?, provider?, input[]} → {provider, model, embeddings[][]}); providers without an embed surface are skipped.
router::transcribe Speech to text: {model?, provider?, audio_base64, mime?, language?, prompt?} → {provider, model, text, segments[]?, language?, duration_secs?}. The provider comes from the named provider, else from the catalog owner of the stt model, else the first provider that declared one.
router::speak Text to speech: {model?, provider?, text, voice?, format?, language?, speed?} → {provider, model, audio_base64, mime, voice?, duration_secs?}. Same resolution over tts models.
router::count_tokens Count prompt tokens: {model, provider?, system_prompt?, tools?, messages, thinking_level?, response_format?} → {provider, model, tokens, estimator}, resolved with the same routing rules as router::chat and forwarded to provider::<id>::count_tokens. Never runs the model and costs nothing; estimator is provider (metering API) or tiktoken (local tokenizer). A provider without the surface is a typed router/no_token_counter error, so callers can fall back to their own estimate.
router::provider::list Registered providers with configured / available status, where each credential comes from (credential_source, credential_ref, credential_error — never a value), and each provider's starting point: default_model and default_thinking_level (see Default model).

Only the read surface is agent-callable (router::models::list / get / supports, router::provider::list); everything else is denied to in-run agents — see Security model.

Error conventions

Errors use stable router/<code> identifiers. Streaming failures return an error object with a user-facing message, the shared kind, retryable, and an optional diagnostic detail. User interfaces should show message and keep code / detail behind technical details; automation should branch on code and retryable, never parse the prose.

The main recovery codes are router/provider_unavailable, router/capacity_exceeded, router/stream_idle_timeout, router/stream_incomplete, router/provider_auth_expired, router/provider_rate_limited, router/context_overflow, and router/provider_rejected.

Provider protocol

Token-gated after the first declare: the response to register carries a registration token, and every later protocol call must present it.

Function Purpose
router::provider::register Self-declaration at attach time; idempotent re-declare with the token.
router::provider::resolve Per-request credential + endpoint resolution (config > secret:// / env:// reference > env > none, see Credentials).
router::provider::update_credential Persist a refreshed credential (OAuth write-back).
router::models::reconcile Replace the provider's catalog slice in one write.

These four are registered with observable: false: their payloads carry the registration token (every one) or a credential (resolve's output, update_credential's input), so an observe of any of them fails with NOT_OBSERVABLE.

The provider worker itself exposes provider::<id>::stream and, when capable, provider::<id>::refresh_models (model discovery), provider::<id>::count_tokens (prompt token counting), provider::<id>::embed (embeddings), and the speech pair provider::<id>::transcribe / provider::<id>::speak.

A worker that serves only embeddings registers provider::<id>::embed and never calls router::provider::register, so it stays out of router::provider::list and out of every model picker; iii-directory's local MiniLM (minilm) is one. A router::embed call that names no provider tries these embed-only surfaces last, alphabetically, found through engine::functions::list in the router's namespace, and only when every registered provider either has no embed surface (Anthropic only, say) or is not configured for good (provider-openai with no API key). Configuring that provider later moves callers to its vector space: memory re-embeds into the model the reply reports. One rule blocks the fallback with router/provider_unavailable, since another embedder would put the caller's vectors in another space, so the caller retries later: a registered provider whose embed has answered before (its registry record keeps an embeds flag) is not found and was live in the last 15 minutes (its registration, its last answered embed, or the first 3 minutes after the router starts), whatever else of it is live (restarting, partly re-registered, or the router just restarted). After that window it no longer blocks: the registry keeps every record, so a removed provider must not hold the fallback for good.

router::chat accepts an optional structured prompt next to the flat system_prompt: system_sections (ordered {text, cache_boundary} blocks) and cache_intent ({surface_digest}, the caller's sha256: identity for the text before the boundary). A sections-only request is flattened into system_prompt (sections joined with \n\n); when both are sent they must agree byte for byte or the request is rejected as invalid_request. Both are forwarded verbatim to provider::<id>::stream alongside the flat string, so a provider that knows nothing about sections keeps reading system_prompt. Cache-aware providers put their prefix marker on the boundary block (Anthropic, Claude Code) or derive a shared prompt_cache_key from the digest (OpenAI, Codex) so every session on the same frozen prefix lands on one cache entry. The digest is a routing hint, never evidence of a hit — read usage.cache_read for that.

Configuration

All operator configuration lives in the engine's llm-router configuration entry — no env vars, no config file. The entry schema is composed at runtime from each registered provider's declaration. The router fetches the complete entry at boot and keeps an in-memory snapshot synchronized by its observer of configuration::on-updated (the entry's configuration:updated events); request handlers never poll configuration::get.

{
  "default_provider": "anthropic",
  "providers": {
    "anthropic": {
      "api_key": "secret://ANTHROPIC_API_KEY",
      "api_url": "https://api.anthropic.com/v1/messages",
      "max_tokens": 8192
    }
  },
  "routing_heuristics": [{ "pattern": "^gpt-", "provider": "openai" }],
  "settings": {
    "stream_timeout_ms": 600000,
    "idle_timeout_ms": 120000,
    "retry_max": 2,
    "output_token_max": 32000
  }
}
Setting Default Meaning
stream_timeout_ms 600000 Hard budget for one streamed turn.
idle_timeout_ms 120000 Max silence between provider frames before the attempt is cut.
retry_max 2 Retries per turn for retryable failures before the first forwarded frame (0–10).
output_token_max unset Optional ceiling on max_output_tokens. Unset forwards each model's own output ceiling, so providers run without an artificial limit; set it only to cap spend.

Pasting a key into a provider's slice is the whole onboarding flow: the router diffs the changed slice, debounces ~2 s, and kicks that provider's provider::<id>::refresh_models discovery; discovered models land in the catalog via router::models::reconcile and show up in router::models::list within seconds — no restart.

Credentials and secret references

The configuration service's default filesystem adapter persists this entry at ./config/<entry-id>.yaml (llm-router unless III_CONFIG_NAME overrides it), a folder meant to be committed, so keep keys out of it: keep the key in the secrets worker, which llm-router depends on, and put a versionable reference in the slice:

  • "api_key": "secret://ANTHROPIC_API_KEY" — encrypted in the secrets worker's vault;
  • "api_key": "env://ANTHROPIC_API_KEY" — the variable in the project's env file (.env, or the secrets worker's env_file for this namespace; else the secrets worker's environment), read by the secrets worker each time, so an edit to the file applies without restarting the router.

Provider keys reuse their env var name; NAME matches ^[A-Za-z_][A-Za-z0-9_.-]{0,127}$. Either way the secret's consumers must list llm-router; the Console's key field does this for you.

A provider's credential is the first of:

  1. the slice's credential object (OAuth write-back via update_credential);
  2. a literal api_key;
  3. an api_key reference (secret:// or env://), resolved through secrets::resolve;
  4. the provider's credential_env_var, read in the router's process;
  5. none.

A reference is never forwarded to a provider as a key. When it does not resolve, the provider is not configured, and the error says why: secret NAME not found in the secrets worker, environment variable NAME is not set in the project's .env or the secrets worker's environment, llm-router is not allowed to read secret NAME; add llm-router to the secret's consumers, or secrets worker is not running. The env var is deliberately not used as a fallback past an explicit reference: you chose the stored secret, so a missing one is reported rather than silently replaced by a different key. Providers built on the shared scaffold skip their own env fallback in that case too, and router::chat fails before dispatch with router/not_configured and the same reason.

router::provider::list entries and the router::provider::resolve response carry credential_source (config | env | secret | none), credential_ref (the secret:// or env:// reference, when the slice uses one; credential_source is secret for both, as the secrets worker resolved it) and credential_error. All three are optional on the wire. The older source field keeps its config | env | none values: a resolved reference reports config there.

Resolved values live only in the router's memory: never in the entry, logs or traces, and a cached value is wiped when replaced. They are re-read:

  • on secrets::on-changed (the router observes it at boot; without a secrets worker the observer waits pending until the function registers). A rotated, deleted or revoked secret then nudges every provider whose slice references it through provider::<id>::on_router_ready, so the provider drops its cached resolve, re-declares and refreshes its models;
  • when a slice that references it changes, before the debounced refresh_models fires, so a pasted reference is resolvable by the time discovery asks for it;
  • when the secrets worker registers after the router, so a router that booted first recovers without a restart;
  • on demand after 5 minutes (10 seconds for a failure), bounding staleness if an event is ever missed. While the secrets worker is unreachable, a value already resolved keeps being served: secrets cannot rotate or be revoked while it is down.

The reference itself is not secret: it is committed, listed and logged. Never paste a key after secret:// or env://.

Operational notes

  • Secret references resolve as llm-router. The secrets worker checks the router's engine-stamped worker identity against the secret's consumers, not the provider's.
  • Env-var credential fallback resolves in the router's process. Without a reference, a provider's credential_env_var (e.g. ANTHROPIC_API_KEY) is read by the llm-router binary, not by the provider worker — launch the router with those variables set, or put a reference in the entry. A key present only in another worker's environment shows up as configured: false. env://NAME is different: the secrets worker reads it from the project's .env, so the router needs no env_file for it.
  • Registration-token recovery. Re-registering a provider id without its original token is rejected (anti-takeover). If a provider durably lost its token, delete the router's registry state (state scope llm-router, key registry) and restart the affected providers to re-bind; pasted credentials in the configuration entry are unaffected.

Security model

The agent-callable surface is governed by the engine-wide permission policy (repo-root iii-permissions.yaml), not a per-worker file. In-run agents may only read the catalog and provider list:

  • Allowed: router::models::list, router::models::get, router::models::supports, router::provider::list.
  • Denied to agents: the chat/spend surface (router::chat, router::complete, router::abort, router::route), the whole provider/credential protocol (router::provider::resolve / register / update_credential, router::models::reconcile), and direct provider::* calls.

Worker-to-worker calls bypass the agent gate, so the harness, context-manager, and provider workers reach the full surface — only in-run agents are restricted. Provider credentials live in the configuration entry, or behind a secret:// or env:// reference in the secrets worker, and are never readable back through any allowed function: router::provider::list reports where a credential comes from, never its value. The provider protocol functions are not observable, so registration tokens and credentials reach no observer (see Provider protocol).

Events

The router hosts three event functions. Each is private (only the router calls it) and returns the event it is called with, so its observers observe the event; none takes args, so observers select events with filters on input.

Function Called when Event Filter on
router::models::on-changed a provider reconciles its catalog slice { "provider": "<id>", "count": <n> } input.provider
router::provider::on-changed the registry changes: a declaration, a discovery result or an availability flip { "provider": "<id>", "op": "register" | "discovery" | "available" | "unavailable" } input.provider, input.op
router::on-ready the router is ready: each observer receives the event once the router has booted and the observer is accepted, once per router start and once more when its worker reconnects within the observer grace period {} —

Delivery is iii's: every call reaches each observer whose filters match, and the router calls a function only while it is observed. The ready event is a call for one observer: the router calls router::on-ready with metadata { "for_observer": "<observer id>" }, and only that observer matches it, so its target receives the Observed envelope like any other event. Observe with capture: ['output'] so the envelope carries the event at output:

iii.registerFunction('my-worker::on-models-changed', async ({ output: event }) => {
  console.log('catalog changed:', event); // { provider, count }
  return {};
});

iii.observe(
  {
    function_id: 'router::models::on-changed',
    filters: { 'input.provider': { $eq: 'anthropic' } },
    capture: ['output'],
  },
  'my-worker::on-models-changed',
);

The router itself observes configuration::on-updated (its entry, router::on_config_changed), engine::on-functions-changed (router::on_functions_changed) and, when the optional secrets worker is installed, secrets::on-changed (router::on_secret_changed). The last observer waits pending until the secrets worker registers its function.

Writing a provider worker

A provider worker must:

  1. Register provider::<id>::stream honouring the channel-writer contract: forward upstream output as AssistantMessageEvent frames into the writer_ref it receives, ending with one terminal frame.
  2. Declare itself at startup via router::provider::register — retrying with backoff until acknowledged (covers provider-before-router boot order) — and re-declare on every router::on-ready event. Observe it once at boot (Rust providers use provider_scaffold::router_client::observe_router_ready with their provider::<id>::on_router_ready handler as the target); the observer outlives router restarts. The declaration may carry icon_svg: one self-contained, monochrome <svg> document (square viewBox, no fixed size, at most 32 KiB). The router echoes it in router::provider::list and the console paints it as a currentColor mask beside the provider's models; a missing or malformed mark falls back to the provider's initial. The declaration may also carry default_models, the provider's recommended starting models in preference order (the current mid-range model first, then its predecessors), and default_thinking_level, the level to pair with it; see Default model.
  3. Resolve credentials per request via router::provider::resolve; never read keys directly. A response with credential_source: "secret" and no credential is an unresolvable secret:// or env:// reference: report its credential_error, do not substitute a key of your own.
  4. Treat closure of its stream channel as cancellation: abort the upstream request and stop writing frames.
  5. Map upstream failures to the shared ErrorKind taxonomy on its error frames. Transport retries (429 / 5xx / connect) are the router's job, not the provider's.

A provider may also register provider::<id>::count_tokens ({model, system_prompt?, tools?, messages, thinking_level?, response_format?} → {model, tokens, estimator}; the last two are the turn's router::chat values, sent only when set) to serve router::count_tokens: an exact count from a provider metering API (estimator: "provider") or a local tokenizer estimate (estimator: "tiktoken"). Providers without it simply make router::count_tokens return a typed router/no_token_counter error for that provider.

Default model

router::provider::list reports one default_model per provider so that a caller with no model choice of its own (a fresh harness::send naming only a provider, a console opening its first chat) starts on something sensible. It is resolved at read time against the provider's current catalog slice:

  1. the model variant first. A variant is the first all-letter id segment after the family (terra in gpt-6.1-terra, sonnet in claude-sonnet-5-5), ranked in the order default_models first names it. Within a ranked variant the newest version in the slice wins, listed or not, so a newer Luna never beats a Terra when Terra is listed first. A model with a ranked variant counts even outside the first preference's family, as long as it has the same vendor/ prefix, so Copilot ranks GPT and Claude models in one list;
  2. ids with no ranked variant come last: the first id in the declared default_models the slice holds, else the slice model sharing the longest id prefix with the first preference, as long as they share the family (the id up to its first -: claude, gpt, codex/gpt). On a tie the highest version wins (gpt-4.1 over gpt-4o), then the plain id over a dated snapshot or a -mini variant;
  3. otherwise absent, and the caller keeps its previous behaviour.

A provider that declares no default_models (local model servers, speech providers) never reports one. default_thinking_level is copied from the declaration; absent means the caller should omit the level and let the provider apply its own default.

A declaration may also carry context_overflow_hint: one or two plain sentences a console shows under a context-overflow failure on one of the provider's models, for causes compacting cannot fix (a self-hosted server started with a small window, say), optionally with a link to the provider's docs. router::provider::list reports it verbatim; absent means the console shows only its generic steps.

A provider that works without a key (a local server started without one) declares credential_optional: true. The router then reports it configured when no credential resolves; a key that is set is still resolved and sent. Absent or false means a key is required.

Thinking levels

router::chat takes thinking_level: off, minimal, low, medium, high or xhigh. Each provider maps a level onto its own knob and warns when a model cannot honour it. off asks for no reasoning at all and is honoured only where the model's descriptor says supports_thinking_off: true (a native switch exists: OpenAI reasoning_effort: none on the models that take it, DeepSeek and GLM thinking.type: disabled, Anthropic Sonnet 5.5 between_tools). false means the model always reasons or the provider has no switch; absent means the provider does not know. Consoles offer, hide or disable the Off choice accordingly. A provider that cannot switch reasoning off sends its lowest effort instead and reports a warning.

Speech providers

A speech provider is the same kind of worker with no chat stream: it declares its models with a speech block and serves one or both speech surfaces. router::models::list hides speech models from chat pickers unless asked (modality: "stt" | "tts" | "any"), and router::chat refuses them with a message that names the right door.

{ "id": "scribe-v1", "provider": "elevenlabs", "context_window": 0, "max_output_tokens": 0,
  "speech": { "modality": "stt", "languages": ["en", "hi"], "streaming": true } }
Function Request Response
provider::<id>::transcribe {model?, audio_base64, mime, language?, prompt?} {model, text, segments[{text, start_secs?, end_secs?}]?, language?, duration_secs?}
provider::<id>::speak {model?, text, voice?, format, language?, speed?} {model, audio_base64, mime, voice?, duration_secs?}

model arrives as the id the caller named, or null for the provider's default. format is what the caller asked for (mp3, wav, pcm16, opus); answer with the container you produced and say so in mime. Credentials resolve through router::provider::resolve exactly as for chat providers.

The first real provider implementing this protocol is provider-anthropic/ — useful as a reference implementation alongside the scripted provider in the integration tests. provider-openai/ follows the same structure for the OpenAI Chat Completions API (native structured output, reasoning_effort). provider-opencode-go/ follows the same structure for the OpenCode Go Chat Completions API (curated model metadata, reasoning_effort).

Local development & testing

cargo test                       # unit suite, no engine needed
cargo test --test integration    # engine-backed suite; self-skips without an engine

The integration suite spawns a throwaway engine per test when iii is on PATH (or III_ENGINE_BIN points at a binary) and covers the chat relay, cancellation, abort, restart recovery, registration token gating, paste-a-key discovery (literal keys, secret:// and env:// references, against a fake secrets worker), and event delivery end to end.

To run the worker locally against an engine:

cargo run -- --url ws://127.0.0.1:49134

--url defaults to ws://127.0.0.1:49134 and honours the III_WS_URL environment variable when the flag is not set. --config is accepted per the standard worker CLI but ignored with a warning — operator config lives in the engine's llm-router configuration entry (see Configuration above).

Search the registry

Find workers, kits and executables, or jump to a page.