context-manager

Turns a raw conversation history plus a target model into a model-ready context: a system prompt and an ordered message list that fits inside the model's usable token budget. It owns what the model sees this turn — token counting, function-result pruning, and history compaction (summarisation) — and nothing else. It is stateless with respect to conversation storage: pass messages in, get results back, persist them yourself (or with session-manager). Summarisation and model-limit lookups go through llm-router when installed; token counting and pruning work standalone.

Install

iii trigger compose::add worker=context-manager

iii trigger compose::add declares the worker in worker-compose.yaml and starts it as part of the Compose project. Runtime configuration lives in the configuration worker (enabled by default in the engine): at boot the worker registers its schema and fetches the authoritative value, and it hot-reloads on change — every field, with no restart (summarizer_timeout_ms is read per call; a lease_dir change rebuilds and swaps the lease store). An optional --config .yaml only seeds the first registration.

Quickstart

Build a budgeted context before every model call:

use iii_sdk::{register_worker, InitOptions, TriggerRequest};
use serde_json::json;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let iii = register_worker("ws://localhost:49134", InitOptions::default());

    let result = iii.trigger(TriggerRequest {
        function_id: "context::assemble".into(),
        payload: json!({
            "messages": [
                { "role": "user", "content": [{ "type": "text", "text": "hi" }], "timestamp": 1 }
            ],
            // Inline limits keep the call standalone; pass only `id`/`provider`
            // to resolve limits through llm-router instead.
            "model": { "id": "claude-sonnet-4",
                       "limits": { "context_window": 200000, "max_output_tokens": 8000 } },
            "system_prompt": "You are a helpful assistant."
        }),
        action: None,
        timeout_ms: Some(30_000),
    }).await?;

    // -> { system_prompt, messages, token_count, usable, model_resolved,
    //      applied: { pruned, pruned_tokens, capped_parts, capped_tokens,
    //                 compacted, summary?, tail_start_index? } }
    println!("{result:#?}");
    Ok(())
}

The compaction round trip: when applied.compacted is true, persist applied.summary and the boundary your storage maps applied.tail_start_index to. On later calls pass only the post-compaction window as messages plus the stored summary as options.previous_summary — the summary is rendered into the system prompt under # Conversation summary, and any further compaction updates it instead of starting over. Callers that skip persistence stay correct at the cost of repeating the bounded summarisation pipeline on every over-budget request.

Before token accounting, context::assemble normalizes images in its cloned model-facing view. A known catalog model receives placeholders unless it explicitly declares vision support. For a known vision model, tool images age after the next assistant response and user images age when a later user turn begins after a response. Inline limits and unresolved models keep images. Live images cost a fixed 4,096-token heuristic budget; base64 bytes are never counted as text. The caller's transcript is unchanged.

Before pruning or compaction runs, assemble unconditionally caps any message-level or inline function result over max_result_tokens (default 20000; 0 disables the pass) to a bounded […result capped: was ~N tokens; middle omitted; re-call {function_id} with narrower arguments if the omitted middle is needed] head+tail view. Text uses provider-visible newline separators and each image costs the fixed 4,096-token allowance — see Configuration below.

Every successful context::assemble response satisfies token_count <= usable. Callers should include the complete tools array and set options.request_overhead_tokens for response-format and provider-specific request fields. If ordinary pruning and compaction are insufficient, assembly replaces oversized function results in its model-facing copy with bounded references to the full result retained in the session transcript. Requests that still cannot fit fail with context/overflow; callers must not issue a provider request in that case.

The other three functions: context::count-tokens (estimate messages + tools + system prompt vs a model), context::prune (replace eligible old function outputs with [output of {function_id} pruned: was ~N tokens; re-call it if still needed] placeholders, no LLM involved), and context::compact (summarise the head, keep a recent tail verbatim — returns ok | busy | empty | overflow).

Configuration

reserved_tokens_cap: 20000     # default reserve = min(cap, pct% of context window)
reserved_pct: 10
tail_turns: 2                  # user+assistant pairs kept verbatim by compaction
protect_recent_tokens: 40000   # newest function-output tokens never pruned
decay_user_turns: 0            # prune after this many subsequent user turns; 0 disables decay
protected_user_turns: 2        # recent user turns exempt from normal pruning; 0 disables this guard
min_free_tokens: 20000         # skip pruning when it would free less
max_output_chars: 2000         # larger outputs are immediately eligible; smaller ones may decay
max_result_tokens: 20000       # per-result ceiling for assemble's unconditional cap pass; 0 disables
lease_ttl_secs: 300            # compaction mutual-exclusion lease TTL
allow_fallback_limits: true    # conservative 8192/1024 when limits can't resolve
summarizer_timeout_ms: 320000  # outer budget for one router::chat summariser call

Other details (and the defaults' single source of truth) live in src/config.rs.

Local development & testing

cargo test                          # unit + BDD (engine scenarios soft-skip)
cargo test --test bdd -- --tags @pure    # no engine required
UPDATE_GOLDENS=1 cargo test --test schemas   # regenerate wire-schema goldens

Switching to a smaller model (MOT-4937)

Automatic assembly and explicit compaction summarize on the selected model and provider; neither silently routes history to a different provider. If the rendered history exceeds the resolved usable input budget, it is consumed in consecutive, UTF-8-safe fragments. Each request includes the summary of earlier fragments as an anchor and is estimated with its system prompt, JSON-escaped user message and request overhead. The final verbatim tail and its original indices are unchanged.

The pipeline allows at most 32 summary calls and shares one deadline across them, limited by summarizer_timeout_ms and the compaction lease lifetime. A failed call, oversized anchor, deadline or call-count limit does not return a partial summary: assembly falls through to its existing emergency reduction and hard-limit check; explicit compaction returns overflow. No user/assistant text is discarded merely to fit a fragment. The estimator remains heuristic, so an upstream overflow can still require a larger-budget model, shorter input, or a new conversation.

Read-only model-switch preview

context::assemble accepts options.preview_only: true for a read-only pre-compaction estimate. It uses the same resolved model budget, media normalization, result caps and normal pruning, but never invokes the summarizer, acquires a compaction lease, or performs emergency reduction. It returns token_count and usable even when over budget. Callers must compare those values: a preview response is not a model-ready request or a guarantee that a future user input will fit. The option defaults to false; normal assembly keeps its existing hard-limit behavior.

The Console checks the full persisted compaction window before a model-picker switch. It asks for confirmation only when that preview is over budget. Cancel keeps the current model; confirmation compacts with the destination model and persists the summary before changing selection. Failed checks or compaction keep the previous selection. The preview includes the session prompt and known tool framing; a later user message, changed configuration, or dynamic hook can change the actual next turn's budget needs.