judge-semif

SemIf provider for the judge hub, running inside the worker: no API key, no Python, no llama-server. SemIf reads runtime-defined decisions from a frozen open LLM: one forward pass, a softmax over the next-token logits of the option letters A–P, no decoding. Its published Qwen3.5-4B baseline agrees with Jev's own references on 0.845 of TypeSafe's public subset (Jev 0.883). This worker runs the same prompt through llama.cpp (the llama-cpp-2 crate) on the CPU, on Metal (macOS) or on Vulkan (Linux x86_64: AMD, NVIDIA and Intel GPUs), picked automatically at start.

Hardware selection

• macOS: Metal is linked in; Apple Silicon GPUs run the model.
• Linux x86_64: llama.cpp's backends are modules loaded at start from the binary's directory (GGML_BACKEND_DL). The Vulkan module loads wherever a Vulkan loader and driver exist (libvulkan.so.1); without them it is skipped and the CPU runs the model, so the same package works on GPU desktops, servers and containers. The CPU module is picked for the host (AVX2, AVX-512, AMX variants). The package ships libllama, libggml, libggml-base, the CPU variants and the Vulkan module (≈77 MB) beside the binary, which finds them through its $ORIGIN runpath.
• Linux aarch64: CPU, statically linked.

gpu_layers: 0 keeps the model on the CPU even when a GPU is present. The chosen device is logged at start as selected inference device.

Install

iii trigger compose::add worker=judge-semif

The model loads only while SemIf is the judge hub's default provider (provider: semif under Settings → Workers → judge): the worker follows the hub's configuration and, when another provider becomes the default, unregisters its functions and releases the model (VRAM included); it loads again when SemIf is selected. The first load downloads the pinned GGUF (qwen3.5-4b: bartowski/Qwen_Qwen3.5-4B-GGUF Q4_K_M, 3.0 GB) into the hf-hub cache ($HF_HOME, default ~/.cache/huggingface). The functions register only once the model answers; until then, and while another provider is the default, the hub reports provider_unavailable, also for calls that name "provider": "semif". To keep it loaded while another provider is the default, turn on Keep every local provider loaded (preload_all) in the judge settings. A hub build that does not expose judge::configuration-id leaves the model loaded from the start. Air-gapped installs set III_SEMIF_GGUF to a local GGUF file.

Make it the hub's default, then call the hub:

iii trigger judge::evaluate --timeout-ms 65000 --json '{
  "timeout_ms": 60000, "provider": "semif",
  "evaluations": [{
    "id": "ticket-4411",
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel."},
    "questions": {
      "department": {"type": "choice", "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
      "urgency": {"type": "score", "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
      "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"}
    }
  }]
}'

How questions map to SemIf

Each question becomes one SemIf decision: {"evidence": state, "criterion": instructions, "options": [...]} in SemIf's frozen system prompt and Qwen chat template (thinking disabled), tokenized by the GGUF vocabulary. Token ids match SemIf's reference tokenizer (tests/prompt.rs), and on the CPU the option probabilities match SemIf's own llama.cpp backend to four decimals.

Contract	SemIf options	Answer
noul	A = criteria.true (or "Yes, the statement holds."), B = criteria.false	noul = P(A)
choice	one per criterion, key: description	argmax, probabilities, confidence
score	level i:	Σ i·p_i, probabilities, confidence, legend

confidence is TypeSafe's (judge_contract::confidence, shared by the local providers): (n·p_max − 1) / (n − 1) for a choice, 1 − the expected distance from the likeliest level over the levels' mean distance from the middle for a score. SemIf's probabilities are conditional on the offered options and uncalibrated. At most 16 options per question. A prompt longer than context_tokens answers payload_too_large: SemIf never truncates evidence.

Every question of one evaluation shares its state, and SemIf puts the evidence first. The worker prefills the prompts' longest common token run once, snapshots it, restores it into up to parallel_questions sequences and decodes their suffixes together in one batch (SemIf's parallel suffixes). The prefix is found on tokens, not re-tokenized text: a state ending in {} merges with the following } two tokens back, which SemIf's own _state_prefix rejects. usage.input_tokens counts the decoded tokens (the shared prefix once).

Speed (Qwen3.5-4B Q4_K_M, i9-14900K, RX 6900 XT)

Per request, 16 questions about one state; parallel_questions 8 unless noted:

	incident (541-token state)	directory block (4.6k-token state, 64 functions)
CPU, 8 threads	3.8 s (6.9 s with parallel 1)	6.6 s
Vulkan	0.79 s (1.18 s with parallel 1)	3.4 s

Before the token-level prefix, the directory block decoded every question in full: 75k tokens and 35.8 s on Vulkan, against 6k tokens and 3.4 s now. Parallel suffixes pay off on short states; long states are dominated by the one-time prefill. On the same card llama.cpp's ROCm backend measured within 10% of Vulkan either way, so the worker ships Vulkan alone. A single short decision takes 67–140 ms on the GPU and 1.1–1.4 s on the CPU.

GPU, reuse and batching change probabilities by up to 0.02 against fresh CPU scoring, as SemIf documents for its own fast paths.

Configuration

Settings → Workers → judge-semif:

Field	Default	Applied
model	qwen3.5-4b	next start
threads	min(8, logical cores)	next start
gpu_layers	all layers when a GPU backend and device exist	next start (0 = CPU)
context_tokens	16384	next start
parallel_questions	8	next start (1 = one question per decode)
max_request_bytes	8388608	new calls
max_timeout_ms	300000	new calls

Building

llama.cpp is compiled from source through crates/llama-runtime: cmake, a C++ compiler and libclang (for bindgen) are required; if libclang lives outside the default search path set LIBCLANG_PATH (and BINDGEN_EXTRA_CLANG_ARGS=-I/include when its builtin headers are not found). Linux x86_64 builds also need the Vulkan loader headers, the SPIR-V headers and glslc (Ubuntu: libvulkan-dev spirv-headers glslc) to compile the Vulkan module (only the module links libvulkan, the binary does not). The build copies the modules and libraries beside the binary, so target/release has the published layout; the release catalog ships them as the artifact's companions. Windows is not published yet.

For the full API, read the hub's reference.