Search

Measure Harness capability, retain reproducible evidence, and expose the E2E dashboard.

binary·Supported platforms: macOS: arm64 · Linux: x64·45 functions·harnesse2e+3·iii

Provider-neutral evaluation hub (judge::evaluate, judge::cancel, and the internal judge::models::list) that forwards typed Noul, Choice and Score questions to the selected judge-<provider> worker; the default provider is a Console setting seeded from JUDGE_PROVIDER (typesafe). Also resolves worker-defined @<name>(id=…) chat mentions for agents (judge::mentions::resolve and a harness pre-generate hook).

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, armv7, x64 · Windows: arm64, x64·8 functions·choiceevaluation+4·iii

TypeSafe evaluation (judge-typesafe::evaluate), model listing (judge-typesafe::models::list) and caller-scoped cancellation (judge-typesafe::cancel), with typed usage, provider errors, per-call retries and worker-owned credentials.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, armv7, x64 · Windows: arm64, x64·8 functions·choiceevaluation+4·iii

Compare live objective metrics for 2-5 root sessions; retain durable prompt experiments as an advanced surface.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, armv7, x64 · Windows: arm64, x64·14 functions·benchmarkevaluation+4·iii

laya provider for the judge hub (judge-laya::evaluate, judge-laya::models::list, judge-laya::cancel) — the ModernBERT typed-decision checkpoints (English, multilingual, typed-decisions) downloaded from the Hugging Face Hub and run in-process with language routing; no API key or external service.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, x64·6 functions·choiceevaluation+5·iii

decider provider for the judge hub (judge-decider::evaluate, judge-decider::models::list, judge-decider::cancel) — decider-4b v2 (a Qwen3.5-4B-Base fine-tune for typed decisions, GGUF) downloaded from the Hugging Face Hub and run in-process through llama.cpp, answering from the next-token logits of the option labels at its fitted temperature; picks Metal (macOS), Vulkan (Linux, when a driver is present) or the CPU automatically; no API key or external service.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, x64·6 functions·choicedecider+7·iii

clef provider for the judge hub (judge-clef::evaluate, judge-clef::models::list, judge-clef::cancel) — Cloudflare's Clef-Flash decision model (a Qwen3.5-9B backbone and its joint schema head in one GGUF) downloaded from the Hugging Face Hub and run in-process through llama.cpp's clef architecture, deciding every question of an evaluation together in one forward pass; picks Metal (macOS), Vulkan (Linux x86_64, when a driver is present) or the CPU automatically; no API key or external service.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, x64·6 functions·choiceclef+7·iii

SemIf provider for the judge hub (judge-semif::evaluate, judge-semif::models::list, judge-semif::cancel) — a frozen open LLM (Qwen3.5-4B GGUF) downloaded from the Hugging Face Hub and run in-process through llama.cpp, answering from the next-token logits of the option letters; picks Metal (macOS), Vulkan (Linux, when a driver is present) or the CPU automatically; no API key or external service.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, x64·6 functions·choiceevaluation+7·iii

OpenAI Decisions evaluation (judge-openai::evaluate), model listing (judge-openai::models::list) and caller-scoped cancellation (judge-openai::cancel), with typed usage, provider errors, per-call retries and worker-owned credentials.

binary·Supported platforms: macOS: arm64, x64 · Linux: arm64, armv7, x64 · Windows: arm64, x64·8 functions·choiceevaluation+4·iii