eval
v0.3.6Session monitor for the Harness (deterministic diagnostics, Jev triage through judge, investigation by the user-chosen LLM, E2E validation plans; inactive until configured).
Install
iii trigger compose::add worker=eval@0.3.6Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
Full markdown
/registry/workers/eval.md?version=0.3.6. Paste it into an LLM prompt or pipe it through curl from a worker.Install
iii trigger compose::add worker=eval@0.3.6Dependencies
Readme
eval
eval monitors finished Harness sessions and proposes Harness improvements
backed by evidence and an E2E validation plan.
The monitor is inactive until configured. It observes, analyzes and
suggests; it never edits code, opens PRs or changes the observed session, and
starts E2E executions only when a person calls eval::start-validation (with a
code directory configured, the investigating LLM
may only call read-only functions: see Code access). The
behavior is specified in
SPECIFICATION.md and the technical plan in
IMPLEMENTATION.md.
How an analysis works
Admission. A terminal
harness::on-turn-completedevent of a root session (or a manualeval::analyze-session) admits one analysis per session turn. Automatic observation admits only the user's chats. Fromsession::getthe session must have kinduser(a record without a kind counts asuser), a chatmetadata.surface(console,slackortelegram, which each client writes on every send) and nometadatakey starting withe2e_. Everything else answersnot_user_chatand can still be analyzed by hand: E2E runs (kinde2e, ore2e_*metadata on the ones recorded before the kind existed), automations (sentinel investigations and the monitor's own, which it opens asautomation; the console stampsconsoleon those too, so the kind is what excludes them), scripted and sub-agent sessions (kinduserbut no surface) and sessions whose record cannot be read. Progress events, descendants and the monitor's own sessions are never admitted; a redelivered event returns the existing analysis.Collection. The monitor waits until the turn is definitive and every descendant has finished (
harness::metrics.complete), then reads every transcript page withinclude_custom: true, failing on malformed pages instead of skipping evidence. The window is the observed turn plus earlier root turns no analysis covered that started while the monitor was already observing (a wake continuation); history from before it is never reported as new. Descendants linked to other turns are listed but not read. Status and tree membership are checked again at the end; a changed session fails the analysis rather than mixing turns.Diagnostics. Deterministic rules run on the complete transcripts before anything is reduced:
repeated_contract_discovery: a successfulengine::functions::infowhose every contract came backunchanged_in_context, each traced to an earlier successful result that supplied that schema. Aregistry-changednotice between the sources and the repeat marks itharness_notice_correlated(correlation, not cause).repeated_tool_error: consecutive calls in one turn to the same target with structurally equal payloads, where the first failure was visible before the retry and both failed with the same explicit code. Corrected arguments, a successful retry, parallel calls in one response, other turns and, in transcripts recorded against a 0.25 engine, its default-namespaceengine::triggers::infoNOT_FOUNDprobe are not flagged.
Each occurrence has a fingerprint (rule, session, turn, call ids).
Triage (Jev).
judge::evaluatewithprovider: typesafeasks one Choice question —needs_investigation,expected_behaviororinsufficient_evidence— over facts computed in code (no transcript text). The effective model, probabilities, confidence, criteria hash andStatsare stored.Routing. The LLM investigates if and only if Jev answered
needs_investigation, for automatic and manual analyses alike (routing.reasonsis["needs_investigation"], else[]). A deterministic finding,insufficient_evidence, a low confidence, insufficient coverage, a manual request and a sample of quiet sessions do not send a session there on their own; the findings stay in the result either way. Analyses recorded under the earlier routing keep theirdiagnostics,insufficient_evidence,low_confidence,coverage_insufficient,audit_sampleandmanual_requestreasons; none of them is produced now.Investigation.
harness::sendopenseval_monitor_<evaluation_id>(kind: automation,metadata.origin: eval_monitor) with the frozen model, an overriding system prompt and a JSON output contract (at most three suggestions; an empty list is valid). Without a code directory (the default) it is one turn with deny-all functions. With one, it reads the code (see Code access). The bundle also carriesmonitor.e2e_scenarios({total, scenarios: [{id, title, summary}]}, frome2e::dashboard::tests-list, one page of at most 100 scenarios, summaries cut at 160 characters, reused for ten minutes per process) and the prompt tells the LLM to setvalidation.scenario_idto one of those ids when it reproduces the problem, and to keepnullwhen a new case is needed. When the E2E does not answer, the list and that instruction are omitted and the analysis goes on. The backend drops any suggestion that cites an entry absent from the evidence shown, cites code that does not exist, or has an incomplete E2E plan, and records why. A message that reaches the monitor's session while it runs continues it in a new turn, and the investigation ends on the session's settled turn.Result.
completedmeans the monitor finished, with or without suggestions. It is never a validated improvement.
States: queued → collecting → judging → (investigating) → completed, or
failed (with the failing stage and code) or cancelled.
A failure's code names the cause. Jev failures are judge_<provider code>
(judge_missing_key, judge_provider_unavailable…), except that HTTP 402 from
the provider is judge_out_of_credits (the message keeps the provider's own
text, and assets.triage_failure keeps code: http and http_status: 402).
The investigation fails with analyst_failed, analyst_step_cap or
analyst_output_invalid; the others are coverage_insufficient,
source_advanced, external_outcome_unknown, deadline and cost_cap (an
automatic investigation that found the daily cost cap reached when it was about
to start).
Each record also keeps three facts for comparing analyses and releases:
supersedes (the earlier evaluation_id of the same turn, set when the
analysis is a reanalysis), harness_version (the version of this worker's
Harness that engine::workers::list reported when a live event admitted the
analysis, read from the engine's default namespace: the turn just ended on
it. A manual analysis, which may be of an older session, has none, and neither
has one admitted while the engine did not answer, which never blocks admission) and
signals (occurrences per deterministic pattern the collection found, keyed
<rule_id>:<target>; empty both when nothing was found and before collection,
and coverage tells which).
Configure and use
The monitor reuses the existing providers; it never receives a TypeSafe or
LLM key. Jev credentials stay in judge-typesafe; the LLM's in its provider.
# 1. Choose the investigating model (validated against router::models::list)
# and keep it paused.
iii call eval::configure --json '{
"enabled": false,
"model": {"model": "<model-id>", "provider": "<provider-id>"}
}'
# 2. Analyze one finished root session by hand (works while paused).
iii call eval::analyze-session --json '{"session_id": "<session>"}'
# 3. Follow it, then read evidence, triage and suggestions.
iii call eval::status --json '{"evaluation_id": "<id>"}'
iii call eval::result --json '{"evaluation_id": "<id>"}'
# 4. Enable automatic observation once the manual results look right.
iii call eval::configure --json '{
"enabled": true,
"model": {"model": "<model-id>", "provider": "<provider-id>"}
}'eval::config shows the configuration and whether the
harness::on-turn-completed observer is active (observer_bound); when it is
false (no harness has registered the function, or the engine refused the
observer, with the reason in observer_error), automatic observation is
unavailable even if enabled.
thinking_level, provider_options (namespaced by the selected provider),
code_repository and daily_cost_cap_usd are optional. A configuration
change applies to new analyses only: each record keeps its model, its code
directory and its config_revision.
Cost
Only LLM cost is in dollars: the investigation's (usage.llm_cost_usd, from
harness::metrics, and known only once the investigation ends) and the
eval::reproduce samples' (cost_usd of each reply). Jev's usage, in a
triage or in classifying a replay's replies, is reported in tokens and never
priced. A missing cost is unknown, never zero, and is never estimated: a
replay sample without a cost, one that failed included (the provider may still
bill a request the bus gave up on), adds nothing to the sums, is counted in the
reproduction's cost_unknown_samples (which makes its cost_usd a lower
bound) and in the day's today_replay_unknown. Each sample is added to the day
as it is saved, so a run that stops partway has already counted what it kept.
The day's spend has two buckets. Capture is what an analysis spends (the
investigation; Jev's triage is in tokens). Replay is what eval::reproduce
spends. Only capture is compared with the daily cap, so a manual replay never
stops automatic observation. The spend persisted before the buckets (one total)
counts as capture for its day.
eval::config returns a cost block to decide before enabling:
today_capture_usdandtoday_unknown: the known capture cost sincesince(the start of the current UTC day: the monitor has no timezone setting), and how many analyses of the day started an investigation but reported no cost (they add nothing to the sum; their cost is unknown). The cost is added to a persisted daily spend as each investigation reports it, andtoday_capture_usdis the larger of that and the stored analyses' sum, so deleting analyses (or retention) does not give the budget back.today_replay_usdandtoday_replay_unknown: the replay bucket, the known cost of the day'seval::reproducesamples and how many had no cost (failed samples included). Reported, never capped.today_usd: both buckets together, everything known to be spent. The cap does not compare it.cap_usdandcapped: the optionaldaily_cost_cap_usdand whethertoday_capture_usd, plus the median cost of each investigation still running, reached it.per_analysis:count,min,medianandmaxof the known cost of the completed analyses that investigated with the configured model, provider and code access (or without it), andunknown, how many of those reported no cost. Without history,countis 0 and the statistics are absent.
daily_cost_cap_usd (a number above 0; absent means no cap, and then the
configuration revision is what it was) pauses automatic observation for the
rest of the UTC day once today_capture_usd reaches it: the turn is not admitted,
eval::on-turn-completed answers cost_cap, and eval::config's
last_rejection records the turn with reason: cost_cap (at_capacity for the
unfinished-analyses cap). A manual eval::analyze-session is never refused by
the cap, and a redelivered event of an already admitted turn is still reused.
The cap is checked twice: at admission and again where the money is spent,
right before an automatic analysis first calls the model (under one lock, so
investigations starting together see each other); one that finds it reached
fails with cost_cap without calling the model; a manual reanalysis is not held
back by the cap and investigates if Jev answers needs_investigation again. A cost is known only when an investigation ends, so the investigations
already running are counted at the median cost of the history. With no history
there is nothing to estimate by and a first burst can overshoot by the
investigations that run at once (the queue runs 8 steps together). Raising or
removing the cap resumes observation at once.
Code access
By default the investigating LLM sees only the evidence bundle. Set
code_repository to the codebase directory (normally the iii workers
repository, e.g. /home/layon/workspaces/workers) and the investigation
becomes a chat with that directory selected:
iii call eval::configure --json '{
"enabled": false,
"model": {"model": "<model-id>", "provider": "<provider-id>"},
"code_repository": "/home/layon/workspaces/workers"
}'- The path must be absolute and an existing directory on the host that runs
eval(checked when it changes; blank means off). The monitor stays disabled by default. It reads the current working tree, uncommitted changes included: there is no pinned copy, no tag lookup and no requirement that the directory is a git clone. To implement a suggestion, create a worktree or work however you prefer. - Each analysis freezes the directory at admission (
record.code_root, echoed ininvestigation.code_root); changing the configuration later never alters a running analysis. - The send carries
{"fs_scope": {"root": "<dir>"}}in the session metadata and inoptions.metadata, exactly what the ADE chat sends for a selected directory: the Harness scopes everycoder::*andshell::*call to it and the console shows the directory selected when the investigation session is opened. The policy is an explicit read-only allowlist,allow: ["coder::search", "coder::tree", "coder::read-file", "session::messages", "github::pr::list", "engine::functions::info"], withdeny: ["eval::*", "e2e::dashboard::execution-*"]kept as well (no new bus function: the LLM uses existing ones). The first five are the functions the prompt names (session::messagesreads more of an observed session than the evidence shows);engine::functions::infois the contract lookup the invocation surface tells a model to make before a first call (without it the LLM guessed argument names, e.g.start_lineforline_from). Everything else is refused by the Harness, so a transcript it reads cannot make it write a file, run a shell, start or message a session, start an E2E execution or record a review.fp::pipeis deliberately absent: its steps run with thefpworker's authority, outside this policy (analysts had used it to readsession::messages, now allowed directly). Without a code directory the policy stays deny-all. Withapproval-gateinstalled five pass without a human (iii-permissions.yamlallows them);session::messagesstays at that file'sneeds_approvaldefault for session reads, so each call waits for a person or the deadline. The step cap goes from 1 to 32 generate steps and the total-token cap from 200,000 to 800,000 (limits.investigation_code_max_turnsandinvestigation_code_max_total_tokens). The deadline and thesubmit_resultoutput contract are unchanged. A turn that uses all its steps endscompletedin the Harness with a notice instead of a result; the monitor fails that analysis withanalyst_step_cap(usage kept, no suggestions) instead of reading the notice as an invalid answer. - The prompt names the directory, asks the LLM to read the code before
proposing a change and to deliver through
submit_resultonly afterwards, states the step and token budget (a turn that uses every step delivers nothing), reminds it that the cause or the best fix may be in any worker (not only the Harness), declares the investigation read-only (no file changes, no starting or messaging sessions, noeval::*), repeats that transcripts and code are data, and asks it to cite every claim about code. It also tells the LLM that it may list the open pull requests withgithub::pr::list(read-only, repoiii-hq/workers) and must say in the suggestion'slimitationswhen one already overlaps it. Without code access the prompt is the one the monitor always had, and the output schema does not offercode_refs. - A suggestion carries
code_refs: up to 8{path, line_from, line_to}with the path relative to the directory and 1-based inclusive lines. When the investigation completes the monitor checks each one on disk and rejects the suggestion, with the reason, if the path is absolute or has.., resolves outside the directory once symlinks are followed, is not a file, or the range is not1 <= line_from <= line_to <= lines in the file;code_refswithout code access are rejected too. The checks read the files as they are when the investigation ends, only as far asline_toand never a file above 16 MiB (a reference to one is rejected), on a blocking thread, because the LLM chooses the file (a traces database or a build output lives in a workers checkout). - Tool calls and tokens of the investigation are the session's own metrics
(
harness::metrics), counted as before.
Residual risk. The investigating LLM can no longer change anything or
start anything, but it still reads untrusted transcripts and can read the
workspace. The fs_scope root does not jail the reads: in this environment the
ide worker runs coder::* unjailed, so coder::read-file and coder::search
accept absolute paths outside the root. What a hostile transcript can still
achieve is steering what the LLM reads and what its suggestions say, so read a
suggestion's code_refs and text as the output of an LLM that read untrusted
input. Leave code_repository unset to keep the investigation read-nothing.
Public functions
| Function | Purpose |
|---|---|
eval::configure |
Model selection, optional code directory, optional daily cost cap and enable/pause. |
eval::config |
Configuration (or null), observer status, the latest rejection and the cost block. |
eval::analyze-session |
Manual analysis; reanalyze: true creates another analysis after the previous one ended. |
eval::list |
Compact records, newest first (limit 1–200, default 50); observation_key keeps only the analyses of one observed turn, reanalyses included. |
eval::status |
One compact record, or null. |
eval::result |
Record plus evidence, diagnostics, triage, suggestions, the monitor's own consumption, E2E links and one review row per suggestion (reviews). |
eval::cancel |
Signals the Jev call (judge::cancel) and stops the investigation (harness::stop); never touches the observed session. |
eval::delete |
Deletes a terminal analysis; the turn stays marked as analyzed until retention. |
eval::attach-validation |
Links baseline and candidate E2E executions to a suggestion and computes the pair's evidence (dry_run: true only looks them up). |
eval::start-validation |
Explicitly starts a baseline and a candidate E2E execution (Docker) of one scenario, pinned to two pushed commits. Spends model tokens. |
eval::review |
set_lifecycle, set_criterion or set_verdict on one suggestion of a terminal analysis. |
eval::reviews |
The stored review rows and, per analysis, its suggestions counted by lifecycle status. |
eval::recurrence |
For a suggestion shipped in a version, how often its patterns appeared per analysis before and from that version. |
eval::reproduce |
Replays a suggestion's decision point: rebuilds the request the model received at that step (optionally edited), samples the next reply N times without running any function and reads the signal in each. dry_run only rebuilds and checks fidelity. See VALIDATION.md. |
Internal: eval::step (queue eval-run, FIFO per analysis, concurrency 8),
eval::on-turn-completed (the target of the worker's observer of
harness::on-turn-completed, delivered the event) and eval::sweep (the
target of its observer of cron::on-tick with args
{ expression: "*/15 * * * * *" }).
Events
eval::on-completed is a private function: eval calls it when an analysis
reaches a terminal status and something observes it; anyone may observe it.
It takes no args; select one analysis with a filter on input.evaluation_id.
Its handler returns the event:
interface CompletedEventV1 {
evaluation_id: string
status: 'completed' | 'failed' | 'cancelled'
timestamp: number // Unix ms
}Delivery is iii's default: every call reaches each observer whose filters
match. completed describes the monitor, not a validated improvement.
iii.observe(
{ function_id: 'eval::on-completed', filters: { 'input.evaluation_id': { $eq: id } }, capture: ['output'] },
'my-worker::on-analysis-done',
)Evidence handling
Snapshots keep, per session, the entry count, the SHA-256 of the collected
JSON (not of stored bytes) and masked previews. Previews favor diagnostic
evidence, the request that started the window, the final answer, notices and
a short tail, within the model-context limit (below); omitted and reduced entries are
counted. Diagnostics take at most 64 KiB of that context (all
of them stay in the snapshot); evidence that does not fit makes coverage
insufficient, and a context above the limit is never sent. Raw result details and opaque reasoning payloads are
dropped; JSON carried as text is decoded before the Harness masker hides
secret-named keys and shortens long strings. Masking is key-based: secrets
in free text are not guaranteed to be removed.
Metrics come from harness::metrics and are cumulative over the session tree
and all turns (metrics_scope: session_tree). The monitor's own Jev Stats
and investigation metrics are stored separately from the observed task.
Unknown cost stays unknown.
Limits
Constants of this version, returned by eval::config as limits so the
console never restates them:
| Limit | Value |
|---|---|
| Analysis budget from admission | 30 min (queue wait, descendants and investigation included) |
| Bus calls for collection | 10 s each, within the budget |
| Jev | 60 s provider timeout, 70 s bus timeout |
| Model context | 192 KiB of serialized JSON (about 50k tokens); diagnostics take at most 64 KiB |
| Assets per analysis | 2 MiB; above it the analysis fails with coverage_insufficient before any model call. The turn record kept in assets.capture.record is the first thing left out to stay under it |
| Investigation | 1 turn, 16,384 output tokens, 200,000 total tokens (the model's own caps still apply) |
| Investigation with a code directory | 32 generate steps, 16,384 output tokens, 800,000 total tokens, only the six read-only functions of the Code access allowlist (investigation_code_max_turns, investigation_code_max_total_tokens) |
| Queue | eval-run, FIFO per analysis, 8 steps at once |
| Unfinished analyses | 500 |
| Retention | 30 days and 1,000 terminal analyses |
Recovery and its limits
Records, indexes and assets live in state scopes eval_monitor,
eval_observation, eval_analysis and eval_analysis_assets; the review rows
in eval_suggestion (below). A record is
saved before its index; startup reconciles missing indexes before the
turn observer is created, and the sweep re-enqueues pending work and
fails analyses past their deadline.
- Assets are written before a stage advances, so a restart repeats collection, never a recorded model answer.
- A Jev call that started before a restart without a saved answer fails with
external_outcome_unknown; it is not repeated automatically (the provider has no durable response cache). Request a reanalysis. - A lost
harness::sendreply is resent with the same idempotency key. - Events emitted while
evalwas offline are not recovered; analyze those sessions manually. - Recovery assumes
statekept its records. The defaultkvadapter flushes to disk everysave_interval_ms(5 s by default), so a crash of the state worker can lose the last writes; there is no exactly-once guarantee under storage loss. - Locks are per process: run a single
evalinstance. - Persisted records reject fields they do not know, so state written by a newer
eval(a capture'srecord, a reproduction'scost_unknown_samples) cannot be read back by an older one. Rolling back is not supported once new analyses or replays exist.
Validation by replay
The first check of a suggestion is a replay of the step where its behavior
happened (VALIDATION.md). Each suggestion carries a check:
the decision_point (an assistant entry of its evidence), the signal (a rule,
contract_rediscovery or repeated_error_call, or a yes/no question Jev
answers per reply) and the proposed change as edits of what the model saw.
Older suggestions have none; eval::reproduce then takes check in the request.
eval::reproduce rebuilds the request the Harness sent at that step: the
window from the durable log (harness::window::build, notices included), the
frozen runtime context, the turn's system prompt and skills baseline (copied
into the analysis as assets.capture, because the Harness keeps only a
session's latest turn record), the agent_call tool and context::assemble.
assets.capture.record also keeps the Harness's whole turn record as read (a
later fork of the session needs it, and the next turn replaces it); when it
would take the assets over their limit it is left out and
assets.capture.record_omitted says why, while the options and digest above
stay.
It counts the result with router::count_tokens (the context manager's estimate
when the provider has no counter) against the recorded usage: exact when the
difference equals the fixed overhead measured at the turn's first step,
approximate otherwise, with the reasons. It then samples router::complete
(one warm-up call, then four at a time), never runs a function, reads each
reply's signal and stores everything in the suggestion's review row
(reproductions[]). extend adds replies; samples: 0 finishes one that
failed. A restart marks running replays failed (interrupted). Not supported
yet: native function exposure, output contracts, and windows the context
manager would prune or compact.
E2E validation
A suggestion carries a reproduction, task invariants, a primary effort metric,
the expected baseline-versus-candidate comparison and non-regression
controls. Turning it into proof is a separate, explicit step: fix the
scenario version and criteria, run baseline and candidate with
harness-e2e, then link both executions:
iii call eval::attach-validation --json '{
"evaluation_id": "<id>", "suggestion_index": 0,
"baseline_execution_id": "<baseline>", "candidate_execution_id": "<candidate>"
}'The link stores what e2e::dashboard::execution-get reports (status, report
availability, evidence errors). Attaching the same pair again refreshes its link
but keeps the first attached_at, which still says when results first existed. Each scenario keeps its run_count and
measures (the execution's own averages) and, when
plan_execution.measurements.cohorts[] has exactly one cohort for it, that
cohort's own aggregates: pass_rate, mean_score, completed_runs,
planned_runs, cost_usd_per_run and total_tokens_per_run (the cohort's
totals divided by its completed runs; the totals also cover failed attempts),
median_wall_time_ms and p50_function_calls. Each is absent when the E2E did
not report it (a count may arrive as a float), never zero. It is a reference,
not a verdict: improvement,
no improvement, regression or inconclusive belong to the E2E comparison and its
criteria.
Review and validation
What people decide about a suggestion lives in its own row, in the
eval_suggestion state scope keyed <evaluation_id>:<suggestion_index>
(SuggestionReviewV1), never in the suggestion, whose schema is what the
investigating LLM must return. The row copies the suggestion's title,
harness_component, plan scenario_id and its patterns (<rule_id>:<target>
of the diagnostics the suggestion's evidence cites), so it outlives the
analysis's retention and eval::delete. A suggestion nobody acted on has no
stored row and reads as new in eval::result.reviews (one row per
suggestion) with lifecycle.history: [].
eval::review {evaluation_id, suggestion_index, action, ...} needs the
analysis to be terminal and the suggestion to exist. The author of every change
is by when the request names one, else the user the eval worker runs as
($USER, else $LOGNAME), else the caller's identity (_caller_worker_id):
a browser's worker id is an opaque uuid, so it comes last. The console sends no
by, so what it records is the host's user.
set_lifecyclewithstatusaccepted,in_progress,shipped,rejectedorduplicate(never back tonew; a shipped suggestion stays shipped, to complete itsprorversion).shippedneeds apr(kept from an earlier status when omitted) and may carry theversionit shipped in, a semantic version like1.8.43;rejectedneeds areason;duplicateneedsduplicate_of.lifecycle.historykeeps{status, at, by, note?}for every change.set_criterionregisters what a validation is judged by, before its results: ametric(signal_per_runwith one of the row'spatterns,pass_rate,cost_usd,duration,tokensorfunction_calls), adirection, amin_effect(the smallest difference between the baseline and candidate means that counts, in the metric's own unit: signals per run, USD, seconds of wall time, function calls or tokens; percentage points, 0-100, forpass_rate.1is one signal per run, never 1%) andmin_runs(1-20 completed runs each side), on the plan's scenario unlessscenario_idnames another. The same criterion again keeps its registration time. Once a pair is attached or a run started, an existing criterion is frozen (criterion_frozen:): one chosen after seeing results proves nothing.set_verdictrecords the person'soutcome(validated_improvement,no_improvement,regressionorinconclusive),rationaleandcontrols_checked, with a snapshot of the criterion.validated_improvementis refused (verdict_refused:) unless a criterion was registered no later than the first moment results could exist (the earliest of each attached execution's ownstarted_at, the attach, and the executions this worker started, also those a restart replaced), so an old pair attached after its criterion does not count as unseen, and the evidence has at leastmin_runsmeasured runs on both sides. The computed outcome (below) is not a precondition: the person may disagree, and both are kept.
Evidence
Whenever a pair is attached (by hand or by a started run) and the suggestion's
scenario is known (the criterion's, else the plan's), the runs of that scenario
in both executions (detail.reports[].report.scenarios[].runs[]) become the
row's evidence: per run its completed flag (the E2E's completion is
completed and technical is valid, both kept as reported, so infrastructure
failures and incomplete runs are listed, never silently dropped), status,
cost, wall time, tokens, function calls, and the occurrences of each of the
row's patterns that diagnostics::detect finds in the run's own transcript
(transcript.messages; a run without transcript has no count, 0 is a
measurement). Per side n is the measured runs with a value of the criterion's
metric and mean their mean. A run is measured when its technical is valid:
one the agent left unfinished (completion: task_incomplete) is the Harness's
result, so it counts, as a fail of the pass rate whatever its status says,
while an infrastructure failure measured nothing and stays out. computed_outcome is decided in code, with a
reason: validated_improvement when the candidate moves the baseline mean by
at least min_effect (in its unit) in the wanted direction, regression when it
moves the wrong way by that much, no_improvement otherwise, and inconclusive
when there is no criterion, a side has no run of the scenario, more than half of
a side's runs did not complete (technical failures and unfinished runs) or a
side has fewer than min_runs measured runs with the metric. Only the root
session of each run is
read (a Docker run's sessions never reach this engine); every metric but
signal_per_run is a mean over the same runs, not the E2E cohort aggregate.
Starting a validation
eval::start-validation {evaluation_id, suggestion_index, scenario_id, candidate_ref, baseline_ref?, runs?, model, provider, criterion} replaces the
manual steps of linking two E2E runs. It runs only when called: it spends model
tokens and Docker time on two executions whose cost depends on the scenario and
runs (a past cohort of the same scenario in the E2E is the best estimate).
With dry_run: true (default false) it stops after the checks that need no
E2E: the refs resolve (step 1), both commits are pushed and differ, and the
scenario and runs are valid (and criterion.min_runs <= runs, when a criterion
is sent; model, provider and criterion are otherwise only required for a
real start). It answers {baseline, candidate, warnings}, each side
{commit, short, branch} (short is 12 characters, branch a remote branch
that holds the commit, the one named like the ref when several do), and
registers nothing, records no run and calls no E2E function. A baseline given
that is not an ancestor of the candidate is a warning, not an error. The
console asks it 400 ms after the last keystroke and when a field loses focus,
shows Resolved · <sha> · pushed to <branch> or the refusal under each field,
and keeps Start disabled until a dry run succeeded for exactly the current
refs, scenario and runs.
- Needs
code_repositoryconfigured (code_repository_required:). The refs are resolved withgitin it:candidate_refandbaseline_refmay be a plain branch, tag or commit name; the baseline defaults to the merge base of the candidate andorigin/main. Each commit must be on a remote branch of the clone (git branch -r --contains, nothing is fetched: rungit fetchafter a push) because the E2E builds the Harness from GitHub. Errors:git_unavailable:,git_ref_invalid:(also when both resolve to the same commit),commit_not_pushed:. - Each stack is the YAML of the E2E's stack built on the
harnesstemplate (e2e::dashboard::stacks-list) with only itsharnesscontainer replaced by{worker: package://harness, commit: <sha>, repository: iii-hq/workers}. Everything else, including thelatestversions the base declares, is the E2E's own (a comparability check on the attached pair shows when two executions resolved different ones). - Nothing changed so far; now the criterion is registered and the run recorded
as
starting(runin the row: both commits, scenario, runs, model, provider,started_at), and two executions start withe2e::dashboard::execution-start(where: docker, one technical retry,runs1-20, default 5, labeleval <evaluation_id> S<n> baseline|candidatewithnthe suggestion number from 1). The row goes torunningwith both execution ids. A refusal ise2e_busy:ore2e_unavailable:; the run is thenfailedwith the error, naming the baseline execution when it had already started (cancel it in the E2E). A start the E2E never answers (e2e_start_unconfirmed:, after 30 s) may still have started an execution, so the run staysstarting, with the baseline id when it had one, and no second start can duplicate it: look for the label in the E2E; the sweep fails the run after 5 minutes. One run per suggestion at a time. - The sweep (every 15 s) polls
e2e::dashboard::executions-list {ids}for running runs. When both executions arecompleted,interrupted,cancelledorfailedthe run isfinishedand the pair is attached with the evidence computed (attached), also when the executions ended badly: the evidence then says why it cannot judge. An E2E that does not answer leaves itfinishedwith the error and the next sweep tries again, for 30 minutes after both ended (finished_at), then it isfailedwithattach_gave_up:and the last error, and the pair can be attached by hand or started again; an execution the E2E no longer lists, or an analysis deleted meanwhile, makes itfailedat once. Astartingrun older than 5 minutes (a restart between the request and its answer, or an E2E that never answered) isfailed. Startup recovery does not wait on the E2E: the first sweep, 15 s later, does.
After the release
eval::recurrence {evaluation_id, suggestion_index} for a suggestion marked
shipped with a version returns, per row pattern, the occurrences, the
analyses with at least one and the occurrences per analysis for the analyses on
Harness versions before that version (before) and from it on
(from_version). Only analyses whose collection finished count (signals is
empty both when nothing was found and before collection), and analyses without a
semantic harness_version are left out and counted in without_version: the
manual ones, among them every reanalysis, so one turn is never counted twice.
Analyses are kept 30 days, so the earlier window is bounded by retention.
Migration from prompt experiments
eval::start, eval::rerun and eval::assert::* are removed, and so is the
session comparison (eval::compare-sessions and its console tab), dropped on
request; the specification still lists it. The prompt experiments' records
stay in the eval_job and eval_session scopes untouched. Old eval::step
messages still queued are skipped (no matching analysis). Finish or cancel
old experiments before upgrading if their results matter.
The console page still shows the previous interface; it will be replaced by the monitor interface in a separate change.
Checks
cargo fmt --manifest-path eval/Cargo.toml --all -- --check
cargo test --locked --manifest-path eval/Cargo.toml --all-targets
cargo clippy --locked --manifest-path eval/Cargo.toml --all-targets -- -D warnings
# Replay the detectors over exported session::messages entries:
cargo run --manifest-path eval/Cargo.toml --example replay -- <session>=<entries.json>tests/flow.rs runs the whole flow over the SDK wire protocol against an
in-memory engine (mocked state, queue, Harness, session-manager, router,
judge and E2E). It proves the monitor's mechanics, not the quality of its
suggestions or any Harness improvement.
API reference (JSON)
{
"functions": [
{
"description": "Analyze the latest definitive turn of a finished root Harness session. Returns the existing analysis of that turn unless reanalyze is true; reanalysis needs the previous analysis to be terminal. Works while automatic observation is paused.",
"metadata": {},
"name": "eval::analyze-session",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"reanalyze": {
"default": false,
"description": "Explicitly create another analysis of an already analyzed turn.",
"type": "boolean"
},
"session_id": {
"type": "string"
}
},
"required": [
"session_id"
],
"title": "AnalyzeSessionRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"evaluation_id": {
"type": "string"
},
"reused": {
"description": "True when an existing analysis of the same session turn was returned.",
"type": "boolean"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"evaluation_id",
"reused",
"status"
],
"title": "AnalyzeSessionResponseV1",
"type": "object"
}
},
{
"description": "Link a baseline and a candidate E2E execution (read through e2e::dashboard::execution-get) to one suggestion of a terminal analysis. The link records report availability and, when the suggestion's scenario is known, the evidence computed from the runs; it is not a verdict and starts no campaign.",
"metadata": {},
"name": "eval::attach-validation",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"baseline_execution_id": {
"type": "string"
},
"candidate_execution_id": {
"type": "string"
},
"dry_run": {
"default": false,
"description": "Look both executions up and return the link without saving it.",
"type": "boolean"
},
"evaluation_id": {
"type": "string"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"baseline_execution_id",
"candidate_execution_id",
"evaluation_id",
"suggestion_index"
],
"title": "AttachValidationRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ComparabilityCheckV1": {
"additionalProperties": false,
"properties": {
"baseline": {
"type": [
"string",
"null"
]
},
"candidate": {
"type": [
"string",
"null"
]
},
"field": {
"type": "string"
},
"matches": {
"type": "boolean"
}
},
"required": [
"field",
"matches"
],
"type": "object"
},
"ComparabilityV1": {
"additionalProperties": false,
"description": "The conditions that must match for a fair comparison; only the Harness version is meant to differ.",
"properties": {
"checks": {
"items": {
"$ref": "#/definitions/ComparabilityCheckV1"
},
"type": "array"
},
"comparable": {
"type": "boolean"
}
},
"required": [
"checks",
"comparable"
],
"type": "object"
},
"E2eAssessmentsV1": {
"additionalProperties": false,
"properties": {
"passed": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"passed",
"total"
],
"type": "object"
},
"E2eExecutionRefV1": {
"additionalProperties": false,
"description": "What `e2e::dashboard::execution-get` reported at lookup time.",
"properties": {
"assessments": {
"anyOf": [
{
"$ref": "#/definitions/E2eAssessmentsV1"
},
{
"type": "null"
}
]
},
"availability": {
"type": [
"string",
"null"
]
},
"conclusion": {
"type": [
"string",
"null"
]
},
"e2e_revision": {
"type": [
"string",
"null"
]
},
"engine_version": {
"type": [
"string",
"null"
]
},
"evidence_error": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_version": {
"type": [
"string",
"null"
]
},
"label": {
"type": [
"string",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"reports": {
"items": {
"$ref": "#/definitions/E2eReportRefV1"
},
"type": "array"
},
"reports_available": {
"description": "True only when the execution reported retained reports.",
"type": "boolean"
},
"scenarios": {
"items": {
"$ref": "#/definitions/E2eScenarioV1"
},
"type": "array"
},
"started_at": {
"description": "When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"status": {
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"reports",
"reports_available",
"scenarios"
],
"type": "object"
},
"E2eMeasureV1": {
"additionalProperties": false,
"properties": {
"average": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"description": "Runs that reported this measure; fewer than `run_count` means some runs reported nothing (never counted as zero).",
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"samples"
],
"type": "object"
},
"E2eReportRefV1": {
"additionalProperties": false,
"properties": {
"available": {
"type": "boolean"
},
"scenario_id": {
"type": "string"
},
"subject_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"available",
"scenario_id"
],
"type": "object"
},
"E2eScenarioV1": {
"additionalProperties": false,
"description": "One scenario's identity and the execution's own averages, as the E2E computed them.",
"properties": {
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"contract_fingerprint": {
"type": [
"string",
"null"
]
},
"cost_usd_per_run": {
"description": "The cohort's total cost divided by its completed runs; the total also covers failed attempts.",
"format": "double",
"type": [
"number",
"null"
]
},
"mean_score": {
"description": "Mean score of the scored runs (0 to 100).",
"format": "double",
"type": [
"number",
"null"
]
},
"measures": {
"additionalProperties": {
"$ref": "#/definitions/E2eMeasureV1"
},
"description": "`function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.",
"type": "object"
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"pass_rate": {
"description": "The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero.\n\nShare of the planned runs that passed (0 to 1).",
"format": "double",
"type": [
"number",
"null"
]
},
"planned_runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"run_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"total_tokens_per_run": {
"description": "The cohort's total tokens divided by its completed runs.",
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"measures",
"run_count",
"scenario_id"
],
"type": "object"
},
"ValidationLinkV1": {
"additionalProperties": false,
"description": "Baseline and candidate executions linked to one suggestion. A link is a reference, not a verdict: the improvement claim belongs to the E2E comparison and its criteria.",
"properties": {
"attached_at": {
"format": "int64",
"type": "integer"
},
"baseline": {
"$ref": "#/definitions/E2eExecutionRefV1"
},
"candidate": {
"$ref": "#/definitions/E2eExecutionRefV1"
},
"comparability": {
"$ref": "#/definitions/ComparabilityV1"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"attached_at",
"baseline",
"candidate",
"comparability",
"suggestion_index"
],
"type": "object"
}
},
"properties": {
"link": {
"$ref": "#/definitions/ValidationLinkV1"
},
"saved": {
"type": "boolean"
}
},
"required": [
"link",
"saved"
],
"title": "AttachValidationResponseV1",
"type": "object"
}
},
{
"description": "Cancel an analysis: signals its Jev call and stops its investigation session. Never touches the observed session or the captured evidence.",
"metadata": {},
"name": "eval::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"cancelled": {
"type": "boolean"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"cancelled",
"status"
],
"title": "EvalCancelResponseV1",
"type": "object"
}
},
{
"description": "Read the monitor configuration (null until configured), whether the harness::on-turn-completed observer is active, the latest rejection (capacity or daily cost cap), the day's known investigation cost against the cap with what one analysis has cost, and, with check_providers, whether the Jev triage provider answers.",
"metadata": {},
"name": "eval::config",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"check_providers": {
"default": false,
"description": "Also ask the triage provider whether it answers (`judge::models::list`).",
"type": "boolean"
}
},
"title": "MonitorStateRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"AnalysisCostStatsV1": {
"additionalProperties": false,
"description": "Cost of the completed analyses that investigated with the configured model, provider and code access (or lack of it).",
"properties": {
"count": {
"description": "Analyses with a known cost; `min`, `median` and `max` come from them.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"max": {
"format": "double",
"type": [
"number",
"null"
]
},
"median": {
"format": "double",
"type": [
"number",
"null"
]
},
"min": {
"format": "double",
"type": [
"number",
"null"
]
},
"unknown": {
"description": "Analyses that investigated but reported no cost.",
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"count",
"unknown"
],
"type": "object"
},
"CapacityRejectionV1": {
"additionalProperties": false,
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"reason": {
"allOf": [
{
"$ref": "#/definitions/RejectionReasonV1"
}
],
"default": "at_capacity",
"description": "Why the turn was not admitted; rows saved before the cost cap existed read as `at_capacity`."
},
"session_id": {
"type": "string"
},
"turn_id": {
"type": "string"
}
},
"required": [
"at",
"session_id",
"turn_id"
],
"type": "object"
},
"MonitorConfigV1": {
"additionalProperties": false,
"properties": {
"code_repository": {
"description": "The codebase directory the investigation reads (see `ConfigureRequestV1`).",
"type": [
"string",
"null"
]
},
"daily_cost_cap_usd": {
"description": "The daily cap on known investigation cost (see `ConfigureRequestV1`).",
"format": "double",
"type": [
"number",
"null"
]
},
"enabled": {
"type": "boolean"
},
"enabled_since": {
"description": "When automatic observation last went from paused to on; analyses use it to bound which earlier turns they may report. Unchanged by edits that keep observation on.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"$ref": "#/definitions/MonitorModelV1"
},
"revision": {
"description": "SHA-256 of the effective configuration; analyses keep the revision they were admitted with.",
"type": "string"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"enabled",
"model",
"revision",
"updated_at"
],
"type": "object"
},
"MonitorCostV1": {
"additionalProperties": false,
"description": "The monitor's own spend. Only the investigation's LLM cost is in dollars; Jev's usage is reported in tokens on each record.",
"properties": {
"cap_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"capped": {
"description": "A cap is set and `today_capture_usd`, plus the median cost of each investigation still running, reached it: automatic observation admits and investigates nothing until the next UTC day.",
"type": "boolean"
},
"per_analysis": {
"allOf": [
{
"$ref": "#/definitions/AnalysisCostStatsV1"
}
],
"description": "What one investigation has cost, from the history."
},
"since": {
"description": "Start (ms since the Unix epoch) of the UTC day the `today_*` values cover. The monitor has no timezone setting, so a day is a UTC day.",
"format": "int64",
"type": "integer"
},
"today_capture_usd": {
"description": "The capture bucket, the only one the cap compares: the larger of the day's persisted capture spend (deleting an analysis does not give it back) and the sum of the stored analyses' `usage.llm_cost_usd`. A day persisted before the buckets counts entirely as capture.",
"format": "double",
"type": "number"
},
"today_replay_unknown": {
"description": "Replay samples of the day without a cost, failed ones included. They add nothing to `today_replay_usd`, but their cost is unknown, not zero.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"today_replay_usd": {
"description": "The replay bucket: the known cost of the day's `eval::reproduce` samples. Never capped.",
"format": "double",
"type": "number"
},
"today_unknown": {
"description": "Analyses of the day whose investigation started but reported no cost. They add nothing to `today_capture_usd`, but their cost is unknown, not zero.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"today_usd": {
"description": "Everything the monitor is known to have spent since `since`: `today_capture_usd` + `today_replay_usd`. The cap does not compare this.",
"format": "double",
"type": "number"
}
},
"required": [
"capped",
"per_analysis",
"since",
"today_capture_usd",
"today_replay_unknown",
"today_replay_usd",
"today_unknown",
"today_usd"
],
"type": "object"
},
"MonitorLimitsV1": {
"additionalProperties": false,
"properties": {
"analysis_budget_ms": {
"description": "Whole-analysis budget from admission, queue wait included.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"assets_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"investigation_code_max_total_tokens": {
"description": "Total-token cap of an investigation that has code access.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"investigation_code_max_turns": {
"description": "Generate-step cap of an investigation that has code access.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"investigation_max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"investigation_max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"investigation_max_turns": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"judge_timeout_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_active_analyses": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"model_context_bytes": {
"description": "Serialized JSON shown to a model (bytes, not tokens).",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"queue_concurrency": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"retention_days": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"retention_max_terminal": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"analysis_budget_ms",
"assets_bytes",
"investigation_code_max_total_tokens",
"investigation_code_max_turns",
"investigation_max_output_tokens",
"investigation_max_total_tokens",
"investigation_max_turns",
"judge_timeout_ms",
"max_active_analyses",
"model_context_bytes",
"queue_concurrency",
"retention_days",
"retention_max_terminal"
],
"type": "object"
},
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"RejectionReasonV1": {
"oneOf": [
{
"description": "The unfinished-analyses cap was reached.",
"enum": [
"at_capacity"
],
"type": "string"
},
{
"description": "The day's known cost reached `daily_cost_cap_usd`.",
"enum": [
"cost_cap"
],
"type": "string"
}
]
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
},
"TriageAvailabilityV1": {
"additionalProperties": false,
"description": "What `judge::models::list` answered for the triage provider. The key stays in the provider; this only reports whether it answers.",
"properties": {
"available": {
"type": "boolean"
},
"checked_at": {
"format": "int64",
"type": "integer"
},
"code": {
"description": "The provider's error code (`missing_key`, `provider_unavailable`…), or `unreachable` when the hub did not answer.",
"type": [
"string",
"null"
]
},
"models": {
"items": {
"type": "string"
},
"type": "array"
},
"provider": {
"type": "string"
}
},
"required": [
"available",
"checked_at",
"models",
"provider"
],
"type": "object"
}
},
"properties": {
"config": {
"anyOf": [
{
"$ref": "#/definitions/MonitorConfigV1"
},
{
"type": "null"
}
],
"description": "`null` until the user configures the monitor."
},
"cost": {
"allOf": [
{
"$ref": "#/definitions/MonitorCostV1"
}
],
"description": "What the investigations cost so far, to decide before enabling."
},
"last_rejection": {
"anyOf": [
{
"$ref": "#/definitions/CapacityRejectionV1"
},
{
"type": "null"
}
],
"description": "The latest turn not admitted: the monitor was at capacity or the daily cost cap was reached (`reason`)."
},
"limits": {
"allOf": [
{
"$ref": "#/definitions/MonitorLimitsV1"
}
],
"description": "The limits this version enforces, so the console never restates them."
},
"observer_bound": {
"description": "True while the `harness::on-turn-completed` observer is active. False while it is pending (no harness has registered the function) or after the engine refused it (`observer_error` says why): automatic observation is unavailable even if `config.enabled` is true.",
"type": "boolean"
},
"observer_error": {
"type": [
"string",
"null"
]
},
"triage": {
"anyOf": [
{
"$ref": "#/definitions/TriageAvailabilityV1"
},
{
"type": "null"
}
],
"description": "Present when the request asked to check the triage provider."
}
},
"required": [
"cost",
"limits",
"observer_bound"
],
"title": "MonitorStateResponseV1",
"type": "object"
}
},
{
"description": "Choose the Harness model/provider the session monitor uses for investigations and enable or pause automatic observation. A new selection is checked against router::models::list; credentials stay in the providers and are never accepted here. An optional code_repository (an absolute directory on this host, normally the iii workers repository) lets the investigation read that code like a chat with the directory selected, with only read-only functions allowed (coder::search, coder::tree, coder::read-file, github::pr::list and engine::functions::info). An optional daily_cost_cap_usd pauses automatic observation for the rest of the UTC day once the known investigation cost reaches it (manual analyses are never refused, and replay spend is reported but not counted). Analyses already admitted keep the configuration they started with.",
"metadata": {},
"name": "eval::configure",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
}
},
"properties": {
"_caller_worker_id": {
"description": "Stamped by the engine on every invocation; callers omit it. Listed so the closed schema still accepts the engine's metadata.",
"type": [
"string",
"null"
],
"writeOnly": true
},
"code_repository": {
"description": "Absolute path of the codebase directory the investigation reads and runs in (normally the iii workers repository, checked out on the monitor's host); absent means no code access.",
"type": [
"string",
"null"
]
},
"daily_cost_cap_usd": {
"description": "US dollars of known investigation cost per UTC day after which automatic observation admits nothing until the next day; absent means no cap. A manual `eval::analyze-session` is never refused by it.",
"format": "double",
"type": [
"number",
"null"
]
},
"enabled": {
"type": "boolean"
},
"model": {
"$ref": "#/definitions/MonitorModelV1"
}
},
"required": [
"enabled",
"model"
],
"title": "ConfigureRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
}
},
"properties": {
"code_repository": {
"description": "The codebase directory the investigation reads (see `ConfigureRequestV1`).",
"type": [
"string",
"null"
]
},
"daily_cost_cap_usd": {
"description": "The daily cap on known investigation cost (see `ConfigureRequestV1`).",
"format": "double",
"type": [
"number",
"null"
]
},
"enabled": {
"type": "boolean"
},
"enabled_since": {
"description": "When automatic observation last went from paused to on; analyses use it to bound which earlier turns they may report. Unchanged by edits that keep observation on.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"$ref": "#/definitions/MonitorModelV1"
},
"revision": {
"description": "SHA-256 of the effective configuration; analyses keep the revision they were admitted with.",
"type": "string"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"enabled",
"model",
"revision",
"updated_at"
],
"title": "MonitorConfigV1",
"type": "object"
}
},
{
"description": "Delete a terminal analysis and its stored previews. The turn stays marked as analyzed until retention, so it is not re-admitted automatically.",
"metadata": {},
"name": "eval::delete",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"deleted": {
"type": "boolean"
}
},
"required": [
"deleted"
],
"title": "EvalDeleteResponseV1",
"type": "object"
}
},
{
"description": "List recent monitor analyses as compact records, newest first, without evidence; observation_key keeps only the analyses of one observed turn.",
"metadata": {},
"name": "eval::list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"limit": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"observation_key": {
"default": null,
"description": "Only the analyses of this observed turn (the `observation_key` of a record), reanalyses included.",
"type": [
"string",
"null"
]
}
},
"title": "EvalListRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"AnalysisCountersV1": {
"additionalProperties": false,
"properties": {
"diagnostics": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"entries": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rejected_suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"sessions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"validations": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"diagnostics",
"entries",
"rejected_suggestions",
"sessions",
"suggestions",
"validations"
],
"type": "object"
},
"AnalysisOriginV1": {
"enum": [
"automatic",
"manual"
],
"type": "string"
},
"AnalysisRecordV1": {
"additionalProperties": false,
"description": "The compact analysis record listed by `eval::list`; evidence and model outputs live separately in the analysis assets.",
"properties": {
"analyst": {
"anyOf": [
{
"$ref": "#/definitions/AnalystRefV1"
},
{
"type": "null"
}
]
},
"code_root": {
"description": "Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.",
"type": [
"string",
"null"
]
},
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"config_revision": {
"type": "string"
},
"counters": {
"$ref": "#/definitions/AnalysisCountersV1"
},
"coverage": {
"anyOf": [
{
"$ref": "#/definitions/CoverageLevelV1"
},
{
"type": "null"
}
]
},
"created_at": {
"format": "int64",
"type": "integer"
},
"criteria_version": {
"type": "string"
},
"deadline": {
"format": "int64",
"type": "integer"
},
"evaluation_id": {
"type": "string"
},
"failure": {
"anyOf": [
{
"$ref": "#/definitions/FailureV1"
},
{
"type": "null"
}
]
},
"harness_version": {
"description": "The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.",
"type": [
"string",
"null"
]
},
"judge_call": {
"anyOf": [
{
"$ref": "#/definitions/JudgeCallV1"
},
{
"type": "null"
}
]
},
"model": {
"allOf": [
{
"$ref": "#/definitions/MonitorModelV1"
}
],
"description": "Frozen at admission: later configuration changes never alter it."
},
"observation_key": {
"type": "string"
},
"observe_since": {
"description": "Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.",
"format": "int64",
"type": "integer"
},
"origin": {
"$ref": "#/definitions/AnalysisOriginV1"
},
"pending_reason": {
"description": "Why collection is still waiting (descendants running, metrics incomplete).",
"type": [
"string",
"null"
]
},
"routing": {
"anyOf": [
{
"$ref": "#/definitions/RoutingV1"
},
{
"type": "null"
}
]
},
"rules_version": {
"type": "string"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"session_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).",
"type": "object"
},
"source_title": {
"description": "The session's title, once collection has read it.",
"type": [
"string",
"null"
]
},
"stages": {
"description": "When each status began, oldest first.",
"items": {
"$ref": "#/definitions/StageTimeV1"
},
"type": "array"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"supersedes": {
"description": "The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.",
"type": [
"string",
"null"
]
},
"turn_id": {
"type": "string"
},
"updated_at": {
"format": "int64",
"type": "integer"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/MonitorUsageV1"
}
],
"description": "The monitor's own consumption, apart from the observed task."
}
},
"required": [
"config_revision",
"counters",
"created_at",
"criteria_version",
"deadline",
"evaluation_id",
"model",
"observation_key",
"observe_since",
"origin",
"rules_version",
"schema_version",
"session_id",
"stages",
"status",
"step",
"turn_id",
"updated_at",
"usage"
],
"type": "object"
},
"AnalystRefV1": {
"additionalProperties": false,
"properties": {
"sent_at": {
"format": "int64",
"type": "integer"
},
"session_id": {
"type": "string"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"sent_at",
"session_id"
],
"type": "object"
},
"CoverageLevelV1": {
"oneOf": [
{
"description": "Every in-scope transcript was read and the model context holds every diagnostic's evidence.",
"enum": [
"complete"
],
"type": "string"
},
{
"description": "Previews omit or shorten entries; the full transcripts were read.",
"enum": [
"partial"
],
"type": "string"
},
{
"description": "Required evidence did not fit; no conclusion of healthy behavior.",
"enum": [
"insufficient"
],
"type": "string"
}
]
},
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"FailureV1": {
"additionalProperties": false,
"description": "The stage that failed and why. `stage` is the status the analysis was in.",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
},
"stage": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"code",
"message",
"stage"
],
"type": "object"
},
"JudgeCallV1": {
"additionalProperties": false,
"description": "A Jev call started by this analysis. Persisted before the call so a restart can tell a lost answer from one never requested.",
"properties": {
"deadline": {
"format": "int64",
"type": "integer"
},
"request_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"deadline",
"request_id",
"started_at"
],
"type": "object"
},
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"MonitorUsageV1": {
"additionalProperties": false,
"description": "Known usage only: absent counters are unknown, never zero.",
"properties": {
"judge_calls": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"judge_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_usage_complete": {
"description": "False when Jev reported incomplete usage or failed.",
"type": "boolean"
},
"llm_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"llm_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"llm_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"judge_calls",
"judge_input_tokens",
"judge_output_tokens",
"judge_usage_complete"
],
"type": "object"
},
"RoutingReasonV1": {
"oneOf": [
{
"enum": [
"needs_investigation"
],
"type": "string"
},
{
"description": "Legacy: no longer produced; kept so stored records still deserialize.",
"enum": [
"diagnostics"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"insufficient_evidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"low_confidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"coverage_insufficient"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"audit_sample"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"manual_request"
],
"type": "string"
}
]
},
"RoutingV1": {
"additionalProperties": false,
"properties": {
"investigate": {
"type": "boolean"
},
"reasons": {
"items": {
"$ref": "#/definitions/RoutingReasonV1"
},
"type": "array"
}
},
"required": [
"investigate",
"reasons"
],
"type": "object"
},
"StageTimeV1": {
"additionalProperties": false,
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"at",
"status"
],
"type": "object"
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
}
},
"properties": {
"evaluations": {
"items": {
"$ref": "#/definitions/AnalysisRecordV1"
},
"type": "array"
}
},
"required": [
"evaluations"
],
"title": "EvalListResponseV1",
"type": "object"
}
},
{
"description": "Called when a session-monitor analysis reaches a terminal status (completed, failed or cancelled). No args; filter on input.evaluation_id or input.status. Completion describes the monitor, not a validated improvement.",
"metadata": {},
"name": "eval::on-completed",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"description": "The `eval::on-completed` event.",
"properties": {
"evaluation_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"timestamp": {
"description": "Unix milliseconds when the analysis ended.",
"format": "int64",
"type": "integer"
}
},
"required": [
"evaluation_id",
"status",
"timestamp"
],
"title": "CompletedEventV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"description": "The `eval::on-completed` event.",
"properties": {
"evaluation_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"timestamp": {
"description": "Unix milliseconds when the analysis ended.",
"format": "int64",
"type": "integer"
}
},
"required": [
"evaluation_id",
"status",
"timestamp"
],
"title": "CompletedEventV1",
"type": "object"
}
},
{
"description": "Internal: admission and wake-up for harness::on-turn-completed.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::on-turn-completed",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Caller": {
"description": "Who made an invocation. `worker_id` / `worker_name` are stamped by the engine from the calling connection; `function_id`, `invocation_id` and `parent` are asserted by the calling SDK.",
"properties": {
"function_id": {
"description": "The handler the call was made from; absent for a top-level call.",
"type": [
"string",
"null"
]
},
"invocation_id": {
"type": [
"string",
"null"
]
},
"parent": {
"anyOf": [
{
"$ref": "#/definitions/ParentCaller"
},
{
"type": "null"
}
],
"description": "The caller of that handler, one level only."
},
"worker_id": {
"type": [
"string",
"null"
]
},
"worker_name": {
"type": [
"string",
"null"
]
}
},
"type": "object"
},
"InvocationStatus": {
"enum": [
"success",
"error"
],
"type": "string"
},
"ObservedError": {
"description": "What a call failed with: [`Observed::error`], [`Call::error`](crate::Call::error).",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
},
"required": [
"code",
"message"
],
"type": "object"
},
"ParentCaller": {
"description": "[`Caller::parent`]: a caller without a parent of its own.",
"properties": {
"function_id": {
"type": [
"string",
"null"
]
},
"invocation_id": {
"type": [
"string",
"null"
]
},
"worker_id": {
"type": [
"string",
"null"
]
},
"worker_name": {
"type": [
"string",
"null"
]
}
},
"type": "object"
},
"WakeEventV1": {
"description": "The `harness::on-turn-completed` event; extra fields are tolerated.",
"properties": {
"parent": {
"default": null
},
"parent_session_id": {
"default": null,
"type": [
"string",
"null"
]
},
"session_id": {
"default": "",
"type": "string"
},
"terminal": {
"default": false,
"type": "boolean"
},
"timestamp": {
"default": null,
"format": "int64",
"type": [
"integer",
"null"
]
},
"turn_id": {
"default": "",
"type": "string"
}
},
"type": "object"
}
},
"description": "What an observer's target receives when iii delivers: the observed call, per the observer's `capture`.\n\n```rust,no_run # use iii_sdk::{Error, IIIClient, Observed, RegisterFunction}; # use serde_json::Value; # fn demo(iii: &IIIClient) { iii.register_function( \"alerts::payment-failed\", RegisterFunction::new(|event: Observed<Value>| { println!(\"{} failed: {:?}\", event.function_id, event.error); Ok::<_, Error>(()) }), ); # } ```",
"properties": {
"activation_id": {
"type": "string"
},
"caller": {
"anyOf": [
{
"$ref": "#/definitions/Caller"
},
{
"type": "null"
}
]
},
"duration_ms": {
"format": "double",
"type": "number"
},
"error": {
"anyOf": [
{
"$ref": "#/definitions/ObservedError"
},
{
"type": "null"
}
]
},
"function_id": {
"type": "string"
},
"input": true,
"invocation_id": {
"type": "string"
},
"metadata": true,
"namespace": {
"type": [
"string",
"null"
]
},
"observer_id": {
"type": "string"
},
"output": {
"anyOf": [
{
"$ref": "#/definitions/WakeEventV1"
},
{
"type": "null"
}
]
},
"started_at_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/definitions/InvocationStatus"
}
},
"required": [
"activation_id",
"duration_ms",
"function_id",
"invocation_id",
"observer_id",
"started_at_ms",
"status"
],
"title": "Observed_for_AnyValue_and_WakeEventV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"WakeOutcomeV1": {
"enum": [
"ignored",
"progress",
"descendant",
"monitor_session",
"disabled",
"stale",
"admitted",
"reused",
"at_capacity",
"cost_cap",
"not_user_chat"
],
"type": "string"
}
},
"properties": {
"evaluation_id": {
"type": [
"string",
"null"
]
},
"outcome": {
"$ref": "#/definitions/WakeOutcomeV1"
}
},
"required": [
"outcome"
],
"title": "WakeResponseV1",
"type": "object"
}
},
{
"description": "For a suggestion marked shipped with its version, how often its patterns were found per analysis on Harness versions before it against from it on. Analyses without a semantic Harness version are left out and counted.",
"metadata": {},
"name": "eval::recurrence",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"evaluation_id",
"suggestion_index"
],
"title": "RecurrenceRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"RecurrencePatternV1": {
"additionalProperties": false,
"description": "One pattern over a set of analyses.",
"properties": {
"analyses_with": {
"description": "Analyses with at least one occurrence.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"occurrences": {
"description": "Occurrences summed over the analyses.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": "string"
},
"per_analysis": {
"description": "Occurrences per analysis; absent when there is no analysis.",
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"analyses_with",
"occurrences",
"pattern"
],
"type": "object"
},
"RecurrenceWindowV1": {
"additionalProperties": false,
"properties": {
"analyses": {
"description": "Analyses whose collection finished and whose Harness version is known.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"patterns": {
"items": {
"$ref": "#/definitions/RecurrencePatternV1"
},
"type": "array"
}
},
"required": [
"analyses",
"patterns"
],
"type": "object"
}
},
"description": "Whether the suggestion's patterns came back: the analyses on Harness versions before the one that shipped it against the ones from it on. Analyses are kept 30 days, so the earlier window is bounded by retention.",
"properties": {
"before": {
"$ref": "#/definitions/RecurrenceWindowV1"
},
"evaluation_id": {
"type": "string"
},
"from_version": {
"$ref": "#/definitions/RecurrenceWindowV1"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"description": "The version the suggestion shipped in, where the windows split.",
"type": "string"
},
"without_version": {
"description": "Analyses left out because they carry no (or no semantic) Harness version.",
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"before",
"evaluation_id",
"from_version",
"suggestion_index",
"version",
"without_version"
],
"title": "RecurrenceResponseV1",
"type": "object"
}
},
{
"description": "Replay the decision point of a suggestion: rebuild the request the model received at that step of the observed turn (optionally with edits that stand for the proposed change), sample the next reply N times (default 20) without running any function, and read the suggestion's signal in each reply. Spends model tokens; with dry_run it only rebuilds the request, checks its fidelity against the recorded usage and reports the original reply. extend adds replies to an earlier reproduction, or with samples 0 finishes one that failed. Results land in eval::result reviews.",
"metadata": {},
"name": "eval::reproduce",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"ReproduceChangeV1": {
"description": "What a reproduction changes in the request.",
"oneOf": [
{
"properties": {
"kind": {
"enum": [
"none"
],
"type": "string"
}
},
"required": [
"kind"
],
"type": "object"
},
{
"properties": {
"kind": {
"enum": [
"proposed"
],
"type": "string"
}
},
"required": [
"kind"
],
"type": "object"
},
{
"properties": {
"edits": {
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"kind": {
"enum": [
"custom"
],
"type": "string"
}
},
"required": [
"edits",
"kind"
],
"type": "object"
}
]
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
}
},
"description": "`eval::reproduce`.",
"properties": {
"_caller_worker_id": {
"default": null,
"description": "Stamped by the engine on every invocation; callers omit it.",
"type": [
"string",
"null"
]
},
"by": {
"default": null,
"type": [
"string",
"null"
]
},
"change": {
"allOf": [
{
"$ref": "#/definitions/ReproduceChangeV1"
}
],
"default": {
"kind": "none"
}
},
"check": {
"anyOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
},
{
"type": "null"
}
],
"default": null,
"description": "The decision point and signal, for a suggestion without a `check` (an older analysis) or to override it."
},
"dry_run": {
"default": false,
"description": "Rebuild the request and check its fidelity without sampling: nothing is spent or stored.",
"type": "boolean"
},
"evaluation_id": {
"type": "string"
},
"extend": {
"default": null,
"description": "Add replies to this earlier reproduction of the suggestion, with its change; with `samples: 0`, only finish what a failure left undone.",
"type": [
"string",
"null"
]
},
"samples": {
"default": null,
"description": "Replies to sample, 1–50; default 20.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"evaluation_id",
"suggestion_index"
],
"title": "ReproduceRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"CriterionV1": {
"additionalProperties": false,
"properties": {
"direction": {
"$ref": "#/definitions/DirectionV1"
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"format": "double",
"type": "number"
},
"min_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": [
"string",
"null"
]
},
"registered_at": {
"format": "int64",
"type": "integer"
},
"registered_by": {
"type": "string"
},
"scenario_id": {
"description": "The scenario whose runs are measured.",
"type": "string"
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs",
"registered_at",
"registered_by",
"scenario_id"
],
"type": "object"
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"EvidenceRunV1": {
"additionalProperties": false,
"description": "One run of the scenario as the E2E reported it. Runs the infrastructure broke are listed with the E2E's own words, never counted.",
"properties": {
"completed": {
"description": "The E2E reported the run complete with a valid technical outcome.",
"type": "boolean"
},
"completion": {
"description": "`completion` as the E2E reported it.",
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.",
"type": [
"object",
"null"
]
},
"status": {
"description": "`passed` or `failed`, the run's task outcome.",
"type": [
"string",
"null"
]
},
"technical": {
"description": "`technical` as the E2E reported it; anything but `valid` is an infrastructure failure.",
"type": [
"string",
"null"
]
},
"total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"completed",
"run_id"
],
"type": "object"
},
"EvidenceSideV1": {
"additionalProperties": false,
"properties": {
"execution_id": {
"type": "string"
},
"mean": {
"description": "Mean of the criterion's metric over those runs; absent without a criterion or without such runs.",
"format": "double",
"type": [
"number",
"null"
]
},
"n": {
"description": "Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"description": "Every run of the scenario the execution holds.",
"items": {
"$ref": "#/definitions/EvidenceRunV1"
},
"type": "array"
}
},
"required": [
"execution_id",
"n",
"runs"
],
"type": "object"
},
"EvidenceV1": {
"additionalProperties": false,
"description": "Computed in code from the attached pair's runs, never by a model. It is what the person's verdict is shown next to.",
"properties": {
"baseline": {
"$ref": "#/definitions/EvidenceSideV1"
},
"candidate": {
"$ref": "#/definitions/EvidenceSideV1"
},
"computed_at": {
"format": "int64",
"type": "integer"
},
"computed_outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"baseline",
"candidate",
"computed_at",
"computed_outcome",
"reason",
"scenario_id"
],
"type": "object"
},
"FidelityLevelV1": {
"oneOf": [
{
"enum": [
"approximate"
],
"type": "string"
},
{
"description": "The provider counts the rebuilt request like the original, up to a fixed per-request overhead measured at the turn's first step.",
"enum": [
"exact"
],
"type": "string"
}
]
},
"FidelityV1": {
"additionalProperties": false,
"description": "How faithfully the request at the decision point was rebuilt.",
"properties": {
"counted_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"estimator": {
"description": "`provider` when the provider counted, else the estimator's name.",
"type": [
"string",
"null"
]
},
"level": {
"$ref": "#/definitions/FidelityLevelV1"
},
"off_ratio": {
"description": "The unexplained difference over the recorded tokens.",
"format": "double",
"type": [
"number",
"null"
]
},
"reasons": {
"default": [],
"description": "Why the rebuild is approximate; empty when exact.",
"items": {
"type": "string"
},
"type": "array"
},
"recorded_tokens": {
"description": "Input tokens of the original step: input + cache read + cache write.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"level"
],
"type": "object"
},
"LifecycleEventV1": {
"additionalProperties": false,
"description": "One change of status, with who made it.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"note": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
}
},
"required": [
"at",
"by",
"status"
],
"type": "object"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"LifecycleV1": {
"additionalProperties": false,
"properties": {
"duplicate_of": {
"description": "What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).",
"type": [
"string",
"null"
]
},
"history": {
"description": "Every change of status, oldest first; empty while `new`.",
"items": {
"$ref": "#/definitions/LifecycleEventV1"
},
"type": "array"
},
"pr": {
"description": "The pull request that implements it (`in_progress`, `shipped`).",
"type": [
"string",
"null"
]
},
"reason": {
"description": "Why it was rejected (`rejected`).",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
},
"version": {
"description": "The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.",
"type": [
"string",
"null"
]
}
},
"required": [
"history",
"status"
],
"type": "object"
},
"ReplyCallV1": {
"additionalProperties": false,
"description": "One function call of a reply.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"payload": {
"description": "The payload as JSON text, cut at 600 characters.",
"type": "string"
},
"target": {
"description": "The function called (through `agent_call` or directly).",
"type": "string"
}
},
"required": [
"payload",
"target"
],
"type": "object"
},
"ReplyUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ReplyV1": {
"additionalProperties": false,
"description": "One reply at the decision point: a sample, or the original step read the same way.",
"properties": {
"calls": {
"items": {
"$ref": "#/definitions/ReplyCallV1"
},
"type": "array"
},
"duration_ms": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"index": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"signal": {
"description": "Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.",
"type": [
"boolean",
"null"
]
},
"text": {
"type": "string"
},
"thinking": {
"description": "The opening of the reply's reasoning and of its text, 400 characters each.",
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/ReplyUsageV1"
}
],
"default": {}
}
},
"required": [
"calls",
"index",
"text",
"thinking"
],
"type": "object"
},
"ReproducePreviewV1": {
"additionalProperties": false,
"description": "What a dry run reports before anything is spent.",
"properties": {
"fidelity": {
"$ref": "#/definitions/FidelityV1"
},
"model": {
"type": "string"
},
"original": {
"$ref": "#/definitions/ReplyV1"
},
"provider": {
"type": [
"string",
"null"
]
},
"samples": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"step_cost_usd": {
"description": "What the original step cost; null when the provider reported nothing.",
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"fidelity",
"model",
"original",
"samples"
],
"type": "object"
},
"ReproductionChangeKindV1": {
"oneOf": [
{
"description": "The context as the model saw it.",
"enum": [
"none"
],
"type": "string"
},
{
"description": "The suggestion's own `check.change`.",
"enum": [
"proposed"
],
"type": "string"
},
{
"description": "Edits a person wrote.",
"enum": [
"custom"
],
"type": "string"
}
]
},
"ReproductionStateV1": {
"enum": [
"running",
"completed",
"failed"
],
"type": "string"
},
"ReproductionV1": {
"additionalProperties": false,
"description": "A replay of the decision point: N replies sampled from the rebuilt request, with or without a change. No function ever runs.",
"properties": {
"by": {
"type": "string"
},
"change": {
"default": [],
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"change_kind": {
"$ref": "#/definitions/ReproductionChangeKindV1"
},
"check": {
"allOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
}
],
"description": "The decision point and signal this replay used."
},
"cost_unknown_samples": {
"default": 0,
"description": "Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost_usd": {
"description": "Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.",
"format": "double",
"type": [
"number",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"fidelity": {
"anyOf": [
{
"$ref": "#/definitions/FidelityV1"
},
{
"type": "null"
}
]
},
"finished_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"judge_input_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"original": {
"anyOf": [
{
"$ref": "#/definitions/ReplyV1"
},
{
"type": "null"
}
],
"description": "The original reply at the decision point, read the same way."
},
"phase": {
"description": "`sampling` or `classifying` while running.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"requested": {
"description": "Replies asked for, extensions included.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"samples": {
"default": [],
"items": {
"$ref": "#/definitions/ReplyV1"
},
"type": "array"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ReproductionStateV1"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"by",
"change_kind",
"check",
"id",
"model",
"requested",
"started_at",
"state",
"updated_at"
],
"type": "object"
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
},
"SuggestionReviewV1": {
"additionalProperties": false,
"description": "Everything people decided about one suggestion, in the `eval_suggestion` state scope. The title, component and patterns are copied from the suggestion so the row outlives the analysis's retention.",
"properties": {
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"evaluation_id": {
"type": "string"
},
"evidence": {
"anyOf": [
{
"$ref": "#/definitions/EvidenceV1"
},
{
"type": "null"
}
]
},
"first_run_at": {
"description": "When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"harness_component": {
"type": "string"
},
"lifecycle": {
"$ref": "#/definitions/LifecycleV1"
},
"patterns": {
"description": "`<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.",
"items": {
"type": "string"
},
"type": "array"
},
"reproductions": {
"description": "Replays of the decision point, oldest first: the base and the tested changes.",
"items": {
"$ref": "#/definitions/ReproductionV1"
},
"type": "array"
},
"run": {
"anyOf": [
{
"$ref": "#/definitions/ValidationRunV1"
},
{
"type": "null"
}
]
},
"scenario_id": {
"description": "The scenario the suggestion's validation plan names.",
"type": [
"string",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"type": "string"
},
"verdict": {
"anyOf": [
{
"$ref": "#/definitions/VerdictV1"
},
{
"type": "null"
}
]
}
},
"required": [
"evaluation_id",
"harness_component",
"lifecycle",
"patterns",
"suggestion_index",
"title"
],
"type": "object"
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
},
"ValidationRunStateV1": {
"oneOf": [
{
"enum": [
"failed"
],
"type": "string"
},
{
"description": "Recorded before the executions are requested.",
"enum": [
"starting"
],
"type": "string"
},
{
"description": "Both executions were requested; the sweep waits for them.",
"enum": [
"running"
],
"type": "string"
},
{
"description": "Both ended; the pair is about to be attached.",
"enum": [
"finished"
],
"type": "string"
},
{
"description": "The pair is attached to the analysis and its evidence computed.",
"enum": [
"attached"
],
"type": "string"
}
]
},
"ValidationRunV1": {
"additionalProperties": false,
"description": "The baseline and candidate E2E executions this worker started for a suggestion (`eval::start-validation`).",
"properties": {
"baseline_commit": {
"description": "The full commits the Harness was built from.",
"type": "string"
},
"baseline_execution_id": {
"description": "Absent until the E2E accepted the execution.",
"type": [
"string",
"null"
]
},
"candidate_commit": {
"type": "string"
},
"candidate_execution_id": {
"type": [
"string",
"null"
]
},
"error": {
"description": "Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).",
"type": [
"string",
"null"
]
},
"finished_at": {
"description": "When both executions were seen ended; the sweep stops trying to attach the pair some time after.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"description": "Runs requested for each side.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ValidationRunStateV1"
}
},
"required": [
"baseline_commit",
"candidate_commit",
"model",
"provider",
"runs",
"scenario_id",
"started_at",
"state"
],
"type": "object"
},
"VerdictV1": {
"additionalProperties": false,
"description": "The person's judgment, kept with the criterion it was made against.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"controls_checked": {
"description": "The non-regression controls of the plan that were checked.",
"items": {
"type": "string"
},
"type": "array"
},
"criterion_snapshot": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"rationale": {
"type": "string"
}
},
"required": [
"at",
"by",
"controls_checked",
"outcome",
"rationale"
],
"type": "object"
}
},
"properties": {
"preview": {
"anyOf": [
{
"$ref": "#/definitions/ReproducePreviewV1"
},
{
"type": "null"
}
]
},
"reproduction_id": {
"type": [
"string",
"null"
]
},
"review": {
"anyOf": [
{
"$ref": "#/definitions/SuggestionReviewV1"
},
{
"type": "null"
}
]
}
},
"title": "ReproduceResponseV1",
"type": "object"
}
},
{
"description": "Read an analysis with its captured evidence, deterministic diagnostics, Jev triage, suggestions, the monitor's own consumption and linked E2E executions, or null. A completed analysis is not a validated improvement.",
"metadata": {},
"name": "eval::result",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"anyOf": [
{
"$ref": "#/definitions/EvalResultResponseV1"
},
{
"type": "null"
}
],
"definitions": {
"AnalysisAssetsV1": {
"additionalProperties": false,
"properties": {
"capture": {
"anyOf": [
{
"$ref": "#/definitions/TurnCaptureV1"
},
{
"type": "null"
}
],
"description": "The observed turn's record, copied when the evidence was captured: what a reproduction rebuilds the request from."
},
"evaluation_id": {
"type": "string"
},
"investigation": {
"anyOf": [
{
"$ref": "#/definitions/InvestigationV1"
},
{
"type": "null"
}
]
},
"snapshot": {
"anyOf": [
{
"$ref": "#/definitions/SnapshotV1"
},
{
"type": "null"
}
]
},
"triage": {
"anyOf": [
{
"$ref": "#/definitions/TriageV1"
},
{
"type": "null"
}
]
},
"triage_failure": {
"anyOf": [
{
"$ref": "#/definitions/TriageFailureV1"
},
{
"type": "null"
}
]
},
"validations": {
"default": [],
"items": {
"$ref": "#/definitions/ValidationLinkV1"
},
"type": "array"
}
},
"required": [
"evaluation_id"
],
"type": "object"
},
"AnalysisCountersV1": {
"additionalProperties": false,
"properties": {
"diagnostics": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"entries": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rejected_suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"sessions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"validations": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"diagnostics",
"entries",
"rejected_suggestions",
"sessions",
"suggestions",
"validations"
],
"type": "object"
},
"AnalysisOriginV1": {
"enum": [
"automatic",
"manual"
],
"type": "string"
},
"AnalysisRecordV1": {
"additionalProperties": false,
"description": "The compact analysis record listed by `eval::list`; evidence and model outputs live separately in the analysis assets.",
"properties": {
"analyst": {
"anyOf": [
{
"$ref": "#/definitions/AnalystRefV1"
},
{
"type": "null"
}
]
},
"code_root": {
"description": "Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.",
"type": [
"string",
"null"
]
},
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"config_revision": {
"type": "string"
},
"counters": {
"$ref": "#/definitions/AnalysisCountersV1"
},
"coverage": {
"anyOf": [
{
"$ref": "#/definitions/CoverageLevelV1"
},
{
"type": "null"
}
]
},
"created_at": {
"format": "int64",
"type": "integer"
},
"criteria_version": {
"type": "string"
},
"deadline": {
"format": "int64",
"type": "integer"
},
"evaluation_id": {
"type": "string"
},
"failure": {
"anyOf": [
{
"$ref": "#/definitions/FailureV1"
},
{
"type": "null"
}
]
},
"harness_version": {
"description": "The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.",
"type": [
"string",
"null"
]
},
"judge_call": {
"anyOf": [
{
"$ref": "#/definitions/JudgeCallV1"
},
{
"type": "null"
}
]
},
"model": {
"allOf": [
{
"$ref": "#/definitions/MonitorModelV1"
}
],
"description": "Frozen at admission: later configuration changes never alter it."
},
"observation_key": {
"type": "string"
},
"observe_since": {
"description": "Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.",
"format": "int64",
"type": "integer"
},
"origin": {
"$ref": "#/definitions/AnalysisOriginV1"
},
"pending_reason": {
"description": "Why collection is still waiting (descendants running, metrics incomplete).",
"type": [
"string",
"null"
]
},
"routing": {
"anyOf": [
{
"$ref": "#/definitions/RoutingV1"
},
{
"type": "null"
}
]
},
"rules_version": {
"type": "string"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"session_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).",
"type": "object"
},
"source_title": {
"description": "The session's title, once collection has read it.",
"type": [
"string",
"null"
]
},
"stages": {
"description": "When each status began, oldest first.",
"items": {
"$ref": "#/definitions/StageTimeV1"
},
"type": "array"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"supersedes": {
"description": "The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.",
"type": [
"string",
"null"
]
},
"turn_id": {
"type": "string"
},
"updated_at": {
"format": "int64",
"type": "integer"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/MonitorUsageV1"
}
],
"description": "The monitor's own consumption, apart from the observed task."
}
},
"required": [
"config_revision",
"counters",
"created_at",
"criteria_version",
"deadline",
"evaluation_id",
"model",
"observation_key",
"observe_since",
"origin",
"rules_version",
"schema_version",
"session_id",
"stages",
"status",
"step",
"turn_id",
"updated_at",
"usage"
],
"type": "object"
},
"AnalystRefV1": {
"additionalProperties": false,
"properties": {
"sent_at": {
"format": "int64",
"type": "integer"
},
"session_id": {
"type": "string"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"sent_at",
"session_id"
],
"type": "object"
},
"Answer": {
"oneOf": [
{
"additionalProperties": false,
"properties": {
"noul": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"noul"
],
"type": "string"
}
},
"required": [
"noul",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"choice": {
"type": "string"
},
"confidence": {
"format": "double",
"type": "number"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"type": {
"enum": [
"choice"
],
"type": "string"
}
},
"required": [
"choice",
"confidence",
"probabilities",
"type"
],
"type": "object"
},
{
"additionalProperties": false,
"properties": {
"confidence": {
"format": "double",
"type": "number"
},
"legend": {
"additionalProperties": {
"$ref": "#/definitions/ScoreLevel"
},
"type": "object"
},
"probabilities": {
"additionalProperties": {
"format": "double",
"type": "number"
},
"type": "object"
},
"score": {
"format": "double",
"type": "number"
},
"type": {
"enum": [
"score"
],
"type": "string"
}
},
"required": [
"confidence",
"legend",
"probabilities",
"score",
"type"
],
"type": "object"
}
]
},
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"CodeRefV1": {
"additionalProperties": false,
"description": "Lines of one file of the code directory.",
"properties": {
"line_from": {
"description": "First line, 1-based and inclusive.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"line_to": {
"description": "Last line, inclusive; at most the file's line count.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"path": {
"description": "Relative to the code directory, e.g. `harness/src/registry.rs` or `context-manager/src/prune.rs`; never absolute and never with `..`.",
"type": "string"
}
},
"required": [
"line_from",
"line_to",
"path"
],
"type": "object"
},
"ComparabilityCheckV1": {
"additionalProperties": false,
"properties": {
"baseline": {
"type": [
"string",
"null"
]
},
"candidate": {
"type": [
"string",
"null"
]
},
"field": {
"type": "string"
},
"matches": {
"type": "boolean"
}
},
"required": [
"field",
"matches"
],
"type": "object"
},
"ComparabilityV1": {
"additionalProperties": false,
"description": "The conditions that must match for a fair comparison; only the Harness version is meant to differ.",
"properties": {
"checks": {
"items": {
"$ref": "#/definitions/ComparabilityCheckV1"
},
"type": "array"
},
"comparable": {
"type": "boolean"
}
},
"required": [
"checks",
"comparable"
],
"type": "object"
},
"ContextSnapshotV1": {
"description": "One generation's context accounting. `free = usable - total`, floored at zero: once provider usage lands, `total` is what was billed, which can exceed the `usable` budget the window was fit into — that budget was derived before the generation from an estimate.",
"properties": {
"categories": {
"$ref": "#/definitions/SnapshotCategoriesV1"
},
"compacted": {
"description": "The context carries a conversation summary: compacted on this step, or anchored on an earlier compaction entry (a previous step, the console's `/compact`). The summary's tokens count in `system_prompt`.",
"type": "boolean"
},
"effective_max_output_tokens": {
"description": "Output allocation `usable` was derived against.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"estimator": {
"description": "Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.",
"type": [
"string",
"null"
]
},
"free": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"prefix_divergences": {
"description": "Already-sent rows this session's requests rewrote, newest last (at most [`MAX_PREFIX_DIVERGENCES`]), carried from step to step.",
"items": {
"$ref": "#/definitions/PrefixDivergenceV1"
},
"type": "array"
},
"prompt_sections_fallback": {
"description": "Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).",
"type": [
"string",
"null"
]
},
"prompt_surface_digest": {
"description": "Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"session_cost_usd": {
"description": "Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.",
"format": "double",
"type": [
"number",
"null"
]
},
"session_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"summarized_head_tokens": {
"description": "Size of the history that summary replaced.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"timestamp": {
"format": "int64",
"type": "integer"
},
"total": {
"description": "Final request estimate: categories plus post-assembly growth.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
},
"usable": {
"description": "The input budget the window was fit into.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"usage": {
"anyOf": [
{
"$ref": "#/definitions/Usage"
},
{
"type": "null"
}
],
"description": "Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed)."
}
},
"required": [
"categories",
"compacted",
"effective_max_output_tokens",
"free",
"model",
"session_id",
"step",
"timestamp",
"total",
"turn_id",
"usable"
],
"type": "object"
},
"CorrelationV1": {
"oneOf": [
{
"enum": [
"unknown"
],
"type": "string"
},
{
"description": "A `registry-changed` notice reached the model between the sources and the repeated call. Temporal correlation, not a proven cause.",
"enum": [
"harness_notice_correlated"
],
"type": "string"
}
]
},
"CoverageLevelV1": {
"oneOf": [
{
"description": "Every in-scope transcript was read and the model context holds every diagnostic's evidence.",
"enum": [
"complete"
],
"type": "string"
},
{
"description": "Previews omit or shorten entries; the full transcripts were read.",
"enum": [
"partial"
],
"type": "string"
},
{
"description": "Required evidence did not fit; no conclusion of healthy behavior.",
"enum": [
"insufficient"
],
"type": "string"
}
]
},
"CoverageV1": {
"additionalProperties": false,
"properties": {
"context_bytes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"diagnostics_in_context": {
"description": "Diagnostics shown to the models; all of them stay in the snapshot.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"entries_read": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"level": {
"$ref": "#/definitions/CoverageLevelV1"
},
"limitations": {
"items": {
"type": "string"
},
"type": "array"
},
"sessions_in_scope": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"sessions_out_of_scope": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"context_bytes",
"diagnostics_in_context",
"entries_read",
"level",
"limitations",
"sessions_in_scope",
"sessions_out_of_scope"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"CriterionV1": {
"additionalProperties": false,
"properties": {
"direction": {
"$ref": "#/definitions/DirectionV1"
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"format": "double",
"type": "number"
},
"min_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": [
"string",
"null"
]
},
"registered_at": {
"format": "int64",
"type": "integer"
},
"registered_by": {
"type": "string"
},
"scenario_id": {
"description": "The scenario whose runs are measured.",
"type": "string"
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs",
"registered_at",
"registered_by",
"scenario_id"
],
"type": "object"
},
"DiagnosticV1": {
"additionalProperties": false,
"description": "One deterministic observation. Model judgments never remove it.",
"properties": {
"correlation": {
"$ref": "#/definitions/CorrelationV1"
},
"evidence": {
"description": "Calls, results and notices, in transcript order.",
"items": {
"$ref": "#/definitions/EntryRefV1"
},
"type": "array"
},
"fingerprint": {
"description": "Rule, session, turn and the call ids of this occurrence.",
"type": "string"
},
"observation": {
"type": "string"
},
"rule_id": {
"type": "string"
},
"rule_version": {
"type": "string"
},
"session_id": {
"type": "string"
},
"target": {
"type": "string"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"correlation",
"evidence",
"fingerprint",
"observation",
"rule_id",
"rule_version",
"session_id",
"target"
],
"type": "object"
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"E2eAssessmentsV1": {
"additionalProperties": false,
"properties": {
"passed": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"passed",
"total"
],
"type": "object"
},
"E2eExecutionRefV1": {
"additionalProperties": false,
"description": "What `e2e::dashboard::execution-get` reported at lookup time.",
"properties": {
"assessments": {
"anyOf": [
{
"$ref": "#/definitions/E2eAssessmentsV1"
},
{
"type": "null"
}
]
},
"availability": {
"type": [
"string",
"null"
]
},
"conclusion": {
"type": [
"string",
"null"
]
},
"e2e_revision": {
"type": [
"string",
"null"
]
},
"engine_version": {
"type": [
"string",
"null"
]
},
"evidence_error": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_version": {
"type": [
"string",
"null"
]
},
"label": {
"type": [
"string",
"null"
]
},
"model": {
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"reports": {
"items": {
"$ref": "#/definitions/E2eReportRefV1"
},
"type": "array"
},
"reports_available": {
"description": "True only when the execution reported retained reports.",
"type": "boolean"
},
"scenarios": {
"items": {
"$ref": "#/definitions/E2eScenarioV1"
},
"type": "array"
},
"started_at": {
"description": "When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"status": {
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"reports",
"reports_available",
"scenarios"
],
"type": "object"
},
"E2eMeasureV1": {
"additionalProperties": false,
"properties": {
"average": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"description": "Runs that reported this measure; fewer than `run_count` means some runs reported nothing (never counted as zero).",
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"samples"
],
"type": "object"
},
"E2eReportRefV1": {
"additionalProperties": false,
"properties": {
"available": {
"type": "boolean"
},
"scenario_id": {
"type": "string"
},
"subject_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"available",
"scenario_id"
],
"type": "object"
},
"E2eScenarioV1": {
"additionalProperties": false,
"description": "One scenario's identity and the execution's own averages, as the E2E computed them.",
"properties": {
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"contract_fingerprint": {
"type": [
"string",
"null"
]
},
"cost_usd_per_run": {
"description": "The cohort's total cost divided by its completed runs; the total also covers failed attempts.",
"format": "double",
"type": [
"number",
"null"
]
},
"mean_score": {
"description": "Mean score of the scored runs (0 to 100).",
"format": "double",
"type": [
"number",
"null"
]
},
"measures": {
"additionalProperties": {
"$ref": "#/definitions/E2eMeasureV1"
},
"description": "`function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.",
"type": "object"
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"pass_rate": {
"description": "The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero.\n\nShare of the planned runs that passed (0 to 1).",
"format": "double",
"type": [
"number",
"null"
]
},
"planned_runs": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"run_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"total_tokens_per_run": {
"description": "The cohort's total tokens divided by its completed runs.",
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"measures",
"run_count",
"scenario_id"
],
"type": "object"
},
"EntryRefV1": {
"additionalProperties": false,
"description": "A transcript entry, addressed the way `session::messages` returns it.",
"properties": {
"entry_id": {
"type": "string"
},
"session_id": {
"type": "string"
}
},
"required": [
"entry_id",
"session_id"
],
"type": "object"
},
"EvalResultResponseV1": {
"additionalProperties": false,
"properties": {
"assets": {
"$ref": "#/definitions/AnalysisAssetsV1"
},
"record": {
"$ref": "#/definitions/AnalysisRecordV1"
},
"reviews": {
"default": [],
"description": "One row per suggestion, in suggestion order: what people decided about it. A suggestion nobody acted on reads as `new`.",
"items": {
"$ref": "#/definitions/SuggestionReviewV1"
},
"type": "array"
}
},
"required": [
"assets",
"record"
],
"type": "object"
},
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"EvidenceRunV1": {
"additionalProperties": false,
"description": "One run of the scenario as the E2E reported it. Runs the infrastructure broke are listed with the E2E's own words, never counted.",
"properties": {
"completed": {
"description": "The E2E reported the run complete with a valid technical outcome.",
"type": "boolean"
},
"completion": {
"description": "`completion` as the E2E reported it.",
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.",
"type": [
"object",
"null"
]
},
"status": {
"description": "`passed` or `failed`, the run's task outcome.",
"type": [
"string",
"null"
]
},
"technical": {
"description": "`technical` as the E2E reported it; anything but `valid` is an infrastructure failure.",
"type": [
"string",
"null"
]
},
"total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"completed",
"run_id"
],
"type": "object"
},
"EvidenceSideV1": {
"additionalProperties": false,
"properties": {
"execution_id": {
"type": "string"
},
"mean": {
"description": "Mean of the criterion's metric over those runs; absent without a criterion or without such runs.",
"format": "double",
"type": [
"number",
"null"
]
},
"n": {
"description": "Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"description": "Every run of the scenario the execution holds.",
"items": {
"$ref": "#/definitions/EvidenceRunV1"
},
"type": "array"
}
},
"required": [
"execution_id",
"n",
"runs"
],
"type": "object"
},
"EvidenceV1": {
"additionalProperties": false,
"description": "Computed in code from the attached pair's runs, never by a model. It is what the person's verdict is shown next to.",
"properties": {
"baseline": {
"$ref": "#/definitions/EvidenceSideV1"
},
"candidate": {
"$ref": "#/definitions/EvidenceSideV1"
},
"computed_at": {
"format": "int64",
"type": "integer"
},
"computed_outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"baseline",
"candidate",
"computed_at",
"computed_outcome",
"reason",
"scenario_id"
],
"type": "object"
},
"ExcludedProbeV1": {
"additionalProperties": false,
"properties": {
"calls": {
"items": {
"type": "string"
},
"type": "array"
},
"code": {
"type": "string"
},
"session_id": {
"type": "string"
},
"target": {
"type": "string"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"calls",
"code",
"session_id",
"target"
],
"type": "object"
},
"FailureV1": {
"additionalProperties": false,
"description": "The stage that failed and why. `stage` is the status the analysis was in.",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
},
"stage": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"code",
"message",
"stage"
],
"type": "object"
},
"FidelityLevelV1": {
"oneOf": [
{
"enum": [
"approximate"
],
"type": "string"
},
{
"description": "The provider counts the rebuilt request like the original, up to a fixed per-request overhead measured at the turn's first step.",
"enum": [
"exact"
],
"type": "string"
}
]
},
"FidelityV1": {
"additionalProperties": false,
"description": "How faithfully the request at the decision point was rebuilt.",
"properties": {
"counted_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"estimator": {
"description": "`provider` when the provider counted, else the estimator's name.",
"type": [
"string",
"null"
]
},
"level": {
"$ref": "#/definitions/FidelityLevelV1"
},
"off_ratio": {
"description": "The unexplained difference over the recorded tokens.",
"format": "double",
"type": [
"number",
"null"
]
},
"reasons": {
"default": [],
"description": "Why the rebuild is approximate; empty when exact.",
"items": {
"type": "string"
},
"type": "array"
},
"recorded_tokens": {
"description": "Input tokens of the original step: input + cache read + cache write.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"level"
],
"type": "object"
},
"InvestigationV1": {
"additionalProperties": false,
"description": "The investigation and its own consumption, kept apart from the observed task's metrics.",
"properties": {
"code_root": {
"description": "The directory the analyst read and ran in; absent when access was off.",
"type": [
"string",
"null"
]
},
"completed_at": {
"format": "int64",
"type": "integer"
},
"effective_model": {
"type": [
"string",
"null"
]
},
"effective_provider": {
"type": [
"string",
"null"
]
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsResponseV1"
},
{
"type": "null"
}
]
},
"rejected": {
"items": {
"$ref": "#/definitions/RejectedSuggestionV1"
},
"type": "array"
},
"requested_model": {
"type": "string"
},
"requested_provider": {
"type": "string"
},
"session_id": {
"type": "string"
},
"signal_assessments": {
"items": {
"$ref": "#/definitions/SignalAssessmentV1"
},
"type": "array"
},
"suggestions": {
"items": {
"$ref": "#/definitions/SuggestionV1"
},
"type": "array"
},
"turn_id": {
"type": "string"
}
},
"required": [
"completed_at",
"rejected",
"requested_model",
"requested_provider",
"session_id",
"signal_assessments",
"suggestions",
"turn_id"
],
"type": "object"
},
"JudgeCallV1": {
"additionalProperties": false,
"description": "A Jev call started by this analysis. Persisted before the call so a restart can tell a lost answer from one never requested.",
"properties": {
"deadline": {
"format": "int64",
"type": "integer"
},
"request_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"deadline",
"request_id",
"started_at"
],
"type": "object"
},
"LifecycleEventV1": {
"additionalProperties": false,
"description": "One change of status, with who made it.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"note": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
}
},
"required": [
"at",
"by",
"status"
],
"type": "object"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"LifecycleV1": {
"additionalProperties": false,
"properties": {
"duplicate_of": {
"description": "What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).",
"type": [
"string",
"null"
]
},
"history": {
"description": "Every change of status, oldest first; empty while `new`.",
"items": {
"$ref": "#/definitions/LifecycleEventV1"
},
"type": "array"
},
"pr": {
"description": "The pull request that implements it (`in_progress`, `shipped`).",
"type": [
"string",
"null"
]
},
"reason": {
"description": "Why it was rejected (`rejected`).",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
},
"version": {
"description": "The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.",
"type": [
"string",
"null"
]
}
},
"required": [
"history",
"status"
],
"type": "object"
},
"MetricsScopeV1": {
"oneOf": [
{
"description": "Cumulative over the root and every descendant, all turns.",
"enum": [
"session_tree"
],
"type": "string"
}
]
},
"ModelUsageV1": {
"additionalProperties": false,
"description": "What the steps one model generated used. A conversation can switch models mid-way, and providers differ in what they report (one may send tokens and no cost, another nothing), so the session's counters alone cannot say which model a gap belongs to.",
"properties": {
"cache_read_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"output_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider": {
"type": "string"
},
"reasoning_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"description": "Model steps (assistant messages) this model generated.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cache_read_tokens",
"cache_write_tokens",
"cost_usd",
"function_calls",
"input_tokens",
"model",
"output_tokens",
"provider",
"reasoning_tokens",
"turns"
],
"type": "object"
},
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"MonitorUsageV1": {
"additionalProperties": false,
"description": "Known usage only: absent counters are unknown, never zero.",
"properties": {
"judge_calls": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"judge_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_usage_complete": {
"description": "False when Jev reported incomplete usage or failed.",
"type": "boolean"
},
"llm_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"llm_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"llm_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"judge_calls",
"judge_input_tokens",
"judge_output_tokens",
"judge_usage_complete"
],
"type": "object"
},
"PrefixDivergenceV1": {
"description": "One generation that rewrote a row an earlier request of the session had already sent: a prefix-matching provider cache (DeepSeek) stops reusing at that row.",
"properties": {
"cache_read": {
"description": "`usage.cache_read` of the rewriting request: what the provider still reused.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"call_id": {
"type": [
"string",
"null"
]
},
"cause": {
"description": "What rewrote the row, as far as the harness can tell: `prune` (context-manager pruned or reduced function results to fit the budget that step) or `system_prompt` (row 0: a compaction summary or a prompt change).",
"type": [
"string",
"null"
]
},
"index": {
"description": "First row (0 = system prompt, then one per message) that differs from the session's previous request. Absent when only the provider's wire mapping changed.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"prev_len": {
"description": "Rows the previous request carried.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"provider_note": {
"description": "The provider's own wire-level finding, cause included, when it reported one (a final-message warning starting with [`PROVIDER_PREFIX_NOTE`]).",
"type": [
"string",
"null"
]
},
"role": {
"type": [
"string",
"null"
]
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turn_id": {
"type": "string"
}
},
"required": [
"step",
"turn_id"
],
"type": "object"
},
"RejectedSuggestionV1": {
"additionalProperties": false,
"properties": {
"index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"title": {
"type": "string"
}
},
"required": [
"index",
"reasons",
"title"
],
"type": "object"
},
"ReplyCallV1": {
"additionalProperties": false,
"description": "One function call of a reply.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"payload": {
"description": "The payload as JSON text, cut at 600 characters.",
"type": "string"
},
"target": {
"description": "The function called (through `agent_call` or directly).",
"type": "string"
}
},
"required": [
"payload",
"target"
],
"type": "object"
},
"ReplyUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ReplyV1": {
"additionalProperties": false,
"description": "One reply at the decision point: a sample, or the original step read the same way.",
"properties": {
"calls": {
"items": {
"$ref": "#/definitions/ReplyCallV1"
},
"type": "array"
},
"duration_ms": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"index": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"signal": {
"description": "Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.",
"type": [
"boolean",
"null"
]
},
"text": {
"type": "string"
},
"thinking": {
"description": "The opening of the reply's reasoning and of its text, 400 characters each.",
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/ReplyUsageV1"
}
],
"default": {}
}
},
"required": [
"calls",
"index",
"text",
"thinking"
],
"type": "object"
},
"ReproductionChangeKindV1": {
"oneOf": [
{
"description": "The context as the model saw it.",
"enum": [
"none"
],
"type": "string"
},
{
"description": "The suggestion's own `check.change`.",
"enum": [
"proposed"
],
"type": "string"
},
{
"description": "Edits a person wrote.",
"enum": [
"custom"
],
"type": "string"
}
]
},
"ReproductionStateV1": {
"enum": [
"running",
"completed",
"failed"
],
"type": "string"
},
"ReproductionV1": {
"additionalProperties": false,
"description": "A replay of the decision point: N replies sampled from the rebuilt request, with or without a change. No function ever runs.",
"properties": {
"by": {
"type": "string"
},
"change": {
"default": [],
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"change_kind": {
"$ref": "#/definitions/ReproductionChangeKindV1"
},
"check": {
"allOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
}
],
"description": "The decision point and signal this replay used."
},
"cost_unknown_samples": {
"default": 0,
"description": "Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost_usd": {
"description": "Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.",
"format": "double",
"type": [
"number",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"fidelity": {
"anyOf": [
{
"$ref": "#/definitions/FidelityV1"
},
{
"type": "null"
}
]
},
"finished_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"judge_input_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"original": {
"anyOf": [
{
"$ref": "#/definitions/ReplyV1"
},
{
"type": "null"
}
],
"description": "The original reply at the decision point, read the same way."
},
"phase": {
"description": "`sampling` or `classifying` while running.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"requested": {
"description": "Replies asked for, extensions included.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"samples": {
"default": [],
"items": {
"$ref": "#/definitions/ReplyV1"
},
"type": "array"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ReproductionStateV1"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"by",
"change_kind",
"check",
"id",
"model",
"requested",
"started_at",
"state",
"updated_at"
],
"type": "object"
},
"RoutingReasonV1": {
"oneOf": [
{
"enum": [
"needs_investigation"
],
"type": "string"
},
{
"description": "Legacy: no longer produced; kept so stored records still deserialize.",
"enum": [
"diagnostics"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"insufficient_evidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"low_confidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"coverage_insufficient"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"audit_sample"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"manual_request"
],
"type": "string"
}
]
},
"RoutingV1": {
"additionalProperties": false,
"properties": {
"investigate": {
"type": "boolean"
},
"reasons": {
"items": {
"$ref": "#/definitions/RoutingReasonV1"
},
"type": "array"
}
},
"required": [
"investigate",
"reasons"
],
"type": "object"
},
"ScoreLevel": {
"anyOf": [
{
"type": "string"
},
{
"additionalProperties": true,
"type": "object"
},
{
"items": true,
"type": "array"
}
],
"description": "Score levels require a description. Unlike instructions, Choice option descriptions and Noul criteria, the provider rejects a null level (HTTP422)."
},
"SessionEvidenceV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"entries": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"in_scope": {
"description": "Out-of-scope descendants belong to earlier turns and are not read.",
"type": "boolean"
},
"json_sha256": {
"description": "SHA-256 of the collected JSON entries (not of the stored bytes).",
"type": [
"string",
"null"
]
},
"omitted_entries": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"parent_turn_id": {
"type": [
"string",
"null"
]
},
"preview": {
"description": "Masked, shortened entries shown to the models.",
"items": true,
"type": "array"
},
"reduced_entries": {
"description": "Preview entries whose strings were truncated or whose secrets were masked.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"session_id": {
"type": "string"
}
},
"required": [
"depth",
"entries",
"in_scope",
"omitted_entries",
"preview",
"reduced_entries",
"session_id"
],
"type": "object"
},
"SessionMetricsResponseV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsageV1"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotalsV1"
},
"traces": {
"anyOf": [
{
"$ref": "#/definitions/SessionTraceMetricsV1"
},
{
"type": "null"
}
],
"description": "Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not."
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionTraceMetricsV1": {
"additionalProperties": false,
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionTraceUsageV1"
},
"type": "array"
},
"duration_ms": {
"description": "Elapsed window from the first observed span to the last observed span.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"description": "Distinct traces across the root session and all descendants.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"by_session",
"duration_ms",
"error_span_count",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionTraceUsageV1": {
"additionalProperties": false,
"properties": {
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duration_ms": {
"description": "Elapsed window from the session's first observed span to its last.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error_span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"session_id": {
"type": "string"
},
"span_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"trace_count": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"depth",
"duration_ms",
"error_span_count",
"session_id",
"span_count",
"trace_count"
],
"type": "object"
},
"SessionUsageTotalsV1": {
"additionalProperties": false,
"properties": {
"by_model": {
"description": "The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.",
"items": {
"$ref": "#/definitions/ModelUsageV1"
},
"type": "array"
},
"cache_read_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cache_read_tokens",
"cache_write_tokens",
"cost_usd",
"function_call_errors",
"function_calls",
"input_tokens",
"output_tokens",
"reasoning_tokens",
"sessions",
"turns"
],
"type": "object"
},
"SessionUsageV1": {
"additionalProperties": false,
"properties": {
"by_model": {
"description": "The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.",
"items": {
"$ref": "#/definitions/ModelUsageV1"
},
"type": "array"
},
"cache_read_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"anyOf": [
{
"$ref": "#/definitions/ContextSnapshotV1"
},
{
"type": "null"
}
],
"description": "The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed."
},
"cost_usd": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"description": "Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cache_read_tokens",
"cache_write_tokens",
"cost_usd",
"depth",
"function_call_errors",
"function_calls",
"input_tokens",
"output_tokens",
"reasoning_tokens",
"session_id",
"turns"
],
"type": "object"
},
"SignalAssessmentV1": {
"additionalProperties": false,
"properties": {
"explanation": {
"type": "string"
},
"fingerprint": {
"description": "The `fingerprint` of a diagnostic in the evidence.",
"type": "string"
},
"verdict": {
"$ref": "#/definitions/SignalVerdictV1"
}
},
"required": [
"explanation",
"fingerprint",
"verdict"
],
"type": "object"
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SignalVerdictV1": {
"enum": [
"likely_expected",
"worth_changing",
"unclear"
],
"type": "string"
},
"SnapshotCategoriesV1": {
"description": "Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.",
"properties": {
"hook_guidance": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"messages": {
"$ref": "#/definitions/SnapshotMessagesV1"
},
"overhead": {
"description": "Provider framing plus response_format / provider_options fields.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"skills": {
"default": 0,
"description": "Selected skill bodies contained in the system prompt.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"system_prompt": {
"description": "Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"tools": {
"description": "Function schemas exposed to the model.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"messages",
"overhead",
"system_prompt",
"tools"
],
"type": "object"
},
"SnapshotMessagesV1": {
"description": "Estimated tokens of the assembled window's messages, by role.",
"properties": {
"assistant": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"custom": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_result": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"user": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assistant",
"custom",
"function_result",
"user"
],
"type": "object"
},
"SnapshotV1": {
"additionalProperties": false,
"properties": {
"captured_at": {
"format": "int64",
"type": "integer"
},
"coverage": {
"$ref": "#/definitions/CoverageV1"
},
"diagnostics": {
"items": {
"$ref": "#/definitions/DiagnosticV1"
},
"type": "array"
},
"e2e_scenario": {
"description": "The harness-e2e scenario the observed session ran, when it came from the E2E (`metadata.e2e_scenario`).",
"type": [
"string",
"null"
]
},
"excluded_probes": {
"description": "Protocol probes that failed as expected and were not reported.",
"items": {
"$ref": "#/definitions/ExcludedProbeV1"
},
"type": "array"
},
"metrics": {
"$ref": "#/definitions/SessionMetricsResponseV1"
},
"metrics_scope": {
"$ref": "#/definitions/MetricsScopeV1"
},
"observed_model": {
"type": [
"string",
"null"
]
},
"observed_provider": {
"type": [
"string",
"null"
]
},
"rules_version": {
"type": "string"
},
"sessions": {
"items": {
"$ref": "#/definitions/SessionEvidenceV1"
},
"type": "array"
},
"source_result_error": {
"type": [
"string",
"null"
]
},
"source_session_id": {
"type": "string"
},
"source_status": {
"$ref": "#/definitions/TurnStatus"
},
"source_stop_reason": {
"type": [
"string",
"null"
]
},
"source_title": {
"type": [
"string",
"null"
]
},
"source_turn_id": {
"type": "string"
},
"window_turn_ids": {
"description": "The observed turn plus earlier root turns no analysis covered yet.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"captured_at",
"coverage",
"diagnostics",
"excluded_probes",
"metrics",
"metrics_scope",
"rules_version",
"sessions",
"source_session_id",
"source_status",
"source_turn_id",
"window_turn_ids"
],
"type": "object"
},
"StageTimeV1": {
"additionalProperties": false,
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"at",
"status"
],
"type": "object"
},
"Stats": {
"additionalProperties": false,
"description": "Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.",
"properties": {
"attempts": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"elapsed_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"questions": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"requests": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"usage_complete": {
"type": "boolean"
}
},
"required": [
"attempts",
"elapsed_ms",
"input_tokens",
"output_tokens",
"questions",
"requests",
"usage_complete"
],
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
},
"SuggestionReviewV1": {
"additionalProperties": false,
"description": "Everything people decided about one suggestion, in the `eval_suggestion` state scope. The title, component and patterns are copied from the suggestion so the row outlives the analysis's retention.",
"properties": {
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"evaluation_id": {
"type": "string"
},
"evidence": {
"anyOf": [
{
"$ref": "#/definitions/EvidenceV1"
},
{
"type": "null"
}
]
},
"first_run_at": {
"description": "When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"harness_component": {
"type": "string"
},
"lifecycle": {
"$ref": "#/definitions/LifecycleV1"
},
"patterns": {
"description": "`<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.",
"items": {
"type": "string"
},
"type": "array"
},
"reproductions": {
"description": "Replays of the decision point, oldest first: the base and the tested changes.",
"items": {
"$ref": "#/definitions/ReproductionV1"
},
"type": "array"
},
"run": {
"anyOf": [
{
"$ref": "#/definitions/ValidationRunV1"
},
{
"type": "null"
}
]
},
"scenario_id": {
"description": "The scenario the suggestion's validation plan names.",
"type": [
"string",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"type": "string"
},
"verdict": {
"anyOf": [
{
"$ref": "#/definitions/VerdictV1"
},
{
"type": "null"
}
]
}
},
"required": [
"evaluation_id",
"harness_component",
"lifecycle",
"patterns",
"suggestion_index",
"title"
],
"type": "object"
},
"SuggestionV1": {
"additionalProperties": false,
"properties": {
"check": {
"anyOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
},
{
"type": "null"
}
],
"description": "How to reproduce the behavior at the step where it happened, and the proposed change as edits of what the model saw there."
},
"code_refs": {
"default": [],
"description": "Code the suggestion rests on, as read in the code directory; only with code access. At most 8; each is checked against the directory.",
"items": {
"$ref": "#/definitions/CodeRefV1"
},
"maxItems": 8,
"type": "array"
},
"evidence": {
"items": {
"$ref": "#/definitions/EntryRefV1"
},
"type": "array"
},
"expected_effect": {
"type": "string"
},
"harness_component": {
"type": "string"
},
"hypothesis": {
"type": "string"
},
"limitations": {
"type": "string"
},
"observation": {
"type": "string"
},
"proposed_change": {
"type": "string"
},
"title": {
"type": "string"
},
"validation": {
"$ref": "#/definitions/ValidationPlanV1"
}
},
"required": [
"evidence",
"expected_effect",
"harness_component",
"hypothesis",
"limitations",
"observation",
"proposed_change",
"title",
"validation"
],
"type": "object"
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
},
"TriageFailureV1": {
"additionalProperties": false,
"properties": {
"code": {
"type": "string"
},
"http_status": {
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"message": {
"type": "string"
},
"request_id": {
"type": "string"
},
"stats": {
"anyOf": [
{
"$ref": "#/definitions/Stats"
},
{
"type": "null"
}
],
"description": "Known usage of the failed call, when the provider returned it."
}
},
"required": [
"code",
"message",
"request_id"
],
"type": "object"
},
"TriageV1": {
"additionalProperties": false,
"properties": {
"answers": {
"additionalProperties": {
"$ref": "#/definitions/Answer"
},
"type": "object"
},
"completed_at": {
"format": "int64",
"type": "integer"
},
"criteria_version": {
"type": "string"
},
"model": {
"description": "The model the provider reports, not the configured default.",
"type": "string"
},
"provider": {
"type": "string"
},
"request_id": {
"type": "string"
},
"stats": {
"$ref": "#/definitions/Stats"
}
},
"required": [
"answers",
"completed_at",
"criteria_version",
"model",
"provider",
"request_id",
"stats"
],
"type": "object"
},
"TurnCaptureV1": {
"additionalProperties": false,
"description": "The observed turn's record as the Harness stored it, which keeps only a session's latest turn.",
"properties": {
"hook_guidance_tokens": {
"default": 0,
"description": "Tokens `pre_generate` hooks added at the turn's last step: above zero, the system prompt the model saw is not stored anywhere.",
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"options": {
"description": "The Harness's `TurnOptions`, unmodified."
},
"prompt_surface_digest": {
"type": [
"string",
"null"
]
},
"record": {
"description": "The whole `TurnRecord` as read when the evidence was captured (it repeats `options`): what a later fork of the session needs, since the Harness overwrites it with the next turn. Absent when it would have taken the assets over their size limit (`record_omitted` says so) and on analyses made before it was kept."
},
"record_omitted": {
"description": "Why `record` was not kept.",
"type": [
"string",
"null"
]
},
"turn_id": {
"type": "string"
}
},
"required": [
"options",
"turn_id"
],
"type": "object"
},
"TurnStatus": {
"description": "The coarse, harness-internal turn lifecycle (harness.md § API Reference). Finer-grained than the session's `status`, which the loop derives from it.",
"enum": [
"running",
"awaiting_functions",
"completed",
"cancelled",
"failed"
],
"type": "string"
},
"Usage": {
"properties": {
"cache_read": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ValidationLinkV1": {
"additionalProperties": false,
"description": "Baseline and candidate executions linked to one suggestion. A link is a reference, not a verdict: the improvement claim belongs to the E2E comparison and its criteria.",
"properties": {
"attached_at": {
"format": "int64",
"type": "integer"
},
"baseline": {
"$ref": "#/definitions/E2eExecutionRefV1"
},
"candidate": {
"$ref": "#/definitions/E2eExecutionRefV1"
},
"comparability": {
"$ref": "#/definitions/ComparabilityV1"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"attached_at",
"baseline",
"candidate",
"comparability",
"suggestion_index"
],
"type": "object"
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
},
"ValidationPlanV1": {
"additionalProperties": false,
"description": "The E2E plan a suggestion must carry. A reviewer can run it without the LLM that wrote it.",
"properties": {
"expectation": {
"type": "string"
},
"invariants": {
"description": "Task-correctness invariants checked by an independent evaluator.",
"items": {
"type": "string"
},
"type": "array"
},
"non_regression_controls": {
"items": {
"type": "string"
},
"type": "array"
},
"primary_metric": {
"description": "The effort metric compared between baseline and candidate.",
"type": "string"
},
"reproduction": {
"type": "string"
},
"scenario_id": {
"description": "An existing harness-e2e scenario, or null when a new case is needed.",
"type": [
"string",
"null"
]
}
},
"required": [
"expectation",
"invariants",
"non_regression_controls",
"primary_metric",
"reproduction"
],
"type": "object"
},
"ValidationRunStateV1": {
"oneOf": [
{
"enum": [
"failed"
],
"type": "string"
},
{
"description": "Recorded before the executions are requested.",
"enum": [
"starting"
],
"type": "string"
},
{
"description": "Both executions were requested; the sweep waits for them.",
"enum": [
"running"
],
"type": "string"
},
{
"description": "Both ended; the pair is about to be attached.",
"enum": [
"finished"
],
"type": "string"
},
{
"description": "The pair is attached to the analysis and its evidence computed.",
"enum": [
"attached"
],
"type": "string"
}
]
},
"ValidationRunV1": {
"additionalProperties": false,
"description": "The baseline and candidate E2E executions this worker started for a suggestion (`eval::start-validation`).",
"properties": {
"baseline_commit": {
"description": "The full commits the Harness was built from.",
"type": "string"
},
"baseline_execution_id": {
"description": "Absent until the E2E accepted the execution.",
"type": [
"string",
"null"
]
},
"candidate_commit": {
"type": "string"
},
"candidate_execution_id": {
"type": [
"string",
"null"
]
},
"error": {
"description": "Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).",
"type": [
"string",
"null"
]
},
"finished_at": {
"description": "When both executions were seen ended; the sweep stops trying to attach the pair some time after.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"description": "Runs requested for each side.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ValidationRunStateV1"
}
},
"required": [
"baseline_commit",
"candidate_commit",
"model",
"provider",
"runs",
"scenario_id",
"started_at",
"state"
],
"type": "object"
},
"VerdictV1": {
"additionalProperties": false,
"description": "The person's judgment, kept with the criterion it was made against.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"controls_checked": {
"description": "The non-regression controls of the plan that were checked.",
"items": {
"type": "string"
},
"type": "array"
},
"criterion_snapshot": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"rationale": {
"type": "string"
}
},
"required": [
"at",
"by",
"controls_checked",
"outcome",
"rationale"
],
"type": "object"
}
},
"title": "Nullable_EvalResultResponseV1"
}
},
{
"description": "Record what a person decided about a suggestion of a terminal analysis: move its lifecycle (new, accepted, in_progress, shipped, rejected, duplicate), register the criterion a validation is judged by, or record a verdict. A verdict of validated_improvement is refused unless the criterion was registered before the first attached pair or run and both sides have min_runs completed runs. The author is `by`, else the eval host's user, else the caller's identity.",
"metadata": {},
"name": "eval::review",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CriterionInputV1": {
"additionalProperties": false,
"description": "What a person commits to before the results exist.",
"properties": {
"direction": {
"allOf": [
{
"$ref": "#/definitions/DirectionV1"
}
],
"description": "The change that counts as an effect, wanted when the candidate moves the baseline mean in this direction."
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"description": "The smallest change of the baseline mean that counts, as an absolute difference in the metric's own unit: signals per run, USD, seconds, calls or tokens; percentage points (0-100) for `pass_rate`. Above 0.",
"format": "double",
"type": "number"
},
"min_runs": {
"description": "Completed runs each side needs for a verdict of improvement (1-20).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"description": "`signal_per_run` only: one of the row's `patterns`.",
"type": [
"string",
"null"
]
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"ReviewActionV1": {
"oneOf": [
{
"description": "Move the suggestion to `status`.",
"enum": [
"set_lifecycle"
],
"type": "string"
},
{
"description": "Register the criterion a validation is judged by, before its results.",
"enum": [
"set_criterion"
],
"type": "string"
},
{
"description": "Record the person's verdict.",
"enum": [
"set_verdict"
],
"type": "string"
}
]
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
}
},
"description": "`eval::review`. The fields of the other actions are ignored. The author is the caller's identity when the engine stamps one, else `by`.",
"properties": {
"_caller_worker_id": {
"default": null,
"description": "Stamped by the engine on every invocation; callers omit it.",
"type": [
"string",
"null"
]
},
"action": {
"$ref": "#/definitions/ReviewActionV1"
},
"by": {
"default": null,
"description": "The author when the caller has no identity.",
"type": [
"string",
"null"
]
},
"controls_checked": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionInputV1"
},
{
"type": "null"
}
],
"default": null,
"description": "`set_criterion`. Refused once a criterion exists and a run started or a pair was attached: a criterion chosen after the results proves nothing."
},
"duplicate_of": {
"default": null,
"type": [
"string",
"null"
]
},
"evaluation_id": {
"type": "string"
},
"note": {
"default": null,
"description": "`set_lifecycle`: shown in the history.",
"type": [
"string",
"null"
]
},
"outcome": {
"anyOf": [
{
"$ref": "#/definitions/ValidationOutcomeV1"
},
{
"type": "null"
}
],
"default": null,
"description": "`set_verdict`. `validated_improvement` is refused unless a criterion was registered before the first attached pair or run, and the evidence has `min_runs` completed runs on both sides."
},
"pr": {
"default": null,
"type": [
"string",
"null"
]
},
"rationale": {
"default": null,
"type": [
"string",
"null"
]
},
"reason": {
"default": null,
"type": [
"string",
"null"
]
},
"scenario_id": {
"default": null,
"description": "`set_criterion`: the scenario to measure; defaults to the one the suggestion's plan names.",
"type": [
"string",
"null"
]
},
"status": {
"anyOf": [
{
"$ref": "#/definitions/LifecycleStatusV1"
},
{
"type": "null"
}
],
"default": null,
"description": "`set_lifecycle`: the status to move to (never back to `new`; a shipped suggestion stays shipped, to complete its `pr` or `version`). `shipped` needs a `pr` (here or already recorded), `rejected` a `reason`, `duplicate` a `duplicate_of`."
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"default": null,
"type": [
"string",
"null"
]
}
},
"required": [
"action",
"evaluation_id",
"suggestion_index"
],
"title": "ReviewRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"CriterionV1": {
"additionalProperties": false,
"properties": {
"direction": {
"$ref": "#/definitions/DirectionV1"
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"format": "double",
"type": "number"
},
"min_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": [
"string",
"null"
]
},
"registered_at": {
"format": "int64",
"type": "integer"
},
"registered_by": {
"type": "string"
},
"scenario_id": {
"description": "The scenario whose runs are measured.",
"type": "string"
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs",
"registered_at",
"registered_by",
"scenario_id"
],
"type": "object"
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"EvidenceRunV1": {
"additionalProperties": false,
"description": "One run of the scenario as the E2E reported it. Runs the infrastructure broke are listed with the E2E's own words, never counted.",
"properties": {
"completed": {
"description": "The E2E reported the run complete with a valid technical outcome.",
"type": "boolean"
},
"completion": {
"description": "`completion` as the E2E reported it.",
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.",
"type": [
"object",
"null"
]
},
"status": {
"description": "`passed` or `failed`, the run's task outcome.",
"type": [
"string",
"null"
]
},
"technical": {
"description": "`technical` as the E2E reported it; anything but `valid` is an infrastructure failure.",
"type": [
"string",
"null"
]
},
"total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"completed",
"run_id"
],
"type": "object"
},
"EvidenceSideV1": {
"additionalProperties": false,
"properties": {
"execution_id": {
"type": "string"
},
"mean": {
"description": "Mean of the criterion's metric over those runs; absent without a criterion or without such runs.",
"format": "double",
"type": [
"number",
"null"
]
},
"n": {
"description": "Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"description": "Every run of the scenario the execution holds.",
"items": {
"$ref": "#/definitions/EvidenceRunV1"
},
"type": "array"
}
},
"required": [
"execution_id",
"n",
"runs"
],
"type": "object"
},
"EvidenceV1": {
"additionalProperties": false,
"description": "Computed in code from the attached pair's runs, never by a model. It is what the person's verdict is shown next to.",
"properties": {
"baseline": {
"$ref": "#/definitions/EvidenceSideV1"
},
"candidate": {
"$ref": "#/definitions/EvidenceSideV1"
},
"computed_at": {
"format": "int64",
"type": "integer"
},
"computed_outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"baseline",
"candidate",
"computed_at",
"computed_outcome",
"reason",
"scenario_id"
],
"type": "object"
},
"FidelityLevelV1": {
"oneOf": [
{
"enum": [
"approximate"
],
"type": "string"
},
{
"description": "The provider counts the rebuilt request like the original, up to a fixed per-request overhead measured at the turn's first step.",
"enum": [
"exact"
],
"type": "string"
}
]
},
"FidelityV1": {
"additionalProperties": false,
"description": "How faithfully the request at the decision point was rebuilt.",
"properties": {
"counted_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"estimator": {
"description": "`provider` when the provider counted, else the estimator's name.",
"type": [
"string",
"null"
]
},
"level": {
"$ref": "#/definitions/FidelityLevelV1"
},
"off_ratio": {
"description": "The unexplained difference over the recorded tokens.",
"format": "double",
"type": [
"number",
"null"
]
},
"reasons": {
"default": [],
"description": "Why the rebuild is approximate; empty when exact.",
"items": {
"type": "string"
},
"type": "array"
},
"recorded_tokens": {
"description": "Input tokens of the original step: input + cache read + cache write.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"level"
],
"type": "object"
},
"LifecycleEventV1": {
"additionalProperties": false,
"description": "One change of status, with who made it.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"note": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
}
},
"required": [
"at",
"by",
"status"
],
"type": "object"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"LifecycleV1": {
"additionalProperties": false,
"properties": {
"duplicate_of": {
"description": "What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).",
"type": [
"string",
"null"
]
},
"history": {
"description": "Every change of status, oldest first; empty while `new`.",
"items": {
"$ref": "#/definitions/LifecycleEventV1"
},
"type": "array"
},
"pr": {
"description": "The pull request that implements it (`in_progress`, `shipped`).",
"type": [
"string",
"null"
]
},
"reason": {
"description": "Why it was rejected (`rejected`).",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
},
"version": {
"description": "The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.",
"type": [
"string",
"null"
]
}
},
"required": [
"history",
"status"
],
"type": "object"
},
"ReplyCallV1": {
"additionalProperties": false,
"description": "One function call of a reply.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"payload": {
"description": "The payload as JSON text, cut at 600 characters.",
"type": "string"
},
"target": {
"description": "The function called (through `agent_call` or directly).",
"type": "string"
}
},
"required": [
"payload",
"target"
],
"type": "object"
},
"ReplyUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ReplyV1": {
"additionalProperties": false,
"description": "One reply at the decision point: a sample, or the original step read the same way.",
"properties": {
"calls": {
"items": {
"$ref": "#/definitions/ReplyCallV1"
},
"type": "array"
},
"duration_ms": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"index": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"signal": {
"description": "Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.",
"type": [
"boolean",
"null"
]
},
"text": {
"type": "string"
},
"thinking": {
"description": "The opening of the reply's reasoning and of its text, 400 characters each.",
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/ReplyUsageV1"
}
],
"default": {}
}
},
"required": [
"calls",
"index",
"text",
"thinking"
],
"type": "object"
},
"ReproductionChangeKindV1": {
"oneOf": [
{
"description": "The context as the model saw it.",
"enum": [
"none"
],
"type": "string"
},
{
"description": "The suggestion's own `check.change`.",
"enum": [
"proposed"
],
"type": "string"
},
{
"description": "Edits a person wrote.",
"enum": [
"custom"
],
"type": "string"
}
]
},
"ReproductionStateV1": {
"enum": [
"running",
"completed",
"failed"
],
"type": "string"
},
"ReproductionV1": {
"additionalProperties": false,
"description": "A replay of the decision point: N replies sampled from the rebuilt request, with or without a change. No function ever runs.",
"properties": {
"by": {
"type": "string"
},
"change": {
"default": [],
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"change_kind": {
"$ref": "#/definitions/ReproductionChangeKindV1"
},
"check": {
"allOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
}
],
"description": "The decision point and signal this replay used."
},
"cost_unknown_samples": {
"default": 0,
"description": "Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost_usd": {
"description": "Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.",
"format": "double",
"type": [
"number",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"fidelity": {
"anyOf": [
{
"$ref": "#/definitions/FidelityV1"
},
{
"type": "null"
}
]
},
"finished_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"judge_input_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"original": {
"anyOf": [
{
"$ref": "#/definitions/ReplyV1"
},
{
"type": "null"
}
],
"description": "The original reply at the decision point, read the same way."
},
"phase": {
"description": "`sampling` or `classifying` while running.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"requested": {
"description": "Replies asked for, extensions included.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"samples": {
"default": [],
"items": {
"$ref": "#/definitions/ReplyV1"
},
"type": "array"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ReproductionStateV1"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"by",
"change_kind",
"check",
"id",
"model",
"requested",
"started_at",
"state",
"updated_at"
],
"type": "object"
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
},
"ValidationRunStateV1": {
"oneOf": [
{
"enum": [
"failed"
],
"type": "string"
},
{
"description": "Recorded before the executions are requested.",
"enum": [
"starting"
],
"type": "string"
},
{
"description": "Both executions were requested; the sweep waits for them.",
"enum": [
"running"
],
"type": "string"
},
{
"description": "Both ended; the pair is about to be attached.",
"enum": [
"finished"
],
"type": "string"
},
{
"description": "The pair is attached to the analysis and its evidence computed.",
"enum": [
"attached"
],
"type": "string"
}
]
},
"ValidationRunV1": {
"additionalProperties": false,
"description": "The baseline and candidate E2E executions this worker started for a suggestion (`eval::start-validation`).",
"properties": {
"baseline_commit": {
"description": "The full commits the Harness was built from.",
"type": "string"
},
"baseline_execution_id": {
"description": "Absent until the E2E accepted the execution.",
"type": [
"string",
"null"
]
},
"candidate_commit": {
"type": "string"
},
"candidate_execution_id": {
"type": [
"string",
"null"
]
},
"error": {
"description": "Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).",
"type": [
"string",
"null"
]
},
"finished_at": {
"description": "When both executions were seen ended; the sweep stops trying to attach the pair some time after.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"description": "Runs requested for each side.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ValidationRunStateV1"
}
},
"required": [
"baseline_commit",
"candidate_commit",
"model",
"provider",
"runs",
"scenario_id",
"started_at",
"state"
],
"type": "object"
},
"VerdictV1": {
"additionalProperties": false,
"description": "The person's judgment, kept with the criterion it was made against.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"controls_checked": {
"description": "The non-regression controls of the plan that were checked.",
"items": {
"type": "string"
},
"type": "array"
},
"criterion_snapshot": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"rationale": {
"type": "string"
}
},
"required": [
"at",
"by",
"controls_checked",
"outcome",
"rationale"
],
"type": "object"
}
},
"description": "Everything people decided about one suggestion, in the `eval_suggestion` state scope. The title, component and patterns are copied from the suggestion so the row outlives the analysis's retention.",
"properties": {
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"evaluation_id": {
"type": "string"
},
"evidence": {
"anyOf": [
{
"$ref": "#/definitions/EvidenceV1"
},
{
"type": "null"
}
]
},
"first_run_at": {
"description": "When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"harness_component": {
"type": "string"
},
"lifecycle": {
"$ref": "#/definitions/LifecycleV1"
},
"patterns": {
"description": "`<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.",
"items": {
"type": "string"
},
"type": "array"
},
"reproductions": {
"description": "Replays of the decision point, oldest first: the base and the tested changes.",
"items": {
"$ref": "#/definitions/ReproductionV1"
},
"type": "array"
},
"run": {
"anyOf": [
{
"$ref": "#/definitions/ValidationRunV1"
},
{
"type": "null"
}
]
},
"scenario_id": {
"description": "The scenario the suggestion's validation plan names.",
"type": [
"string",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"type": "string"
},
"verdict": {
"anyOf": [
{
"$ref": "#/definitions/VerdictV1"
},
{
"type": "null"
}
]
}
},
"required": [
"evaluation_id",
"harness_component",
"lifecycle",
"patterns",
"suggestion_index",
"title"
],
"title": "SuggestionReviewV1",
"type": "object"
}
},
{
"description": "List the stored suggestion reviews (lifecycle, criterion, run, evidence, verdict) and, per analysis with suggestions, how many sit at each lifecycle status.",
"metadata": {},
"name": "eval::reviews",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"default": null,
"description": "Only this analysis; absent means every analysis.",
"type": [
"string",
"null"
]
}
},
"title": "ReviewsRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"CriterionV1": {
"additionalProperties": false,
"properties": {
"direction": {
"$ref": "#/definitions/DirectionV1"
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"format": "double",
"type": "number"
},
"min_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": [
"string",
"null"
]
},
"registered_at": {
"format": "int64",
"type": "integer"
},
"registered_by": {
"type": "string"
},
"scenario_id": {
"description": "The scenario whose runs are measured.",
"type": "string"
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs",
"registered_at",
"registered_by",
"scenario_id"
],
"type": "object"
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"EvidenceRunV1": {
"additionalProperties": false,
"description": "One run of the scenario as the E2E reported it. Runs the infrastructure broke are listed with the E2E's own words, never counted.",
"properties": {
"completed": {
"description": "The E2E reported the run complete with a valid technical outcome.",
"type": "boolean"
},
"completion": {
"description": "`completion` as the E2E reported it.",
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.",
"type": [
"object",
"null"
]
},
"status": {
"description": "`passed` or `failed`, the run's task outcome.",
"type": [
"string",
"null"
]
},
"technical": {
"description": "`technical` as the E2E reported it; anything but `valid` is an infrastructure failure.",
"type": [
"string",
"null"
]
},
"total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"completed",
"run_id"
],
"type": "object"
},
"EvidenceSideV1": {
"additionalProperties": false,
"properties": {
"execution_id": {
"type": "string"
},
"mean": {
"description": "Mean of the criterion's metric over those runs; absent without a criterion or without such runs.",
"format": "double",
"type": [
"number",
"null"
]
},
"n": {
"description": "Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"description": "Every run of the scenario the execution holds.",
"items": {
"$ref": "#/definitions/EvidenceRunV1"
},
"type": "array"
}
},
"required": [
"execution_id",
"n",
"runs"
],
"type": "object"
},
"EvidenceV1": {
"additionalProperties": false,
"description": "Computed in code from the attached pair's runs, never by a model. It is what the person's verdict is shown next to.",
"properties": {
"baseline": {
"$ref": "#/definitions/EvidenceSideV1"
},
"candidate": {
"$ref": "#/definitions/EvidenceSideV1"
},
"computed_at": {
"format": "int64",
"type": "integer"
},
"computed_outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"baseline",
"candidate",
"computed_at",
"computed_outcome",
"reason",
"scenario_id"
],
"type": "object"
},
"FidelityLevelV1": {
"oneOf": [
{
"enum": [
"approximate"
],
"type": "string"
},
{
"description": "The provider counts the rebuilt request like the original, up to a fixed per-request overhead measured at the turn's first step.",
"enum": [
"exact"
],
"type": "string"
}
]
},
"FidelityV1": {
"additionalProperties": false,
"description": "How faithfully the request at the decision point was rebuilt.",
"properties": {
"counted_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"estimator": {
"description": "`provider` when the provider counted, else the estimator's name.",
"type": [
"string",
"null"
]
},
"level": {
"$ref": "#/definitions/FidelityLevelV1"
},
"off_ratio": {
"description": "The unexplained difference over the recorded tokens.",
"format": "double",
"type": [
"number",
"null"
]
},
"reasons": {
"default": [],
"description": "Why the rebuild is approximate; empty when exact.",
"items": {
"type": "string"
},
"type": "array"
},
"recorded_tokens": {
"description": "Input tokens of the original step: input + cache read + cache write.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"level"
],
"type": "object"
},
"LifecycleEventV1": {
"additionalProperties": false,
"description": "One change of status, with who made it.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"note": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
}
},
"required": [
"at",
"by",
"status"
],
"type": "object"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"LifecycleV1": {
"additionalProperties": false,
"properties": {
"duplicate_of": {
"description": "What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).",
"type": [
"string",
"null"
]
},
"history": {
"description": "Every change of status, oldest first; empty while `new`.",
"items": {
"$ref": "#/definitions/LifecycleEventV1"
},
"type": "array"
},
"pr": {
"description": "The pull request that implements it (`in_progress`, `shipped`).",
"type": [
"string",
"null"
]
},
"reason": {
"description": "Why it was rejected (`rejected`).",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
},
"version": {
"description": "The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.",
"type": [
"string",
"null"
]
}
},
"required": [
"history",
"status"
],
"type": "object"
},
"ReplyCallV1": {
"additionalProperties": false,
"description": "One function call of a reply.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"payload": {
"description": "The payload as JSON text, cut at 600 characters.",
"type": "string"
},
"target": {
"description": "The function called (through `agent_call` or directly).",
"type": "string"
}
},
"required": [
"payload",
"target"
],
"type": "object"
},
"ReplyUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ReplyV1": {
"additionalProperties": false,
"description": "One reply at the decision point: a sample, or the original step read the same way.",
"properties": {
"calls": {
"items": {
"$ref": "#/definitions/ReplyCallV1"
},
"type": "array"
},
"duration_ms": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"index": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"signal": {
"description": "Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.",
"type": [
"boolean",
"null"
]
},
"text": {
"type": "string"
},
"thinking": {
"description": "The opening of the reply's reasoning and of its text, 400 characters each.",
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/ReplyUsageV1"
}
],
"default": {}
}
},
"required": [
"calls",
"index",
"text",
"thinking"
],
"type": "object"
},
"ReproductionChangeKindV1": {
"oneOf": [
{
"description": "The context as the model saw it.",
"enum": [
"none"
],
"type": "string"
},
{
"description": "The suggestion's own `check.change`.",
"enum": [
"proposed"
],
"type": "string"
},
{
"description": "Edits a person wrote.",
"enum": [
"custom"
],
"type": "string"
}
]
},
"ReproductionStateV1": {
"enum": [
"running",
"completed",
"failed"
],
"type": "string"
},
"ReproductionV1": {
"additionalProperties": false,
"description": "A replay of the decision point: N replies sampled from the rebuilt request, with or without a change. No function ever runs.",
"properties": {
"by": {
"type": "string"
},
"change": {
"default": [],
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"change_kind": {
"$ref": "#/definitions/ReproductionChangeKindV1"
},
"check": {
"allOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
}
],
"description": "The decision point and signal this replay used."
},
"cost_unknown_samples": {
"default": 0,
"description": "Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost_usd": {
"description": "Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.",
"format": "double",
"type": [
"number",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"fidelity": {
"anyOf": [
{
"$ref": "#/definitions/FidelityV1"
},
{
"type": "null"
}
]
},
"finished_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"judge_input_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"original": {
"anyOf": [
{
"$ref": "#/definitions/ReplyV1"
},
{
"type": "null"
}
],
"description": "The original reply at the decision point, read the same way."
},
"phase": {
"description": "`sampling` or `classifying` while running.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"requested": {
"description": "Replies asked for, extensions included.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"samples": {
"default": [],
"items": {
"$ref": "#/definitions/ReplyV1"
},
"type": "array"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ReproductionStateV1"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"by",
"change_kind",
"check",
"id",
"model",
"requested",
"started_at",
"state",
"updated_at"
],
"type": "object"
},
"ReviewSummaryV1": {
"additionalProperties": false,
"description": "Suggestions of one analysis by lifecycle status; a suggestion nobody acted on counts as `new`.",
"properties": {
"accepted": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"duplicate": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"evaluation_id": {
"type": "string"
},
"in_progress": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"new": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rejected": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"shipped": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"accepted",
"duplicate",
"evaluation_id",
"in_progress",
"new",
"rejected",
"shipped",
"suggestions"
],
"type": "object"
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
},
"SuggestionReviewV1": {
"additionalProperties": false,
"description": "Everything people decided about one suggestion, in the `eval_suggestion` state scope. The title, component and patterns are copied from the suggestion so the row outlives the analysis's retention.",
"properties": {
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"evaluation_id": {
"type": "string"
},
"evidence": {
"anyOf": [
{
"$ref": "#/definitions/EvidenceV1"
},
{
"type": "null"
}
]
},
"first_run_at": {
"description": "When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"harness_component": {
"type": "string"
},
"lifecycle": {
"$ref": "#/definitions/LifecycleV1"
},
"patterns": {
"description": "`<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.",
"items": {
"type": "string"
},
"type": "array"
},
"reproductions": {
"description": "Replays of the decision point, oldest first: the base and the tested changes.",
"items": {
"$ref": "#/definitions/ReproductionV1"
},
"type": "array"
},
"run": {
"anyOf": [
{
"$ref": "#/definitions/ValidationRunV1"
},
{
"type": "null"
}
]
},
"scenario_id": {
"description": "The scenario the suggestion's validation plan names.",
"type": [
"string",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"type": "string"
},
"verdict": {
"anyOf": [
{
"$ref": "#/definitions/VerdictV1"
},
{
"type": "null"
}
]
}
},
"required": [
"evaluation_id",
"harness_component",
"lifecycle",
"patterns",
"suggestion_index",
"title"
],
"type": "object"
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
},
"ValidationRunStateV1": {
"oneOf": [
{
"enum": [
"failed"
],
"type": "string"
},
{
"description": "Recorded before the executions are requested.",
"enum": [
"starting"
],
"type": "string"
},
{
"description": "Both executions were requested; the sweep waits for them.",
"enum": [
"running"
],
"type": "string"
},
{
"description": "Both ended; the pair is about to be attached.",
"enum": [
"finished"
],
"type": "string"
},
{
"description": "The pair is attached to the analysis and its evidence computed.",
"enum": [
"attached"
],
"type": "string"
}
]
},
"ValidationRunV1": {
"additionalProperties": false,
"description": "The baseline and candidate E2E executions this worker started for a suggestion (`eval::start-validation`).",
"properties": {
"baseline_commit": {
"description": "The full commits the Harness was built from.",
"type": "string"
},
"baseline_execution_id": {
"description": "Absent until the E2E accepted the execution.",
"type": [
"string",
"null"
]
},
"candidate_commit": {
"type": "string"
},
"candidate_execution_id": {
"type": [
"string",
"null"
]
},
"error": {
"description": "Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).",
"type": [
"string",
"null"
]
},
"finished_at": {
"description": "When both executions were seen ended; the sweep stops trying to attach the pair some time after.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"description": "Runs requested for each side.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ValidationRunStateV1"
}
},
"required": [
"baseline_commit",
"candidate_commit",
"model",
"provider",
"runs",
"scenario_id",
"started_at",
"state"
],
"type": "object"
},
"VerdictV1": {
"additionalProperties": false,
"description": "The person's judgment, kept with the criterion it was made against.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"controls_checked": {
"description": "The non-regression controls of the plan that were checked.",
"items": {
"type": "string"
},
"type": "array"
},
"criterion_snapshot": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"rationale": {
"type": "string"
}
},
"required": [
"at",
"by",
"controls_checked",
"outcome",
"rationale"
],
"type": "object"
}
},
"properties": {
"reviews": {
"description": "The stored rows: suggestions somebody acted on or validated.",
"items": {
"$ref": "#/definitions/SuggestionReviewV1"
},
"type": "array"
},
"summaries": {
"description": "One per analysis that has suggestions, newest first.",
"items": {
"$ref": "#/definitions/ReviewSummaryV1"
},
"type": "array"
}
},
"required": [
"reviews",
"summaries"
],
"title": "ReviewsResponseV1",
"type": "object"
}
},
{
"description": "Explicitly start a baseline and a candidate E2E execution for one suggestion. Both commits (candidate_ref, and baseline_ref or the merge base with origin/main) are resolved with git in the configured code_repository and must be on a remote branch; each stack is the E2E's `harness` template with only the Harness pinned to its commit, run in Docker (runs 1-20, default 5). The criterion is registered before anything starts; the sweep attaches the pair when both finish and computes the evidence. Spends model tokens, and never runs unless called. With dry_run: true it only resolves and checks the refs, scenario and runs and answers with the resolved commits (`baseline`, `candidate`, `warnings`); it registers nothing and calls no E2E function.",
"metadata": {},
"name": "eval::start-validation",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CriterionInputV1": {
"additionalProperties": false,
"description": "What a person commits to before the results exist.",
"properties": {
"direction": {
"allOf": [
{
"$ref": "#/definitions/DirectionV1"
}
],
"description": "The change that counts as an effect, wanted when the candidate moves the baseline mean in this direction."
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"description": "The smallest change of the baseline mean that counts, as an absolute difference in the metric's own unit: signals per run, USD, seconds, calls or tokens; percentage points (0-100) for `pass_rate`. Above 0.",
"format": "double",
"type": "number"
},
"min_runs": {
"description": "Completed runs each side needs for a verdict of improvement (1-20).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"description": "`signal_per_run` only: one of the row's `patterns`.",
"type": [
"string",
"null"
]
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
}
},
"description": "`eval::start-validation`: builds a baseline and a candidate stack from the E2E's `harness` template pinned to two pushed commits and starts both executions in Docker. Spends model tokens, so it only ever runs on an explicit request. With `dry_run` it only resolves and checks the refs.",
"properties": {
"_caller_worker_id": {
"default": null,
"description": "Stamped by the engine on every invocation; callers omit it.",
"type": [
"string",
"null"
]
},
"baseline_ref": {
"default": null,
"description": "Defaults to the merge base of the candidate and `origin/main`.",
"type": [
"string",
"null"
]
},
"by": {
"default": null,
"description": "The author: this name wins over the host's user and the caller's identity.",
"type": [
"string",
"null"
]
},
"candidate_ref": {
"description": "A ref of `code_repository` (a branch, a tag or a commit) that is on a remote branch.",
"type": "string"
},
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionInputV1"
},
{
"type": "null"
}
],
"default": null,
"description": "Registered first, before any execution starts. Required unless `dry_run`, which only checks it against `runs` when present."
},
"dry_run": {
"default": false,
"description": "Resolve and check the refs, the scenario and the runs exactly as a start does, and answer with the commits they resolve to. Registers nothing, calls no E2E function and spends nothing.",
"type": "boolean"
},
"evaluation_id": {
"type": "string"
},
"model": {
"default": "",
"description": "Required unless `dry_run`.",
"type": "string"
},
"provider": {
"default": "",
"description": "Required unless `dry_run`.",
"type": "string"
},
"runs": {
"default": null,
"description": "Runs of the scenario for each side; defaults to 5 (1-20).",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"scenario_id": {
"description": "The harness-e2e scenario to run.",
"type": "string"
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"candidate_ref",
"evaluation_id",
"scenario_id",
"suggestion_index"
],
"title": "StartValidationRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"anyOf": [
{
"$ref": "#/definitions/SuggestionReviewV1"
},
{
"$ref": "#/definitions/ValidationResolutionV1"
}
],
"definitions": {
"ChangeEditV1": {
"additionalProperties": false,
"description": "One edit of what the model saw. `target` is an entry id of the conversation before the decision point, or `system_prompt`. An edit either replaces `find` with `replace` in the target's text or removes the entry.",
"properties": {
"find": {
"type": [
"string",
"null"
]
},
"remove": {
"type": "boolean"
},
"replace": {
"type": [
"string",
"null"
]
},
"target": {
"type": "string"
}
},
"required": [
"target"
],
"type": "object"
},
"CriterionMetricV1": {
"description": "What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).",
"oneOf": [
{
"enum": [
"pass_rate",
"cost_usd",
"duration",
"tokens",
"function_calls"
],
"type": "string"
},
{
"description": "Occurrences of one of the suggestion's patterns per run, counted by the deterministic detectors over each run's transcript.",
"enum": [
"signal_per_run"
],
"type": "string"
}
]
},
"CriterionV1": {
"additionalProperties": false,
"properties": {
"direction": {
"$ref": "#/definitions/DirectionV1"
},
"metric": {
"$ref": "#/definitions/CriterionMetricV1"
},
"min_effect": {
"format": "double",
"type": "number"
},
"min_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pattern": {
"type": [
"string",
"null"
]
},
"registered_at": {
"format": "int64",
"type": "integer"
},
"registered_by": {
"type": "string"
},
"scenario_id": {
"description": "The scenario whose runs are measured.",
"type": "string"
}
},
"required": [
"direction",
"metric",
"min_effect",
"min_runs",
"registered_at",
"registered_by",
"scenario_id"
],
"type": "object"
},
"DirectionV1": {
"enum": [
"decrease",
"increase"
],
"type": "string"
},
"EvidenceRunV1": {
"additionalProperties": false,
"description": "One run of the scenario as the E2E reported it. Runs the infrastructure broke are listed with the E2E's own words, never counted.",
"properties": {
"completed": {
"description": "The E2E reported the run complete with a valid technical outcome.",
"type": "boolean"
},
"completion": {
"description": "`completion` as the E2E reported it.",
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.",
"type": [
"object",
"null"
]
},
"status": {
"description": "`passed` or `failed`, the run's task outcome.",
"type": [
"string",
"null"
]
},
"technical": {
"description": "`technical` as the E2E reported it; anything but `valid` is an infrastructure failure.",
"type": [
"string",
"null"
]
},
"total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"completed",
"run_id"
],
"type": "object"
},
"EvidenceSideV1": {
"additionalProperties": false,
"properties": {
"execution_id": {
"type": "string"
},
"mean": {
"description": "Mean of the criterion's metric over those runs; absent without a criterion or without such runs.",
"format": "double",
"type": [
"number",
"null"
]
},
"n": {
"description": "Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"runs": {
"description": "Every run of the scenario the execution holds.",
"items": {
"$ref": "#/definitions/EvidenceRunV1"
},
"type": "array"
}
},
"required": [
"execution_id",
"n",
"runs"
],
"type": "object"
},
"EvidenceV1": {
"additionalProperties": false,
"description": "Computed in code from the attached pair's runs, never by a model. It is what the person's verdict is shown next to.",
"properties": {
"baseline": {
"$ref": "#/definitions/EvidenceSideV1"
},
"candidate": {
"$ref": "#/definitions/EvidenceSideV1"
},
"computed_at": {
"format": "int64",
"type": "integer"
},
"computed_outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"baseline",
"candidate",
"computed_at",
"computed_outcome",
"reason",
"scenario_id"
],
"type": "object"
},
"FidelityLevelV1": {
"oneOf": [
{
"enum": [
"approximate"
],
"type": "string"
},
{
"description": "The provider counts the rebuilt request like the original, up to a fixed per-request overhead measured at the turn's first step.",
"enum": [
"exact"
],
"type": "string"
}
]
},
"FidelityV1": {
"additionalProperties": false,
"description": "How faithfully the request at the decision point was rebuilt.",
"properties": {
"counted_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"estimator": {
"description": "`provider` when the provider counted, else the estimator's name.",
"type": [
"string",
"null"
]
},
"level": {
"$ref": "#/definitions/FidelityLevelV1"
},
"off_ratio": {
"description": "The unexplained difference over the recorded tokens.",
"format": "double",
"type": [
"number",
"null"
]
},
"reasons": {
"default": [],
"description": "Why the rebuild is approximate; empty when exact.",
"items": {
"type": "string"
},
"type": "array"
},
"recorded_tokens": {
"description": "Input tokens of the original step: input + cache read + cache write.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"level"
],
"type": "object"
},
"LifecycleEventV1": {
"additionalProperties": false,
"description": "One change of status, with who made it.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"note": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
}
},
"required": [
"at",
"by",
"status"
],
"type": "object"
},
"LifecycleStatusV1": {
"description": "Where a suggestion stands. Only people move it: the monitor never does.",
"enum": [
"new",
"accepted",
"in_progress",
"shipped",
"rejected",
"duplicate"
],
"type": "string"
},
"LifecycleV1": {
"additionalProperties": false,
"properties": {
"duplicate_of": {
"description": "What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).",
"type": [
"string",
"null"
]
},
"history": {
"description": "Every change of status, oldest first; empty while `new`.",
"items": {
"$ref": "#/definitions/LifecycleEventV1"
},
"type": "array"
},
"pr": {
"description": "The pull request that implements it (`in_progress`, `shipped`).",
"type": [
"string",
"null"
]
},
"reason": {
"description": "Why it was rejected (`rejected`).",
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/LifecycleStatusV1"
},
"version": {
"description": "The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.",
"type": [
"string",
"null"
]
}
},
"required": [
"history",
"status"
],
"type": "object"
},
"ReplyCallV1": {
"additionalProperties": false,
"description": "One function call of a reply.",
"properties": {
"description": {
"type": [
"string",
"null"
]
},
"payload": {
"description": "The payload as JSON text, cut at 600 characters.",
"type": "string"
},
"target": {
"description": "The function called (through `agent_call` or directly).",
"type": "string"
}
},
"required": [
"payload",
"target"
],
"type": "object"
},
"ReplyUsageV1": {
"additionalProperties": false,
"properties": {
"cache_read_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ReplyV1": {
"additionalProperties": false,
"description": "One reply at the decision point: a sample, or the original step read the same way.",
"properties": {
"calls": {
"items": {
"$ref": "#/definitions/ReplyCallV1"
},
"type": "array"
},
"duration_ms": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"error": {
"type": [
"string",
"null"
]
},
"index": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"signal": {
"description": "Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.",
"type": [
"boolean",
"null"
]
},
"text": {
"type": "string"
},
"thinking": {
"description": "The opening of the reply's reasoning and of its text, 400 characters each.",
"type": "string"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/ReplyUsageV1"
}
],
"default": {}
}
},
"required": [
"calls",
"index",
"text",
"thinking"
],
"type": "object"
},
"ReproductionChangeKindV1": {
"oneOf": [
{
"description": "The context as the model saw it.",
"enum": [
"none"
],
"type": "string"
},
{
"description": "The suggestion's own `check.change`.",
"enum": [
"proposed"
],
"type": "string"
},
{
"description": "Edits a person wrote.",
"enum": [
"custom"
],
"type": "string"
}
]
},
"ReproductionStateV1": {
"enum": [
"running",
"completed",
"failed"
],
"type": "string"
},
"ReproductionV1": {
"additionalProperties": false,
"description": "A replay of the decision point: N replies sampled from the rebuilt request, with or without a change. No function ever runs.",
"properties": {
"by": {
"type": "string"
},
"change": {
"default": [],
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"change_kind": {
"$ref": "#/definitions/ReproductionChangeKindV1"
},
"check": {
"allOf": [
{
"$ref": "#/definitions/SuggestionCheckV1"
}
],
"description": "The decision point and signal this replay used."
},
"cost_unknown_samples": {
"default": 0,
"description": "Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"cost_usd": {
"description": "Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.",
"format": "double",
"type": [
"number",
"null"
]
},
"error": {
"type": [
"string",
"null"
]
},
"fidelity": {
"anyOf": [
{
"$ref": "#/definitions/FidelityV1"
},
{
"type": "null"
}
]
},
"finished_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"id": {
"type": "string"
},
"judge_input_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"default": 0,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"original": {
"anyOf": [
{
"$ref": "#/definitions/ReplyV1"
},
{
"type": "null"
}
],
"description": "The original reply at the decision point, read the same way."
},
"phase": {
"description": "`sampling` or `classifying` while running.",
"type": [
"string",
"null"
]
},
"provider": {
"type": [
"string",
"null"
]
},
"requested": {
"description": "Replies asked for, extensions included.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"samples": {
"default": [],
"items": {
"$ref": "#/definitions/ReplyV1"
},
"type": "array"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ReproductionStateV1"
},
"updated_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"by",
"change_kind",
"check",
"id",
"model",
"requested",
"started_at",
"state",
"updated_at"
],
"type": "object"
},
"ResolvedCommitV1": {
"additionalProperties": false,
"description": "One side of a validation, resolved to the commit the E2E would build.",
"properties": {
"branch": {
"description": "A remote branch that holds the commit, as `git branch -r` names it (`origin/feat`); the one named like the ref when several do.",
"type": "string"
},
"commit": {
"description": "The full commit.",
"type": "string"
},
"short": {
"description": "The first 12 characters of `commit`.",
"type": "string"
}
},
"required": [
"branch",
"commit",
"short"
],
"type": "object"
},
"SignalRuleV1": {
"oneOf": [
{
"description": "The reply asks `engine::functions::info` only for contracts an earlier result in the context already supplied.",
"enum": [
"contract_rediscovery"
],
"type": "string"
},
{
"description": "The reply repeats the last call that failed: same target, equal payload.",
"enum": [
"repeated_error_call"
],
"type": "string"
}
]
},
"SignalV1": {
"additionalProperties": false,
"description": "How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.",
"properties": {
"question": {
"type": [
"string",
"null"
]
},
"rule": {
"anyOf": [
{
"$ref": "#/definitions/SignalRuleV1"
},
{
"type": "null"
}
]
}
},
"type": "object"
},
"SuggestionCheckV1": {
"additionalProperties": false,
"description": "How a suggestion is checked at the step where its behavior happened.",
"properties": {
"change": {
"default": [],
"description": "The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.",
"items": {
"$ref": "#/definitions/ChangeEditV1"
},
"type": "array"
},
"decision_point": {
"description": "The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.",
"type": "string"
},
"signal": {
"$ref": "#/definitions/SignalV1"
}
},
"required": [
"decision_point",
"signal"
],
"type": "object"
},
"SuggestionReviewV1": {
"additionalProperties": false,
"description": "Everything people decided about one suggestion, in the `eval_suggestion` state scope. The title, component and patterns are copied from the suggestion so the row outlives the analysis's retention.",
"properties": {
"criterion": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"evaluation_id": {
"type": "string"
},
"evidence": {
"anyOf": [
{
"$ref": "#/definitions/EvidenceV1"
},
{
"type": "null"
}
]
},
"first_run_at": {
"description": "When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"harness_component": {
"type": "string"
},
"lifecycle": {
"$ref": "#/definitions/LifecycleV1"
},
"patterns": {
"description": "`<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.",
"items": {
"type": "string"
},
"type": "array"
},
"reproductions": {
"description": "Replays of the decision point, oldest first: the base and the tested changes.",
"items": {
"$ref": "#/definitions/ReproductionV1"
},
"type": "array"
},
"run": {
"anyOf": [
{
"$ref": "#/definitions/ValidationRunV1"
},
{
"type": "null"
}
]
},
"scenario_id": {
"description": "The scenario the suggestion's validation plan names.",
"type": [
"string",
"null"
]
},
"suggestion_index": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"title": {
"type": "string"
},
"verdict": {
"anyOf": [
{
"$ref": "#/definitions/VerdictV1"
},
{
"type": "null"
}
]
}
},
"required": [
"evaluation_id",
"harness_component",
"lifecycle",
"patterns",
"suggestion_index",
"title"
],
"type": "object"
},
"ValidationOutcomeV1": {
"oneOf": [
{
"description": "The candidate moved the metric by at least `min_effect` in the wanted direction, with `min_runs` completed runs each side.",
"enum": [
"validated_improvement"
],
"type": "string"
},
{
"description": "The metric moved less than `min_effect`.",
"enum": [
"no_improvement"
],
"type": "string"
},
{
"description": "The metric moved by at least `min_effect` the wrong way.",
"enum": [
"regression"
],
"type": "string"
},
{
"description": "Too few completed runs, missing data or no criterion.",
"enum": [
"inconclusive"
],
"type": "string"
}
]
},
"ValidationResolutionV1": {
"additionalProperties": false,
"description": "What `eval::start-validation` answers to a `dry_run`: the commits both refs resolve to, every start check passed.",
"properties": {
"baseline": {
"$ref": "#/definitions/ResolvedCommitV1"
},
"candidate": {
"$ref": "#/definitions/ResolvedCommitV1"
},
"warnings": {
"description": "What a start would go ahead with but a person should know.",
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"baseline",
"candidate",
"warnings"
],
"type": "object"
},
"ValidationRunStateV1": {
"oneOf": [
{
"enum": [
"failed"
],
"type": "string"
},
{
"description": "Recorded before the executions are requested.",
"enum": [
"starting"
],
"type": "string"
},
{
"description": "Both executions were requested; the sweep waits for them.",
"enum": [
"running"
],
"type": "string"
},
{
"description": "Both ended; the pair is about to be attached.",
"enum": [
"finished"
],
"type": "string"
},
{
"description": "The pair is attached to the analysis and its evidence computed.",
"enum": [
"attached"
],
"type": "string"
}
]
},
"ValidationRunV1": {
"additionalProperties": false,
"description": "The baseline and candidate E2E executions this worker started for a suggestion (`eval::start-validation`).",
"properties": {
"baseline_commit": {
"description": "The full commits the Harness was built from.",
"type": "string"
},
"baseline_execution_id": {
"description": "Absent until the E2E accepted the execution.",
"type": [
"string",
"null"
]
},
"candidate_commit": {
"type": "string"
},
"candidate_execution_id": {
"type": [
"string",
"null"
]
},
"error": {
"description": "Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).",
"type": [
"string",
"null"
]
},
"finished_at": {
"description": "When both executions were seen ended; the sweep stops trying to attach the pair some time after.",
"format": "int64",
"type": [
"integer",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"description": "Runs requested for each side.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
},
"state": {
"$ref": "#/definitions/ValidationRunStateV1"
}
},
"required": [
"baseline_commit",
"candidate_commit",
"model",
"provider",
"runs",
"scenario_id",
"started_at",
"state"
],
"type": "object"
},
"VerdictV1": {
"additionalProperties": false,
"description": "The person's judgment, kept with the criterion it was made against.",
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"by": {
"type": "string"
},
"controls_checked": {
"description": "The non-regression controls of the plan that were checked.",
"items": {
"type": "string"
},
"type": "array"
},
"criterion_snapshot": {
"anyOf": [
{
"$ref": "#/definitions/CriterionV1"
},
{
"type": "null"
}
]
},
"outcome": {
"$ref": "#/definitions/ValidationOutcomeV1"
},
"rationale": {
"type": "string"
}
},
"required": [
"at",
"by",
"controls_checked",
"outcome",
"rationale"
],
"type": "object"
}
},
"description": "The suggestion's row once the executions started, or, for a `dry_run`, the resolution.",
"title": "StartValidationResponseV1"
}
},
{
"description": "Read one analysis record (status, counters, failure stage) or null.",
"metadata": {},
"name": "eval::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
}
},
"required": [
"evaluation_id"
],
"title": "EvaluationIdRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"anyOf": [
{
"$ref": "#/definitions/AnalysisRecordV1"
},
{
"type": "null"
}
],
"definitions": {
"AnalysisCountersV1": {
"additionalProperties": false,
"properties": {
"diagnostics": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"entries": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rejected_suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"sessions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"suggestions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"validations": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"diagnostics",
"entries",
"rejected_suggestions",
"sessions",
"suggestions",
"validations"
],
"type": "object"
},
"AnalysisOriginV1": {
"enum": [
"automatic",
"manual"
],
"type": "string"
},
"AnalysisRecordV1": {
"additionalProperties": false,
"description": "The compact analysis record listed by `eval::list`; evidence and model outputs live separately in the analysis assets.",
"properties": {
"analyst": {
"anyOf": [
{
"$ref": "#/definitions/AnalystRefV1"
},
{
"type": "null"
}
]
},
"code_root": {
"description": "Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.",
"type": [
"string",
"null"
]
},
"completed_at": {
"format": "int64",
"type": [
"integer",
"null"
]
},
"config_revision": {
"type": "string"
},
"counters": {
"$ref": "#/definitions/AnalysisCountersV1"
},
"coverage": {
"anyOf": [
{
"$ref": "#/definitions/CoverageLevelV1"
},
{
"type": "null"
}
]
},
"created_at": {
"format": "int64",
"type": "integer"
},
"criteria_version": {
"type": "string"
},
"deadline": {
"format": "int64",
"type": "integer"
},
"evaluation_id": {
"type": "string"
},
"failure": {
"anyOf": [
{
"$ref": "#/definitions/FailureV1"
},
{
"type": "null"
}
]
},
"harness_version": {
"description": "The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.",
"type": [
"string",
"null"
]
},
"judge_call": {
"anyOf": [
{
"$ref": "#/definitions/JudgeCallV1"
},
{
"type": "null"
}
]
},
"model": {
"allOf": [
{
"$ref": "#/definitions/MonitorModelV1"
}
],
"description": "Frozen at admission: later configuration changes never alter it."
},
"observation_key": {
"type": "string"
},
"observe_since": {
"description": "Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.",
"format": "int64",
"type": "integer"
},
"origin": {
"$ref": "#/definitions/AnalysisOriginV1"
},
"pending_reason": {
"description": "Why collection is still waiting (descendants running, metrics incomplete).",
"type": [
"string",
"null"
]
},
"routing": {
"anyOf": [
{
"$ref": "#/definitions/RoutingV1"
},
{
"type": "null"
}
]
},
"rules_version": {
"type": "string"
},
"schema_version": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"session_id": {
"type": "string"
},
"signals": {
"additionalProperties": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"description": "Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).",
"type": "object"
},
"source_title": {
"description": "The session's title, once collection has read it.",
"type": [
"string",
"null"
]
},
"stages": {
"description": "When each status began, oldest first.",
"items": {
"$ref": "#/definitions/StageTimeV1"
},
"type": "array"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"supersedes": {
"description": "The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.",
"type": [
"string",
"null"
]
},
"turn_id": {
"type": "string"
},
"updated_at": {
"format": "int64",
"type": "integer"
},
"usage": {
"allOf": [
{
"$ref": "#/definitions/MonitorUsageV1"
}
],
"description": "The monitor's own consumption, apart from the observed task."
}
},
"required": [
"config_revision",
"counters",
"created_at",
"criteria_version",
"deadline",
"evaluation_id",
"model",
"observation_key",
"observe_since",
"origin",
"rules_version",
"schema_version",
"session_id",
"stages",
"status",
"step",
"turn_id",
"updated_at",
"usage"
],
"type": "object"
},
"AnalystRefV1": {
"additionalProperties": false,
"properties": {
"sent_at": {
"format": "int64",
"type": "integer"
},
"session_id": {
"type": "string"
},
"turn_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"sent_at",
"session_id"
],
"type": "object"
},
"CoverageLevelV1": {
"oneOf": [
{
"description": "Every in-scope transcript was read and the model context holds every diagnostic's evidence.",
"enum": [
"complete"
],
"type": "string"
},
{
"description": "Previews omit or shorten entries; the full transcripts were read.",
"enum": [
"partial"
],
"type": "string"
},
{
"description": "Required evidence did not fit; no conclusion of healthy behavior.",
"enum": [
"insufficient"
],
"type": "string"
}
]
},
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
},
"FailureV1": {
"additionalProperties": false,
"description": "The stage that failed and why. `stage` is the status the analysis was in.",
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
},
"stage": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"code",
"message",
"stage"
],
"type": "object"
},
"JudgeCallV1": {
"additionalProperties": false,
"description": "A Jev call started by this analysis. Persisted before the call so a restart can tell a lost answer from one never requested.",
"properties": {
"deadline": {
"format": "int64",
"type": "integer"
},
"request_id": {
"type": "string"
},
"started_at": {
"format": "int64",
"type": "integer"
}
},
"required": [
"deadline",
"request_id",
"started_at"
],
"type": "object"
},
"MonitorModelV1": {
"additionalProperties": false,
"description": "The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.",
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"provider_options": {
"additionalProperties": true,
"description": "Provider-native options, namespaced by the selected provider id.",
"type": [
"object",
"null"
]
},
"thinking_level": {
"anyOf": [
{
"$ref": "#/definitions/ThinkingLevel"
},
{
"type": "null"
}
]
}
},
"required": [
"model",
"provider"
],
"type": "object"
},
"MonitorUsageV1": {
"additionalProperties": false,
"description": "Known usage only: absent counters are unknown, never zero.",
"properties": {
"judge_calls": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"judge_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"judge_usage_complete": {
"description": "False when Jev reported incomplete usage or failed.",
"type": "boolean"
},
"llm_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"llm_input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"llm_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"judge_calls",
"judge_input_tokens",
"judge_output_tokens",
"judge_usage_complete"
],
"type": "object"
},
"RoutingReasonV1": {
"oneOf": [
{
"enum": [
"needs_investigation"
],
"type": "string"
},
{
"description": "Legacy: no longer produced; kept so stored records still deserialize.",
"enum": [
"diagnostics"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"insufficient_evidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"low_confidence"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"coverage_insufficient"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"audit_sample"
],
"type": "string"
},
{
"description": "Legacy: no longer produced.",
"enum": [
"manual_request"
],
"type": "string"
}
]
},
"RoutingV1": {
"additionalProperties": false,
"properties": {
"investigate": {
"type": "boolean"
},
"reasons": {
"items": {
"$ref": "#/definitions/RoutingReasonV1"
},
"type": "array"
}
},
"required": [
"investigate",
"reasons"
],
"type": "object"
},
"StageTimeV1": {
"additionalProperties": false,
"properties": {
"at": {
"format": "int64",
"type": "integer"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"at",
"status"
],
"type": "object"
},
"ThinkingLevel": {
"oneOf": [
{
"enum": [
"minimal",
"low",
"medium",
"high",
"xhigh"
],
"type": "string"
},
{
"description": "No reasoning at all; honoured only where the model advertises `supports_thinking_off` (elsewhere the provider falls back to its own lowest setting, or omits the parameter, and warns).",
"enum": [
"off"
],
"type": "string"
}
]
}
},
"title": "Nullable_AnalysisRecordV1"
}
},
{
"description": "Internal durable analysis step.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::step",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"evaluation_id": {
"type": "string"
},
"step": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"evaluation_id",
"step"
],
"title": "StepRequestV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"EvalStatusV1": {
"enum": [
"queued",
"collecting",
"judging",
"investigating",
"completed",
"failed",
"cancelled"
],
"type": "string"
}
},
"properties": {
"skipped": {
"type": "boolean"
},
"status": {
"$ref": "#/definitions/EvalStatusV1"
}
},
"required": [
"skipped",
"status"
],
"title": "StepResponseV1",
"type": "object"
}
},
{
"description": "Internal recovery, deadline and retention sweep.",
"metadata": {
"internal": true,
"trace_hidden": true
},
"name": "eval::sweep",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"scheduled_at": {
"default": null,
"format": "int64",
"type": [
"integer",
"null"
]
},
"scheduled_time": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "SweepEventV1",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"properties": {
"expired": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"reconciled": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"requeued": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"retained_deleted": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"expired",
"reconciled",
"requeued",
"retained_deleted"
],
"title": "SweepResponseV1",
"type": "object"
}
},
{
"description": "Serve the eval worker's injected console UI assets (the target of its ade::ui::asset observers; the console calls it through each asset's observer with {path}).",
"metadata": {
"internal": true
},
"name": "eval::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Input of the content function: the console asks for one asset by path.",
"properties": {
"path": {
"description": "The asset path from the observer's args (e.g. `state/page.js`).",
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentInput",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Output of the content function.",
"properties": {
"content": {
"description": "The asset source, verbatim.",
"type": "string"
},
"content_type": {
"description": "MIME type the console should serve the asset with.",
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResult",
"type": "object"
}
}
]
}