Skip to content
Sign in

eval

v0.2.24

Session monitor for the Harness (deterministic diagnostics, Jev triage through judge, investigation by the user-chosen LLM, E2E validation plans; inactive until configured).

Install

bash
iii trigger compose::add worker=eval@0.2.24

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions19

eval::analyze-session

Function

Analyze the latest definitive turn of a finished root Harness session. Returns the existing analysis of that turn unless reanalyze is true; reanalysis needs the previous analysis to be terminal. Works while automatic observation is paused.

Request
  • reanalyzeboolean

    Explicitly create another analysis of an already analyzed turn.

  • session_idstringrequired
Response
  • evaluation_idstringrequired
  • reusedbooleanrequired

    True when an existing analysis of the same session turn was returned.

  • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …

eval::attach-validation

Function

Link a baseline and a candidate E2E execution (read through e2e::dashboard::execution-get) to one suggestion of a terminal analysis. The link records report availability and, when the suggestion's scenario is known, the evidence computed from the runs; it is not a verdict and starts no campaign.

Request
  • baseline_execution_idstringrequired
  • candidate_execution_idstringrequired
  • dry_runboolean

    Look both executions up and return the link without saving it.

  • evaluation_idstringrequired
  • suggestion_indexinteger· uintrequiredmin 0
Response
  • linkobjectrequired

    Baseline and candidate executions linked to one suggestion. A link is a reference, not a verdict: the improvement claim belongs to the E2E comparison and its criteria.

    • attached_atinteger· int64required
    • baselineobjectrequired

      What `e2e::dashboard::execution-get` reported at lookup time.

      • assessmentsany of
        Show 2 nested
        any of (2)
        Variant 1
        • passedinteger· uint32requiredmin 0
        • totalinteger· uint32requiredmin 0
        Variant 2
        valuenull
      • availabilitystring
      • conclusionstring
      • e2e_revisionstring
      • engine_versionstring
      • evidence_errorstring
      • execution_idstringrequired
      • harness_versionstring
      • labelstring
      • modelstring
      • providerstring
      • reportsobject[]required
        Show 3 nested
        • availablebooleanrequired
        • scenario_idstringrequired
        • subject_idstring
      • reports_availablebooleanrequired

        True only when the execution reported retained reports.

      • scenariosobject[]required
        Show 13 nested
        • behavior_sha256string
        • completed_runsinteger· uint32min 0
        • contract_fingerprintstring
        • cost_usd_per_runnumber· double

          The cohort's total cost divided by its completed runs; the total also covers failed attempts.

        • mean_scorenumber· double

          Mean score of the scored runs (0 to 100).

        • measuresobjectrequired

          `function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.

        • median_wall_time_msnumber· double
        • p50_function_callsnumber· double
        • pass_ratenumber· double

          The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero. Share of the planned runs that passed (0 to 1).

        • planned_runsinteger· uint32min 0
        • run_countinteger· uint32requiredmin 0
        • scenario_idstringrequired
        • total_tokens_per_runnumber· double

          The cohort's total tokens divided by its completed runs.

      • started_atinteger· int64

        When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.

      • statusstring
    • candidateobjectrequired

      What `e2e::dashboard::execution-get` reported at lookup time.

      • assessmentsany of
        Show 2 nested
        any of (2)
        Variant 1
        • passedinteger· uint32requiredmin 0
        • totalinteger· uint32requiredmin 0
        Variant 2
        valuenull
      • availabilitystring
      • conclusionstring
      • e2e_revisionstring
      • engine_versionstring
      • evidence_errorstring
      • execution_idstringrequired
      • harness_versionstring
      • labelstring
      • modelstring
      • providerstring
      • reportsobject[]required
        Show 3 nested
        • availablebooleanrequired
        • scenario_idstringrequired
        • subject_idstring
      • reports_availablebooleanrequired

        True only when the execution reported retained reports.

      • scenariosobject[]required
        Show 13 nested
        • behavior_sha256string
        • completed_runsinteger· uint32min 0
        • contract_fingerprintstring
        • cost_usd_per_runnumber· double

          The cohort's total cost divided by its completed runs; the total also covers failed attempts.

        • mean_scorenumber· double

          Mean score of the scored runs (0 to 100).

        • measuresobjectrequired

          `function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.

        • median_wall_time_msnumber· double
        • p50_function_callsnumber· double
        • pass_ratenumber· double

          The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero. Share of the planned runs that passed (0 to 1).

        • planned_runsinteger· uint32min 0
        • run_countinteger· uint32requiredmin 0
        • scenario_idstringrequired
        • total_tokens_per_runnumber· double

          The cohort's total tokens divided by its completed runs.

      • started_atinteger· int64

        When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.

      • statusstring
    • comparabilityobjectrequired

      The conditions that must match for a fair comparison; only the Harness version is meant to differ.

      • checksobject[]required
        Show 4 nested
        • baselinestring
        • candidatestring
        • fieldstringrequired
        • matchesbooleanrequired
      • comparablebooleanrequired
    • suggestion_indexinteger· uintrequiredmin 0
  • savedbooleanrequired

eval::cancel

Function

Cancel an analysis: signals its Jev call and stops its investigation session. Never touches the observed session or the captured evidence.

Request
  • evaluation_idstringrequired
Response
  • cancelledbooleanrequired
  • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …

eval::completed

Function

A session-monitor analysis reached a terminal status (completed, failed or cancelled). Completion describes the monitor, not a validated improvement.

Request
  • evaluation_idstring
Response
valueunknown

eval::config

Function

Read the monitor configuration (null until configured), whether the harness::turn-completed observation trigger was requested, the latest rejection (capacity or daily cost cap), the day's known investigation cost against the cap with what one analysis has cost, and, with check_providers, whether the Jev triage provider answers.

Request
  • check_providersboolean

    Also ask the triage provider whether it answers (`judge::models::list`).

Response
  • configany of

    `null` until the user configures the monitor.

    any of (2)
    Variant 1
    • code_repositorystring

      The codebase directory the investigation reads (see `ConfigureRequestV1`).

    • daily_cost_cap_usdnumber· double

      The daily cap on known investigation cost (see `ConfigureRequestV1`).

    • enabledbooleanrequired
    • enabled_sinceinteger· int64

      When automatic observation last went from paused to on; analyses use it to bound which earlier turns they may report. Unchanged by edits that keep observation on.

    • modelobjectrequired

      The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.

      Show 4 nested
      • modelstringrequired
      • providerstringrequired
      • provider_optionsobject

        Provider-native options, namespaced by the selected provider id.

      • thinking_levelany of
        Show 2 nested
        any of (2)
        Variant 1
        one of (2)
        Variant 1
        valuestringenum minimal, low, medium, high, xhigh
        Variant 2
        valuestringenum off
        Variant 2
        valuenull
    • revisionstringrequired

      SHA-256 of the effective configuration; analyses keep the revision they were admitted with.

    • updated_atinteger· int64required
    Variant 2
    valuenull
  • costall ofrequired

    What the investigations cost so far, to decide before enabling.

    all of (1)
    Variant 1
    • cap_usdnumber· double
    • cappedbooleanrequired

      A cap is set and `today_capture_usd`, plus the median cost of each investigation still running, reached it: automatic observation admits and investigates nothing until the next UTC day.

    • per_analysisall ofrequired

      What one investigation has cost, from the history.

      Show 1 nested
      all of (1)
      Variant 1
      • countinteger· uint32requiredmin 0

        Analyses with a known cost; `min`, `median` and `max` come from them.

      • maxnumber· double
      • mediannumber· double
      • minnumber· double
      • unknowninteger· uint32requiredmin 0

        Analyses that investigated but reported no cost.

    • sinceinteger· int64required

      Start (ms since the Unix epoch) of the UTC day the `today_*` values cover. The monitor has no timezone setting, so a day is a UTC day.

    • today_capture_usdnumber· doublerequired

      The capture bucket, the only one the cap compares: the larger of the day's persisted capture spend (deleting an analysis does not give it back) and the sum of the stored analyses' `usage.llm_cost_usd`. A day persisted before the buckets counts entirely as capture.

    • today_replay_unknowninteger· uint32requiredmin 0

      Replay samples of the day without a cost, failed ones included. They add nothing to `today_replay_usd`, but their cost is unknown, not zero.

    • today_replay_usdnumber· doublerequired

      The replay bucket: the known cost of the day's `eval::reproduce` samples. Never capped.

    • today_unknowninteger· uint32requiredmin 0

      Analyses of the day whose investigation started but reported no cost. They add nothing to `today_capture_usd`, but their cost is unknown, not zero.

    • today_usdnumber· doublerequired

      Everything the monitor is known to have spent since `since`: `today_capture_usd` + `today_replay_usd`. The cap does not compare this.

  • last_rejectionany of

    The latest turn not admitted: the monitor was at capacity or the daily cost cap was reached (`reason`).

    any of (2)
    Variant 1
    • atinteger· int64required
    • reasonall of

      Why the turn was not admitted; rows saved before the cost cap existed read as `at_capacity`.

      Show 1 nested
      all of (1)
      Variant 1
      one of (2)
      Variant 1
      valuestringenum at_capacity
      Variant 2
      valuestringenum cost_cap
    • session_idstringrequired
    • turn_idstringrequired
    Variant 2
    valuenull
  • limitsall ofrequired

    The limits this version enforces, so the console never restates them.

    all of (1)
    Variant 1
    • analysis_budget_msinteger· uint64requiredmin 0

      Whole-analysis budget from admission, queue wait included.

    • assets_bytesinteger· uint64requiredmin 0
    • investigation_code_max_total_tokensinteger· uint64requiredmin 0

      Total-token cap of an investigation that has code access.

    • investigation_code_max_turnsinteger· uint32requiredmin 0

      Generate-step cap of an investigation that has code access.

    • investigation_max_output_tokensinteger· uint64requiredmin 0
    • investigation_max_total_tokensinteger· uint64requiredmin 0
    • investigation_max_turnsinteger· uint32requiredmin 0
    • judge_timeout_msinteger· uint64requiredmin 0
    • max_active_analysesinteger· uint32requiredmin 0
    • model_context_bytesinteger· uint64requiredmin 0

      Serialized JSON shown to a model (bytes, not tokens).

    • queue_concurrencyinteger· uint32requiredmin 0
    • retention_daysinteger· uint32requiredmin 0
    • retention_max_terminalinteger· uint32requiredmin 0
  • observer_boundbooleanrequired

    False when the `harness::turn-completed` binding was refused locally: automatic observation is unavailable even if `config.enabled` is true. The SDK does not wait for the engine's acknowledgement, so true means the binding was requested, not confirmed.

  • observer_errorstring
  • triageany of

    Present when the request asked to check the triage provider.

    any of (2)
    Variant 1
    • availablebooleanrequired
    • checked_atinteger· int64required
    • codestring

      The provider's error code (`missing_key`, `provider_unavailable`…), or `unreachable` when the hub did not answer.

    • modelsstring[]required
    • providerstringrequired
    Variant 2
    valuenull

eval::configure

Function

Choose the Harness model/provider the session monitor uses for investigations and enable or pause automatic observation. A new selection is checked against router::models::list; credentials stay in the providers and are never accepted here. An optional code_repository (an absolute directory on this host, normally the iii workers repository) lets the investigation read that code like a chat with the directory selected, with only read-only functions allowed (coder::search, coder::tree, coder::read-file, github::pr::list and engine::functions::info). An optional daily_cost_cap_usd pauses automatic observation for the rest of the UTC day once the known investigation cost reaches it (manual analyses are never refused, and replay spend is reported but not counted). Analyses already admitted keep the configuration they started with.

Request
  • _caller_worker_idstring

    Stamped by the engine on every invocation; callers omit it. Listed so the closed schema still accepts the engine's metadata.

  • code_repositorystring

    Absolute path of the codebase directory the investigation reads and runs in (normally the iii workers repository, checked out on the monitor's host); absent means no code access.

  • daily_cost_cap_usdnumber· double

    US dollars of known investigation cost per UTC day after which automatic observation admits nothing until the next day; absent means no cap. A manual `eval::analyze-session` is never refused by it.

  • enabledbooleanrequired
  • modelobjectrequired

    The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.

    • modelstringrequired
    • providerstringrequired
    • provider_optionsobject

      Provider-native options, namespaced by the selected provider id.

    • thinking_levelany of
      any of (2)
      Variant 1
      one of (2)
      Variant 1
      valuestringenum minimal, low, medium, high, xhigh
      Variant 2
      valuestringenum off
      Variant 2
      valuenull
Response
  • code_repositorystring

    The codebase directory the investigation reads (see `ConfigureRequestV1`).

  • daily_cost_cap_usdnumber· double

    The daily cap on known investigation cost (see `ConfigureRequestV1`).

  • enabledbooleanrequired
  • enabled_sinceinteger· int64

    When automatic observation last went from paused to on; analyses use it to bound which earlier turns they may report. Unchanged by edits that keep observation on.

  • modelobjectrequired

    The Harness model the user chose for investigations. Credentials stay in the provider worker; this request never carries a key.

    • modelstringrequired
    • providerstringrequired
    • provider_optionsobject

      Provider-native options, namespaced by the selected provider id.

    • thinking_levelany of
      any of (2)
      Variant 1
      one of (2)
      Variant 1
      valuestringenum minimal, low, medium, high, xhigh
      Variant 2
      valuestringenum off
      Variant 2
      valuenull
  • revisionstringrequired

    SHA-256 of the effective configuration; analyses keep the revision they were admitted with.

  • updated_atinteger· int64required

eval::delete

Function

Delete a terminal analysis and its stored previews. The turn stays marked as analyzed until retention, so it is not re-admitted automatically.

Request
  • evaluation_idstringrequired
Response
  • deletedbooleanrequired

eval::list

Function

List recent monitor analyses as compact records, newest first, without evidence; observation_key keeps only the analyses of one observed turn.

Request
  • limitinteger· uint32min 0
  • observation_keystring

    Only the analyses of this observed turn (the `observation_key` of a record), reanalyses included.

Response
  • evaluationsobject[]required
    • analystany of
      any of (2)
      Variant 1
      • sent_atinteger· int64required
      • session_idstringrequired
      • turn_idstring
      Variant 2
      valuenull
    • code_rootstring

      Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.

    • completed_atinteger· int64
    • config_revisionstringrequired
    • countersobjectrequired
      • diagnosticsinteger· uint32requiredmin 0
      • entriesinteger· uint32requiredmin 0
      • rejected_suggestionsinteger· uint32requiredmin 0
      • sessionsinteger· uint32requiredmin 0
      • suggestionsinteger· uint32requiredmin 0
      • validationsinteger· uint32requiredmin 0
    • coverageany of
      any of (2)
      Variant 1
      one of (3)
      Variant 1
      valuestringenum complete
      Variant 2
      valuestringenum partial
      Variant 3
      valuestringenum insufficient
      Variant 2
      valuenull
    • created_atinteger· int64required
    • criteria_versionstringrequired
    • deadlineinteger· int64required
    • evaluation_idstringrequired
    • failureany of
      any of (2)
      Variant 1
      • codestringrequired
      • messagestringrequired
      • stagestringrequiredenum queued, collecting, judging, investigating, completed, failed, …
      Variant 2
      valuenull
    • harness_versionstring

      The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.

    • judge_callany of
      any of (2)
      Variant 1
      • deadlineinteger· int64required
      • request_idstringrequired
      • started_atinteger· int64required
      Variant 2
      valuenull
    • modelall ofrequired

      Frozen at admission: later configuration changes never alter it.

      all of (1)
      Variant 1
      • modelstringrequired
      • providerstringrequired
      • provider_optionsobject

        Provider-native options, namespaced by the selected provider id.

      • thinking_levelany of
        Show 2 nested
        any of (2)
        Variant 1
        one of (2)
        Variant 1
        valuestringenum minimal, low, medium, high, xhigh
        Variant 2
        valuestringenum off
        Variant 2
        valuenull
    • observation_keystringrequired
    • observe_sinceinteger· int64required

      Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.

    • originstringrequiredenum automatic, manual
    • pending_reasonstring

      Why collection is still waiting (descendants running, metrics incomplete).

    • routingany of
      any of (2)
      Variant 1
      • investigatebooleanrequired
      • reasonsone of[]required
        Show 7 nested
        one of (7)
        Variant 1
        valuestringenum needs_investigation
        Variant 2
        valuestringenum diagnostics
        Variant 3
        valuestringenum insufficient_evidence
        Variant 4
        valuestringenum low_confidence
        Variant 5
        valuestringenum coverage_insufficient
        Variant 6
        valuestringenum audit_sample
        Variant 7
        valuestringenum manual_request
      Variant 2
      valuenull
    • rules_versionstringrequired
    • schema_versioninteger· uint32requiredmin 0
    • session_idstringrequired
    • signalsobject

      Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).

    • source_titlestring

      The session's title, once collection has read it.

    • stagesobject[]required

      When each status began, oldest first.

      • atinteger· int64required
      • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
    • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
    • stepinteger· uint64requiredmin 0
    • supersedesstring

      The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.

    • turn_idstringrequired
    • updated_atinteger· int64required
    • usageall ofrequired

      The monitor's own consumption, apart from the observed task.

      all of (1)
      Variant 1
      • judge_callsinteger· uint32requiredmin 0
      • judge_input_tokensinteger· uint64requiredmin 0
      • judge_output_tokensinteger· uint64requiredmin 0
      • judge_usage_completebooleanrequired

        False when Jev reported incomplete usage or failed.

      • llm_cost_usdnumber· double
      • llm_input_tokensinteger· uint64min 0
      • llm_output_tokensinteger· uint64min 0

eval::on-turn-completed

Function

Internal harness turn-completed admission and wake-up.

Request
  • parentunknown
  • parent_session_idstring
  • session_idstring
  • terminalboolean
  • timestampinteger· int64
  • turn_idstring
Response
  • evaluation_idstring
  • outcomestringrequiredenum ignored, progress, descendant, monitor_session, disabled, stale, …

eval::recurrence

Function

For a suggestion marked shipped with its version, how often its patterns were found per analysis on Harness versions before it against from it on. Analyses without a semantic Harness version are left out and counted.

Request
  • evaluation_idstringrequired
  • suggestion_indexinteger· uintrequiredmin 0
Response
  • beforeobjectrequired
    • analysesinteger· uint32requiredmin 0

      Analyses whose collection finished and whose Harness version is known.

    • patternsobject[]required
      • analyses_withinteger· uint32requiredmin 0

        Analyses with at least one occurrence.

      • occurrencesinteger· uint32requiredmin 0

        Occurrences summed over the analyses.

      • patternstringrequired
      • per_analysisnumber· double

        Occurrences per analysis; absent when there is no analysis.

  • evaluation_idstringrequired
  • from_versionobjectrequired
    • analysesinteger· uint32requiredmin 0

      Analyses whose collection finished and whose Harness version is known.

    • patternsobject[]required
      • analyses_withinteger· uint32requiredmin 0

        Analyses with at least one occurrence.

      • occurrencesinteger· uint32requiredmin 0

        Occurrences summed over the analyses.

      • patternstringrequired
      • per_analysisnumber· double

        Occurrences per analysis; absent when there is no analysis.

  • suggestion_indexinteger· uintrequiredmin 0
  • versionstringrequired

    The version the suggestion shipped in, where the windows split.

  • without_versioninteger· uint32requiredmin 0

    Analyses left out because they carry no (or no semantic) Harness version.

eval::reproduce

Function

Replay the decision point of a suggestion: rebuild the request the model received at that step of the observed turn (optionally with edits that stand for the proposed change), sample the next reply N times (default 20) without running any function, and read the suggestion's signal in each reply. Spends model tokens; with dry_run it only rebuilds the request, checks its fidelity against the recorded usage and reports the original reply. extend adds replies to an earlier reproduction, or with samples 0 finishes one that failed. Results land in eval::result reviews.

Request
  • _caller_worker_idstring

    Stamped by the engine on every invocation; callers omit it.

  • bystring
  • changeall of
    all of (1)
    Variant 1
    one of (3)
    Variant 1
    • kindstringrequiredenum none
    Variant 2
    • kindstringrequiredenum proposed
    Variant 3
    • editsobject[]required
      Show 4 nested
      • findstring
      • removeboolean
      • replacestring
      • targetstringrequired
    • kindstringrequiredenum custom
  • checkany of

    The decision point and signal, for a suggestion without a `check` (an older analysis) or to override it.

    any of (2)
    Variant 1
    • changeobject[]

      The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

      Show 4 nested
      • findstring
      • removeboolean
      • replacestring
      • targetstringrequired
    • decision_pointstringrequired

      The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

    • signalobjectrequired

      How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

      Show 2 nested
      • questionstring
      • ruleany of
        Show 2 nested
        any of (2)
        Variant 1
        one of (2)
        Variant 1
        valuestringenum contract_rediscovery
        Variant 2
        valuestringenum repeated_error_call
        Variant 2
        valuenull
    Variant 2
    valuenull
  • dry_runboolean

    Rebuild the request and check its fidelity without sampling: nothing is spent or stored.

  • evaluation_idstringrequired
  • extendstring

    Add replies to this earlier reproduction of the suggestion, with its change; with `samples: 0`, only finish what a failure left undone.

  • samplesinteger· uint32min 0

    Replies to sample, 1–50; default 20.

  • suggestion_indexinteger· uintrequiredmin 0
Response
  • previewany of
    any of (2)
    Variant 1
    • fidelityobjectrequired

      How faithfully the request at the decision point was rebuilt.

      Show 6 nested
      • counted_tokensinteger· uint64min 0
      • estimatorstring

        `provider` when the provider counted, else the estimator's name.

      • levelone ofrequired
        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum approximate
        Variant 2
        valuestringenum exact
      • off_rationumber· double

        The unexplained difference over the recorded tokens.

      • reasonsstring[]

        Why the rebuild is approximate; empty when exact.

      • recorded_tokensinteger· uint64min 0

        Input tokens of the original step: input + cache read + cache write.

    • modelstringrequired
    • originalobjectrequired

      One reply at the decision point: a sample, or the original step read the same way.

      Show 8 nested
      • callsobject[]required
        Show 3 nested
        • descriptionstring
        • payloadstringrequired

          The payload as JSON text, cut at 600 characters.

        • targetstringrequired

          The function called (through `agent_trigger` or directly).

      • duration_msinteger· uint64min 0
      • errorstring
      • indexinteger· uint32requiredmin 0
      • signalboolean

        Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

      • textstringrequired
      • thinkingstringrequired

        The opening of the reply's reasoning and of its text, 400 characters each.

      • usageall of
        Show 1 nested
        all of (1)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
    • providerstring
    • samplesinteger· uint32requiredmin 0
    • step_cost_usdnumber· double

      What the original step cost; null when the provider reported nothing.

    Variant 2
    valuenull
  • reproduction_idstring
  • reviewany of
    any of (2)
    Variant 1
    • criterionany of
      Show 2 nested
      any of (2)
      Variant 1
      • directionstringrequiredenum decrease, increase
      • metricone ofrequired

        What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
        Variant 2
        valuestringenum signal_per_run
      • min_effectnumber· doublerequired
      • min_runsinteger· uint32requiredmin 0
      • patternstring
      • registered_atinteger· int64required
      • registered_bystringrequired
      • scenario_idstringrequired

        The scenario whose runs are measured.

      Variant 2
      valuenull
    • evaluation_idstringrequired
    • evidenceany of
      Show 2 nested
      any of (2)
      Variant 1
      • baselineobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • candidateobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • computed_atinteger· int64required
      • computed_outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • reasonstringrequired
      • scenario_idstringrequired
      Variant 2
      valuenull
    • first_run_atinteger· int64

      When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.

    • harness_componentstringrequired
    • lifecycleobjectrequired
      Show 6 nested
      • duplicate_ofstring

        What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).

      • historyobject[]required

        Every change of status, oldest first; empty while `new`.

        Show 4 nested
        • atinteger· int64required
        • bystringrequired
        • notestring
        • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

          Where a suggestion stands. Only people move it: the monitor never does.

      • prstring

        The pull request that implements it (`in_progress`, `shipped`).

      • reasonstring

        Why it was rejected (`rejected`).

      • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

        Where a suggestion stands. Only people move it: the monitor never does.

      • versionstring

        The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.

    • patternsstring[]required

      `<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.

    • reproductionsobject[]

      Replays of the decision point, oldest first: the base and the tested changes.

      Show 21 nested
      • bystringrequired
      • changeobject[]
        Show 4 nested
        • findstring
        • removeboolean
        • replacestring
        • targetstringrequired
      • change_kindone ofrequired
        Show 3 nested
        one of (3)
        Variant 1
        valuestringenum none
        Variant 2
        valuestringenum proposed
        Variant 3
        valuestringenum custom
      • checkall ofrequired

        The decision point and signal this replay used.

        Show 1 nested
        all of (1)
        Variant 1
        • changeobject[]

          The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

          Show 4 nested
          • findstring
          • removeboolean
          • replacestring
          • targetstringrequired
        • decision_pointstringrequired

          The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

        • signalobjectrequired

          How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

          Show 2 nested
          • questionstring
          • ruleany of
            Show 2 nested
            any of (2)
            Variant 1
            one of (2)
            Variant 1
            valuestringenum contract_rediscovery
            Variant 2
            valuestringenum repeated_error_call
            Variant 2
            valuenull
      • cost_unknown_samplesinteger· uint32min 0

        Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.

      • cost_usdnumber· double

        Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.

      • errorstring
      • fidelityany of
        Show 2 nested
        any of (2)
        Variant 1
        • counted_tokensinteger· uint64min 0
        • estimatorstring

          `provider` when the provider counted, else the estimator's name.

        • levelone ofrequired
          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum approximate
          Variant 2
          valuestringenum exact
        • off_rationumber· double

          The unexplained difference over the recorded tokens.

        • reasonsstring[]

          Why the rebuild is approximate; empty when exact.

        • recorded_tokensinteger· uint64min 0

          Input tokens of the original step: input + cache read + cache write.

        Variant 2
        valuenull
      • finished_atinteger· int64
      • idstringrequired
      • judge_input_tokensinteger· uint64min 0
      • judge_output_tokensinteger· uint64min 0
      • modelstringrequired
      • originalany of

        The original reply at the decision point, read the same way.

        Show 2 nested
        any of (2)
        Variant 1
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
        Variant 2
        valuenull
      • phasestring

        `sampling` or `classifying` while running.

      • providerstring
      • requestedinteger· uint32requiredmin 0

        Replies asked for, extensions included.

      • samplesobject[]
        Show 8 nested
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
      • started_atinteger· int64required
      • statestringrequiredenum running, completed, failed
      • updated_atinteger· int64required
    • runany of
      Show 2 nested
      any of (2)
      Variant 1
      • baseline_commitstringrequired

        The full commits the Harness was built from.

      • baseline_execution_idstring

        Absent until the E2E accepted the execution.

      • candidate_commitstringrequired
      • candidate_execution_idstring
      • errorstring

        Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).

      • finished_atinteger· int64

        When both executions were seen ended; the sweep stops trying to attach the pair some time after.

      • modelstringrequired
      • providerstringrequired
      • runsinteger· uint32requiredmin 0

        Runs requested for each side.

      • scenario_idstringrequired
      • started_atinteger· int64required
      • stateone ofrequired
        Show 5 nested
        one of (5)
        Variant 1
        valuestringenum failed
        Variant 2
        valuestringenum starting
        Variant 3
        valuestringenum running
        Variant 4
        valuestringenum finished
        Variant 5
        valuestringenum attached
      Variant 2
      valuenull
    • scenario_idstring

      The scenario the suggestion's validation plan names.

    • suggestion_indexinteger· uintrequiredmin 0
    • titlestringrequired
    • verdictany of
      Show 2 nested
      any of (2)
      Variant 1
      • atinteger· int64required
      • bystringrequired
      • controls_checkedstring[]required

        The non-regression controls of the plan that were checked.

      • criterion_snapshotany of
        Show 2 nested
        any of (2)
        Variant 1
        • directionstringrequiredenum decrease, increase
        • metricone ofrequired

          What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
          Variant 2
          valuestringenum signal_per_run
        • min_effectnumber· doublerequired
        • min_runsinteger· uint32requiredmin 0
        • patternstring
        • registered_atinteger· int64required
        • registered_bystringrequired
        • scenario_idstringrequired

          The scenario whose runs are measured.

        Variant 2
        valuenull
      • outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • rationalestringrequired
      Variant 2
      valuenull
    Variant 2
    valuenull

eval::result

Function

Read an analysis with its captured evidence, deterministic diagnostics, Jev triage, suggestions, the monitor's own consumption and linked E2E executions, or null. A completed analysis is not a validated improvement.

Request
  • evaluation_idstringrequired
Response
any of (2)
Variant 1
  • assetsobjectrequired
    • captureany of

      The observed turn's record, copied when the evidence was captured: what a reproduction rebuilds the request from.

      Show 2 nested
      any of (2)
      Variant 1
      • hook_guidance_tokensinteger· uint64min 0

        Tokens `pre_generate` hooks added at the turn's last step: above zero, the system prompt the model saw is not stored anywhere.

      • optionsunknownrequired

        The Harness's `TurnOptions`, unmodified.

      • prompt_surface_digeststring
      • recordunknown

        The whole `TurnRecord` as read when the evidence was captured (it repeats `options`): what a later fork of the session needs, since the Harness overwrites it with the next turn. Absent when it would have taken the assets over their size limit (`record_omitted` says so) and on analyses made before it was kept.

      • record_omittedstring

        Why `record` was not kept.

      • turn_idstringrequired
      Variant 2
      valuenull
    • evaluation_idstringrequired
    • investigationany of
      Show 2 nested
      any of (2)
      Variant 1
      • code_rootstring

        The directory the analyst read and ran in; absent when access was off.

      • completed_atinteger· int64required
      • effective_modelstring
      • effective_providerstring
      • metricsany of
        Show 2 nested
        any of (2)
        Variant 1
        • by_sessionobject[]required
          Show 14 nested
          • by_modelobject[]

            The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.

            Show 10 nested
            • cache_read_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cache_write_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cost_usdnumber· doublerequired

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • function_callsinteger· uint64requiredmin 0
            • input_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • modelstringrequired
            • output_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • providerstringrequired
            • reasoning_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • turnsinteger· uint64requiredmin 0

              Model steps (assistant messages) this model generated.

          • cache_read_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cache_write_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • contextany of

            The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.

            Show 2 nested
            any of (2)
            Variant 1
            • categoriesobjectrequired

              Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.

              Show 6 nested
              • hook_guidanceinteger· uint64min 0
              • messagesobjectrequired

                Estimated tokens of the assembled window's messages, by role.

                Show 4 nested
                • assistantinteger· uint64requiredmin 0
                • custominteger· uint64requiredmin 0
                • function_resultinteger· uint64requiredmin 0
                • userinteger· uint64requiredmin 0
              • overheadinteger· uint64requiredmin 0

                Provider framing plus response_format / provider_options fields.

              • skillsinteger· uint64min 0

                Selected skill bodies contained in the system prompt.

              • system_promptinteger· uint64requiredmin 0

                Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.

              • toolsinteger· uint64requiredmin 0

                Function schemas exposed to the model.

            • compactedbooleanrequired

              The context carries a conversation summary: compacted on this step, or anchored on an earlier compaction entry (a previous step, the console's `/compact`). The summary's tokens count in `system_prompt`.

            • effective_max_output_tokensinteger· uint64requiredmin 0

              Output allocation `usable` was derived against.

            • estimatorstring

              Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.

            • freeinteger· uint64requiredmin 0
            • modelstringrequired
            • prefix_divergencesobject[]

              Already-sent rows this session's requests rewrote, newest last (at most [`MAX_PREFIX_DIVERGENCES`]), carried from step to step.

              Show 9 nested
              • cache_readinteger· uint64min 0

                `usage.cache_read` of the rewriting request: what the provider still reused.

              • call_idstring
              • causestring

                What rewrote the row, as far as the harness can tell: `prune` (context-manager pruned or reduced function results to fit the budget that step) or `system_prompt` (row 0: a compaction summary or a prompt change).

              • indexinteger· uint64min 0

                First row (0 = system prompt, then one per message) that differs from the session's previous request. Absent when only the provider's wire mapping changed.

              • prev_leninteger· uint64min 0

                Rows the previous request carried.

              • provider_notestring

                The provider's own wire-level finding, cause included, when it reported one (a final-message warning starting with [`PROVIDER_PREFIX_NOTE`]).

              • rolestring
              • stepinteger· uint64requiredmin 0
              • turn_idstringrequired
            • prompt_sections_fallbackstring

              Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).

            • prompt_surface_digeststring

              Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.

            • providerstring
            • session_cost_usdnumber· double

              Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.

            • session_idstringrequired
            • stepinteger· uint64requiredmin 0
            • summarized_head_tokensinteger· uint64min 0

              Size of the history that summary replaced.

            • timestampinteger· int64required
            • totalinteger· uint64requiredmin 0

              Final request estimate: categories plus post-assembly growth.

            • turn_idstringrequired
            • usableinteger· uint64requiredmin 0

              The input budget the window was fit into.

            • usageany of

              Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).

              Show 2 nested
              any of (2)
              Variant 1
              • cache_readinteger· uint64min 0
              • cache_writeinteger· uint64min 0
              • cost_usdnumber· double
              • inputinteger· uint64min 0
              • outputinteger· uint64min 0
              • reasoninginteger· uint64min 0
              Variant 2
              valuenull
            Variant 2
            valuenull
          • cost_usdnumber· doublerequired

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • depthinteger· uint32requiredmin 0
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • output_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • parent_session_idstring
          • reasoning_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • session_idstringrequired
          • turnsinteger· uint64requiredmin 0
        • completebooleanrequired
        • root_session_idstringrequired
        • totalsobjectrequired
          Show 11 nested
          • by_modelobject[]

            The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.

            Show 10 nested
            • cache_read_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cache_write_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cost_usdnumber· doublerequired

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • function_callsinteger· uint64requiredmin 0
            • input_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • modelstringrequired
            • output_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • providerstringrequired
            • reasoning_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • turnsinteger· uint64requiredmin 0

              Model steps (assistant messages) this model generated.

          • cache_read_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cache_write_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cost_usdnumber· doublerequired

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • output_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • reasoning_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • sessionsinteger· uint64requiredmin 0
          • turnsinteger· uint64requiredmin 0
        • tracesany of

          Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

          Show 2 nested
          any of (2)
          Variant 1
          • by_sessionobject[]required
            Show 7 nested
            • depthinteger· uint32requiredmin 0
            • duration_msinteger· uint64requiredmin 0

              Elapsed window from the session's first observed span to its last.

            • error_span_countinteger· uint64requiredmin 0
            • parent_session_idstring
            • session_idstringrequired
            • span_countinteger· uint64requiredmin 0
            • trace_countinteger· uint64requiredmin 0
          • duration_msinteger· uint64requiredmin 0

            Elapsed window from the first observed span to the last observed span.

          • error_span_countinteger· uint64requiredmin 0
          • span_countinteger· uint64requiredmin 0
          • trace_countinteger· uint64requiredmin 0

            Distinct traces across the root session and all descendants.

          Variant 2
          valuenull
        Variant 2
        valuenull
      • rejectedobject[]required
        Show 3 nested
        • indexinteger· uintrequiredmin 0
        • reasonsstring[]required
        • titlestringrequired
      • requested_modelstringrequired
      • requested_providerstringrequired
      • session_idstringrequired
      • signal_assessmentsobject[]required
        Show 3 nested
        • explanationstringrequired
        • fingerprintstringrequired

          The `fingerprint` of a diagnostic in the evidence.

        • verdictstringrequiredenum likely_expected, worth_changing, unclear
      • suggestionsobject[]required
        Show 11 nested
        • checkany of

          How to reproduce the behavior at the step where it happened, and the proposed change as edits of what the model saw there.

          Show 2 nested
          any of (2)
          Variant 1
          • changeobject[]

            The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

            Show 4 nested
            • findstring
            • removeboolean
            • replacestring
            • targetstringrequired
          • decision_pointstringrequired

            The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

          • signalobjectrequired

            How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

            Show 2 nested
            • questionstring
            • ruleany of
              Show 2 nested
              any of (2)
              Variant 1
              one of (2)
              Variant 1
              valuestringenum contract_rediscovery
              Variant 2
              valuestringenum repeated_error_call
              Variant 2
              valuenull
          Variant 2
          valuenull
        • code_refsobject[]max 8 items

          Code the suggestion rests on, as read in the code directory; only with code access. At most 8; each is checked against the directory.

          Show 3 nested
          • line_frominteger· uint32requiredmin 0

            First line, 1-based and inclusive.

          • line_tointeger· uint32requiredmin 0

            Last line, inclusive; at most the file's line count.

          • pathstringrequired

            Relative to the code directory, e.g. `harness/src/registry.rs` or `context-manager/src/prune.rs`; never absolute and never with `..`.

        • evidenceobject[]required
          Show 2 nested
          • entry_idstringrequired
          • session_idstringrequired
        • expected_effectstringrequired
        • harness_componentstringrequired
        • hypothesisstringrequired
        • limitationsstringrequired
        • observationstringrequired
        • proposed_changestringrequired
        • titlestringrequired
        • validationobjectrequired

          The E2E plan a suggestion must carry. A reviewer can run it without the LLM that wrote it.

          Show 6 nested
          • expectationstringrequired
          • invariantsstring[]required

            Task-correctness invariants checked by an independent evaluator.

          • non_regression_controlsstring[]required
          • primary_metricstringrequired

            The effort metric compared between baseline and candidate.

          • reproductionstringrequired
          • scenario_idstring

            An existing harness-e2e scenario, or null when a new case is needed.

      • turn_idstringrequired
      Variant 2
      valuenull
    • snapshotany of
      Show 2 nested
      any of (2)
      Variant 1
      • captured_atinteger· int64required
      • coverageobjectrequired
        Show 7 nested
        • context_bytesinteger· uint32requiredmin 0
        • diagnostics_in_contextinteger· uint32requiredmin 0

          Diagnostics shown to the models; all of them stay in the snapshot.

        • entries_readinteger· uint32requiredmin 0
        • levelone ofrequired
          Show 3 nested
          one of (3)
          Variant 1
          valuestringenum complete
          Variant 2
          valuestringenum partial
          Variant 3
          valuestringenum insufficient
        • limitationsstring[]required
        • sessions_in_scopeinteger· uint32requiredmin 0
        • sessions_out_of_scopeinteger· uint32requiredmin 0
      • diagnosticsobject[]required
        Show 9 nested
        • correlationone ofrequired
          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum unknown
          Variant 2
          valuestringenum harness_notice_correlated
        • evidenceobject[]required

          Calls, results and notices, in transcript order.

          Show 2 nested
          • entry_idstringrequired
          • session_idstringrequired
        • fingerprintstringrequired

          Rule, session, turn and the call ids of this occurrence.

        • observationstringrequired
        • rule_idstringrequired
        • rule_versionstringrequired
        • session_idstringrequired
        • targetstringrequired
        • turn_idstring
      • e2e_scenariostring

        The harness-e2e scenario the observed session ran, when it came from the E2E (`metadata.e2e_scenario`).

      • excluded_probesobject[]required

        Protocol probes that failed as expected and were not reported.

        Show 5 nested
        • callsstring[]required
        • codestringrequired
        • session_idstringrequired
        • targetstringrequired
        • turn_idstring
      • metricsobjectrequired
        Show 5 nested
        • by_sessionobject[]required
          Show 14 nested
          • by_modelobject[]

            The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.

            Show 10 nested
            • cache_read_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cache_write_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cost_usdnumber· doublerequired

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • function_callsinteger· uint64requiredmin 0
            • input_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • modelstringrequired
            • output_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • providerstringrequired
            • reasoning_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • turnsinteger· uint64requiredmin 0

              Model steps (assistant messages) this model generated.

          • cache_read_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cache_write_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • contextany of

            The session's latest per-generation context snapshot (categories, budget, usage) — absent for sessions that have not generated since snapshots landed.

            Show 2 nested
            any of (2)
            Variant 1
            • categoriesobjectrequired

              Where the request's tokens sit. Categories are assembly-time estimates; `hook_guidance` is the measured growth after assembly (pre-generate hook appends and orphan-repair patches), 0 when the request left assembly unchanged.

              Show 6 nested
              • hook_guidanceinteger· uint64min 0
              • messagesobjectrequired

                Estimated tokens of the assembled window's messages, by role.

                Show 4 nested
                • assistantinteger· uint64requiredmin 0
                • custominteger· uint64requiredmin 0
                • function_resultinteger· uint64requiredmin 0
                • userinteger· uint64requiredmin 0
              • overheadinteger· uint64requiredmin 0

                Provider framing plus response_format / provider_options fields.

              • skillsinteger· uint64min 0

                Selected skill bodies contained in the system prompt.

              • system_promptinteger· uint64requiredmin 0

                Final assembled system prompt excluding tokens attributed to `skills`: identity, per-step aids, and any compaction summary section.

              • toolsinteger· uint64requiredmin 0

                Function schemas exposed to the model.

            • compactedbooleanrequired

              The context carries a conversation summary: compacted on this step, or anchored on an earlier compaction entry (a previous step, the console's `/compact`). The summary's tokens count in `system_prompt`.

            • effective_max_output_tokensinteger· uint64requiredmin 0

              Output allocation `usable` was derived against.

            • estimatorstring

              Which estimator produced the numbers (`heuristic` until the context-manager resolves a real tokenizer). Absent when the context-manager predates the breakdown response.

            • freeinteger· uint64requiredmin 0
            • modelstringrequired
            • prefix_divergencesobject[]

              Already-sent rows this session's requests rewrote, newest last (at most [`MAX_PREFIX_DIVERGENCES`]), carried from step to step.

              Show 9 nested
              • cache_readinteger· uint64min 0

                `usage.cache_read` of the rewriting request: what the provider still reused.

              • call_idstring
              • causestring

                What rewrote the row, as far as the harness can tell: `prune` (context-manager pruned or reduced function results to fit the budget that step) or `system_prompt` (row 0: a compaction summary or a prompt change).

              • indexinteger· uint64min 0

                First row (0 = system prompt, then one per message) that differs from the session's previous request. Absent when only the provider's wire mapping changed.

              • prev_leninteger· uint64min 0

                Rows the previous request carried.

              • provider_notestring

                The provider's own wire-level finding, cause included, when it reported one (a final-message warning starting with [`PROVIDER_PREFIX_NOTE`]).

              • rolestring
              • stepinteger· uint64requiredmin 0
              • turn_idstringrequired
            • prompt_sections_fallbackstring

              Why no shared prefix went out: `disabled` (config), `no_stable_prefix` (empty identity prompt) or `prefix_rewritten` (a pre_generate hook replaced the prompt head; its prompt still wins, it just shares nothing).

            • prompt_surface_digeststring

              Identity of the shared prompt prefix this generation went out with (`sha256:` of the stable section) — present only when the request carried `system_sections`. A local identity, never a provider hit; `usage.cache_read` is the reuse signal.

            • providerstring
            • session_cost_usdnumber· double

              Running cost of the whole session in USD, accumulated across every generation step. `usage.cost_usd` is one step's bill — on providers with steep cache discounts the per-step number swings two orders of magnitude, so a chip showing it alone reads as a bouncing total.

            • session_idstringrequired
            • stepinteger· uint64requiredmin 0
            • summarized_head_tokensinteger· uint64min 0

              Size of the history that summary replaced.

            • timestampinteger· int64required
            • totalinteger· uint64requiredmin 0

              Final request estimate: categories plus post-assembly growth.

            • turn_idstringrequired
            • usableinteger· uint64requiredmin 0

              The input budget the window was fit into.

            • usageany of

              Actual provider usage for this generation, stamped after the terminal frame; absent when the provider returned none (or the generation never completed).

              Show 2 nested
              any of (2)
              Variant 1
              • cache_readinteger· uint64min 0
              • cache_writeinteger· uint64min 0
              • cost_usdnumber· double
              • inputinteger· uint64min 0
              • outputinteger· uint64min 0
              • reasoninginteger· uint64min 0
              Variant 2
              valuenull
            Variant 2
            valuenull
          • cost_usdnumber· doublerequired

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • depthinteger· uint32requiredmin 0
          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • output_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • parent_session_idstring
          • reasoning_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • session_idstringrequired
          • turnsinteger· uint64requiredmin 0
        • completebooleanrequired
        • root_session_idstringrequired
        • totalsobjectrequired
          Show 11 nested
          • by_modelobject[]

            The same counters per model that generated a step, ordered by provider then model. Absent when nothing generated, and from a harness that predates it.

            Show 10 nested
            • cache_read_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cache_write_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • cost_usdnumber· doublerequired

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • function_callsinteger· uint64requiredmin 0
            • input_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • modelstringrequired
            • output_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • providerstringrequired
            • reasoning_tokensinteger· uint64requiredmin 0

              Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

            • turnsinteger· uint64requiredmin 0

              Model steps (assistant messages) this model generated.

          • cache_read_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cache_write_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • cost_usdnumber· doublerequired

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • function_call_errorsinteger· uint64requiredmin 0
          • function_callsinteger· uint64requiredmin 0
          • input_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • output_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • reasoning_tokensinteger· uint64requiredmin 0

            Summed over every generation that reported it. A single generation without the counter (a stream the provider cut before usage came back) is left out of the sum. `null` means no generation reported it or more than one did not — never a measured zero, so the key is always present and a reader can tell the two apart.

          • sessionsinteger· uint64requiredmin 0
          • turnsinteger· uint64requiredmin 0
        • tracesany of

          Trace/span aggregates when the engine's in-memory observability exporter is available. Usage metrics remain available when it is not.

          Show 2 nested
          any of (2)
          Variant 1
          • by_sessionobject[]required
            Show 7 nested
            • depthinteger· uint32requiredmin 0
            • duration_msinteger· uint64requiredmin 0

              Elapsed window from the session's first observed span to its last.

            • error_span_countinteger· uint64requiredmin 0
            • parent_session_idstring
            • session_idstringrequired
            • span_countinteger· uint64requiredmin 0
            • trace_countinteger· uint64requiredmin 0
          • duration_msinteger· uint64requiredmin 0

            Elapsed window from the first observed span to the last observed span.

          • error_span_countinteger· uint64requiredmin 0
          • span_countinteger· uint64requiredmin 0
          • trace_countinteger· uint64requiredmin 0

            Distinct traces across the root session and all descendants.

          Variant 2
          valuenull
      • metrics_scopeone ofrequired
        Show 1 nested
        one of (1)
        Variant 1
        valuestringenum session_tree
      • observed_modelstring
      • observed_providerstring
      • rules_versionstringrequired
      • sessionsobject[]required
        Show 10 nested
        • depthinteger· uint32requiredmin 0
        • entriesinteger· uint32requiredmin 0
        • in_scopebooleanrequired

          Out-of-scope descendants belong to earlier turns and are not read.

        • json_sha256string

          SHA-256 of the collected JSON entries (not of the stored bytes).

        • omitted_entriesinteger· uint32requiredmin 0
        • parent_session_idstring
        • parent_turn_idstring
        • previewunknown[]required

          Masked, shortened entries shown to the models.

        • reduced_entriesinteger· uint32requiredmin 0

          Preview entries whose strings were truncated or whose secrets were masked.

        • session_idstringrequired
      • source_result_errorstring
      • source_session_idstringrequired
      • source_statusstringrequiredenum running, awaiting_functions, completed, cancelled, failed

        The coarse, harness-internal turn lifecycle (harness.md § API Reference). Finer-grained than the session's `status`, which the loop derives from it.

      • source_stop_reasonstring
      • source_titlestring
      • source_turn_idstringrequired
      • window_turn_idsstring[]required

        The observed turn plus earlier root turns no analysis covered yet.

      Variant 2
      valuenull
    • triageany of
      Show 2 nested
      any of (2)
      Variant 1
      • answersobjectrequired
      • completed_atinteger· int64required
      • criteria_versionstringrequired
      • modelstringrequired

        The model the provider reports, not the configured default.

      • providerstringrequired
      • request_idstringrequired
      • statsobjectrequired

        Known accepted usage, never a billing estimate. Unreported counters or failure/cancellation make usage_complete false.

        Show 7 nested
        • attemptsinteger· uintrequiredmin 0
        • elapsed_msinteger· uint64requiredmin 0
        • input_tokensinteger· uint64requiredmin 0
        • output_tokensinteger· uint64requiredmin 0
        • questionsinteger· uintrequiredmin 0
        • requestsinteger· uintrequiredmin 0
        • usage_completebooleanrequired
      Variant 2
      valuenull
    • triage_failureany of
      Show 2 nested
      any of (2)
      Variant 1
      • codestringrequired
      • http_statusinteger· uint16min 0
      • messagestringrequired
      • request_idstringrequired
      • statsany of

        Known usage of the failed call, when the provider returned it.

        Show 2 nested
        any of (2)
        Variant 1
        • attemptsinteger· uintrequiredmin 0
        • elapsed_msinteger· uint64requiredmin 0
        • input_tokensinteger· uint64requiredmin 0
        • output_tokensinteger· uint64requiredmin 0
        • questionsinteger· uintrequiredmin 0
        • requestsinteger· uintrequiredmin 0
        • usage_completebooleanrequired
        Variant 2
        valuenull
      Variant 2
      valuenull
    • validationsobject[]
      Show 5 nested
      • attached_atinteger· int64required
      • baselineobjectrequired

        What `e2e::dashboard::execution-get` reported at lookup time.

        Show 16 nested
        • assessmentsany of
          Show 2 nested
          any of (2)
          Variant 1
          • passedinteger· uint32requiredmin 0
          • totalinteger· uint32requiredmin 0
          Variant 2
          valuenull
        • availabilitystring
        • conclusionstring
        • e2e_revisionstring
        • engine_versionstring
        • evidence_errorstring
        • execution_idstringrequired
        • harness_versionstring
        • labelstring
        • modelstring
        • providerstring
        • reportsobject[]required
          Show 3 nested
          • availablebooleanrequired
          • scenario_idstringrequired
          • subject_idstring
        • reports_availablebooleanrequired

          True only when the execution reported retained reports.

        • scenariosobject[]required
          Show 13 nested
          • behavior_sha256string
          • completed_runsinteger· uint32min 0
          • contract_fingerprintstring
          • cost_usd_per_runnumber· double

            The cohort's total cost divided by its completed runs; the total also covers failed attempts.

          • mean_scorenumber· double

            Mean score of the scored runs (0 to 100).

          • measuresobjectrequired

            `function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.

          • median_wall_time_msnumber· double
          • p50_function_callsnumber· double
          • pass_ratenumber· double

            The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero. Share of the planned runs that passed (0 to 1).

          • planned_runsinteger· uint32min 0
          • run_countinteger· uint32requiredmin 0
          • scenario_idstringrequired
          • total_tokens_per_runnumber· double

            The cohort's total tokens divided by its completed runs.

        • started_atinteger· int64

          When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.

        • statusstring
      • candidateobjectrequired

        What `e2e::dashboard::execution-get` reported at lookup time.

        Show 16 nested
        • assessmentsany of
          Show 2 nested
          any of (2)
          Variant 1
          • passedinteger· uint32requiredmin 0
          • totalinteger· uint32requiredmin 0
          Variant 2
          valuenull
        • availabilitystring
        • conclusionstring
        • e2e_revisionstring
        • engine_versionstring
        • evidence_errorstring
        • execution_idstringrequired
        • harness_versionstring
        • labelstring
        • modelstring
        • providerstring
        • reportsobject[]required
          Show 3 nested
          • availablebooleanrequired
          • scenario_idstringrequired
          • subject_idstring
        • reports_availablebooleanrequired

          True only when the execution reported retained reports.

        • scenariosobject[]required
          Show 13 nested
          • behavior_sha256string
          • completed_runsinteger· uint32min 0
          • contract_fingerprintstring
          • cost_usd_per_runnumber· double

            The cohort's total cost divided by its completed runs; the total also covers failed attempts.

          • mean_scorenumber· double

            Mean score of the scored runs (0 to 100).

          • measuresobjectrequired

            `function_calls`, `function_call_errors`, `tokens`, `duration_seconds` and `cost_usd`, when reported.

          • median_wall_time_msnumber· double
          • p50_function_callsnumber· double
          • pass_ratenumber· double

            The scenario's cohort in `plan_execution.measurements.cohorts[]`, when the execution has exactly one for it. Each value below is absent when the E2E did not report it, never zero. Share of the planned runs that passed (0 to 1).

          • planned_runsinteger· uint32min 0
          • run_countinteger· uint32requiredmin 0
          • scenario_idstringrequired
          • total_tokens_per_runnumber· double

            The cohort's total tokens divided by its completed runs.

        • started_atinteger· int64

          When the E2E started the execution (ms since the Unix epoch): its results cannot have existed before.

        • statusstring
      • comparabilityobjectrequired

        The conditions that must match for a fair comparison; only the Harness version is meant to differ.

        Show 2 nested
        • checksobject[]required
          Show 4 nested
          • baselinestring
          • candidatestring
          • fieldstringrequired
          • matchesbooleanrequired
        • comparablebooleanrequired
      • suggestion_indexinteger· uintrequiredmin 0
  • recordobjectrequired

    The compact analysis record listed by `eval::list`; evidence and model outputs live separately in the analysis assets.

    • analystany of
      Show 2 nested
      any of (2)
      Variant 1
      • sent_atinteger· int64required
      • session_idstringrequired
      • turn_idstring
      Variant 2
      valuenull
    • code_rootstring

      Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.

    • completed_atinteger· int64
    • config_revisionstringrequired
    • countersobjectrequired
      Show 6 nested
      • diagnosticsinteger· uint32requiredmin 0
      • entriesinteger· uint32requiredmin 0
      • rejected_suggestionsinteger· uint32requiredmin 0
      • sessionsinteger· uint32requiredmin 0
      • suggestionsinteger· uint32requiredmin 0
      • validationsinteger· uint32requiredmin 0
    • coverageany of
      Show 2 nested
      any of (2)
      Variant 1
      one of (3)
      Variant 1
      valuestringenum complete
      Variant 2
      valuestringenum partial
      Variant 3
      valuestringenum insufficient
      Variant 2
      valuenull
    • created_atinteger· int64required
    • criteria_versionstringrequired
    • deadlineinteger· int64required
    • evaluation_idstringrequired
    • failureany of
      Show 2 nested
      any of (2)
      Variant 1
      • codestringrequired
      • messagestringrequired
      • stagestringrequiredenum queued, collecting, judging, investigating, completed, failed, …
      Variant 2
      valuenull
    • harness_versionstring

      The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.

    • judge_callany of
      Show 2 nested
      any of (2)
      Variant 1
      • deadlineinteger· int64required
      • request_idstringrequired
      • started_atinteger· int64required
      Variant 2
      valuenull
    • modelall ofrequired

      Frozen at admission: later configuration changes never alter it.

      Show 1 nested
      all of (1)
      Variant 1
      • modelstringrequired
      • providerstringrequired
      • provider_optionsobject

        Provider-native options, namespaced by the selected provider id.

      • thinking_levelany of
        Show 2 nested
        any of (2)
        Variant 1
        one of (2)
        Variant 1
        valuestringenum minimal, low, medium, high, xhigh
        Variant 2
        valuestringenum off
        Variant 2
        valuenull
    • observation_keystringrequired
    • observe_sinceinteger· int64required

      Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.

    • originstringrequiredenum automatic, manual
    • pending_reasonstring

      Why collection is still waiting (descendants running, metrics incomplete).

    • routingany of
      Show 2 nested
      any of (2)
      Variant 1
      • investigatebooleanrequired
      • reasonsone of[]required
        Show 7 nested
        one of (7)
        Variant 1
        valuestringenum needs_investigation
        Variant 2
        valuestringenum diagnostics
        Variant 3
        valuestringenum insufficient_evidence
        Variant 4
        valuestringenum low_confidence
        Variant 5
        valuestringenum coverage_insufficient
        Variant 6
        valuestringenum audit_sample
        Variant 7
        valuestringenum manual_request
      Variant 2
      valuenull
    • rules_versionstringrequired
    • schema_versioninteger· uint32requiredmin 0
    • session_idstringrequired
    • signalsobject

      Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).

    • source_titlestring

      The session's title, once collection has read it.

    • stagesobject[]required

      When each status began, oldest first.

      Show 2 nested
      • atinteger· int64required
      • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
    • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
    • stepinteger· uint64requiredmin 0
    • supersedesstring

      The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.

    • turn_idstringrequired
    • updated_atinteger· int64required
    • usageall ofrequired

      The monitor's own consumption, apart from the observed task.

      Show 1 nested
      all of (1)
      Variant 1
      • judge_callsinteger· uint32requiredmin 0
      • judge_input_tokensinteger· uint64requiredmin 0
      • judge_output_tokensinteger· uint64requiredmin 0
      • judge_usage_completebooleanrequired

        False when Jev reported incomplete usage or failed.

      • llm_cost_usdnumber· double
      • llm_input_tokensinteger· uint64min 0
      • llm_output_tokensinteger· uint64min 0
  • reviewsobject[]

    One row per suggestion, in suggestion order: what people decided about it. A suggestion nobody acted on reads as `new`.

    • criterionany of
      Show 2 nested
      any of (2)
      Variant 1
      • directionstringrequiredenum decrease, increase
      • metricone ofrequired

        What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
        Variant 2
        valuestringenum signal_per_run
      • min_effectnumber· doublerequired
      • min_runsinteger· uint32requiredmin 0
      • patternstring
      • registered_atinteger· int64required
      • registered_bystringrequired
      • scenario_idstringrequired

        The scenario whose runs are measured.

      Variant 2
      valuenull
    • evaluation_idstringrequired
    • evidenceany of
      Show 2 nested
      any of (2)
      Variant 1
      • baselineobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • candidateobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • computed_atinteger· int64required
      • computed_outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • reasonstringrequired
      • scenario_idstringrequired
      Variant 2
      valuenull
    • first_run_atinteger· int64

      When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.

    • harness_componentstringrequired
    • lifecycleobjectrequired
      Show 6 nested
      • duplicate_ofstring

        What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).

      • historyobject[]required

        Every change of status, oldest first; empty while `new`.

        Show 4 nested
        • atinteger· int64required
        • bystringrequired
        • notestring
        • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

          Where a suggestion stands. Only people move it: the monitor never does.

      • prstring

        The pull request that implements it (`in_progress`, `shipped`).

      • reasonstring

        Why it was rejected (`rejected`).

      • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

        Where a suggestion stands. Only people move it: the monitor never does.

      • versionstring

        The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.

    • patternsstring[]required

      `<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.

    • reproductionsobject[]

      Replays of the decision point, oldest first: the base and the tested changes.

      Show 21 nested
      • bystringrequired
      • changeobject[]
        Show 4 nested
        • findstring
        • removeboolean
        • replacestring
        • targetstringrequired
      • change_kindone ofrequired
        Show 3 nested
        one of (3)
        Variant 1
        valuestringenum none
        Variant 2
        valuestringenum proposed
        Variant 3
        valuestringenum custom
      • checkall ofrequired

        The decision point and signal this replay used.

        Show 1 nested
        all of (1)
        Variant 1
        • changeobject[]

          The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

          Show 4 nested
          • findstring
          • removeboolean
          • replacestring
          • targetstringrequired
        • decision_pointstringrequired

          The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

        • signalobjectrequired

          How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

          Show 2 nested
          • questionstring
          • ruleany of
            Show 2 nested
            any of (2)
            Variant 1
            one of (2)
            Variant 1
            valuestringenum contract_rediscovery
            Variant 2
            valuestringenum repeated_error_call
            Variant 2
            valuenull
      • cost_unknown_samplesinteger· uint32min 0

        Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.

      • cost_usdnumber· double

        Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.

      • errorstring
      • fidelityany of
        Show 2 nested
        any of (2)
        Variant 1
        • counted_tokensinteger· uint64min 0
        • estimatorstring

          `provider` when the provider counted, else the estimator's name.

        • levelone ofrequired
          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum approximate
          Variant 2
          valuestringenum exact
        • off_rationumber· double

          The unexplained difference over the recorded tokens.

        • reasonsstring[]

          Why the rebuild is approximate; empty when exact.

        • recorded_tokensinteger· uint64min 0

          Input tokens of the original step: input + cache read + cache write.

        Variant 2
        valuenull
      • finished_atinteger· int64
      • idstringrequired
      • judge_input_tokensinteger· uint64min 0
      • judge_output_tokensinteger· uint64min 0
      • modelstringrequired
      • originalany of

        The original reply at the decision point, read the same way.

        Show 2 nested
        any of (2)
        Variant 1
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
        Variant 2
        valuenull
      • phasestring

        `sampling` or `classifying` while running.

      • providerstring
      • requestedinteger· uint32requiredmin 0

        Replies asked for, extensions included.

      • samplesobject[]
        Show 8 nested
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
      • started_atinteger· int64required
      • statestringrequiredenum running, completed, failed
      • updated_atinteger· int64required
    • runany of
      Show 2 nested
      any of (2)
      Variant 1
      • baseline_commitstringrequired

        The full commits the Harness was built from.

      • baseline_execution_idstring

        Absent until the E2E accepted the execution.

      • candidate_commitstringrequired
      • candidate_execution_idstring
      • errorstring

        Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).

      • finished_atinteger· int64

        When both executions were seen ended; the sweep stops trying to attach the pair some time after.

      • modelstringrequired
      • providerstringrequired
      • runsinteger· uint32requiredmin 0

        Runs requested for each side.

      • scenario_idstringrequired
      • started_atinteger· int64required
      • stateone ofrequired
        Show 5 nested
        one of (5)
        Variant 1
        valuestringenum failed
        Variant 2
        valuestringenum starting
        Variant 3
        valuestringenum running
        Variant 4
        valuestringenum finished
        Variant 5
        valuestringenum attached
      Variant 2
      valuenull
    • scenario_idstring

      The scenario the suggestion's validation plan names.

    • suggestion_indexinteger· uintrequiredmin 0
    • titlestringrequired
    • verdictany of
      Show 2 nested
      any of (2)
      Variant 1
      • atinteger· int64required
      • bystringrequired
      • controls_checkedstring[]required

        The non-regression controls of the plan that were checked.

      • criterion_snapshotany of
        Show 2 nested
        any of (2)
        Variant 1
        • directionstringrequiredenum decrease, increase
        • metricone ofrequired

          What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
          Variant 2
          valuestringenum signal_per_run
        • min_effectnumber· doublerequired
        • min_runsinteger· uint32requiredmin 0
        • patternstring
        • registered_atinteger· int64required
        • registered_bystringrequired
        • scenario_idstringrequired

          The scenario whose runs are measured.

        Variant 2
        valuenull
      • outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • rationalestringrequired
      Variant 2
      valuenull
Variant 2
valuenull

eval::review

Function

Record what a person decided about a suggestion of a terminal analysis: move its lifecycle (new, accepted, in_progress, shipped, rejected, duplicate), register the criterion a validation is judged by, or record a verdict. A verdict of validated_improvement is refused unless the criterion was registered before the first attached pair or run and both sides have min_runs completed runs. The author is `by`, else the eval host's user, else the caller's identity.

Request
  • _caller_worker_idstring

    Stamped by the engine on every invocation; callers omit it.

  • actionone ofrequired
    one of (3)
    Variant 1
    valuestringenum set_lifecycle
    Variant 2
    valuestringenum set_criterion
    Variant 3
    valuestringenum set_verdict
  • bystring

    The author when the caller has no identity.

  • controls_checkedstring[]
  • criterionany of

    `set_criterion`. Refused once a criterion exists and a run started or a pair was attached: a criterion chosen after the results proves nothing.

    any of (2)
    Variant 1
    • directionall ofrequired

      The change that counts as an effect, wanted when the candidate moves the baseline mean in this direction.

      Show 1 nested
      all of (1)
      Variant 1
      valuestringenum decrease, increase
    • metricone ofrequired

      What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

      Show 2 nested
      one of (2)
      Variant 1
      valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
      Variant 2
      valuestringenum signal_per_run
    • min_effectnumber· doublerequired

      The smallest change of the baseline mean that counts, as an absolute difference in the metric's own unit: signals per run, USD, seconds, calls or tokens; percentage points (0-100) for `pass_rate`. Above 0.

    • min_runsinteger· uint32requiredmin 0

      Completed runs each side needs for a verdict of improvement (1-20).

    • patternstring

      `signal_per_run` only: one of the row's `patterns`.

    Variant 2
    valuenull
  • duplicate_ofstring
  • evaluation_idstringrequired
  • notestring

    `set_lifecycle`: shown in the history.

  • outcomeany of

    `set_verdict`. `validated_improvement` is refused unless a criterion was registered before the first attached pair or run, and the evidence has `min_runs` completed runs on both sides.

    any of (2)
    Variant 1
    one of (4)
    Variant 1
    valuestringenum validated_improvement
    Variant 2
    valuestringenum no_improvement
    Variant 3
    valuestringenum regression
    Variant 4
    valuestringenum inconclusive
    Variant 2
    valuenull
  • prstring
  • rationalestring
  • reasonstring
  • scenario_idstring

    `set_criterion`: the scenario to measure; defaults to the one the suggestion's plan names.

  • statusany of

    `set_lifecycle`: the status to move to (never back to `new`; a shipped suggestion stays shipped, to complete its `pr` or `version`). `shipped` needs a `pr` (here or already recorded), `rejected` a `reason`, `duplicate` a `duplicate_of`.

    any of (2)
    Variant 1
    valuestringenum new, accepted, in_progress, shipped, rejected, duplicate
    Variant 2
    valuenull
  • suggestion_indexinteger· uintrequiredmin 0
  • versionstring
Response
  • criterionany of
    any of (2)
    Variant 1
    • directionstringrequiredenum decrease, increase
    • metricone ofrequired

      What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

      Show 2 nested
      one of (2)
      Variant 1
      valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
      Variant 2
      valuestringenum signal_per_run
    • min_effectnumber· doublerequired
    • min_runsinteger· uint32requiredmin 0
    • patternstring
    • registered_atinteger· int64required
    • registered_bystringrequired
    • scenario_idstringrequired

      The scenario whose runs are measured.

    Variant 2
    valuenull
  • evaluation_idstringrequired
  • evidenceany of
    any of (2)
    Variant 1
    • baselineobjectrequired
      Show 4 nested
      • execution_idstringrequired
      • meannumber· double

        Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

      • ninteger· uint32requiredmin 0

        Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

      • runsobject[]required

        Every run of the scenario the execution holds.

        Show 10 nested
        • completedbooleanrequired

          The E2E reported the run complete with a valid technical outcome.

        • completionstring

          `completion` as the E2E reported it.

        • cost_usdnumber· double
        • function_callsnumber· double
        • run_idstringrequired
        • signalsobject

          Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

        • statusstring

          `passed` or `failed`, the run's task outcome.

        • technicalstring

          `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

        • total_tokensnumber· double
        • wall_time_msnumber· double
    • candidateobjectrequired
      Show 4 nested
      • execution_idstringrequired
      • meannumber· double

        Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

      • ninteger· uint32requiredmin 0

        Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

      • runsobject[]required

        Every run of the scenario the execution holds.

        Show 10 nested
        • completedbooleanrequired

          The E2E reported the run complete with a valid technical outcome.

        • completionstring

          `completion` as the E2E reported it.

        • cost_usdnumber· double
        • function_callsnumber· double
        • run_idstringrequired
        • signalsobject

          Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

        • statusstring

          `passed` or `failed`, the run's task outcome.

        • technicalstring

          `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

        • total_tokensnumber· double
        • wall_time_msnumber· double
    • computed_atinteger· int64required
    • computed_outcomeone ofrequired
      Show 4 nested
      one of (4)
      Variant 1
      valuestringenum validated_improvement
      Variant 2
      valuestringenum no_improvement
      Variant 3
      valuestringenum regression
      Variant 4
      valuestringenum inconclusive
    • reasonstringrequired
    • scenario_idstringrequired
    Variant 2
    valuenull
  • first_run_atinteger· int64

    When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.

  • harness_componentstringrequired
  • lifecycleobjectrequired
    • duplicate_ofstring

      What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).

    • historyobject[]required

      Every change of status, oldest first; empty while `new`.

      • atinteger· int64required
      • bystringrequired
      • notestring
      • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

        Where a suggestion stands. Only people move it: the monitor never does.

    • prstring

      The pull request that implements it (`in_progress`, `shipped`).

    • reasonstring

      Why it was rejected (`rejected`).

    • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

      Where a suggestion stands. Only people move it: the monitor never does.

    • versionstring

      The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.

  • patternsstring[]required

    `<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.

  • reproductionsobject[]

    Replays of the decision point, oldest first: the base and the tested changes.

    • bystringrequired
    • changeobject[]
      • findstring
      • removeboolean
      • replacestring
      • targetstringrequired
    • change_kindone ofrequired
      one of (3)
      Variant 1
      valuestringenum none
      Variant 2
      valuestringenum proposed
      Variant 3
      valuestringenum custom
    • checkall ofrequired

      The decision point and signal this replay used.

      all of (1)
      Variant 1
      • changeobject[]

        The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

        Show 4 nested
        • findstring
        • removeboolean
        • replacestring
        • targetstringrequired
      • decision_pointstringrequired

        The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

      • signalobjectrequired

        How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

        Show 2 nested
        • questionstring
        • ruleany of
          Show 2 nested
          any of (2)
          Variant 1
          one of (2)
          Variant 1
          valuestringenum contract_rediscovery
          Variant 2
          valuestringenum repeated_error_call
          Variant 2
          valuenull
    • cost_unknown_samplesinteger· uint32min 0

      Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.

    • cost_usdnumber· double

      Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.

    • errorstring
    • fidelityany of
      any of (2)
      Variant 1
      • counted_tokensinteger· uint64min 0
      • estimatorstring

        `provider` when the provider counted, else the estimator's name.

      • levelone ofrequired
        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum approximate
        Variant 2
        valuestringenum exact
      • off_rationumber· double

        The unexplained difference over the recorded tokens.

      • reasonsstring[]

        Why the rebuild is approximate; empty when exact.

      • recorded_tokensinteger· uint64min 0

        Input tokens of the original step: input + cache read + cache write.

      Variant 2
      valuenull
    • finished_atinteger· int64
    • idstringrequired
    • judge_input_tokensinteger· uint64min 0
    • judge_output_tokensinteger· uint64min 0
    • modelstringrequired
    • originalany of

      The original reply at the decision point, read the same way.

      any of (2)
      Variant 1
      • callsobject[]required
        Show 3 nested
        • descriptionstring
        • payloadstringrequired

          The payload as JSON text, cut at 600 characters.

        • targetstringrequired

          The function called (through `agent_trigger` or directly).

      • duration_msinteger· uint64min 0
      • errorstring
      • indexinteger· uint32requiredmin 0
      • signalboolean

        Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

      • textstringrequired
      • thinkingstringrequired

        The opening of the reply's reasoning and of its text, 400 characters each.

      • usageall of
        Show 1 nested
        all of (1)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
      Variant 2
      valuenull
    • phasestring

      `sampling` or `classifying` while running.

    • providerstring
    • requestedinteger· uint32requiredmin 0

      Replies asked for, extensions included.

    • samplesobject[]
      • callsobject[]required
        Show 3 nested
        • descriptionstring
        • payloadstringrequired

          The payload as JSON text, cut at 600 characters.

        • targetstringrequired

          The function called (through `agent_trigger` or directly).

      • duration_msinteger· uint64min 0
      • errorstring
      • indexinteger· uint32requiredmin 0
      • signalboolean

        Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

      • textstringrequired
      • thinkingstringrequired

        The opening of the reply's reasoning and of its text, 400 characters each.

      • usageall of
        Show 1 nested
        all of (1)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
    • started_atinteger· int64required
    • statestringrequiredenum running, completed, failed
    • updated_atinteger· int64required
  • runany of
    any of (2)
    Variant 1
    • baseline_commitstringrequired

      The full commits the Harness was built from.

    • baseline_execution_idstring

      Absent until the E2E accepted the execution.

    • candidate_commitstringrequired
    • candidate_execution_idstring
    • errorstring

      Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).

    • finished_atinteger· int64

      When both executions were seen ended; the sweep stops trying to attach the pair some time after.

    • modelstringrequired
    • providerstringrequired
    • runsinteger· uint32requiredmin 0

      Runs requested for each side.

    • scenario_idstringrequired
    • started_atinteger· int64required
    • stateone ofrequired
      Show 5 nested
      one of (5)
      Variant 1
      valuestringenum failed
      Variant 2
      valuestringenum starting
      Variant 3
      valuestringenum running
      Variant 4
      valuestringenum finished
      Variant 5
      valuestringenum attached
    Variant 2
    valuenull
  • scenario_idstring

    The scenario the suggestion's validation plan names.

  • suggestion_indexinteger· uintrequiredmin 0
  • titlestringrequired
  • verdictany of
    any of (2)
    Variant 1
    • atinteger· int64required
    • bystringrequired
    • controls_checkedstring[]required

      The non-regression controls of the plan that were checked.

    • criterion_snapshotany of
      Show 2 nested
      any of (2)
      Variant 1
      • directionstringrequiredenum decrease, increase
      • metricone ofrequired

        What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
        Variant 2
        valuestringenum signal_per_run
      • min_effectnumber· doublerequired
      • min_runsinteger· uint32requiredmin 0
      • patternstring
      • registered_atinteger· int64required
      • registered_bystringrequired
      • scenario_idstringrequired

        The scenario whose runs are measured.

      Variant 2
      valuenull
    • outcomeone ofrequired
      Show 4 nested
      one of (4)
      Variant 1
      valuestringenum validated_improvement
      Variant 2
      valuestringenum no_improvement
      Variant 3
      valuestringenum regression
      Variant 4
      valuestringenum inconclusive
    • rationalestringrequired
    Variant 2
    valuenull

eval::reviews

Function

List the stored suggestion reviews (lifecycle, criterion, run, evidence, verdict) and, per analysis with suggestions, how many sit at each lifecycle status.

Request
  • evaluation_idstring

    Only this analysis; absent means every analysis.

Response
  • reviewsobject[]required

    The stored rows: suggestions somebody acted on or validated.

    • criterionany of
      any of (2)
      Variant 1
      • directionstringrequiredenum decrease, increase
      • metricone ofrequired

        What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
        Variant 2
        valuestringenum signal_per_run
      • min_effectnumber· doublerequired
      • min_runsinteger· uint32requiredmin 0
      • patternstring
      • registered_atinteger· int64required
      • registered_bystringrequired
      • scenario_idstringrequired

        The scenario whose runs are measured.

      Variant 2
      valuenull
    • evaluation_idstringrequired
    • evidenceany of
      any of (2)
      Variant 1
      • baselineobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • candidateobjectrequired
        Show 4 nested
        • execution_idstringrequired
        • meannumber· double

          Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

        • ninteger· uint32requiredmin 0

          Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

        • runsobject[]required

          Every run of the scenario the execution holds.

          Show 10 nested
          • completedbooleanrequired

            The E2E reported the run complete with a valid technical outcome.

          • completionstring

            `completion` as the E2E reported it.

          • cost_usdnumber· double
          • function_callsnumber· double
          • run_idstringrequired
          • signalsobject

            Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

          • statusstring

            `passed` or `failed`, the run's task outcome.

          • technicalstring

            `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

          • total_tokensnumber· double
          • wall_time_msnumber· double
      • computed_atinteger· int64required
      • computed_outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • reasonstringrequired
      • scenario_idstringrequired
      Variant 2
      valuenull
    • first_run_atinteger· int64

      When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.

    • harness_componentstringrequired
    • lifecycleobjectrequired
      • duplicate_ofstring

        What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).

      • historyobject[]required

        Every change of status, oldest first; empty while `new`.

        Show 4 nested
        • atinteger· int64required
        • bystringrequired
        • notestring
        • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

          Where a suggestion stands. Only people move it: the monitor never does.

      • prstring

        The pull request that implements it (`in_progress`, `shipped`).

      • reasonstring

        Why it was rejected (`rejected`).

      • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

        Where a suggestion stands. Only people move it: the monitor never does.

      • versionstring

        The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.

    • patternsstring[]required

      `<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.

    • reproductionsobject[]

      Replays of the decision point, oldest first: the base and the tested changes.

      • bystringrequired
      • changeobject[]
        Show 4 nested
        • findstring
        • removeboolean
        • replacestring
        • targetstringrequired
      • change_kindone ofrequired
        Show 3 nested
        one of (3)
        Variant 1
        valuestringenum none
        Variant 2
        valuestringenum proposed
        Variant 3
        valuestringenum custom
      • checkall ofrequired

        The decision point and signal this replay used.

        Show 1 nested
        all of (1)
        Variant 1
        • changeobject[]

          The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

          Show 4 nested
          • findstring
          • removeboolean
          • replacestring
          • targetstringrequired
        • decision_pointstringrequired

          The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

        • signalobjectrequired

          How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

          Show 2 nested
          • questionstring
          • ruleany of
            Show 2 nested
            any of (2)
            Variant 1
            one of (2)
            Variant 1
            valuestringenum contract_rediscovery
            Variant 2
            valuestringenum repeated_error_call
            Variant 2
            valuenull
      • cost_unknown_samplesinteger· uint32min 0

        Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.

      • cost_usdnumber· double

        Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.

      • errorstring
      • fidelityany of
        Show 2 nested
        any of (2)
        Variant 1
        • counted_tokensinteger· uint64min 0
        • estimatorstring

          `provider` when the provider counted, else the estimator's name.

        • levelone ofrequired
          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum approximate
          Variant 2
          valuestringenum exact
        • off_rationumber· double

          The unexplained difference over the recorded tokens.

        • reasonsstring[]

          Why the rebuild is approximate; empty when exact.

        • recorded_tokensinteger· uint64min 0

          Input tokens of the original step: input + cache read + cache write.

        Variant 2
        valuenull
      • finished_atinteger· int64
      • idstringrequired
      • judge_input_tokensinteger· uint64min 0
      • judge_output_tokensinteger· uint64min 0
      • modelstringrequired
      • originalany of

        The original reply at the decision point, read the same way.

        Show 2 nested
        any of (2)
        Variant 1
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
        Variant 2
        valuenull
      • phasestring

        `sampling` or `classifying` while running.

      • providerstring
      • requestedinteger· uint32requiredmin 0

        Replies asked for, extensions included.

      • samplesobject[]
        Show 8 nested
        • callsobject[]required
          Show 3 nested
          • descriptionstring
          • payloadstringrequired

            The payload as JSON text, cut at 600 characters.

          • targetstringrequired

            The function called (through `agent_trigger` or directly).

        • duration_msinteger· uint64min 0
        • errorstring
        • indexinteger· uint32requiredmin 0
        • signalboolean

          Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

        • textstringrequired
        • thinkingstringrequired

          The opening of the reply's reasoning and of its text, 400 characters each.

        • usageall of
          Show 1 nested
          all of (1)
          Variant 1
          • cache_read_tokensinteger· uint64min 0
          • cache_write_tokensinteger· uint64min 0
          • cost_usdnumber· double
          • input_tokensinteger· uint64min 0
          • output_tokensinteger· uint64min 0
      • started_atinteger· int64required
      • statestringrequiredenum running, completed, failed
      • updated_atinteger· int64required
    • runany of
      any of (2)
      Variant 1
      • baseline_commitstringrequired

        The full commits the Harness was built from.

      • baseline_execution_idstring

        Absent until the E2E accepted the execution.

      • candidate_commitstringrequired
      • candidate_execution_idstring
      • errorstring

        Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).

      • finished_atinteger· int64

        When both executions were seen ended; the sweep stops trying to attach the pair some time after.

      • modelstringrequired
      • providerstringrequired
      • runsinteger· uint32requiredmin 0

        Runs requested for each side.

      • scenario_idstringrequired
      • started_atinteger· int64required
      • stateone ofrequired
        Show 5 nested
        one of (5)
        Variant 1
        valuestringenum failed
        Variant 2
        valuestringenum starting
        Variant 3
        valuestringenum running
        Variant 4
        valuestringenum finished
        Variant 5
        valuestringenum attached
      Variant 2
      valuenull
    • scenario_idstring

      The scenario the suggestion's validation plan names.

    • suggestion_indexinteger· uintrequiredmin 0
    • titlestringrequired
    • verdictany of
      any of (2)
      Variant 1
      • atinteger· int64required
      • bystringrequired
      • controls_checkedstring[]required

        The non-regression controls of the plan that were checked.

      • criterion_snapshotany of
        Show 2 nested
        any of (2)
        Variant 1
        • directionstringrequiredenum decrease, increase
        • metricone ofrequired

          What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

          Show 2 nested
          one of (2)
          Variant 1
          valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
          Variant 2
          valuestringenum signal_per_run
        • min_effectnumber· doublerequired
        • min_runsinteger· uint32requiredmin 0
        • patternstring
        • registered_atinteger· int64required
        • registered_bystringrequired
        • scenario_idstringrequired

          The scenario whose runs are measured.

        Variant 2
        valuenull
      • outcomeone ofrequired
        Show 4 nested
        one of (4)
        Variant 1
        valuestringenum validated_improvement
        Variant 2
        valuestringenum no_improvement
        Variant 3
        valuestringenum regression
        Variant 4
        valuestringenum inconclusive
      • rationalestringrequired
      Variant 2
      valuenull
  • summariesobject[]required

    One per analysis that has suggestions, newest first.

    • acceptedinteger· uint32requiredmin 0
    • duplicateinteger· uint32requiredmin 0
    • evaluation_idstringrequired
    • in_progressinteger· uint32requiredmin 0
    • newinteger· uint32requiredmin 0
    • rejectedinteger· uint32requiredmin 0
    • shippedinteger· uint32requiredmin 0
    • suggestionsinteger· uint32requiredmin 0

eval::start-validation

Function

Explicitly start a baseline and a candidate E2E execution for one suggestion. Both commits (candidate_ref, and baseline_ref or the merge base with origin/main) are resolved with git in the configured code_repository and must be on a remote branch; each stack is the E2E's `harness` template with only the Harness pinned to its commit, run in Docker (runs 1-20, default 5). The criterion is registered before anything starts; the sweep attaches the pair when both finish and computes the evidence. Spends model tokens, and never runs unless called. With dry_run: true it only resolves and checks the refs, scenario and runs and answers with the resolved commits (`baseline`, `candidate`, `warnings`); it registers nothing and calls no E2E function.

Request
  • _caller_worker_idstring

    Stamped by the engine on every invocation; callers omit it.

  • baseline_refstring

    Defaults to the merge base of the candidate and `origin/main`.

  • bystring

    The author: this name wins over the host's user and the caller's identity.

  • candidate_refstringrequired

    A ref of `code_repository` (a branch, a tag or a commit) that is on a remote branch.

  • criterionany of

    Registered first, before any execution starts. Required unless `dry_run`, which only checks it against `runs` when present.

    any of (2)
    Variant 1
    • directionall ofrequired

      The change that counts as an effect, wanted when the candidate moves the baseline mean in this direction.

      Show 1 nested
      all of (1)
      Variant 1
      valuestringenum decrease, increase
    • metricone ofrequired

      What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

      Show 2 nested
      one of (2)
      Variant 1
      valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
      Variant 2
      valuestringenum signal_per_run
    • min_effectnumber· doublerequired

      The smallest change of the baseline mean that counts, as an absolute difference in the metric's own unit: signals per run, USD, seconds, calls or tokens; percentage points (0-100) for `pass_rate`. Above 0.

    • min_runsinteger· uint32requiredmin 0

      Completed runs each side needs for a verdict of improvement (1-20).

    • patternstring

      `signal_per_run` only: one of the row's `patterns`.

    Variant 2
    valuenull
  • dry_runboolean

    Resolve and check the refs, the scenario and the runs exactly as a start does, and answer with the commits they resolve to. Registers nothing, calls no E2E function and spends nothing.

  • evaluation_idstringrequired
  • modelstring

    Required unless `dry_run`.

  • providerstring

    Required unless `dry_run`.

  • runsinteger· uint32min 0

    Runs of the scenario for each side; defaults to 5 (1-20).

  • scenario_idstringrequired

    The harness-e2e scenario to run.

  • suggestion_indexinteger· uintrequiredmin 0
Response
any of (2)
Variant 1
  • criterionany of
    any of (2)
    Variant 1
    • directionstringrequiredenum decrease, increase
    • metricone ofrequired

      What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

      Show 2 nested
      one of (2)
      Variant 1
      valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
      Variant 2
      valuestringenum signal_per_run
    • min_effectnumber· doublerequired
    • min_runsinteger· uint32requiredmin 0
    • patternstring
    • registered_atinteger· int64required
    • registered_bystringrequired
    • scenario_idstringrequired

      The scenario whose runs are measured.

    Variant 2
    valuenull
  • evaluation_idstringrequired
  • evidenceany of
    any of (2)
    Variant 1
    • baselineobjectrequired
      Show 4 nested
      • execution_idstringrequired
      • meannumber· double

        Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

      • ninteger· uint32requiredmin 0

        Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

      • runsobject[]required

        Every run of the scenario the execution holds.

        Show 10 nested
        • completedbooleanrequired

          The E2E reported the run complete with a valid technical outcome.

        • completionstring

          `completion` as the E2E reported it.

        • cost_usdnumber· double
        • function_callsnumber· double
        • run_idstringrequired
        • signalsobject

          Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

        • statusstring

          `passed` or `failed`, the run's task outcome.

        • technicalstring

          `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

        • total_tokensnumber· double
        • wall_time_msnumber· double
    • candidateobjectrequired
      Show 4 nested
      • execution_idstringrequired
      • meannumber· double

        Mean of the criterion's metric over those runs; absent without a criterion or without such runs.

      • ninteger· uint32requiredmin 0

        Measurable runs (see `EvidenceRunV1::measurable`) with a value of the criterion's metric (every measurable run while there is no criterion).

      • runsobject[]required

        Every run of the scenario the execution holds.

        Show 10 nested
        • completedbooleanrequired

          The E2E reported the run complete with a valid technical outcome.

        • completionstring

          `completion` as the E2E reported it.

        • cost_usdnumber· double
        • function_callsnumber· double
        • run_idstringrequired
        • signalsobject

          Occurrences of each of the row's patterns the detectors found in this run's transcript (0 is a measurement); absent when the report held no transcript.

        • statusstring

          `passed` or `failed`, the run's task outcome.

        • technicalstring

          `technical` as the E2E reported it; anything but `valid` is an infrastructure failure.

        • total_tokensnumber· double
        • wall_time_msnumber· double
    • computed_atinteger· int64required
    • computed_outcomeone ofrequired
      Show 4 nested
      one of (4)
      Variant 1
      valuestringenum validated_improvement
      Variant 2
      valuestringenum no_improvement
      Variant 3
      valuestringenum regression
      Variant 4
      valuestringenum inconclusive
    • reasonstringrequired
    • scenario_idstringrequired
    Variant 2
    valuenull
  • first_run_atinteger· int64

    When the first execution this worker started for the suggestion began, kept when a new start replaces `run`: results of it may have been seen.

  • harness_componentstringrequired
  • lifecycleobjectrequired
    • duplicate_ofstring

      What it duplicates: a pull request, or another `<evaluation_id>:<index>` (`duplicate`).

    • historyobject[]required

      Every change of status, oldest first; empty while `new`.

      Show 4 nested
      • atinteger· int64required
      • bystringrequired
      • notestring
      • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

        Where a suggestion stands. Only people move it: the monitor never does.

    • prstring

      The pull request that implements it (`in_progress`, `shipped`).

    • reasonstring

      Why it was rejected (`rejected`).

    • statusstringrequiredenum new, accepted, in_progress, shipped, rejected, duplicate

      Where a suggestion stands. Only people move it: the monitor never does.

    • versionstring

      The Harness version that shipped it, a semantic version like `1.8.43`; the recurrence query splits the analyses at it.

  • patternsstring[]required

    `<rule_id>:<target>` of the diagnostics the suggestion's evidence cites: what a validation counts and a release is judged by.

  • reproductionsobject[]

    Replays of the decision point, oldest first: the base and the tested changes.

    • bystringrequired
    • changeobject[]
      Show 4 nested
      • findstring
      • removeboolean
      • replacestring
      • targetstringrequired
    • change_kindone ofrequired
      Show 3 nested
      one of (3)
      Variant 1
      valuestringenum none
      Variant 2
      valuestringenum proposed
      Variant 3
      valuestringenum custom
    • checkall ofrequired

      The decision point and signal this replay used.

      Show 1 nested
      all of (1)
      Variant 1
      • changeobject[]

        The proposed change as edits of what the model saw before that step; empty when it cannot be expressed as text.

        Show 4 nested
        • findstring
        • removeboolean
        • replacestring
        • targetstringrequired
      • decision_pointstringrequired

        The assistant entry where the behavior happened (`…_<step>_assistant`), one of the suggestion's `evidence` entries.

      • signalobjectrequired

        How to recognize the behavior in one reply: exactly one of a rule computed in code or a yes/no question Jev answers about the reply.

        Show 2 nested
        • questionstring
        • ruleany of
          Show 2 nested
          any of (2)
          Variant 1
          one of (2)
          Variant 1
          valuestringenum contract_rediscovery
          Variant 2
          valuestringenum repeated_error_call
          Variant 2
          valuenull
    • cost_unknown_samplesinteger· uint32min 0

      Samples without a reported cost, failed ones included (the provider may have billed a request that never answered): unknown, not in `cost_usd`, and never estimated.

    • cost_usdnumber· double

      Known cost of the samples; null while no reply reported one. Only a lower bound while `cost_unknown_samples` is above zero. Jev's calls are in tokens below, never priced.

    • errorstring
    • fidelityany of
      Show 2 nested
      any of (2)
      Variant 1
      • counted_tokensinteger· uint64min 0
      • estimatorstring

        `provider` when the provider counted, else the estimator's name.

      • levelone ofrequired
        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum approximate
        Variant 2
        valuestringenum exact
      • off_rationumber· double

        The unexplained difference over the recorded tokens.

      • reasonsstring[]

        Why the rebuild is approximate; empty when exact.

      • recorded_tokensinteger· uint64min 0

        Input tokens of the original step: input + cache read + cache write.

      Variant 2
      valuenull
    • finished_atinteger· int64
    • idstringrequired
    • judge_input_tokensinteger· uint64min 0
    • judge_output_tokensinteger· uint64min 0
    • modelstringrequired
    • originalany of

      The original reply at the decision point, read the same way.

      Show 2 nested
      any of (2)
      Variant 1
      • callsobject[]required
        Show 3 nested
        • descriptionstring
        • payloadstringrequired

          The payload as JSON text, cut at 600 characters.

        • targetstringrequired

          The function called (through `agent_trigger` or directly).

      • duration_msinteger· uint64min 0
      • errorstring
      • indexinteger· uint32requiredmin 0
      • signalboolean

        Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

      • textstringrequired
      • thinkingstringrequired

        The opening of the reply's reasoning and of its text, 400 characters each.

      • usageall of
        Show 1 nested
        all of (1)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
      Variant 2
      valuenull
    • phasestring

      `sampling` or `classifying` while running.

    • providerstring
    • requestedinteger· uint32requiredmin 0

      Replies asked for, extensions included.

    • samplesobject[]
      Show 8 nested
      • callsobject[]required
        Show 3 nested
        • descriptionstring
        • payloadstringrequired

          The payload as JSON text, cut at 600 characters.

        • targetstringrequired

          The function called (through `agent_trigger` or directly).

      • duration_msinteger· uint64min 0
      • errorstring
      • indexinteger· uint32requiredmin 0
      • signalboolean

        Whether the signal shows in this reply; null when Jev answered unclear, the reply failed or it was not classified yet.

      • textstringrequired
      • thinkingstringrequired

        The opening of the reply's reasoning and of its text, 400 characters each.

      • usageall of
        Show 1 nested
        all of (1)
        Variant 1
        • cache_read_tokensinteger· uint64min 0
        • cache_write_tokensinteger· uint64min 0
        • cost_usdnumber· double
        • input_tokensinteger· uint64min 0
        • output_tokensinteger· uint64min 0
    • started_atinteger· int64required
    • statestringrequiredenum running, completed, failed
    • updated_atinteger· int64required
  • runany of
    any of (2)
    Variant 1
    • baseline_commitstringrequired

      The full commits the Harness was built from.

    • baseline_execution_idstring

      Absent until the E2E accepted the execution.

    • candidate_commitstringrequired
    • candidate_execution_idstring
    • errorstring

      Why it failed, or why attaching is being retried; starts with a stable code (`e2e_unavailable:`, `e2e_busy:`…).

    • finished_atinteger· int64

      When both executions were seen ended; the sweep stops trying to attach the pair some time after.

    • modelstringrequired
    • providerstringrequired
    • runsinteger· uint32requiredmin 0

      Runs requested for each side.

    • scenario_idstringrequired
    • started_atinteger· int64required
    • stateone ofrequired
      Show 5 nested
      one of (5)
      Variant 1
      valuestringenum failed
      Variant 2
      valuestringenum starting
      Variant 3
      valuestringenum running
      Variant 4
      valuestringenum finished
      Variant 5
      valuestringenum attached
    Variant 2
    valuenull
  • scenario_idstring

    The scenario the suggestion's validation plan names.

  • suggestion_indexinteger· uintrequiredmin 0
  • titlestringrequired
  • verdictany of
    any of (2)
    Variant 1
    • atinteger· int64required
    • bystringrequired
    • controls_checkedstring[]required

      The non-regression controls of the plan that were checked.

    • criterion_snapshotany of
      Show 2 nested
      any of (2)
      Variant 1
      • directionstringrequiredenum decrease, increase
      • metricone ofrequired

        What a validation measures. Every metric but `signal_per_run` is the mean over the runs of the scenario the infrastructure did not break (`pass_rate` is the share that passed, an unfinished run counting as not passed; `duration` is the wall time in milliseconds).

        Show 2 nested
        one of (2)
        Variant 1
        valuestringenum pass_rate, cost_usd, duration, tokens, function_calls
        Variant 2
        valuestringenum signal_per_run
      • min_effectnumber· doublerequired
      • min_runsinteger· uint32requiredmin 0
      • patternstring
      • registered_atinteger· int64required
      • registered_bystringrequired
      • scenario_idstringrequired

        The scenario whose runs are measured.

      Variant 2
      valuenull
    • outcomeone ofrequired
      Show 4 nested
      one of (4)
      Variant 1
      valuestringenum validated_improvement
      Variant 2
      valuestringenum no_improvement
      Variant 3
      valuestringenum regression
      Variant 4
      valuestringenum inconclusive
    • rationalestringrequired
    Variant 2
    valuenull
Variant 2
  • baselineobjectrequired

    One side of a validation, resolved to the commit the E2E would build.

    • branchstringrequired

      A remote branch that holds the commit, as `git branch -r` names it (`origin/feat`); the one named like the ref when several do.

    • commitstringrequired

      The full commit.

    • shortstringrequired

      The first 12 characters of `commit`.

  • candidateobjectrequired

    One side of a validation, resolved to the commit the E2E would build.

    • branchstringrequired

      A remote branch that holds the commit, as `git branch -r` names it (`origin/feat`); the one named like the ref when several do.

    • commitstringrequired

      The full commit.

    • shortstringrequired

      The first 12 characters of `commit`.

  • warningsstring[]required

    What a start would go ahead with but a person should know.

eval::status

Function

Read one analysis record (status, counters, failure stage) or null.

Request
  • evaluation_idstringrequired
Response
any of (2)
Variant 1
  • analystany of
    any of (2)
    Variant 1
    • sent_atinteger· int64required
    • session_idstringrequired
    • turn_idstring
    Variant 2
    valuenull
  • code_rootstring

    Frozen at admission with the model: the directory the investigation reads and runs in, absent when code access was off.

  • completed_atinteger· int64
  • config_revisionstringrequired
  • countersobjectrequired
    • diagnosticsinteger· uint32requiredmin 0
    • entriesinteger· uint32requiredmin 0
    • rejected_suggestionsinteger· uint32requiredmin 0
    • sessionsinteger· uint32requiredmin 0
    • suggestionsinteger· uint32requiredmin 0
    • validationsinteger· uint32requiredmin 0
  • coverageany of
    any of (2)
    Variant 1
    one of (3)
    Variant 1
    valuestringenum complete
    Variant 2
    valuestringenum partial
    Variant 3
    valuestringenum insufficient
    Variant 2
    valuenull
  • created_atinteger· int64required
  • criteria_versionstringrequired
  • deadlineinteger· int64required
  • evaluation_idstringrequired
  • failureany of
    any of (2)
    Variant 1
    • codestringrequired
    • messagestringrequired
    • stagestringrequiredenum queued, collecting, judging, investigating, completed, failed, …
    Variant 2
    valuenull
  • harness_versionstring

    The Harness version the engine reported when a live event admitted the analysis (the turn just ended on it); absent on a manual analysis, which may be of an older session, and when the engine did not say.

  • judge_callany of
    any of (2)
    Variant 1
    • deadlineinteger· int64required
    • request_idstringrequired
    • started_atinteger· int64required
    Variant 2
    valuenull
  • modelall ofrequired

    Frozen at admission: later configuration changes never alter it.

    all of (1)
    Variant 1
    • modelstringrequired
    • providerstringrequired
    • provider_optionsobject

      Provider-native options, namespaced by the selected provider id.

    • thinking_levelany of
      Show 2 nested
      any of (2)
      Variant 1
      one of (2)
      Variant 1
      valuestringenum minimal, low, medium, high, xhigh
      Variant 2
      valuestringenum off
      Variant 2
      valuenull
  • observation_keystringrequired
  • observe_sinceinteger· int64required

    Earlier root turns join the window only if they started after this instant: the monitor was observing then. Turns from before the monitor was enabled are never reported as new.

  • originstringrequiredenum automatic, manual
  • pending_reasonstring

    Why collection is still waiting (descendants running, metrics incomplete).

  • routingany of
    any of (2)
    Variant 1
    • investigatebooleanrequired
    • reasonsone of[]required
      Show 7 nested
      one of (7)
      Variant 1
      valuestringenum needs_investigation
      Variant 2
      valuestringenum diagnostics
      Variant 3
      valuestringenum insufficient_evidence
      Variant 4
      valuestringenum low_confidence
      Variant 5
      valuestringenum coverage_insufficient
      Variant 6
      valuestringenum audit_sample
      Variant 7
      valuestringenum manual_request
    Variant 2
    valuenull
  • rules_versionstringrequired
  • schema_versioninteger· uint32requiredmin 0
  • session_idstringrequired
  • signalsobject

    Occurrences of each deterministic pattern the collection found, keyed `<rule_id>:<target>` (a target may itself contain colons). Empty both when nothing was found and before collection (`coverage` is set once collection read the evidence).

  • source_titlestring

    The session's title, once collection has read it.

  • stagesobject[]required

    When each status began, oldest first.

    • atinteger· int64required
    • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
  • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …
  • stepinteger· uint64requiredmin 0
  • supersedesstring

    The earlier analysis of the same turn this reanalysis was requested over (same `observation_key`); absent on a first analysis.

  • turn_idstringrequired
  • updated_atinteger· int64required
  • usageall ofrequired

    The monitor's own consumption, apart from the observed task.

    all of (1)
    Variant 1
    • judge_callsinteger· uint32requiredmin 0
    • judge_input_tokensinteger· uint64requiredmin 0
    • judge_output_tokensinteger· uint64requiredmin 0
    • judge_usage_completebooleanrequired

      False when Jev reported incomplete usage or failed.

    • llm_cost_usdnumber· double
    • llm_input_tokensinteger· uint64min 0
    • llm_output_tokensinteger· uint64min 0
Variant 2
valuenull

eval::step

Function

Internal durable analysis step.

Request
  • evaluation_idstringrequired
  • stepinteger· uint64requiredmin 0
Response
  • skippedbooleanrequired
  • statusstringrequiredenum queued, collecting, judging, investigating, completed, failed, …

eval::sweep

Function

Internal recovery, deadline and retention sweep.

Request
  • scheduled_atinteger· int64
  • scheduled_timestring
Response
  • expiredinteger· uint64requiredmin 0
  • reconciledinteger· uint64requiredmin 0
  • requeuedinteger· uint64requiredmin 0
  • retained_deletedinteger· uint64requiredmin 0

eval::ui-content

Function

Serve the eval worker's injected console UI assets (content function for its console:script / console:style / console:module triggers).

Request
  • pathstringrequired

    The asset path from the trigger config (e.g. `state/page.js`).

Response
  • contentstringrequired

    The asset source, verbatim.

  • content_typestringrequired

    MIME type the console should serve the asset with.

Search the registry

Find workers, kits and executables, or jump to a page.