harness-e2e

v0.19.0

Measure Harness capability, retain reproducible evidence, and expose the E2E dashboard.

license: Apache-2.0Published 3d agoharness e2e evaluation benchmark dashboard

Install

bash

iii trigger compose::add worker=harness-e2e

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions45

e2e::archive

Function

Persist a completed execution through private iii storage and idempotent history.

Request

execution_idstringrequired
retention_classstringrequiredenum temporary, pull_request, longitudinal, canonical

Response

archiveobjectrequired
- archive_idstringrequired
- created_atstringrequired
- execution_idstringrequired
- expires_atstring
- identity_sha256stringrequired
- manifestobjectrequired
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
- manifest_backupobjectrequired
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
- retention_classstringrequiredenum temporary, pull_request, longitudinal, canonical
duplicate_ingestionbooleanrequired
historyobjectrequired
- archiveobjectrequired
  archive_idstringrequired
  created_atstringrequired
  execution_idstringrequired
  expires_atstring
  identity_sha256stringrequired
  manifestobjectrequired
  Show 6 nestedHide nested
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
  manifest_backupobjectrequired
  Show 6 nestedHide nested
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
  retention_classstringrequiredenum temporary, pull_request, longitudinal, canonical
- baseline_comparablebooleanrequired
- case_countinteger· uint32requiredmin 0
- e2e_repositorystringrequired
- e2e_revisionstringrequired
- execution_idstringrequired
- identity_sha256stringrequired
- ingestion_idstringrequired
- lanestringrequired
- occurred_atstringrequired
- passedbooleanrequired
- result_contract_sha256stringrequired
- stack_modestringrequired
- subject_modelstringrequired
- subject_providerstringrequired
- subject_revisionstringrequired

e2e::archive-head

Function

Verify availability and immutable metadata for an archived execution.

Request

execution_idstringrequired

Response

archiveobjectrequired
- archive_idstringrequired
- created_atstringrequired
- execution_idstringrequired
- expires_atstring
- identity_sha256stringrequired
- manifestobjectrequired
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
- manifest_backupobjectrequired
  bucketstringrequired
  keystringrequired
  media_typestringrequired
  sha256stringrequired
  size_bytesinteger· uint64requiredmin 0
  uristringrequired
- retention_classstringrequiredenum temporary, pull_request, longitudinal, canonical
availabilitystringrequiredenum available, expired
verified_atstringrequired

e2e::cancel

Function

Request idempotent cancellation and cleanup of an E2E execution.

Request

execution_idstringrequired

Response

acceptedbooleanrequired
execution_idstringrequired
phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …

e2e::dashboard::attempt-get

Function

Read the persisted projection for one logical-run attempt.

Request

attempt_idstringrequired
execution_idstringrequired
run_idstringrequired

Response

completionstringrequiredenum completed, task_incomplete, undetermined
criteriaobject[]
- awardedinteger· uint8min 0
- descriptionstring
- gateboolean
  A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell "no gate" from a report older than the field.
- gate_minimuminteger· uint8min 0
  The points a gate needs to count as passed when fewer than `possible` already mean the task's purpose was met. Absent: full points.
- idstringrequired
- labelstring
  The criterion as a short statement; absent in reports older than the field and for criteria that state none. Readers fall back to `description`.
- possibleinteger· uint8requiredmin 0
- reasonstringrequired
dimensionsobject[]required
- dimensionstringrequiredenum deliverable, structural_integrity, efficiency, robustness, e2e_infrastructure
- passedboolean
- signalsunknownrequired
efficiencyany of
any of (2)
Variant 1
child_sessionsinteger· uint64min 0
child_turnsinteger· uint64min 0
cost_usdnumber· double
critical_path_msinteger· uint64min 0
effective_fan_outinteger· uint64min 0
function_call_errorsinteger· uint64min 0
function_callsinteger· uint64min 0
input_tokensinteger· uint64min 0
observed_complexityobjectrequired
Show 11 nestedHide nested
ambiguity_levelinteger· uint64min 0
artifact_countinteger· uint64min 0
coordination_edgesinteger· uint64min 0
dependency_depthinteger· uint64min 0
external_systemsinteger· uint64min 0
parallel_branchesinteger· uint64min 0
planning_depthinteger· uint64min 0
state_transitionsinteger· uint64min 0
unavailableobject
validation_loopsinteger· uint64min 0
wake_cyclesinteger· uint64min 0
observed_workinteger· uint64min 0
output_tokensinteger· uint64min 0
root_turnsinteger· uint64min 0
technical_attemptsinteger· uint32requiredmin 0
total_tokensinteger· uint64min 0
transient_resumesinteger· uint64min 0
unavailableobject
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
wall_time_msinteger· uint64requiredmin 0
Variant 2
valuenull
failuresobject[]required
- codestring
- contaminationall of
  all of (1)
  Variant 1
  valuestringenum none, case, execution
- domainstringrequiredenum subject, resource, e2e_infrastructure
- messagestringrequired
- phasestringrequiredenum setup, execute, collect, evaluate, cleanup
- retry_scopeall of
  all of (1)
  Variant 1
  valuestringenum none, same_slot
- statusall ofrequired
  The failure's original classification, independent of the primary run status.
  all of (1)
  Variant 1
  valuestringenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
technicalstringrequiredenum valid, technical_invalid

e2e::dashboard::catalog-get

Function

Read models and scenarios when the execution dialog opens.

Request

urlstring

Response

docker_parallel_groupsinteger· uintrequiredmin 0
Groups a Docker execution runs at once, across executions.
modelsobject[]required
- idstringrequired
- providerstringrequired
scenario_groupsstring[][]required
Scenarios that run only together, in this order: picking one runs all.
scenariosstring[]required
urlstringrequired

e2e::dashboard::credential-delete

Function

Delete a provider credential this Console set; answers with the list.

Request

namestringrequired

Response

credentialsobject[]required
By name, whether each is set and where from; never a value.
- namestringrequired
- providersstring[]required
  The providers that read it, as the catalog says.
- setbooleanrequired
- sourcestring
  `console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.

e2e::dashboard::credential-set

Function

Set a provider credential (an environment variable's name and its value) for Docker executions; answers with the list, never a value.

Request

namestringrequired
An environment variable's name (OPENAI_API_KEY).
secretstringrequired
The value, under a key the iii SDKs redact from the invocation payloads they record in traces (`iii.invocation.input`): `value` would be kept there as sent.

Response

credentialsobject[]required
By name, whether each is set and where from; never a value.
- namestringrequired
- providersstring[]required
  The providers that read it, as the catalog says.
- setbooleanrequired
- sourcestring
  `console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.

e2e::dashboard::credentials-list

Function

List the provider credentials Docker executions receive: each by name, whether it is set and where from, never its value.

Request

Empty object

Response

credentialsobject[]required
By name, whether each is set and where from; never a value.
- namestringrequired
- providersstring[]required
  The providers that read it, as the catalog says.
- setbooleanrequired
- sourcestring
  `console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.

e2e::dashboard::evaluated-versions-list

Function

List immutable evaluated-system versions and their exact evaluation cohorts.

Request

cohort_idstring

Response

cohortsobject[]required
- idstringrequired
- lanestringrequired
- subject_modelstringrequired
- subject_providerstringrequired
revisionstringrequired
versionsobject[]required
- cohort_idstringrequired
- completed_atstringrequired
- execution_countinteger· uintrequiredmin 0
- idstringrequired
- labelstringrequired
- stack_modestringrequired

e2e::dashboard::evidence-read

Function

Read one evidence file or deliverable screenshot a retained run's report declares, as base64 (10 MB at most).

Request

execution_idstringrequired
The native execution whose report declares the file.
pathstringrequired
A path the report declares for one of its runs (evidence or a deliverable).
pointerstring
A declared screenshot's JSON Pointer inside that deliverable; only the image is returned.

Response

base64stringrequired
media_typestringrequired

e2e::dashboard::execution-cancel

Function

Stop an execution where it runs: no next scenario is admitted and the running one is cancelled; in Docker the running groups stop; on GitHub `gh run cancel`, and what finished is imported when the run ends.

Request

execution_idstringrequired

Response

Empty object

e2e::dashboard::execution-delete

Function

Delete one finished native run, or one finished execution together with its native runs.

Request

execution_idstringrequired

Response

Empty object

e2e::dashboard::execution-get

Function

Read one retained execution summary and its diagnostic report.

Request

execution_idstringrequired

Response

detailunknownrequired
manifestobjectrequired
- executionsunknown[]required
- last_updatestringrequired
- modestringrequired
- next_cursorstring
- repo_urlstringrequired
- retentionobjectrequired
  detailsinteger· uintrequiredmin 0
  summariesinteger· uintrequiredmin 0
- totalinteger· uintrequiredmin 0

e2e::dashboard::execution-rename

Function

Name one execution; an empty label restores its default name.

Request

execution_idstringrequired
labelstringrequired
Empty restores the default name.

Response

Empty object

e2e::dashboard::execution-slot-rerun

Function

Run one scenario of a finished execution again where it ran: on this harness, in Docker as the next attempt, or on GitHub by re-running its group's job (followed and imported again when the run ends); the last attempt counts. Answers with its id and runs in the background.

Request

execution_idstringrequired
scenario_idstringrequired
The scenario to run again; a sequential group runs again whole.

Response

Empty object

e2e::dashboard::execution-start

Function

Start an execution from its parameters on this harness, in Docker on a stack, or on GitHub (dispatching the exact-stack workflow, followed and imported when the run ends); answers with its id and runs in the background.

Request

labelstring
Empty or absent uses the default name.
parametersall ofrequired
What to run; a previous execution's parameters run it again.
all of (1)
Variant 1
agentstring
Agent profile the subject ran under.
modelstringrequired
providerstringrequired
runsinteger· uint32requiredmin 0
scenariosstring[]required
stackany of
The stack it ran on in Docker or on GitHub; this harness runs on its own. Running it again offers it as recorded.
Show 2 nestedHide nested
any of (2)
Variant 1
namestringrequired
A repository stack's id, a local one's name, or what a contract names.
sha256string
`sha256:` of `yaml`; the worker sets it, a request never does.
yamlstringrequired
The YAML as sent; once imported, the final `stack.yaml` its contract recorded (the iii release and the runner pinned).
Variant 2
valuenull
suiteany of
The suite it ran; absent for an execution from before suites.
Show 2 nestedHide nested
any of (2)
Variant 1
idstring
Absent for an unnamed suite.
labelstring
sha256string
`profile_sha256` of the snapshot; the runner sets it, a request never does.
Variant 2
valuenull
technical_retriesinteger· uint8requiredmin 0
whereall of
Where it ran; absent for an execution from before Docker, which ran on this harness.
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum harness, docker, github

Response

Empty object

e2e::dashboard::executions-list

Function

List compact retained execution summaries with server-side filters and cursor pagination.

Request

cursorstring
eventstring
idsstring[]
ids_csvstring
limitinteger· uint16min 0
querystring
statusstring

Response

executionsunknown[]required
last_updatestringrequired
modestringrequired
next_cursorstring
repo_urlstringrequired
retentionobjectrequired
- detailsinteger· uintrequiredmin 0
- summariesinteger· uintrequiredmin 0
totalinteger· uintrequiredmin 0

e2e::dashboard::github-run-contracts

Function

Read the suite, subject, profile and runner of listed GitHub runs from their contract artifacts.

Request

repositorystring
runsobject[]required
The runs of one listed page whose contract is not read yet.
- run_attemptinteger· uint64requiredmin 0
- run_idinteger· uint64requiredmin 0

Response

Empty object

e2e::dashboard::github-run-import

Function

Import one exact-stack workflow run from GitHub as an execution; answers at once and imports in the background.

Request

repositorystring
run_idinteger· uint64requiredmin 0

Response

Empty object

e2e::dashboard::github-runs-list

Function

List completed exact-stack workflow runs on GitHub with their local import; suites and subjects already read come along.

Request

pageinteger· uint32min 0
repositorystring

Response

Empty object

e2e::dashboard::github-status-get

Function

Tell whether the GitHub CLI on this worker's machine is installed and signed in, so executions can be dispatched to the repository's exact-stack workflow; never fails for a missing or signed-out gh.

Request

Empty object

Response

accountstring
messagestring
How to fix it when not ready.
readybooleanrequired
repositorystringrequired

e2e::dashboard::iii-releases-list

Function

List the newest iii CLI releases (iii-hq/iii tags iii/v*, alphas left out), each with its date and whether it publishes the CLI, and the newest release candidate `iii: latest` installs; with a version, whether that release exists. Kept ten minutes.

Request

versionstring
A version typed by hand, e.g. `0.24.2`: whether iii-hq/iii has the release `iii/v<version>` and it publishes the CLI.

Response

checkedany of
The version asked about.
any of (2)
Variant 1
releaseany of
Its release; none when iii-hq/iii has no `iii/v<version>`.
Show 2 nestedHide nested
any of (2)
Variant 1
clibooleanrequired
Whether it publishes `iii-x86_64-unknown-linux-gnu.tar.gz`, the CLI a group installs.
prereleasebooleanrequired
published_atstring
versionstringrequired
The tag without `iii/v`.
Variant 2
valuenull
versionstringrequired
Variant 2
valuenull
latest_candidatestring
What `iii: latest` installs: the newest `X.Y.Z-rc.N` tag.
releasesobject[]required
The newest releases, stable and pre-release, as GitHub lists them; alphas (`iii-alpha/…`) and drafts left out.
- clibooleanrequired
  Whether it publishes `iii-x86_64-unknown-linux-gnu.tar.gz`, the CLI a group installs.
- prereleasebooleanrequired
- published_atstring
- versionstringrequired
  The tag without `iii/v`.

e2e::dashboard::run-cancel

Function

Cancel the active local E2E execution.

Request

Empty object

Response

defaultsobjectrequired
- modelstringrequired
- providerstringrequired
- runsinteger· uint32requiredmin 0
- seedinteger· uint64min 0
- technical_retriesinteger· uint8requiredmin 0
- urlstringrequired
jobany of
any of (2)
Variant 1
completed_atstringrequired
errorstringrequired
idstringrequired
labelstringrequired
logstringrequired
log_frominteger· uint64requiredmin 0
log_offsetinteger· uint64requiredmin 0
log_truncatedbooleanrequired
requestobjectrequired
Show 8 nestedHide nested
labelstring
modelstringrequired
providerstringrequired
runsinteger· uint32requiredmin 0
scenariosstring[]required
seedinteger· uint64min 0
technical_retriesinteger· uint8requiredmin 0
urlstringrequired
returncodeinteger· int32
started_atstringrequired
statusstringrequiredenum running, cancelling, cancelled, completed, failed
Variant 2
valuenull

e2e::dashboard::stack-create

Function

Create a stack of this Console as a copy of another stack, or from its YAML, refused only as stack-update refuses it.

Request

fromstring
The stack it starts as a copy of: one of the repository or of this Console.
labelstring
Empty or absent names a copy after its stack; a stack of YAML needs one.
yamlstring
The whole stack as YAML, kept exactly as written; refused and warned about as `stack-update` does.

Response

containersobject[]required
- commitstring
- namestringrequired
- versionstring
- workerstring
  The worker reference as written: `package://…`, `path://…` or other.
idstringrequired
iiistring
The iii CLI release it installs.
labelstringrequired
sourcestringrequired
`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.
templatestring
The iii-hq/templates project it starts from.
updated_atstring
warningsstring[]required
What the executor or Compose may not do as written; never blocking.
yamlstringrequired

e2e::dashboard::stack-delete

Function

Delete a stack of this Console.

Request

stack_idstringrequired

Response

Empty object

e2e::dashboard::stack-preview

Function

Read a stack's YAML as stack-create and stack-update would, without saving it: what it declares (the template's revision apart) and its warnings, or why it would be refused.

Request

yamlstringrequired
A stack's YAML as a draft holds it; nothing is saved.

Response

any of (2)

Variant 1

containersobject[]required
- commitstring
- namestringrequired
- versionstring
- workerstring
  The worker reference as written: `package://…`, `path://…` or other.
iiistring
The iii CLI release it installs.
templateany of
The iii-hq/templates project it starts from.
any of (2)
Variant 1
idstringrequired
revisionstring
A commit, tag or branch of iii-hq/templates; none follows `main`.
Variant 2
valuenull
warningsstring[]required
What the executor or Compose may not do as written; never blocking.

Variant 2

refusedstringrequired
Why `stack-create` and `stack-update` would refuse it.

e2e::dashboard::stack-templates-list

Function

List the iii-hq/templates projects a stack can start from at a revision (a commit, tag or branch; main when none), each with the workers its worker-compose.yaml declares and the oldest iii it runs on; kept ten minutes.

Request

revisionstring
A commit, tag or branch of iii-hq/templates; `main` when absent.

Response

refstringrequired
The revision asked for: `main` unless another commit, tag or branch was.
repositorystringrequired
`iii-hq/templates`.
revisionstringrequired
The commit it was at when it was read.
templatesobject[]required
In the order `iii/template.yaml` lists them; a listed folder that does not exist is left out.
- descriptionstringrequired
- idstringrequired
  The folder under `iii/`, what a stack's `template:` names.
- min_iii_versionstring
  The oldest iii release it runs on, when its `template.yaml` says.
- namestringrequired
- notestring
  Why it declares no workers.
- workersobject[]required
  The containers its `worker-compose.yaml` declares with a `worker:`.
  namestringrequired
  The container's name.
  versionstring
  The version it pins, when it pins one.
  workerstringrequired
  The worker as written: `package://…` or `path://…`.

e2e::dashboard::stack-update

Function

Change the name or the YAML of a stack of this Console; refused only when the YAML does not parse or declares no containers.

Request

labelstring
stack_idstringrequired
yamlstring
The whole stack as YAML, kept exactly as written.

Response

containersobject[]required
- commitstring
- namestringrequired
- versionstring
- workerstring
  The worker reference as written: `package://…`, `path://…` or other.
idstringrequired
iiistring
The iii CLI release it installs.
labelstringrequired
sourcestringrequired
`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.
templatestring
The iii-hq/templates project it starts from.
updated_atstring
warningsstring[]required
What the executor or Compose may not do as written; never blocking.
yamlstringrequired

e2e::dashboard::stacks-list

Function

List the stacks an execution can run on: the repository's (read-only) and this Console's, each with its YAML, what it declares and its warnings.

Request

Empty object

Response

stacksobject[]required
The repository's stacks, then this Console's.
- containersobject[]required
  commitstring
  namestringrequired
  versionstring
  workerstring
  The worker reference as written: `package://…`, `path://…` or other.
- idstringrequired
- iiistring
  The iii CLI release it installs.
- labelstringrequired
- sourcestringrequired
  `repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.
- templatestring
  The iii-hq/templates project it starts from.
- updated_atstring
- warningsstring[]required
  What the executor or Compose may not do as written; never blocking.
- yamlstringrequired

e2e::dashboard::suite-create

Function

Create a suite of this Console as a copy of another suite, or holding the tests given.

Request

fromstring
The suite it starts as a copy of: one of the master plan or of this Console.
labelstring
Empty or absent names a copy after its suite; a suite of tests needs one.
repetitionsinteger· uint32min 0
Runs of each test: the copied suite's, else 1.
technical_retriesinteger· uint8min 0
Technical retries a crash gets: the copied suite's, else 1.
testsstring[]
The test ids it holds; sequential groups are completed.

Response

idstringrequired
labelstringrequired
purposestringrequired
repetitionsinteger· uint32requiredmin 0
scenariosstring[]required
sha256string
Digest of the snapshot this runner materializes it to; what an execution of it records.
sourcestringrequired
`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.
technical_retriesinteger· uint8requiredmin 0
updated_atstring
warningsstring[]
What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.

e2e::dashboard::suite-delete

Function

Delete a suite of this Console.

Request

suite_idstringrequired

Response

Empty object

e2e::dashboard::suite-update

Function

Change the name, scenarios, runs or technical retries of a suite of this Console.

Request

labelstring
repetitionsinteger· uint32min 0
scenariosstring[]
suite_idstringrequired
technical_retriesinteger· uint8min 0

Response

idstringrequired
labelstringrequired
purposestringrequired
repetitionsinteger· uint32requiredmin 0
scenariosstring[]required
sha256string
Digest of the snapshot this runner materializes it to; what an execution of it records.
sourcestringrequired
`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.
technical_retriesinteger· uint8requiredmin 0
updated_atstring
warningsstring[]
What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.

e2e::dashboard::suites-list

Function

List the suites an execution can run: the master plan's (read-only) and this Console's.

Request

Empty object

Response

suitesobject[]required
The master plan's suites, then this Console's.
- idstringrequired
- labelstringrequired
- purposestringrequired
- repetitionsinteger· uint32requiredmin 0
- scenariosstring[]required
- sha256string
  Digest of the snapshot this runner materializes it to; what an execution of it records.
- sourcestringrequired
  `repository` for a suite of the master plan (read-only), `local` for one this Console keeps.
- technical_retriesinteger· uint8requiredmin 0
- updated_atstring
- warningsstring[]
  What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.

e2e::dashboard::test-history-get

Function

Read local metric history for one test version, with provider-grouped execution models, without comparison actions.

Request

case_idstring
cursorstring
executionsstring[]
Only these executions, at most two: the runs an A × B compares, found wherever they are in the history.
limitinteger· uint16min 0
resultstring
subject_modelstring
subject_providerstring
system_version_idstring
test_idstring
test_versionstring

Response

available_versionsobject[]required
- execution_countinteger· uintrequiredmin 0
- last_seenstring
- observation_countinteger· uintrequiredmin 0
  The runs a test history lists: one per execution and case.
- run_countinteger· uintrequiredmin 0
- versionstringrequired
casesstring[]required
current_versionstring
The current definition's digest, which may have no executions yet.
next_cursorstring
observationsobject[]required
- agent_profilestring
  The agent profile the subject ran with, when it had one.
- assessment_profile_sha256stringrequired
- assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
- behavior_sha256stringrequired
- case_idstringrequired
- cohort_idstringrequired
- completed_atstringrequired
- contract_sha256stringrequired
- engine_revisionstring
- evaluated_version_idstring
- execution_idstringrequired
- execution_labelstring
  That execution's name, when it has one.
- harness_revisionstring
- mean_scorenumber· double
- median_cost_usdnumber· double
- median_duration_secondsnumber· double
- median_function_call_errorsnumber· double
- median_function_callsnumber· double
- median_tokensnumber· double
- median_turnsnumber· double
- plan_execution_idstring
  The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.
- run_countinteger· uintrequiredmin 0
- runsobject[]required
  attempt_idstringrequired
  completionstringrequiredenum completed, task_incomplete, undetermined
  detailsany of
  Absent when the run's native results could not be read.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  calls_by_workerobject[]required
  Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.
  Show 2 nestedHide nested
  callsinteger· uint64requiredmin 0
  workerstringrequired
  child_sessionsinteger· uintrequiredmin 0
  criteriaobject[]required
  Every criterion the run was scored on, in contract order.
  Show 6 nestedHide nested
  awardednumber· double
  Absent when nobody evaluated the criterion.
  descriptionstring
  The criterion as it was worded for this run; absent in older reports.
  gateboolean
  Whether this criterion blocked completion; absent in older reports.
  idstringrequired
  possiblenumber· doublerequired
  reasonstringrequired
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  sessionsobject[]required
  The root session first, each session followed by the ones it spawned.
  Show 8 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  labelstring
  The agent profile the session was spawned with, when the root's transcript names it.
  parent_session_idstring
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  Variant 2
  valuenull
  duration_secondsnumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  run_idstringrequired
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  turnsnumber· double
- scored_runsinteger· uintrequiredmin 0
- seedinteger· uint64min 0
- stack_modestringrequired
- statusstringrequired
- subject_modelstringrequired
- subject_providerstringrequired
- system_labelstringrequired
- system_revisionstring
- system_version_idstring
seriesobject[]required
- assessment_profile_sha256stringrequired
- behavior_sha256stringrequired
- case_idstringrequired
- cohort_idstringrequired
- contract_sha256stringrequired
- engine_revisionstring
- execution_countinteger· uintrequiredmin 0
- harness_revisionstring
- idstringrequired
- mean_scorenumber· double
- median_cost_usdnumber· double
- median_duration_secondsnumber· double
- median_function_call_errorsnumber· double
- median_function_callsnumber· double
- median_tokensnumber· double
- median_turnsnumber· double
- run_countinteger· uintrequiredmin 0
- seedinteger· uint64min 0
- stack_modestringrequired
- subject_modelstringrequired
- subject_providerstringrequired
- system_labelstringrequired
- system_revisionstring
- system_version_idstring
subject_modelsobject[]required
- modelsstring[]required
- providerstringrequired
subjectsstring[]required
systemsobject[]required
- idstringrequired
- labelstringrequired
test_idstringrequired
test_versionstringrequired
The definition whose executions are shown (the latest with evidence by default).
totalinteger· uintrequiredmin 0

e2e::dashboard::test-version-get

Function

Read one test version across two immutable evaluated-system versions.

Request

cohort_idstringrequired
from_version_idstringrequired
test_idstringrequired
test_versionstringrequired
to_version_idstringrequired

Response

compatibilitystringrequired
compatibility_reasonsstring[]required
deltaobjectrequired
- cost_usdnumber· double
- duration_secondsnumber· double
- scorenumber· double
- tokensnumber· double
fromany of
any of (2)
Variant 1
assessment_summaryobjectrequired
Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
Show 7 nestedHide nested
assessment_countinteger· uintrequiredmin 0
assessment_outcomesobjectrequired
Show 6 nestedHide nested
errorinteger· uintrequiredmin 0
failedinteger· uintrequiredmin 0
not_evaluatedinteger· uintrequiredmin 0
partialinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
unavailableinteger· uintrequiredmin 0
asset_countinteger· uintrequiredmin 0
asset_validation_outcomesobjectrequired
Show 10 nestedHide nested
invalidinteger· uintrequiredmin 0
malformedinteger· uintrequiredmin 0
not_evaluatedinteger· uintrequiredmin 0
not_producedinteger· uintrequiredmin 0
oversizedinteger· uintrequiredmin 0
removed_during_cleanupinteger· uintrequiredmin 0
unexpectedinteger· uintrequiredmin 0
unreadableinteger· uintrequiredmin 0
unsafe_pathinteger· uintrequiredmin 0
validinteger· uintrequiredmin 0
evidence_reference_countinteger· uintrequiredmin 0
run_countinteger· uintrequiredmin 0
system_statusesobjectrequired
Show 6 nestedHide nested
hard_gate_failedinteger· uintrequiredmin 0
infrastructure_errorinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
resource_limitinteger· uintrequiredmin 0
subject_errorinteger· uintrequiredmin 0
unavailableinteger· uintrequiredmin 0
case_countinteger· uintrequiredmin 0
evaluated_version_idstringrequired
execution_countinteger· uintrequiredmin 0
mean_scorenumber· double
median_cost_usdnumber· double
median_duration_secondsnumber· double
median_tokensnumber· double
outcomesobjectrequired
Show 4 nestedHide nested
hard_gate_failedinteger· uintrequiredmin 0
infra_failedinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
technical_failedinteger· uintrequiredmin 0
pass_ratenumber· double
samplesobjectrequired
Show 5 nestedHide nested
cost_usdinteger· uintrequiredmin 0
duration_secondsinteger· uintrequiredmin 0
scoreinteger· uintrequiredmin 0
tokensinteger· uintrequiredmin 0
turnsinteger· uintrequiredmin 0
scored_runsinteger· uintrequiredmin 0
total_runsinteger· uintrequiredmin 0
Variant 2
valuenull
from_observationsobject[]required
- agent_profilestring
  The agent profile the subject ran with, when it had one.
- assessment_profile_sha256stringrequired
- assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
- behavior_sha256stringrequired
- case_idstringrequired
- cohort_idstringrequired
- completed_atstringrequired
- contract_sha256stringrequired
- engine_revisionstring
- evaluated_version_idstring
- execution_idstringrequired
- execution_labelstring
  That execution's name, when it has one.
- harness_revisionstring
- mean_scorenumber· double
- median_cost_usdnumber· double
- median_duration_secondsnumber· double
- median_function_call_errorsnumber· double
- median_function_callsnumber· double
- median_tokensnumber· double
- median_turnsnumber· double
- plan_execution_idstring
  The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.
- run_countinteger· uintrequiredmin 0
- runsobject[]required
  attempt_idstringrequired
  completionstringrequiredenum completed, task_incomplete, undetermined
  detailsany of
  Absent when the run's native results could not be read.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  calls_by_workerobject[]required
  Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.
  Show 2 nestedHide nested
  callsinteger· uint64requiredmin 0
  workerstringrequired
  child_sessionsinteger· uintrequiredmin 0
  criteriaobject[]required
  Every criterion the run was scored on, in contract order.
  Show 6 nestedHide nested
  awardednumber· double
  Absent when nobody evaluated the criterion.
  descriptionstring
  The criterion as it was worded for this run; absent in older reports.
  gateboolean
  Whether this criterion blocked completion; absent in older reports.
  idstringrequired
  possiblenumber· doublerequired
  reasonstringrequired
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  sessionsobject[]required
  The root session first, each session followed by the ones it spawned.
  Show 8 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  labelstring
  The agent profile the session was spawned with, when the root's transcript names it.
  parent_session_idstring
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  Variant 2
  valuenull
  duration_secondsnumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  run_idstringrequired
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  turnsnumber· double
- scored_runsinteger· uintrequiredmin 0
- seedinteger· uint64min 0
- stack_modestringrequired
- statusstringrequired
- subject_modelstringrequired
- subject_providerstringrequired
- system_labelstringrequired
- system_revisionstring
- system_version_idstring
test_idstringrequired
test_versionstringrequired
toany of
any of (2)
Variant 1
assessment_summaryobjectrequired
Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
Show 7 nestedHide nested
assessment_countinteger· uintrequiredmin 0
assessment_outcomesobjectrequired
Show 6 nestedHide nested
errorinteger· uintrequiredmin 0
failedinteger· uintrequiredmin 0
not_evaluatedinteger· uintrequiredmin 0
partialinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
unavailableinteger· uintrequiredmin 0
asset_countinteger· uintrequiredmin 0
asset_validation_outcomesobjectrequired
Show 10 nestedHide nested
invalidinteger· uintrequiredmin 0
malformedinteger· uintrequiredmin 0
not_evaluatedinteger· uintrequiredmin 0
not_producedinteger· uintrequiredmin 0
oversizedinteger· uintrequiredmin 0
removed_during_cleanupinteger· uintrequiredmin 0
unexpectedinteger· uintrequiredmin 0
unreadableinteger· uintrequiredmin 0
unsafe_pathinteger· uintrequiredmin 0
validinteger· uintrequiredmin 0
evidence_reference_countinteger· uintrequiredmin 0
run_countinteger· uintrequiredmin 0
system_statusesobjectrequired
Show 6 nestedHide nested
hard_gate_failedinteger· uintrequiredmin 0
infrastructure_errorinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
resource_limitinteger· uintrequiredmin 0
subject_errorinteger· uintrequiredmin 0
unavailableinteger· uintrequiredmin 0
case_countinteger· uintrequiredmin 0
evaluated_version_idstringrequired
execution_countinteger· uintrequiredmin 0
mean_scorenumber· double
median_cost_usdnumber· double
median_duration_secondsnumber· double
median_tokensnumber· double
outcomesobjectrequired
Show 4 nestedHide nested
hard_gate_failedinteger· uintrequiredmin 0
infra_failedinteger· uintrequiredmin 0
passedinteger· uintrequiredmin 0
technical_failedinteger· uintrequiredmin 0
pass_ratenumber· double
samplesobjectrequired
Show 5 nestedHide nested
cost_usdinteger· uintrequiredmin 0
duration_secondsinteger· uintrequiredmin 0
scoreinteger· uintrequiredmin 0
tokensinteger· uintrequiredmin 0
turnsinteger· uintrequiredmin 0
scored_runsinteger· uintrequiredmin 0
total_runsinteger· uintrequiredmin 0
Variant 2
valuenull
to_observationsobject[]required
- agent_profilestring
  The agent profile the subject ran with, when it had one.
- assessment_profile_sha256stringrequired
- assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
- behavior_sha256stringrequired
- case_idstringrequired
- cohort_idstringrequired
- completed_atstringrequired
- contract_sha256stringrequired
- engine_revisionstring
- evaluated_version_idstring
- execution_idstringrequired
- execution_labelstring
  That execution's name, when it has one.
- harness_revisionstring
- mean_scorenumber· double
- median_cost_usdnumber· double
- median_duration_secondsnumber· double
- median_function_call_errorsnumber· double
- median_function_callsnumber· double
- median_tokensnumber· double
- median_turnsnumber· double
- plan_execution_idstring
  The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.
- run_countinteger· uintrequiredmin 0
- runsobject[]required
  attempt_idstringrequired
  completionstringrequiredenum completed, task_incomplete, undetermined
  detailsany of
  Absent when the run's native results could not be read.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  calls_by_workerobject[]required
  Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.
  Show 2 nestedHide nested
  callsinteger· uint64requiredmin 0
  workerstringrequired
  child_sessionsinteger· uintrequiredmin 0
  criteriaobject[]required
  Every criterion the run was scored on, in contract order.
  Show 6 nestedHide nested
  awardednumber· double
  Absent when nobody evaluated the criterion.
  descriptionstring
  The criterion as it was worded for this run; absent in older reports.
  gateboolean
  Whether this criterion blocked completion; absent in older reports.
  idstringrequired
  possiblenumber· doublerequired
  reasonstringrequired
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  sessionsobject[]required
  The root session first, each session followed by the ones it spawned.
  Show 8 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  labelstring
  The agent profile the session was spawned with, when the root's transcript names it.
  parent_session_idstring
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  Variant 2
  valuenull
  duration_secondsnumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  run_idstringrequired
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  turnsnumber· double
- scored_runsinteger· uintrequiredmin 0
- seedinteger· uint64min 0
- stack_modestringrequired
- statusstringrequired
- subject_modelstringrequired
- subject_providerstringrequired
- system_labelstringrequired
- system_revisionstring
- system_version_idstring

e2e::dashboard::tests-list

Function

List the versioned test catalog with compact A/B results and stable pagination.

Request

cohort_idstring
cursorstring
from_version_idstring
limitinteger· uint16min 0
querystring
to_version_idstring

Response

next_cursorstring
revisionstringrequired
rowsobject[]required
- available_versionsobject[]required
  execution_countinteger· uintrequiredmin 0
  last_seenstring
  observation_countinteger· uintrequiredmin 0
  The runs a test history lists: one per execution and case.
  run_countinteger· uintrequiredmin 0
  versionstringrequired
- calibrationany of
  any of (2)
  Variant 1
  compatible_sample_countinteger· uintrequiredmin 0
  maturitystringrequired
  Variant 2
  valuenull
- characterizationany of
  any of (2)
  Variant 1
  human_horizonall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  basisall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum unknown, author_estimate, measured
  max_minutesinteger· uint32min 0
  min_minutesinteger· uint32min 0
  realismall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  executionall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum synthetic, realistic_simulator, frozen_real_artifact
  shadowall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum none, read_only
  Variant 2
  valuenull
- current_versionstring
- last_runany of
  The most recent retained run, on any definition.
  any of (2)
  Variant 1
  atstringrequired
  When its execution completed.
  completionstringrequiredenum completed, task_incomplete, undetermined
  definitionone ofrequired
  Which definition a run evaluated, next to the test's current one.
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum current
  Variant 2
  valuestringenum previous
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  Variant 2
  valuenull
- lifecyclestringrequired
- recent_scoresnumber[]· doublerequired
  The scores of the last runs, oldest first; null where a run has none.
- resultany of
  any of (2)
  Variant 1
  compatibilitystringrequired
  compatibility_reasonsstring[]required
  deltaobjectrequired
  Show 4 nestedHide nested
  cost_usdnumber· double
  duration_secondsnumber· double
  scorenumber· double
  tokensnumber· double
  fromany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  Show 7 nestedHide nested
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  case_countinteger· uintrequiredmin 0
  evaluated_version_idstringrequired
  execution_countinteger· uintrequiredmin 0
  mean_scorenumber· double
  median_cost_usdnumber· double
  median_duration_secondsnumber· double
  median_tokensnumber· double
  outcomesobjectrequired
  Show 4 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infra_failedinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  technical_failedinteger· uintrequiredmin 0
  pass_ratenumber· double
  samplesobjectrequired
  Show 5 nestedHide nested
  cost_usdinteger· uintrequiredmin 0
  duration_secondsinteger· uintrequiredmin 0
  scoreinteger· uintrequiredmin 0
  tokensinteger· uintrequiredmin 0
  turnsinteger· uintrequiredmin 0
  scored_runsinteger· uintrequiredmin 0
  total_runsinteger· uintrequiredmin 0
  Variant 2
  valuenull
  from_observationsobject[]required
  Show 32 nestedHide nested
  agent_profilestring
  The agent profile the subject ran with, when it had one.
  assessment_profile_sha256stringrequired
  assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  Show 7 nestedHide nested
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  behavior_sha256stringrequired
  case_idstringrequired
  cohort_idstringrequired
  completed_atstringrequired
  contract_sha256stringrequired
  engine_revisionstring
  evaluated_version_idstring
  execution_idstringrequired
  execution_labelstring
  That execution's name, when it has one.
  harness_revisionstring
  mean_scorenumber· double
  median_cost_usdnumber· double
  median_duration_secondsnumber· double
  median_function_call_errorsnumber· double
  median_function_callsnumber· double
  median_tokensnumber· double
  median_turnsnumber· double
  plan_execution_idstring
  The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.
  run_countinteger· uintrequiredmin 0
  runsobject[]required
  Show 10 nestedHide nested
  attempt_idstringrequired
  completionstringrequiredenum completed, task_incomplete, undetermined
  detailsany of
  Absent when the run's native results could not be read.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  calls_by_workerobject[]required
  Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.
  Show 2 nestedHide nested
  callsinteger· uint64requiredmin 0
  workerstringrequired
  child_sessionsinteger· uintrequiredmin 0
  criteriaobject[]required
  Every criterion the run was scored on, in contract order.
  Show 6 nestedHide nested
  awardednumber· double
  Absent when nobody evaluated the criterion.
  descriptionstring
  The criterion as it was worded for this run; absent in older reports.
  gateboolean
  Whether this criterion blocked completion; absent in older reports.
  idstringrequired
  possiblenumber· doublerequired
  reasonstringrequired
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  sessionsobject[]required
  The root session first, each session followed by the ones it spawned.
  Show 8 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  labelstring
  The agent profile the session was spawned with, when the root's transcript names it.
  parent_session_idstring
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  Variant 2
  valuenull
  duration_secondsnumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  run_idstringrequired
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  turnsnumber· double
  scored_runsinteger· uintrequiredmin 0
  seedinteger· uint64min 0
  stack_modestringrequired
  statusstringrequired
  subject_modelstringrequired
  subject_providerstringrequired
  system_labelstringrequired
  system_revisionstring
  system_version_idstring
  test_idstringrequired
  test_versionstringrequired
  toany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  Show 7 nestedHide nested
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  case_countinteger· uintrequiredmin 0
  evaluated_version_idstringrequired
  execution_countinteger· uintrequiredmin 0
  mean_scorenumber· double
  median_cost_usdnumber· double
  median_duration_secondsnumber· double
  median_tokensnumber· double
  outcomesobjectrequired
  Show 4 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infra_failedinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  technical_failedinteger· uintrequiredmin 0
  pass_ratenumber· double
  samplesobjectrequired
  Show 5 nestedHide nested
  cost_usdinteger· uintrequiredmin 0
  duration_secondsinteger· uintrequiredmin 0
  scoreinteger· uintrequiredmin 0
  tokensinteger· uintrequiredmin 0
  turnsinteger· uintrequiredmin 0
  scored_runsinteger· uintrequiredmin 0
  total_runsinteger· uintrequiredmin 0
  Variant 2
  valuenull
  to_observationsobject[]required
  Show 32 nestedHide nested
  agent_profilestring
  The agent profile the subject ran with, when it had one.
  assessment_profile_sha256stringrequired
  assessment_summaryobjectrequired
  Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.
  Show 7 nestedHide nested
  assessment_countinteger· uintrequiredmin 0
  assessment_outcomesobjectrequired
  Show 6 nestedHide nested
  errorinteger· uintrequiredmin 0
  failedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  partialinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  asset_countinteger· uintrequiredmin 0
  asset_validation_outcomesobjectrequired
  Show 10 nestedHide nested
  invalidinteger· uintrequiredmin 0
  malformedinteger· uintrequiredmin 0
  not_evaluatedinteger· uintrequiredmin 0
  not_producedinteger· uintrequiredmin 0
  oversizedinteger· uintrequiredmin 0
  removed_during_cleanupinteger· uintrequiredmin 0
  unexpectedinteger· uintrequiredmin 0
  unreadableinteger· uintrequiredmin 0
  unsafe_pathinteger· uintrequiredmin 0
  validinteger· uintrequiredmin 0
  evidence_reference_countinteger· uintrequiredmin 0
  run_countinteger· uintrequiredmin 0
  system_statusesobjectrequired
  Show 6 nestedHide nested
  hard_gate_failedinteger· uintrequiredmin 0
  infrastructure_errorinteger· uintrequiredmin 0
  passedinteger· uintrequiredmin 0
  resource_limitinteger· uintrequiredmin 0
  subject_errorinteger· uintrequiredmin 0
  unavailableinteger· uintrequiredmin 0
  behavior_sha256stringrequired
  case_idstringrequired
  cohort_idstringrequired
  completed_atstringrequired
  contract_sha256stringrequired
  engine_revisionstring
  evaluated_version_idstring
  execution_idstringrequired
  execution_labelstring
  That execution's name, when it has one.
  harness_revisionstring
  mean_scorenumber· double
  median_cost_usdnumber· double
  median_duration_secondsnumber· double
  median_function_call_errorsnumber· double
  median_function_callsnumber· double
  median_tokensnumber· double
  median_turnsnumber· double
  plan_execution_idstring
  The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.
  run_countinteger· uintrequiredmin 0
  runsobject[]required
  Show 10 nestedHide nested
  attempt_idstringrequired
  completionstringrequiredenum completed, task_incomplete, undetermined
  detailsany of
  Absent when the run's native results could not be read.
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  cache_read_tokensinteger· uint64min 0
  cache_write_tokensinteger· uint64min 0
  calls_by_workerobject[]required
  Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.
  Show 2 nestedHide nested
  callsinteger· uint64requiredmin 0
  workerstringrequired
  child_sessionsinteger· uintrequiredmin 0
  criteriaobject[]required
  Every criterion the run was scored on, in contract order.
  Show 6 nestedHide nested
  awardednumber· double
  Absent when nobody evaluated the criterion.
  descriptionstring
  The criterion as it was worded for this run; absent in older reports.
  gateboolean
  Whether this criterion blocked completion; absent in older reports.
  idstringrequired
  possiblenumber· doublerequired
  reasonstringrequired
  input_tokensinteger· uint64min 0
  output_tokensinteger· uint64min 0
  sessionsobject[]required
  The root session first, each session followed by the ones it spawned.
  Show 8 nestedHide nested
  depthinteger· uint32requiredmin 0
  duration_msinteger· uint64min 0
  function_call_errorsinteger· uint64requiredmin 0
  function_callsinteger· uint64requiredmin 0
  labelstring
  The agent profile the session was spawned with, when the root's transcript names it.
  parent_session_idstring
  session_idstringrequired
  turnsinteger· uint64requiredmin 0
  Variant 2
  valuenull
  duration_secondsnumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  run_idstringrequired
  scorenumber· double
  statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
  turnsnumber· double
  scored_runsinteger· uintrequiredmin 0
  seedinteger· uint64min 0
  stack_modestringrequired
  statusstringrequired
  subject_modelstringrequired
  subject_providerstringrequired
  system_labelstringrequired
  system_revisionstring
  system_version_idstring
  Variant 2
  valuenull
- runs_currentinteger· uintrequiredmin 0
  Runs retained on the current definition.
- runs_totalinteger· uintrequiredmin 0
  Runs retained on every definition.
- selected_versionstring
- specany of
  any of (2)
  Variant 1
  areaany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  valuestringenum core-loop, data-safety, tool-adaptation, scheduling, coding, coordination, …
  Variant 2
  valuenull
  criteriaobject[]required
  Show 8 nestedHide nested
  descriptionstringrequired
  dimensionstringrequiredenum deliverable, structural_integrity, efficiency, robustness, e2e_infrastructure
  gatebooleanrequired
  idstringrequired
  kindstringrequiredenum required_check, signal, asset_validation
  labelstring
  policystringrequiredenum hard_gate, advisory
  weightinteger· uint8requiredmin 0
  denied_functionsstring[]required
  disciplinesstring[]enum frontend, backend, data, infra, testing
  executionobjectrequired
  Show 5 nestedHide nested
  max_output_tokensinteger· uint64min 0
  max_total_tokensinteger· uint64min 0
  Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.
  max_turnsinteger· uint32min 0
  Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.
  max_validation_retriesinteger· uint32min 0
  Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.
  stuck_timeout_secondsinteger· uint64requiredmin 0
  Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.
  promptstringrequired
  The prompt as materialized for a representative run: run-scoped paths resolve against a catalog namespace, never a real execution.
  protectsstring
  What the test protects and what breaks when it fails.
  summarystring
  The task in plain words; absent when the scenario states none.
  titlestring
  Human-readable test name; absent on responses from older runners.
  workersstring[]enum ade, auth, browser, canvas, compose, context-manager, …
  The iii workers the test protects; empty when it protects the harness alone.
  Variant 2
  valuenull
- test_idstringrequired
totalinteger· uintrequiredmin 0

e2e::dashboard::trends-get

Function

Read how one suite moves over time on one provider, model and profile (or `any` model or profile, on one line): every series, the stacks of the selected one, and a point per execution measured over its technically valid runs.

Request

modelstring
A model, or `any`: the suite's executions on every model, one line.
profilestring
Null or empty: no profile; `any`: every profile, one line; absent: the latest series that fits the rest.
providerstring
Left open (with the provider) when the model is `any`.
sincestring
Only executions started at or after this RFC 3339 instant; the series and the stacks' workers are still read over all time.
stackstring
A stack name, `not_recorded` or `any`; absent, or one the series never ran on: `any`.
suitestring
A suite id, or the sha256 of an unsaved suite.
untilstring
Only executions started at or before this RFC 3339 instant.

Response

pointsobject[]required
Oldest first.
- countedinteger· uintrequiredmin 0
  Technically valid runs.
- enginestring
- execution_idstringrequired
- labelstring
- measuresany of
  Null when nothing counted.
  any of (2)
  Variant 1
  completedinteger· uintrequiredmin 0
  duration_ms_meannumber· double
  function_call_errorsnumber· double
  function_callsnumber· double
  function_calls_meannumber· double
  input_tokens_meannumber· double
  plannedinteger· uintrequiredmin 0
  The runs planned (tests × runs per test), else the runs observed: what `completed` counts.
  score_meannumber· double
  turns_meannumber· double
  Variant 2
  valuenull
- modelstringrequired
- plannedstring[]
- profilestring
- providerstringrequired
  Its own model and profile: under `any` they differ between points.
- reasonstring
  When nothing counted: the execution's error, else why.
- runsinteger· uintrequiredmin 0
  Runs observed.
- sourceobjectrequired
  kindstringrequired
  `local`, `docker` or `github`.
  release_control_execution_idstring
  run_attemptinteger· uint32min 0
  run_idinteger· uint64min 0
  urlstring
- stackobjectrequired
  matched_by_workersbooleanrequired
  Not recorded, joined by its workers' names.
  namestring
- started_atstringrequired
- testsobject[]required
  Planned tests first, in plan order, then any other that ran.
  behavior_sha256string
  idstringrequired
  scorenumber· double
  Mean of the counted runs' scores.
  statestringrequired
  `scored`, `no_score`, `technical_invalid` or `not_run`.
- workersobject
  Identity per worker, runner and engine left out; null: not recorded.
selectedany of
The request's series, else the one with the latest execution; its provider and model, or its profile, are `any` when the request left them open.
any of (2)
Variant 1
modelstringrequired
profilestring
providerstringrequired
suitestringrequired
Variant 2
valuenull
seriesobject[]required
Every series, latest execution first.
- executionsinteger· uintrequiredmin 0
- latest_atstringrequired
- modelstringrequired
- profilestring
- providerstringrequired
- suitestringrequired
- suite_labelstringrequired
- wherestring[]requiredenum harness, docker, github
stackstringrequired
The stack filter applied.
stacksobject[]required
Of the selected series, then `not_recorded` (if any) and `any`.
- executionsinteger· uintrequiredmin 0
- namestringrequired

e2e::dashboard::version-compare

Function

Link to what changed between two builds of iii, the runner or a worker (release tags or commits) on GitHub, with how many commits apart they are when gh can tell.

Request

basestringrequired
A version, or a `@sha7` identity (`*` marks uncommitted changes).
headstringrequired
namestringrequired
`iii`, `harness-e2e` or a worker of iii-hq/workers.

Response

total_commitsinteger· uint64min 0
Null when `gh` is missing or GitHub answers an error.
urlstringrequired

e2e::dashboard::worker-resolve

Function

What the iii registry resolves a worker to: its newest release and the workers it brings, or the registry's error; kept ten minutes.

Request

workerstringrequired
A worker's name in the iii registry, e.g. `harness`.

Response

any of (2)

Variant 1

dependenciesstring[]required
Every worker its graph brings, engine built-ins left out.
namestringrequired
versionstringrequired
Its newest release.

Variant 2

errorobjectrequired
- codestringrequired
  The registry's code, e.g. `worker_not_found`.
- messagestringrequired

e2e::results-get

Function

Get the immutable result and manifest of an E2E execution.

Request

execution_idstringrequired

Response

archiveany of
any of (2)
Variant 1
archive_idstringrequired
created_atstringrequired
execution_idstringrequired
expires_atstring
identity_sha256stringrequired
manifestobjectrequired
Show 6 nestedHide nested
bucketstringrequired
keystringrequired
media_typestringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
uristringrequired
manifest_backupobjectrequired
Show 6 nestedHide nested
bucketstringrequired
keystringrequired
media_typestringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
uristringrequired
retention_classstringrequiredenum temporary, pull_request, longitudinal, canonical
Variant 2
valuenull
execution_idstringrequired
manifestany of
any of (2)
Variant 1
control_planeobjectrequired
Show 1 nestedHide nested
functionsobject[]required
Show 4 nestedHide nested
function_idstringrequired
request_schemaunknownrequired
response_schemaunknownrequired
sha256stringrequired
executionobjectrequired
Show 4 nestedHide nested
completed_atstringrequired
execution_idstringrequired
lanestringrequired
started_atstringrequired
observation_contractany of
Show 2 nestedHide nested
any of (2)
Variant 1
attemptinteger· uint32requiredmin 0
correlationobjectrequired
Show 3 nestedHide nested
deployment_idstringrequired
operation_idstringrequired
systemstringrequired
modeobjectrequired
Show 2 nestedHide nested
decisionstringrequiredenum observe_only
environmentstringrequiredenum demonstration
planobjectrequired
Show 4 nestedHide nested
catalog_sha256stringrequired
idstringrequired
revisionstringrequired
sha256stringrequired
runnerobjectrequired
Show 3 nestedHide nested
namestringrequired
revisionstringrequired
versionstringrequired
selected_casesobject[]required
Show 6 nestedHide nested
behavior_sha256stringrequired
case_idstringrequired
contract_sha256stringrequired
inputs_sha256stringrequired
scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
seedinteger· uint64requiredmin 0
targetobjectrequired
Show 3 nestedHide nested
applicationstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
versionstringrequired
Variant 2
valuenull
subjectobjectrequired
Show 7 nestedHide nested
agentany of
Show 2 nestedHide nested
any of (2)
Variant 1
configuration_sha256stringrequired
idstringrequired
Variant 2
valuenull
context_windowinteger· uint64requiredmin 0
max_output_tokensinteger· uint64requiredmin 0
modelstringrequired
providerstringrequired
supports_toolsboolean
supports_visionboolean
system_under_testobjectrequired
Show 7 nestedHide nested
contract_hashesobjectrequired
e2e_repositorystringrequired
e2e_revisionstringrequired
engine_revisionstring
engine_versionstringrequired
harness_versionstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
worker_contractsobject[]
Show 3 nestedHide nested
function_idstringrequired
request_schema_sha256stringrequired
response_schema_sha256stringrequired
Variant 2
valuenull
observationany of
any of (2)
Variant 1
evidenceobjectrequired
Show 3 nestedHide nested
artifactsobject[]
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
manifest_sha256string
results_sha256string
identityobjectrequired
Show 4 nestedHide nested
executionobjectrequired
Show 6 nestedHide nested
attemptinteger· uint32requiredmin 0
completed_atstringrequired
idstringrequired
request_sha256stringrequired
run_contract_sha256stringrequired
started_atstringrequired
planobjectrequired
Show 4 nestedHide nested
catalog_sha256stringrequired
idstringrequired
revisionstringrequired
sha256stringrequired
runnerobjectrequired
Show 3 nestedHide nested
namestringrequired
revisionstringrequired
versionstringrequired
targetobjectrequired
Show 3 nestedHide nested
applicationstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
versionstringrequired
modeobjectrequired
Show 2 nestedHide nested
decisionstringrequiredenum observe_only
environmentstringrequiredenum demonstration
outcomeobjectrequired
Show 5 nestedHide nested
control_phasestringrequired
data_availabilitystringrequiredenum complete, partial, unavailable
errorstring
objectivestringrequiredenum passed, hard_gate_failed, technical_failed, infrastructure_failed, cancelled, unsupported_plan
passedboolean
provenanceobjectrequired
Show 3 nestedHide nested
subject_modelstringrequired
subject_providerstringrequired
system_under_testany of
Show 2 nestedHide nested
any of (2)
Variant 1
contract_hashesobjectrequired
e2e_repositorystringrequired
e2e_revisionstringrequired
engine_revisionstring
engine_versionstringrequired
harness_versionstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
Variant 2
valuenull
samplesobject[]required
Show 12 nestedHide nested
attempt_idstringrequired
behavior_sha256string
case_idstringrequired
data_availabilitystringrequiredenum complete, partial, unavailable
derivationsobject[]
Show 4 nestedHide nested
formulastringrequired
formula_versionstringrequired
metricstringrequired
originstringrequiredenum observed, derived_from_observed
metric_valuesobject
metricsany of
Show 2 nestedHide nested
any of (2)
Variant 1
child_sessionsinteger· uint64min 0
child_turnsinteger· uint64min 0
cost_usdnumber· double
critical_path_msinteger· uint64min 0
effective_fan_outinteger· uint64min 0
function_call_errorsinteger· uint64min 0
function_callsinteger· uint64min 0
input_tokensinteger· uint64min 0
observed_complexityobjectrequired
Show 11 nestedHide nested
ambiguity_levelinteger· uint64min 0
artifact_countinteger· uint64min 0
coordination_edgesinteger· uint64min 0
dependency_depthinteger· uint64min 0
external_systemsinteger· uint64min 0
parallel_branchesinteger· uint64min 0
planning_depthinteger· uint64min 0
state_transitionsinteger· uint64min 0
unavailableobject
validation_loopsinteger· uint64min 0
wake_cyclesinteger· uint64min 0
observed_workinteger· uint64min 0
output_tokensinteger· uint64min 0
root_turnsinteger· uint64min 0
technical_attemptsinteger· uint32requiredmin 0
total_tokensinteger· uint64min 0
transient_resumesinteger· uint64min 0
unavailableobject
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
wall_time_msinteger· uint64requiredmin 0
Variant 2
valuenull
originstringrequiredenum observed, derived_from_observed
run_idstringrequired
scenario_idstringrequired
seedinteger· uint64requiredmin 0
statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
schemastringrequired
Variant 2
valuenull
observation_artifactany of
any of (2)
Variant 1
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
Variant 2
valuenull
phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
reportany of
any of (2)
Variant 1
assessment_contractobjectrequired
Show 1 nestedHide nested
runsobject[]required
Show 5 nestedHide nested
assessmentsobject[]
Show 9 nestedHide nested
criterion_idstringrequired
dimensionstringrequiredenum deliverable, structural_integrity, efficiency, robustness, e2e_infrastructure
evidenceobject[]
Show 3 nestedHide nested
artifact_idstringrequired
artifact_sha256stringrequired
locatorstring
kindstringrequiredenum required_check, signal, asset_validation
outcomestringrequiredenum passed, failed, partial, not_evaluated, unavailable, error
policystringrequiredenum hard_gate, advisory
scoreany of
Show 2 nestedHide nested
any of (2)
Variant 1
awardedinteger· uint8requiredmin 0
possibleinteger· uint8requiredmin 0
Variant 2
valuenull
summarystringrequired
targetobjectrequired
Show 2 nestedHide nested
idstringrequired
kindstringrequiredenum criterion
assetsobject[]
Show 4 nestedHide nested
asset_idstringrequired
evidenceobject[]
Show 3 nestedHide nested
artifact_idstringrequired
artifact_sha256stringrequired
locatorstring
outcomestringrequiredenum valid, invalid, malformed, oversized, not_produced, unreadable, …
summarystringrequired
attempt_idstringrequired
run_idstringrequired
system_statusstringrequiredenum unavailable, passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
engine_revisionstring
executionobjectrequired
Show 4 nestedHide nested
completed_atstringrequired
execution_idstringrequired
lanestringrequired
started_atstringrequired
manifestany of
Show 2 nestedHide nested
any of (2)
Variant 1
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
Variant 2
valuenull
objective_outcomestringrequiredenum passed, failed, inconclusive
observation_contractany of
Show 2 nestedHide nested
any of (2)
Variant 1
attemptinteger· uint32requiredmin 0
correlationobjectrequired
Show 3 nestedHide nested
deployment_idstringrequired
operation_idstringrequired
systemstringrequired
modeobjectrequired
Show 2 nestedHide nested
decisionstringrequiredenum observe_only
environmentstringrequiredenum demonstration
planobjectrequired
Show 4 nestedHide nested
catalog_sha256stringrequired
idstringrequired
revisionstringrequired
sha256stringrequired
runnerobjectrequired
Show 3 nestedHide nested
namestringrequired
revisionstringrequired
versionstringrequired
selected_casesobject[]required
Show 6 nestedHide nested
behavior_sha256stringrequired
case_idstringrequired
contract_sha256stringrequired
inputs_sha256stringrequired
scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
seedinteger· uint64requiredmin 0
targetobjectrequired
Show 3 nestedHide nested
applicationstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
versionstringrequired
Variant 2
valuenull
passedbooleanrequired
persistence_errorsstring[]
redactionall of
Show 1 nestedHide nested
all of (1)
Variant 1
redacted_fieldsinteger· uint32requiredmin 0
redacted_valuesinteger· uint32requiredmin 0
rulesstring[]required
report_statestringrequiredenum complete, partial
result_contract_sha256stringrequired
scenariosobject[]required
Show 15 nestedHide nested
aggregateobjectrequired
Show 23 nestedHide nested
completed_runsinteger· uint32requiredmin 0
completion_evidence_coveragenumber· double
completion_ratenumber· double
costobjectrequired
Show 2 nestedHide nested
subject_usdnumber· double
total_usdnumber· double
deferred_runsinteger· uint32requiredmin 0
execution_reliabilitynumber· double
failed_attempt_tokensinteger· uint64min 0
mean_scorenumber· double
Mean score of the technically valid runs that have one; null when none.
observed_runsinteger· uint32requiredmin 0
pass_ratenumber· doublerequired
passed_runsinteger· uint32requiredmin 0
planned_runsinteger· uint32requiredmin 0
required_passesinteger· uint32requiredmin 0
robustnessobjectrequired
Show 13 nestedHide nested
cost_per_successful_deliverablenumber· double
deliverable_success_ratenumber· double
eligiblebooleanrequired
flaky_ratenumber· double
median_wall_time_msnumber· double
minimum_sample_sizeinteger· uint32requiredmin 0
p95_wall_time_msinteger· uint64min 0
sample_sizeinteger· uint32requiredmin 0
structural_integrity_ratenumber· double
tail_minimum_sample_sizeinteger· uint32requiredmin 0
technical_failure_ratenumber· double
unavailableobject
wall_time_variancenumber· double
scored_runsinteger· uint32requiredmin 0
Technically valid runs that have a score.
task_incomplete_runsinteger· uint32requiredmin 0
technical_failuresinteger· uint32requiredmin 0
technical_invalid_runsinteger· uint32requiredmin 0
technical_valid_runsinteger· uint32requiredmin 0
tokens_completed_p50number· double
tokens_per_completionnumber· double
total_tokens_consumedinteger· uint64min 0
undetermined_runsinteger· uint32requiredmin 0
areaany of
Show 2 nestedHide nested
any of (2)
Variant 1
valuestringenum core-loop, data-safety, tool-adaptation, scheduling, coding, coordination, …
Variant 2
valuenull
behavior_sha256string
Digest of the definition the case was materialized from. Absent only when no case could be materialized.
caseany of
Show 2 nestedHide nested
any of (2)
Variant 1
behavior_sha256stringrequired
Digest of the scenario definition this case was materialized from: prompt, execution policy, criteria, and the case-independent contract. Two cases with the same digest were evaluated by the same definition. Sealed by `ScenarioId::materialize`; empty only before sealing.
case_idstringrequired
characterizationall of
Show 1 nestedHide nested
all of (1)
Variant 1
human_horizonall of
Show 1 nestedHide nested
all of (1)
Variant 1
basisall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum unknown, author_estimate, measured
max_minutesinteger· uint32min 0
min_minutesinteger· uint32min 0
realismall of
Show 1 nestedHide nested
all of (1)
Variant 1
executionall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum synthetic, realistic_simulator, frozen_real_artifact
shadowall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum none, read_only
deliverable_contractobjectrequired
Show 4 nestedHide nested
artifactsobject[]required
Show 5 nestedHide nested
idstringrequired
kindstringrequired
max_size_bytesinteger· uint64requiredmin 0
media_typestringrequired
schemaunknownrequired
capture_before_cleanupbooleanrequired
invariantsobject[]required
Show 2 nestedHide nested
descriptionstringrequired
idstringrequired
provenance_requiredbooleanrequired
inputsunknownrequired
inputs_sha256stringrequired
required_capabilitiesstring[]requiredenum browser::interactive, cross_repo_contract_simulator::v1, curl, docker, e2e::adaptive-flow-v1, e2e::control-plane-v1, …
scenario_idstringrequired
seedinteger· uint64requiredmin 0
Variant 2
valuenull
case_idstring
deferral_reasonstring
disciplinesstring[]enum frontend, backend, data, infra, testing
execution_policyobjectrequired
Show 5 nestedHide nested
max_output_tokensinteger· uint64min 0
max_total_tokensinteger· uint64min 0
Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.
max_turnsinteger· uint32min 0
Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.
max_validation_retriesinteger· uint32min 0
Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.
stuck_timeout_secondsinteger· uint64requiredmin 0
Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.
passedbooleanrequired
protectsstring
What the test protects and what breaks when it fails.
runsobject[]required
Show 26 nestedHide nested
attempt_idstringrequired
attempt_numberinteger· uint32requiredmin 0
auditany of
Advisory behavioral audit over the captured transcript and metrics. Never contributes to score, status, gates, or longitudinal inputs.
Show 2 nestedHide nested
any of (2)
Variant 1
flagsobject[]required
Show 6 nestedHide nested
confidencenumber· double
evidenceobject[]
Show 3 nestedHide nested
detailstringrequired
Sanitized, bounded description. Never contains secret material.
function_idstring
message_indexinteger· uintmin 0
Index of the transcript message the observation anchors to.
kindone ofrequired
Show 4 nestedHide nested
one of (4)
Variant 1
valuestringenum verifier_tampering
Variant 2
valuestringenum out_of_scope_session_access
Variant 3
valuestringenum secret_exposure
Variant 4
valuestringenum destructive_action
severitystringrequiredenum info, warning, critical
sourcestringrequiredenum deterministic
summarystringrequired
Variant 2
valuenull
completionstringrequiredenum completed, task_incomplete, undetermined
costobjectrequired
Show 2 nestedHide nested
subject_usdnumber· double
total_usdnumber· double
criteriaobject[]required
Show 8 nestedHide nested
awardedinteger· uint8min 0
descriptionstring
gateboolean
A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell "no gate" from a report older than the field.
gate_minimuminteger· uint8min 0
The points a gate needs to count as passed when fewer than `possible` already mean the task's purpose was met. Absent: full points.
idstringrequired
labelstring
The criterion as a short statement; absent in reports older than the field and for criteria that state none. Readers fall back to `description`.
possibleinteger· uint8requiredmin 0
reasonstringrequired
deliverablesobject[]
Show 13 nestedHide nested
artifactany of
Show 2 nestedHide nested
any of (2)
Variant 1
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
Variant 2
valuenull
content_formatstringrequiredenum json, text_utf8
content_sha256stringrequired
content_size_bytesinteger· uint64requiredmin 0
idstringrequired
invariantsobject[]required
Show 3 nestedHide nested
idstringrequired
passedbooleanrequired
reasonstringrequired
kindstringrequired
media_typestringrequired
previewunknownrequired
provenanceobject[]required
Show 3 nestedHide nested
kindstringrequired
relationstringrequired
source_idstringrequired
provenance_validbooleanrequired
schema_validbooleanrequired
screenshotsobject[]
The screenshots embedded in the content, so a reader can list them without opening the artifact. Empty when there are none.
Show 5 nestedHide nested
captionstringrequired
The capture's caption from its `captures.json`, else the file's path without the attempt folder it was packed under.
media_typestringrequired
pointerstringrequired
JSON Pointer to the embedded file inside the deliverable: `/attachments/board-desktop.png`.
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
dimensionsobject[]
Show 3 nestedHide nested
dimensionstringrequiredenum deliverable, structural_integrity, efficiency, robustness, e2e_infrastructure
passedboolean
signalsunknownrequired
efficiencyany of
Show 2 nestedHide nested
any of (2)
Variant 1
child_sessionsinteger· uint64min 0
child_turnsinteger· uint64min 0
cost_usdnumber· double
critical_path_msinteger· uint64min 0
effective_fan_outinteger· uint64min 0
function_call_errorsinteger· uint64min 0
function_callsinteger· uint64min 0
input_tokensinteger· uint64min 0
observed_complexityobjectrequired
Show 11 nestedHide nested
ambiguity_levelinteger· uint64min 0
artifact_countinteger· uint64min 0
coordination_edgesinteger· uint64min 0
dependency_depthinteger· uint64min 0
external_systemsinteger· uint64min 0
parallel_branchesinteger· uint64min 0
planning_depthinteger· uint64min 0
state_transitionsinteger· uint64min 0
unavailableobject
validation_loopsinteger· uint64min 0
wake_cyclesinteger· uint64min 0
observed_workinteger· uint64min 0
output_tokensinteger· uint64min 0
root_turnsinteger· uint64min 0
technical_attemptsinteger· uint32requiredmin 0
total_tokensinteger· uint64min 0
transient_resumesinteger· uint64min 0
unavailableobject
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
wall_time_msinteger· uint64requiredmin 0
Variant 2
valuenull
evaluatorsobjectrequired
Show 1 nestedHide nested
completionstringrequiredenum not_required, pending, available, unavailable
evidenceobject[]
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
failuresobject[]
Show 7 nestedHide nested
codestring
contaminationall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum none, case, execution
domainstringrequiredenum subject, resource, e2e_infrastructure
messagestringrequired
phasestringrequiredenum setup, execute, collect, evaluate, cleanup
retry_scopeall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum none, same_slot
statusall ofrequired
The failure's original classification, independent of the primary run status.
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
metricsany of
Show 2 nestedHide nested
any of (2)
Variant 1
by_sessionobject[]required
Show 16 nestedHide nested
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextunknown
cost_usdnumber· double
depthinteger· uint32requiredmin 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
parent_session_idstring
reasoning_tokensinteger· uint64min 0
session_idstringrequired
transient_resumesinteger· uint64min 0
turnsinteger· uint64requiredmin 0
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
completebooleanrequired
root_session_idstringrequired
totalsobjectrequired
Show 13 nestedHide nested
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64requiredmin 0
transient_resumesinteger· uint64min 0
turnsinteger· uint64requiredmin 0
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
tracesunknown
Variant 2
valuenull
promptstringrequired
retry_attemptsobject[]
Show 20 nestedHide nested
attempt_idstringrequired
attempt_numberinteger· uint32requiredmin 0
completionstringrequiredenum completed, task_incomplete, undetermined
costobjectrequired
Show 2 nestedHide nested
subject_usdnumber· double
total_usdnumber· double
deliverablesobject[]
Show 13 nestedHide nested
artifactany of
Show 2 nestedHide nested
any of (2)
Variant 1
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
Variant 2
valuenull
content_formatstringrequiredenum json, text_utf8
content_sha256stringrequired
content_size_bytesinteger· uint64requiredmin 0
idstringrequired
invariantsobject[]required
Show 3 nestedHide nested
idstringrequired
passedbooleanrequired
reasonstringrequired
kindstringrequired
media_typestringrequired
previewunknownrequired
provenanceobject[]required
Show 3 nestedHide nested
kindstringrequired
relationstringrequired
source_idstringrequired
provenance_validbooleanrequired
schema_validbooleanrequired
screenshotsobject[]
The screenshots embedded in the content, so a reader can list them without opening the artifact. Empty when there are none.
Show 5 nestedHide nested
captionstringrequired
The capture's caption from its `captures.json`, else the file's path without the attempt folder it was packed under.
media_typestringrequired
pointerstringrequired
JSON Pointer to the embedded file inside the deliverable: `/attachments/board-desktop.png`.
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
dimensionsobject[]
Show 3 nestedHide nested
dimensionstringrequiredenum deliverable, structural_integrity, efficiency, robustness, e2e_infrastructure
passedboolean
signalsunknownrequired
efficiencyany of
Show 2 nestedHide nested
any of (2)
Variant 1
child_sessionsinteger· uint64min 0
child_turnsinteger· uint64min 0
cost_usdnumber· double
critical_path_msinteger· uint64min 0
effective_fan_outinteger· uint64min 0
function_call_errorsinteger· uint64min 0
function_callsinteger· uint64min 0
input_tokensinteger· uint64min 0
observed_complexityobjectrequired
Show 11 nestedHide nested
ambiguity_levelinteger· uint64min 0
artifact_countinteger· uint64min 0
coordination_edgesinteger· uint64min 0
dependency_depthinteger· uint64min 0
external_systemsinteger· uint64min 0
parallel_branchesinteger· uint64min 0
planning_depthinteger· uint64min 0
state_transitionsinteger· uint64min 0
unavailableobject
validation_loopsinteger· uint64min 0
wake_cyclesinteger· uint64min 0
observed_workinteger· uint64min 0
output_tokensinteger· uint64min 0
root_turnsinteger· uint64min 0
technical_attemptsinteger· uint32requiredmin 0
total_tokensinteger· uint64min 0
transient_resumesinteger· uint64min 0
unavailableobject
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
wall_time_msinteger· uint64requiredmin 0
Variant 2
valuenull
evaluatorsobjectrequired
Show 1 nestedHide nested
completionstringrequiredenum not_required, pending, available, unavailable
evidenceobject[]
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
failuresobject[]required
Show 7 nestedHide nested
codestring
contaminationall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum none, case, execution
domainstringrequiredenum subject, resource, e2e_infrastructure
messagestringrequired
phasestringrequiredenum setup, execute, collect, evaluate, cleanup
retry_scopeall of
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum none, same_slot
statusall ofrequired
The failure's original classification, independent of the primary run status.
Show 1 nestedHide nested
all of (1)
Variant 1
valuestringenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
metricsany of
Show 2 nestedHide nested
any of (2)
Variant 1
by_sessionobject[]required
Show 16 nestedHide nested
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
contextunknown
cost_usdnumber· double
depthinteger· uint32requiredmin 0
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
parent_session_idstring
reasoning_tokensinteger· uint64min 0
session_idstringrequired
transient_resumesinteger· uint64min 0
turnsinteger· uint64requiredmin 0
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
completebooleanrequired
root_session_idstringrequired
totalsobjectrequired
Show 13 nestedHide nested
cache_read_tokensinteger· uint64min 0
cache_write_tokensinteger· uint64min 0
cost_usdnumber· double
function_call_errorsinteger· uint64requiredmin 0
function_callsinteger· uint64requiredmin 0
input_tokensinteger· uint64min 0
output_tokensinteger· uint64min 0
reasoning_tokensinteger· uint64min 0
sessionsinteger· uint64requiredmin 0
transient_resumesinteger· uint64min 0
turnsinteger· uint64requiredmin 0
validation_retriesinteger· uint64min 0
wake_resumesinteger· uint64min 0
tracesunknown
Variant 2
valuenull
run_idstringrequired
scenario_flowany of
Show 2 nestedHide nested
any of (2)
Variant 1
checkpointobjectrequired
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
cleanupobjectrequired
Show 3 nestedHide nested
duration_msinteger· uint64requiredmin 0
failurestring
statusstringrequiredenum succeeded, failed
definition_sha256stringrequired
snapshotunknownrequired
Variant 2
valuenull
scoreinteger· uint8min 0
The points the attempt's evaluated criteria awarded, as on the run.
semantic_testsobject[]
Show 21 nestedHide nested
activationone ofrequired
Show 3 nestedHide nested
one of (3)
Variant 1
policystringrequiredenum always
Variant 2
conditionsobject[]required
Show 3 nestedHide nested
equalsbooleanrequired
node_idstringrequired
portstringrequired
policystringrequiredenum all
Variant 3
conditionsobject[]required
Show 3 nestedHide nested
equalsbooleanrequired
node_idstringrequired
portstringrequired
policystringrequiredenum any
assetsobject[]
Show 9 nestedHide nested
artifactobjectrequired
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
content_sha256stringrequired
idstringrequired
kindstringrequired
media_typestringrequired
namespaced_idstringrequired
previewunknownrequired
preview_truncatedbooleanrequired
size_bytesinteger· uint64requiredmin 0
completed_atstring
cost_usdnumber· double
dependenciesstring[]required
dependency_policystringrequiredenum succeeded, terminal
duration_msinteger· uint64requiredmin 0
evaluationsobject[]
Show 5 nestedHide nested
evidence_idsstring[]
idstringrequired
outcomestringrequiredenum passed, failed, advisory, not_evaluated
scorenumber· double
summarystringrequired
failuresobject[]
Show 3 nestedHide nested
messagestringrequired
phasestringrequiredenum preflight, execute, capture, persist, evaluate, cleanup, …
technicalbooleanrequired
hard_gatesobject[]
Show 4 nestedHide nested
evidence_idsstring[]
idstringrequired
passedbooleanrequired
reasonstringrequired
harness_session_idstring
metricsunknown
node_idstringrequired
outputsobject
redactionall of
Show 1 nestedHide nested
all of (1)
Variant 1
redacted_fieldsinteger· uint32requiredmin 0
redacted_valuesinteger· uint32requiredmin 0
rulesstring[]required
requiredbooleanrequired
skip_reasonstring
started_atstring
statusstringrequiredenum pending, running, succeeded, hard_gate_failed, failed, skipped, …
step_typestringrequired
transcriptunknown
session_idstringrequired
statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
technicalstringrequiredenum valid, technical_invalid
transcriptunknown
wall_time_msinteger· uint64requiredmin 0
run_idstringrequired
scenario_flowany of
Evidence-only flow identity emitted by Rust. This value is never accepted as runner input and cannot reconstruct executable configuration.
Show 2 nestedHide nested
any of (2)
Variant 1
checkpointobjectrequired
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
cleanupobjectrequired
Show 3 nestedHide nested
duration_msinteger· uint64requiredmin 0
failurestring
statusstringrequiredenum succeeded, failed
definition_sha256stringrequired
snapshotunknownrequired
Variant 2
valuenull
scenario_measurementsobject[]
Show 4 nestedHide nested
idstringrequired
originstringrequiredenum observed, derived_from_observed
unitstringrequired
valuenumber· doublerequired
scoreinteger· uint8min 0
semantic_testsobject[]
Semantic tests executed inside a code-defined composite scenario. Product requests, polling and captures stay inside these meaningful test units.
Show 21 nestedHide nested
activationone ofrequired
Show 3 nestedHide nested
one of (3)
Variant 1
policystringrequiredenum always
Variant 2
conditionsobject[]required
Show 3 nestedHide nested
equalsbooleanrequired
node_idstringrequired
portstringrequired
policystringrequiredenum all
Variant 3
conditionsobject[]required
Show 3 nestedHide nested
equalsbooleanrequired
node_idstringrequired
portstringrequired
policystringrequiredenum any
assetsobject[]
Show 9 nestedHide nested
artifactobjectrequired
Show 6 nestedHide nested
idstringrequired
kindstringrequired
media_typestringrequired
pathstringrequired
sha256stringrequired
size_bytesinteger· uint64requiredmin 0
content_sha256stringrequired
idstringrequired
kindstringrequired
media_typestringrequired
namespaced_idstringrequired
previewunknownrequired
preview_truncatedbooleanrequired
size_bytesinteger· uint64requiredmin 0
completed_atstring
cost_usdnumber· double
dependenciesstring[]required
dependency_policystringrequiredenum succeeded, terminal
duration_msinteger· uint64requiredmin 0
evaluationsobject[]
Show 5 nestedHide nested
evidence_idsstring[]
idstringrequired
outcomestringrequiredenum passed, failed, advisory, not_evaluated
scorenumber· double
summarystringrequired
failuresobject[]
Show 3 nestedHide nested
messagestringrequired
phasestringrequiredenum preflight, execute, capture, persist, evaluate, cleanup, …
technicalbooleanrequired
hard_gatesobject[]
Show 4 nestedHide nested
evidence_idsstring[]
idstringrequired
passedbooleanrequired
reasonstringrequired
harness_session_idstring
metricsunknown
node_idstringrequired
outputsobject
redactionall of
Show 1 nestedHide nested
all of (1)
Variant 1
redacted_fieldsinteger· uint32requiredmin 0
redacted_valuesinteger· uint32requiredmin 0
rulesstring[]required
requiredbooleanrequired
skip_reasonstring
started_atstring
statusstringrequiredenum pending, running, succeeded, hard_gate_failed, failed, skipped, …
step_typestringrequired
transcriptunknown
session_idstringrequired
statusstringrequiredenum passed, hard_gate_failed, subject_error, resource_limit, infrastructure_error
technicalstringrequiredenum valid, technical_invalid
transcriptunknown
wall_time_msinteger· uint64requiredmin 0
worker_contractsobject[]
Exact request/response schemas observed after this scenario's setup. Run-scoped fixture functions cannot be observed in the suite preflight, so attempts retain them here and the suite folds them into the manifest.
Show 3 nestedHide nested
function_idstringrequired
request_schema_sha256stringrequired
response_schema_sha256stringrequired
scenario_idstringrequired
summarystring
titlestring
Presentation captured when this result was produced; absent in older results.
workersstring[]enum ade, auth, browser, canvas, compose, context-manager, …
The iii workers the test protects; empty when it protects the harness alone.
slot_start_deadline_secondsinteger· uint64min 0
Soft scheduling deadline: prevents starting slots after this interval.
subjectobjectrequired
Show 7 nestedHide nested
agentany of
Show 2 nestedHide nested
any of (2)
Variant 1
configuration_sha256stringrequired
idstringrequired
Variant 2
valuenull
context_windowinteger· uint64requiredmin 0
max_output_tokensinteger· uint64requiredmin 0
modelstringrequired
providerstringrequired
supports_toolsboolean
supports_visionboolean
system_under_testobjectrequired
Show 7 nestedHide nested
contract_hashesobjectrequired
e2e_repositorystringrequired
e2e_revisionstringrequired
engine_revisionstring
engine_versionstringrequired
harness_versionstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
Variant 2
valuenull
request_sha256stringrequired
result_pathstring
run_contract_sha256string

e2e::results-list

Function

List durable E2E executions using comparable identity filters.

Request

lanestring
limitinteger· uint16min 0
scenario_idany of
any of (2)
Variant 1
valuestringenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
Variant 2
valuenull

Response

executionsobject[]required
- active_attemptany of
  any of (2)
  Variant 1
  attempt_idstringrequired
  run_idstring
  scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
  session_idstringrequired
  Variant 2
  valuenull
- cancel_requestedbooleanrequired
- errorstring
- execution_idstringrequired
- journal_progressobjectrequired
  active_attempt_idstring
  attempts_finishedinteger· uint64requiredmin 0
  attempts_startedinteger· uint64requiredmin 0
  committed_eventsinteger· uint64requiredmin 0
  last_event_sha256string
  phasestring
  planned_slotsinteger· uint64requiredmin 0
  runs_committedinteger· uint64requiredmin 0
  slots_deferredinteger· uint64requiredmin 0
  subject_observations_committedinteger· uint64requiredmin 0
  terminalbooleanrequired
  terminal_reasonstring
  terminal_stateany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  valuestringenum completed, failed, cancelled, needs_reconciliation, unsupported
  Variant 2
  valuenull
- phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
- terminalbooleanrequired
- transitionsobject[]required
  atstringrequired
  phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
  reasonstringrequired
- updated_atstringrequired

e2e::run

Function

Admit an asynchronous, idempotent E2E execution.

Request

agentstring
idempotency_keystringrequired
labelstring
lanestring
modelstringrequired
progress_interval_secondsinteger· uint64min 0
providerstringrequired
rotating_seedsinteger[]· uint64min 0
run_contractany of
any of (2)
Variant 1
attemptinteger· uint32requiredmin 0
correlationobjectrequired
Show 3 nestedHide nested
deployment_idstringrequired
operation_idstringrequired
systemstringrequired
modeobjectrequired
Show 2 nestedHide nested
decisionstringrequiredenum observe_only
environmentstringrequiredenum demonstration
planobjectrequired
Show 4 nestedHide nested
catalog_sha256stringrequired
idstringrequired
revisionstringrequired
sha256stringrequired
runnerobjectrequired
Show 3 nestedHide nested
namestringrequired
revisionstringrequired
versionstringrequired
selected_casesobject[]required
Show 6 nestedHide nested
behavior_sha256stringrequired
case_idstringrequired
contract_sha256stringrequired
inputs_sha256stringrequired
scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
seedinteger· uint64requiredmin 0
targetobjectrequired
Show 3 nestedHide nested
applicationstringrequired
stackone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
modestringrequiredenum source
workers_repositorystringrequired
workers_revisionstringrequired
Variant 2
modestringrequiredenum registry
stack_lock_digeststringrequired
stack_versionsobjectrequired
versionstringrequired
Variant 2
valuenull
runsinteger· uint32min 0
scenariosstring[]enum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
seedinteger· uint64min 0
slot_start_deadline_secondsinteger· uint64min 0
Soft scheduling budget, resolved before request hashing/admission.
technical_retriesinteger· uint8min 0

Response

duplicatebooleanrequired
execution_idstringrequired
phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
request_sha256stringrequired
run_contract_sha256string

e2e::scenarios-list

Function

List every built-in E2E test with its immutable contract, capabilities, complexity, and materialized case.

Request

seedinteger· uint64min 0
Optional deterministic seed used to materialize each listed test case. Omit it to use each scenario's canonical seed.

Response

catalog_sha256stringrequired
runnerobjectrequired
- namestringrequired
- revisionstringrequired
- versionstringrequired
scenariosobject[]required
- areaany of
  any of (2)
  Variant 1
  valuestringenum core-loop, data-safety, tool-adaptation, scheduling, coding, coordination, …
  Variant 2
  valuenull
- behavior_sha256stringrequired
- case_idstringrequired
- characterizationobjectrequired
  human_horizonall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  basisall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum unknown, author_estimate, measured
  max_minutesinteger· uint32min 0
  min_minutesinteger· uint32min 0
  realismall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  executionall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum synthetic, realistic_simulator, frozen_real_artifact
  shadowall of
  Show 1 nestedHide nested
  all of (1)
  Variant 1
  valuestringenum none, read_only
- contract_sha256stringrequired
- deliverable_contractobjectrequired
  artifactsobject[]required
  Show 5 nestedHide nested
  idstringrequired
  kindstringrequired
  max_size_bytesinteger· uint64requiredmin 0
  media_typestringrequired
  schemaunknownrequired
  capture_before_cleanupbooleanrequired
  invariantsobject[]required
  Show 2 nestedHide nested
  descriptionstringrequired
  idstringrequired
  provenance_requiredbooleanrequired
- disciplinesstring[]enum frontend, backend, data, infra, testing
- inputs_sha256stringrequired
- protectsstring
- required_capabilitiesstring[]requiredenum browser::interactive, cross_repo_contract_simulator::v1, curl, docker, e2e::adaptive-flow-v1, e2e::control-plane-v1, …
- resource_envelopeobjectrequired
  executionobjectrequired
  Show 5 nestedHide nested
  max_output_tokensinteger· uint64min 0
  max_total_tokensinteger· uint64min 0
  Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.
  max_turnsinteger· uint32min 0
  Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.
  max_validation_retriesinteger· uint32min 0
  Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.
  stuck_timeout_secondsinteger· uint64requiredmin 0
  Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.
  workflowany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  max_cost_usdnumber· double
  max_nodesinteger· uint16requiredmin 0
  max_parallelinteger· uint16requiredmin 0
  max_total_tokensinteger· uint64min 0
  step_timeout_secondsinteger· uint64requiredmin 0
  technical_retriesinteger· uint8requiredmin 0
  workflow_timeout_secondsinteger· uint64requiredmin 0
  Variant 2
  valuenull
- scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
- seedinteger· uint64requiredmin 0
- summarystring
- titlestring
- workersstring[]enum ade, auth, browser, canvas, compose, context-manager, …
  The iii workers the test protects; empty when it protects the harness alone.
schemastringrequired

e2e::status

Function

Read the durable phase and progress of an E2E execution.

Request

execution_idstringrequired

Response

active_attemptany of
any of (2)
Variant 1
attempt_idstringrequired
run_idstring
scenario_idstringrequiredenum registry_planning, registry_implementation, registry_environment, registry_verification, kanban_c1_foundation, kanban_c2_persistence, …
session_idstringrequired
Variant 2
valuenull
cancel_requestedbooleanrequired
errorstring
execution_idstringrequired
journal_progressobjectrequired
- active_attempt_idstring
- attempts_finishedinteger· uint64requiredmin 0
- attempts_startedinteger· uint64requiredmin 0
- committed_eventsinteger· uint64requiredmin 0
- last_event_sha256string
- phasestring
- planned_slotsinteger· uint64requiredmin 0
- runs_committedinteger· uint64requiredmin 0
- slots_deferredinteger· uint64requiredmin 0
- subject_observations_committedinteger· uint64requiredmin 0
- terminalbooleanrequired
- terminal_reasonstring
- terminal_stateany of
  any of (2)
  Variant 1
  valuestringenum completed, failed, cancelled, needs_reconciliation, unsupported
  Variant 2
  valuenull
phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
terminalbooleanrequired
transitionsobject[]required
- atstringrequired
- phasestringrequiredenum requested, admitted, preflighting, materializing, setting_up, executing, …
- reasonstringrequired
updated_atstringrequired

e2e::ui-content

Function

Serve the Harness E2E injectable Console assets.

Request

pathstringrequired

Response

contentstringrequired
content_typestringrequired

Triggers1

e2e::dashboard::changed

Trigger

A local E2E execution started, produced progress, or reached a terminal state.

Invocation

Empty object

Return

valueunknown

Install

Functions45

Triggers1

Search the registry