harness-e2e
v0.15.2Measure Harness capability, retain reproducible evidence, and expose the E2E dashboard.
Install
bash
iii trigger compose::add worker=harness-e2e@0.15.2Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.
Agent-ready brief for v0.15.2
Install, config, dependencies, readme, skills and API reference, all in one place. Fetch as agent-context.md for an LLM to consume.
Full markdown
The same content rendered as discrete blocks below is exposed as a single markdown document at
/registry/workers/harness-e2e.md?version=0.15.2. Paste it into an LLM prompt or pipe it through curl from a worker.Install
iii trigger compose::add worker=harness-e2e@0.15.2Configuration
iii-config.yaml
- control_database: harness_e2e
control_namespace: harness-e2e-control
data_dir: ~/.iii/data/harness-e2eDependencies
Dependency · 3Version
Readme
README.md
Harness E2E
harness-e2e measures what a Harness stack can execute: correct deliverables,
structural integrity, bounded work, and repeatable outcomes.
The repository stays independent of the workers source tree. Discovery,
execution, observation, state access, and cleanup go through functions
registered in iii. The product input is an already-running iii stack.
One binary, harness-e2e, does both jobs. Compose starts it as the
asynchronous e2e::* control plane and the injectable Console page. The same
binary also runs scenarios and inspects reports from the command line.
Score
A run's score is the sum of the points its evaluated criteria awarded.
- A criterion nobody evaluated adds nothing.
- Scores are not normalized or rescaled.
- Criterion points stay independent of completion and resource limits.
- Criteria do not veto the score or approve a run.
- Completion, technical validity, artifact evidence, and runtime controls are reported separately.
- Infrastructure and execution failures still fail the CLI.
Native criteria keep awards that were already measured when a later check
cannot run. That later check has no award and stays not_evaluated. An
incomplete criterion set has no total score. A product failure remains
technically valid. An infrastructure failure invalidates the run and keeps the
observations already recorded.
Every score and every audit flag is deterministic. No scenario calls a second model to judge the first.
Requirements
shell_coder_sandbox, chess_engine_build, and trend_blog prepare their
reviewed fixture from an embedded Git bundle. They need Git. They do not need
a fixture checkout or HARNESS_E2E_FIXTURE_PATH. Each attempt uses a private
workspace, and temporary source checkouts are removed after their contents are
read.
chess_engine_build also needs a Compose daemon, a running iii Console, and an
interactive browser. It grades a run-scoped chess Worker through live functions,
the Console injectable-UI manifest, and the playable page at
#/worker/<worker>/chess, then preserves screenshots of the real Console with
the functional evidence. Setup installs the pinned iii SDK with npm before the
run; a cold npm cache needs Registry access.
typescript_chat_service carries its own frozen skeleton. It needs Node 22.6
or newer on the runner host: the subject's TypeScript application runs through
Node type stripping, both in the public suite and in the runner-owned
behavioral probe.
The trending-topics build needs Linux amd64, Docker, Git, Python 3, Node, and access to its pinned fixture. See runtime and controls.
The engineering handoff uses a protected disposable checkout of a pinned
revision of iii-hq/e2e-fixture.
Develop
From the repository root:
pnpm --dir dashboard install --frozen-lockfile
pnpm --dir dashboard typecheck
pnpm --dir dashboard lint
pnpm --dir dashboard test
pnpm --dir dashboard build
cargo test --locked --all-targets
cargo clippy --locked --all-targets -- -D warnings
node --test tests/dashboard/*.test.cjs
HARNESS_E2E_BIN="$PWD/target/debug/harness-e2e" python3 -m unittest discover -s tests/python -p 'test_*.py'The Rust build writes dashboard/dist-console/page.js and styles.css, then
embeds both in the worker. Node and pnpm must be on PATH.
List scenarios and their definition digests:
cargo run --locked --bin harness-e2e -- list
cargo run --locked --bin harness-e2e -- catalogCommands
| Command | What it does |
|---|---|
(none) / worker |
Run the Compose-managed control-plane service. |
list |
Print every scenario id. |
catalog |
Print the materialized scenario catalog. |
run |
Execute one or more scenarios against a running stack. |
report |
Summarize a saved results.json. |
models |
List models registered in the running stack. |
test-plan list |
List suites and their coverage. |
test-plan materialize |
Expand one suite into campaigns, groups, and cases. |
--manifest |
Print the Registry worker manifest as JSON. |
Run one scenario against an existing stack:
cargo run --locked --bin harness-e2e -- run \
--url ws://127.0.0.1:49134 \
--model codex/gpt-5.6-luna \
--provider openai-codex \
--scenario todo_worker_simpleScenarios
Every scenario is a built-in module under src/scenarios/. The module owns
its prompt, setup, deterministic evaluator, and cleanup. Its id is the same
in the CLI, the worker catalog, the campaign runner, the dashboard, and the
canonical result artifacts. Plans name the scenarios they run.
Featured suites:
- Registry:
registry_planning,registry_implementation,registry_environment, andregistry_verification. Each has its own atomic validations. - Linkly tutorial:
linkly_tutorialruns the seven tutorial chapters plus a project-restart guard as one scripted dialogue on one Harness session, against thelinkly-agenticscaffold's Compose stack, and scores twenty-two deterministic checks.scripts/linkly_stack.pyprepares that stack. - Trending topics: an isolated per-attempt Git remote and an independent Playwright acceptance check against the delivered SHA. Design is free. Screenshots are evidence, not an aesthetic score.
config/test-plan.json groups the catalog into modules (state, context,
wakes, coordination, software engineering, integration, security, adaptive
operations, continuous engineering, and the incremental Kanban application)
plus a diagnostic set that suites can opt into.
Test plans
config/test-plan.json is the executable plan. Rust materializes its suites and execution rules from that source and from the native scenario contracts. There is no generated catalog to keep in sync.
| Suite | Purpose |
|---|---|
regression |
Daily runtime, recovery, context and safety checks; one technical retry where safe. |
software-engineering |
Kanban, Registry delivery, the trending-topics blog, Linkly, Alertmanager route migration and a playable chess Worker. |
pr |
Four essential checks of a candidate stack before merging a change. |
after-release |
Five essential checks of the published stack. |
A suite is only what to test: scenarios, runs of each and technical retries. In the Console these suites are read-only; Suites copies any of them into a suite of the Console to edit, and Run tests runs a suite with a model on the current Harness. See suites.
cargo run --locked -- test-plan list
cargo run --locked -- test-plan materialize --suite software-engineeringThe software-engineering suite runs each selected case once, with no
technical retries: the seven Kanban cases, Registry implementation and verification,
the trending-topics build, the Linkly tutorial, Alertmanager route migration,
and the playable chess Worker in the Console. That is 13 cases and 13 planned runs,
in 12 execution groups. Registry
implementation and verification share a group, in that order, so verification
receives the implementation delivery. Trending topics runs in
case-trending-topics-build. Linkly runs its eight exchanges in
case-linkly-tutorial, using a fresh pinned linkly-agentic scaffold as its
Compose project. Baseline worker versions come from the resolved stack contract.
Alertmanager runs in its own group, needs Go 1.25+, and has no turn or token
ceiling.
Release Control
Operational campaigns run through
.github/workflows/exact-stack-e2e.yml.
This repository does not publish independent daily, weekly, post-deploy, or
fault-stress dispatch workflows.
A dispatch names what to test, where, and with whom:
| Input | Meaning |
|---|---|
suite |
A suite id from config/test-plan.json, or one suite as JSON. |
stack |
A stack in stacks/ by name (default default), or stack YAML. |
model |
provider/model the subject runs. |
profile |
Optional Directory agent profile the subject runs as. |
execution_id |
Optional Release Control execution. Without it the run only produces artifacts. |
A stack is an iii Compose project plus iii (a release, or latest for the
newest iii/v* release candidate, as Release Control resolves it) and an
optional template
(<id> or <id>@<revision> of iii-hq/templates). Credentials never go in a
stack: the executor stamps the namespace, the runner's data directory, the
model's provider, what the suite needs and the private env file per group.
Scripts always come from the dispatched ref.
Preparation resolves the rest, once:
scripts/prepare_execution.py dispatchreads the inputs intoexecution.json,stack.yamlandplan.json;runtimeresolvesiiiand the template.runnerresolves the stack's ownharness-e2ewithiii compose build, pins that release in the stack and fetches it. That binary materializes the suite (suite.json, also kept asprofile.json), so the suite always comes from the runner every group runs, and the finalizer aggregates with it. The runner's identity in the reports is its revision.contractswrites one contract per campaign.- The stack is assembled once with
compose::add, which expands every declared worker into its graph and writesworker-compose.lock; it gets the groups' provider credentials and one retry. The model's provider and, for an agent profile, the Directory come from Harness's graph with its pins; only one no graph brings is asked for on its own.lockputs that project and lock into every contract. Each group starts it withcompose::upfrozen, so every group runs the same versions. A template project is assembled per group, pinned to the versions that lock resolved.
The contract artifact carries the suite snapshot, the final stack.yaml, the
lock, the model, the profile and the iii release.
Campaign manifests never select or rotate seeds. They keep replay-safe turns separate from scripted dialogue and composite flows, persist a summary for every group, and are advisory while their longitudinal history is being calibrated. Release Control owns scheduling. The executor keeps the result advisory and archives each materialized group through the environment-owned durable archiver.
Partial GitHub reruns reuse the contract artifact produced by prepare, kept
for 90 days. The finalizer matches group artifacts to the completed jobs'
execution times and keeps successful groups from earlier attempts. A rerun
that produces no artifact stays missing evidence; an older artifact does not
replace it.
scripts/report_execution.py posts observations to Release Control's run
ledger over OIDC: materialized before anything runs, one shard per
campaign group, and a summary from the finalizer. Runs come from
results.json, or from journal checkpoints when a group died before writing
one. A group that produced neither still reports that fact.
workers supplies versioned stack components. It does not orchestrate
campaigns.
Executor image
Every phase runs in one image of tools, ghcr.io/iii-hq/harness-e2e:tools-<first 12 hex of the Dockerfile's sha256> (Dockerfile): git, curl, jq,
gh, Python 3 with pip and PyYAML, Node 24 with pnpm, Go 1.25, Rust 1.98.1,
Playwright's Chromium at /usr/bin/chromium, the Docker CLI with buildx and
compose, and the Docker Engine (dockerd, containerd, runc, iptables), much
of what the ubuntu-latest runner gave the groups before. Bases, the Ubuntu
archive snapshot and every download are pinned, so one Dockerfile is one set
of tools. It holds no scripts:
scripts/run_in_image.sh <phase> mounts the
checkout at the same path, runs as the caller's uid with
no-new-privileges, passes the phase's environment through by name, and
runs scripts/executor.sh prepare [materialize|assemble|fixtures], group, package or finalize [restore|aggregate] there. prepare fixtures checks out what the groups
start from and no package brings (the Kanban fixture, the Linkly templates,
the stack's template, the Registry sources and the trending topics fixture)
below target/, three tries each, and group routes the fixture repositories
a scenario clones to those checkouts; finalize lays each campaign's groups
out from the group bundles the execution selected (linked, not copied) before
aggregating them. Only prepare gets a GITHUB_TOKEN: a group's subject has
a shell. Interrupted, the wrapper stops its container; a group whose image or
container never started still writes its failure.json.
No phase mounts the host's Docker socket or joins the host's network. Each
container has a network of its own on Docker's default bridge, where a group's
engine listens on 49134 without taking a host port. A group container runs
a Docker daemon of its own: privileged, in a cgroup namespace of its own, it
starts as root with
an anonymous volume, labelled like the container, at /var/lib/docker;
executor.sh group starts dockerd there, runs the group as the caller's uid
(whose group owns the daemon's socket) and stops the daemon after it, which
stops its containers. Every container a scenario starts (Registry's runner,
Kanban's, trending topics') is that daemon's, in the group's network, so the
application the Registry fixture publishes on 127.0.0.1 is where its
screenshots look, and it goes with the group's container and its volume
(docker rm -fv for one left behind). Each group starts with no image and
pulls what its scenarios run.
This keeps groups from colliding, not from the host. The group's user
reaches root in its container through the daemon's socket, and the container
is privileged, so that root is root-equivalent on the host, as the host's
socket was: a subject can start a privileged container with the host's
devices, mount the host's disk and read what is there (the host daemon's
container configurations, with a concurrent prepare's GITHUB_TOKEN, a
worker's provider_env_file, gh's credentials). From the default bridge it
also reaches the host's ports on the bridge's gateway (a local Console on
3113, iii on 49134) and other groups' containers.
The workflow keeps on the runner what needs it: the GitHub App token for the
private fixture sources, artifacts, the OIDC reports, gh, packaging, and
removing a cancelled phase's container before anything is reported or
packaged.
executor-image.yml publishes a tag
from main when the Dockerfile changes (or by hand) and never rebuilds an
existing one. A tag that is not published yet is built where it runs, with a
warning. resolution.json records the image the execution was prepared in,
the registry digest or the tag when it was built locally, and the reports
carry it as identity.executor_image.
Agent profile and project template
A dispatch can name an existing Directory agent profile (profile), such as
console-ui. console-ui is Console UI Engineer. The runner reads that profile from
the group's Directory and sends its id as agent to e2e::run. The profile,
its parents, skills, functions, and model or provider must exist in that
stack. The runner downloads the stack's versioned skill bundles into an
isolated Directory and waits up to 120 seconds. A profile that is still
missing fails resolution. A profile model overrides the plan model;
provider::model also selects its provider. Results record the resolved
subject model and the profile configuration hash. Run again resolves the
same profile id in its test stack. Comparisons stay manual in Release Control.
Omitting profile keeps the built-in agent.
A project template belongs to the stack and is independent of the suite and the agent profile:
template: harness # or harness@<revision>The runner reads iii/template.yaml in iii-hq/templates at that revision
(main when none is given), resolves it to one commit before sharding, and
records that commit as identity.template.revision. Every group uses that
source. The selected Compose project supplies the base. The versions the
execution's stack lock resolved override its package selectors, extra packages
enter the lock, and local workers stay local.
Template skills replace whole downloaded namespaces, and its agent files take
precedence over downloaded profiles. Machine-global profiles and skills are
unused when a template or agent override is selected. The runner also enables
the campaign's provider when the project does not already have it.
Scenarios, prompts, permissions, fixtures, seeds, and repetitions stay the same. The evaluated agent applies to ordinary sessions and to workflow or adaptive steps. Evaluators stay the same. Linkly keeps its pinned task scaffold and container roles; the selected template's base and agent assets are applied separately. With no template, the run keeps the existing generated stack (or the required fixture).
To measure a profile, compare the same suite, template commit, model, and stack
with and without profile. Changing the template as well measures the
combined effect. Non-Compose templates and templates that ask for an
interactive language choice are rejected before boot.
Console
Build the worker and its page from the repository root:
cargo build --locked --bin harness-e2eWhen Console connects to the same iii namespace, the worker registers the page
assets and the e2e::dashboard::* functions for read, suites, stacks, run,
status, and cancellation. The page lives under #/ext/harness-e2e and exposes
Tests, Executions, Suites, and Stacks. See dashboard/README.md.
The running Harness must publish request and response schemas compatible with the current typed surface. Missing or incompatible fields fail preflight. There is no payload-version compatibility mode.
The page loads through iii: 25 compact summaries on the first overview page, one complete report when an execution is opened, only the selected pair for comparison, and the model and scenario catalog when the run dialog opens. Filtering and cursor pagination run on the server. Transport failures stay visible. The trusted publisher still writes the bounded JSON report archive used by CI. It does not publish a standalone Harness E2E web application.
Import executions from GitHub
On Executions, Import from GitHub lists the completed runs of the
exact-stack-e2e.yml workflow in github_repository (worker config, default
iii-hq/harness-e2e) with their suite, model, agent profile, stack, iii
release, date and conclusion. The worker calls the gh CLI, so sign it in once with
gh auth login; its errors are shown as they come.
Import answers at once with an execution in the importing state; the
worker downloads the run's highest-attempt bundle into its data directory and
installs every group's native run as an ordinary retained run. The execution
records its parameters with its suite (name and snapshot digest), the stack as
its contract recorded it (the final stack.yaml and its digest, which Run
again offers as recorded), the workers each group resolved and observed, and
its GitHub origin. Contracts from before the workflow stated its execution (plan and
profile only) import too. A group that left only failure.json is kept as a slot with
that error. Importing a run again replaces the runs of the earlier import; the
execution keeps its name. Imported and local executions are the same record:
lists, reports, evidence and renaming treat them alike.
Run in Docker
Run tests with Where: Docker runs the execution as the exact-stack
workflow does, from this worker, with no checkout of this repository: the
worker embeds the Dockerfile (whose digest names the executor
image) and scripts/, and runs every phase through
scripts/run_in_image.sh from docker-executions/<execution id>/ under its
data directory:
inputs.json: the dispatch (DISPATCH_*): a master-plan suite run as it is by its id, any other suite whole as JSON, the stack's YAML, the model and the agent profile.checkout/: what the wrapper mounts, with the Dockerfile and the scripts, frozen for the execution, andtarget/, every phase's work;target/artifacts/keeps the bundles under the workflow's artifact names (e2e-contract-<id>-gh-1,e2e-observation-<id>-<campaign>-<group>-gh-<n>,e2e-observation-<id>-gh-<n>).logs/: each phase's output.
prepare materialize, assemble and fixtures run once, then one group
container per group, docker_parallel_groups (2) at a time across executions,
each packaged, then finalize, whose root bundle is imported by the code that
imports a GitHub run, from the folder. The root links its groups' bundles
rather than copying them, and once an attempt is finalized the roots before it
go, before it is imported. Cancel stops the running groups' containers (never
a package or finalize), then finalizes and imports what finished. Running
a scenario again runs its groups in new containers as the execution's next
attempt, with the same contract, lock, scripts and image, finalizes and
imports again: a group's last attempt that ended counts, and one cancelled or
cut by a restart leaves the attempt before it counted. A worker that restarts
removes an active Docker execution's containers, interrupts what did not
finish and imports what did. A worker older than this release cannot read a
Docker execution and drops it from its database, as it drops any row it cannot
read.
Every group runs on a network of its own with a Docker daemon of its own (see Executor image): it publishes no port on this host, so its stack never collides with this machine's iii (its Console binds 3113) or another group's. It can still reach them, and root on this host, as that section says.
Worker configuration:
provider_env_file: an env file with the provider credentials the GitHub groups receive (DEEPSEEK_API_KEY,ZAI_API_KEY,TYPESAFE_API_KEY), passed toprepare assembleand every group with--env-file; never logged or copied into the execution's folder. Without one, the execution says its providers start without credentials.scripts_dir: a checkout'sscripts/to run instead of the embedded ones, copied into each new execution, so an edited script takes effect on the next.docker_parallel_groups: groups at once, 2 by default.docker_registry_mirrors: a pull-through cache per registry for the Docker daemon each group runs, which starts with no image, as{mcr.microsoft.com: http://172.17.0.1:5001}: the daemon pulls from the mirror first and from the registry when the mirror fails. It reaches the groups asHARNESS_E2E_REGISTRY_MIRRORS(REGISTRY=URL ...), whichscripts/run_in_image.sh groupalso takes from its caller.
Kanban's and trending topics' images come from mcr.microsoft.com, which
dockerd --registry-mirror does not cover (it mirrors Docker Hub alone); a
group's daemon writes each mirror into /etc/docker/certs.d/<registry>/hosts.toml
instead. One registry:2 per upstream serves as the cache, listening on the
Docker bridge's gateway (docker network inspect bridge --format '{{(index .IPAM.Config 0).Gateway}}'), where every group container reaches this host:
docker run -d --name mcr-cache --restart unless-stopped -p 172.17.0.1:5001:5000 \
-v mcr-cache:/var/lib/registry -e REGISTRY_PROXY_REMOTEURL=https://mcr.microsoft.com registry:2
docker run -d --name hub-cache --restart unless-stopped -p 172.17.0.1:5000:5000 \
-v hub-cache:/var/lib/registry -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2The phases get none of the worker's environment but where Docker and
TMPDIR are.
prepare fixtures gets the worker's GITHUB_TOKEN, or the signed-in gh's,
for the private Registry and trending topics sources.
Worker
Release Control names the exact project roots. This repository writes only the
root configuration and passes worker@version references to compose::add.
iii resolves the Registry graph, writes the project topology, and reconciles
containers. Every execution starts an empty Engine and a dedicated Compose
daemon, then runs compose::add, compose::up, compose::status, and
compose::down. Each execution uses one isolated namespace for Compose and
for the project functions it starts.
Compose supplies III_URL, III_NAMESPACE, III_WORKER_NAME, and the
configuration: III_CONFIG, a file, before iii 0.24.3; III_CONFIG_NAME, the
configuration-service entry read with configuration::get, from 0.24.3 on.
The first three and one of those two are mandatory. The configuration holds the
execution-specific evidence directory and the separate control-plane database
namespace. Start worker-compose.control.yaml before worker-compose.yaml.
The control file provisions the single-connection harness_e2e SQLite pool
and disables SQL history. The Harness exits when that database or its schema
is unavailable. The subject namespace never receives the database client or
its filesystem path.
Publication validates the locally built binary through a path:// Compose
container before upload. Published campaigns use only exact Registry package
versions. Provider secrets go to temporary permission-restricted env_file
files and never appear in contract, Compose, evidence, or archive artifacts.
The worker exposes:
e2e::run,e2e::status,e2e::cancele2e::results-get,e2e::results-list,e2e::comparee2e::scenarios-liste2e::archive,e2e::archive-head,e2e::archive-restoree2e::history-list,e2e::retention-sweep
Subject policies deny e2e::*.
Live worker check
tests/live_worker.rs talks to a worker registered in a real engine, so those
tests are ignored by default. They check that the worker publishes this
revision's catalog (ids, seeds, cases, definition digests), that it refuses
malformed requests, and, when a subject is named, that one minimal_path
execution passes admission, setup, the subject turn, capture, evaluation, and
persistence. The resulting report is readable through e2e::results-get:
HARNESS_E2E_LIVE_URL=ws://127.0.0.1:49134 HARNESS_E2E_LIVE_NAMESPACE=my-project \
HARNESS_E2E_LIVE_PROVIDER=deepseek HARNESS_E2E_LIVE_MODEL=deepseek-flash \
cargo test --test live_worker -- --ignoredSeven scenarios state host paths in their prompts, so their definition digests
match only when the test process has the worker's HARNESS_E2E_RUN_DIR,
TMPDIR, and HARNESS_E2E_*_FIXTURE_PATH values. Export the same environment
the Compose file gives the worker. The scenario run leaves one execution
labelled live worker validation in the worker's storage.
Storage
Durable artifacts are chunked through storage::*. Admissions, executions,
runs, attempts, and artifact references are written through the control-plane
database::* worker. Execution records keep compact dashboard summaries and
observations, so lists and history do not load native reports.
Storage has no version number and no migration step. Every table records the
fingerprint of the statements that create it. At start, the worker recreates
tables whose fingerprint moved, in one transaction. It keeps the execution
records, local suites and stacks, and receipts it can still read, and rebuilds
run projections from the native bundles. Rows it cannot read, missing bundles,
and tables this binary no longer writes (such as the retired history_* import
tables and the saved_plans of the retired baseline/candidate plans) are
dropped and logged as warnings. Nothing is reconstructed as a scored result. A
report written under another results contract is read with a warning.
The Console's suites live in local_suites, its stacks in local_stacks, and
executions (run here or imported from GitHub) in saved_plan_executions. A
suite, stack or execution this binary cannot read is deleted on the next read.
An execution a retired plan ran stays, without a suite.
Native bundles keep full reports, manifests, and transcripts, loaded on demand. The runner has no S3, GCS, R2, SQL-driver, or Harness dependency.
Weekly Stress materializes deterministic fault plans and evaluates journals
from a protected supervisor. Lane promotion is governed by
config/policies/cutover.json.
Observation
The runner waits for a session tree to finish by binding
harness::turn-completed to an internal sink, e2e::on-turn-completed, before
harness::send. That sink is not a control-plane verb: it is not registered
with e2e::run, e2e::status, or e2e::cancel, and it does not appear in
e2e::scenarios-list. Subject policies already deny e2e::*.
A 15-second watchdog samples harness::metrics and one root
harness::status for stuck detection, heartbeat logs, and e2e::cancel. If
the trigger type is missing from engine::triggers::list, the run is
unsupported infrastructure. There is no fallback that polls harness::status
or harness::metrics. After the tree completes, the runner still collects
terminal status, metrics, transcripts, and deliverables.
Comparison
Every completed execution records the subject and E2E revisions, observed wire
contracts, definition digest, materialized inputs, seed, policies, artifacts,
and raw structural evidence. e2e::compare takes two distinct completed
execution ids (from_execution_id and to_execution_id) and writes
comparisons/<comparison-id>/e2e-delta.json plus e2e-summary.md. Numeric
deltas stay disabled when the case set or the canonical contract differs.
Deliverable, structural, technical, cost, latency, turn, and retry deltas stay independent. A case is repeatable after five local runs meet the deliverable, structural, and technical thresholds. Cost and wall time are observed metrics and are compared only inside a compatible baseline and candidate cohort.
Deterministic assessment has one payload shape, written only to results.json.
Scenario contracts are the only versioned domain. Before cleanup, asset
capture applies explicit safety limits and writes an unversioned sidecar with
the canonical deterministic validation portion, which is aggregated into
results.json.
Repository
| Path | Owns |
|---|---|
src/ |
Runner, wire adapters, scenarios, evaluation, longitudinal comparison, and the E2E control worker. |
config/ |
Comparison and cutover policies, fault profiles, and the master test plan. |
stacks/ |
Stacks a campaign runs on: iii Compose projects plus the iii release. |
tests/ |
Test-only fixtures, golden wire schemas, and the Node and Python suites. |
schemas/ |
Public contracts for generated E2E artifacts. |
dashboard/ |
React, TypeScript, Vite, and Tailwind Console page embedded in the worker. |
Generated reports, transcripts, logs, and deliverables stay outside Git.
The crate may depend on the iii SDK and on generic libraries. It must not
declare a path or Git dependency on workers, Harness, or another product
crate. Contract compatibility is established at runtime from
engine::functions::list and engine::functions::info. The checked-in schemas
are parity fixtures, not a linked product API.
Package boundary
This repository executes exact-stack test plans and publishes immutable
harness-e2e Registry releases from main. A dispatch names a suite, a stack
and a model; the stack is assembled and locked once, and the contract carries
that lock, so a campaign can state afterwards exactly what it ran.
Releases
cut-release.yml is dispatched by hand with bump set to patch, minor or
major. It writes the next version into Cargo.toml and Cargo.lock, commits
that on main and pushes the tag harness-e2e/v<version>. The tag runs
release.yml: it checks that the tag, the manifest and the commit agree, builds
the dashboard once and the binaries for x86_64-unknown-linux-gnu and
aarch64-apple-darwin, creates the GitHub release, collects the typed
interface in an isolated engine, publishes the Registry candidate to next
and promotes it to latest with a compare-and-swap on the previous latest.
The Registry job runs in the workers-registry-next environment; required
reviewers on that environment turn it into a manual approval. Versions are
plain semver; the -experimental suffix ended with 0.11.19.
The root iii.worker.yaml is the public manifest for local iii worker
development and package compatibility. The root worker-compose.yaml is a
normal public Compose document. Release Control and post-prepare workflow
phases read neither source contract.
API reference (JSON)
agent-api-reference.json
{
"functions": [
{
"description": "Persist a completed execution through private iii storage and idempotent history.",
"metadata": {
"contract": {
"capabilities": [
"archive"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
}
},
"properties": {
"execution_id": {
"type": "string"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"execution_id",
"retention_class"
],
"title": "ArchiveExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"baseline_comparable": {
"type": "boolean"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"result_contract_sha256": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"baseline_comparable",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"result_contract_sha256",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"duplicate_ingestion": {
"type": "boolean"
},
"history": {
"$ref": "#/definitions/HistoryRecord"
}
},
"required": [
"archive",
"duplicate_ingestion",
"history"
],
"title": "ArchiveResponse",
"type": "object"
}
},
{
"description": "Verify availability and immutable metadata for an archived execution.",
"metadata": {
"contract": {
"capabilities": [
"archive-head"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-head",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"verified_at": {
"type": "string"
}
},
"required": [
"archive",
"availability",
"verified_at"
],
"title": "ArchiveHeadResponse",
"type": "object"
}
},
{
"description": "Restore and verify an archived execution from its immutable manifest.",
"metadata": {
"contract": {
"capabilities": [
"archive-restore"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::archive-restore",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArchiveAvailability": {
"enum": [
"available",
"expired"
],
"type": "string"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"availability": {
"$ref": "#/definitions/ArchiveAvailability"
},
"restored_files": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"restored_from_backup": {
"type": "boolean"
},
"restored_root": {
"type": [
"string",
"null"
]
}
},
"required": [
"archive",
"availability",
"restored_files",
"restored_from_backup"
],
"title": "ArchiveRestoreResponse",
"type": "object"
}
},
{
"description": "Request idempotent cancellation and cleanup of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"cancel"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"accepted": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
}
},
"required": [
"accepted",
"execution_id",
"phase"
],
"title": "CancelResponse",
"type": "object"
}
},
{
"description": "Compare two completed executions when their identities are eligible.",
"metadata": {
"contract": {
"capabilities": [
"compare"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::compare",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ComparisonPolicy": {
"properties": {
"baseline_id": {
"description": "Identifier of the reviewed baseline these thresholds came from, when one was loaded. Absent when the code-default thresholds are in effect.",
"type": [
"string",
"null"
]
},
"regression": {
"allOf": [
{
"$ref": "#/definitions/RegressionThresholds"
}
],
"default": {
"cost_increase_ratio": 0.2,
"deliverable_success_drop": 0.03,
"p95_turns_increase_ratio": 0.25,
"structural_integrity_drop": 0.02,
"technical_failure_increase": 0.02,
"wall_time_increase_ratio": 0.2
}
},
"scenario_overrides": {
"additionalProperties": {
"$ref": "#/definitions/ThresholdOverrides"
},
"description": "Per-scenario partial threshold overrides carried from the reviewed baseline.",
"type": "object"
}
},
"type": "object"
},
"RegressionThresholds": {
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": "number"
},
"deliverable_success_drop": {
"format": "double",
"type": "number"
},
"p95_turns_increase_ratio": {
"format": "double",
"type": "number"
},
"structural_integrity_drop": {
"format": "double",
"type": "number"
},
"technical_failure_increase": {
"format": "double",
"type": "number"
},
"wall_time_increase_ratio": {
"format": "double",
"type": "number"
}
},
"required": [
"cost_increase_ratio",
"deliverable_success_drop",
"p95_turns_increase_ratio",
"structural_integrity_drop",
"technical_failure_increase",
"wall_time_increase_ratio"
],
"type": "object"
},
"ThresholdOverrides": {
"additionalProperties": false,
"description": "Partial per-scenario replacement for [`RegressionThresholds`]: a named field replaces the baseline value, an omitted field inherits it.",
"properties": {
"cost_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity_drop": {
"format": "double",
"type": [
"number",
"null"
]
},
"technical_failure_increase": {
"format": "double",
"type": [
"number",
"null"
]
},
"wall_time_increase_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
}
},
"properties": {
"baseline": {
"default": null,
"description": "Reviewed baseline file to load thresholds from. Omit to use the checked-in `config/baselines/default.json` when it exists, falling back to the code-default thresholds. Ignored when `policy` is set.",
"type": [
"string",
"null"
]
},
"from_execution_id": {
"type": "string"
},
"policy": {
"anyOf": [
{
"$ref": "#/definitions/ComparisonPolicy"
},
{
"type": "null"
}
],
"default": null,
"description": "Explicit policy override. Omit to gate on the reviewed baseline."
},
"to_execution_id": {
"type": "string"
}
},
"required": [
"from_execution_id",
"to_execution_id"
],
"title": "CompareRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"BenchmarkDelta": {
"properties": {
"consumption": {
"additionalProperties": {
"$ref": "#/definitions/DeltaValue"
},
"default": {},
"description": "Descriptive only; these deltas never create a regression gate.",
"type": "object"
},
"deliverable_success_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"flaky_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"mean_score": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p50_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_cost_usd": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_turns": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"p95_wall_time_ms": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"retry_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"structural_integrity_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"technical_failure_rate": {
"anyOf": [
{
"$ref": "#/definitions/DeltaValue"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"type": "object"
},
"CaseCohortKey": {
"properties": {
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"execution": {
"$ref": "#/definitions/ExecutionCohortIdentity"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"behavior_sha256",
"case_id",
"contract_sha256",
"execution",
"inputs_sha256",
"scenario_id",
"seed"
],
"type": "object"
},
"CaseComparison": {
"properties": {
"delta": {
"$ref": "#/definitions/BenchmarkDelta"
},
"from": {
"$ref": "#/definitions/CaseMetrics"
},
"from_run_ids": {
"items": {
"type": "string"
},
"type": "array"
},
"key": {
"$ref": "#/definitions/CaseCohortKey"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to": {
"$ref": "#/definitions/CaseMetrics"
},
"to_run_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"delta",
"from",
"from_run_ids",
"key",
"regressions",
"to",
"to_run_ids"
],
"type": "object"
},
"CaseMetrics": {
"properties": {
"consumption": {
"allOf": [
{
"$ref": "#/definitions/ConsumptionMetrics"
}
],
"default": {
"completed_runs": 0,
"observed_runs": 0,
"p50_function_calls": null,
"p50_total_tokens": null,
"p95_function_calls": null,
"p95_total_tokens": null,
"tokens_per_completion": null,
"total_tokens_consumed": null,
"unavailable": {}
},
"description": "Includes every observed run, even those excluded from quality comparison."
},
"deliverable_success": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"excluded_infrastructure_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"included_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"retry_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"structural_integrity": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"technical_failure": {
"anyOf": [
{
"$ref": "#/definitions/RateEstimate"
},
{
"type": "null"
}
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"excluded_infrastructure_runs",
"included_runs"
],
"type": "object"
},
"CohortAudit": {
"properties": {
"excluded_cases": {
"items": {
"$ref": "#/definitions/ExcludedCase"
},
"type": "array"
},
"excluded_runs": {
"items": {
"$ref": "#/definitions/ExcludedRun"
},
"type": "array"
},
"included_case_ids": {
"items": {
"type": "string"
},
"type": "array"
}
},
"required": [
"excluded_cases",
"excluded_runs",
"included_case_ids"
],
"type": "object"
},
"ComparisonSide": {
"enum": [
"from",
"to"
],
"type": "string"
},
"ComparisonSummary": {
"properties": {
"baseline_id": {
"description": "Reviewed baseline the regression thresholds came from; absent when the code-default thresholds gated this comparison.",
"type": [
"string",
"null"
]
},
"cases": {
"items": {
"$ref": "#/definitions/CaseComparison"
},
"type": "array"
},
"cohort": {
"$ref": "#/definitions/CohortAudit"
},
"comparable": {
"type": "boolean"
},
"comparison_id": {
"type": "string"
},
"from_execution_id": {
"type": "string"
},
"from_revision": {
"type": [
"string",
"null"
]
},
"gate_passed": {
"type": "boolean"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"regressions": {
"items": {
"$ref": "#/definitions/RegressionSignal"
},
"type": "array"
},
"to_execution_id": {
"type": "string"
},
"to_revision": {
"type": [
"string",
"null"
]
}
},
"required": [
"cases",
"cohort",
"comparable",
"comparison_id",
"from_execution_id",
"gate_passed",
"reasons",
"regressions",
"to_execution_id"
],
"type": "object"
},
"ConsumptionMetrics": {
"description": "Subject-side usage across all observed attempts.",
"properties": {
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"p50_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"p50_total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"p95_total_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens_per_completion": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_tokens_consumed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"completed_runs",
"observed_runs",
"unavailable"
],
"type": "object"
},
"DeltaValue": {
"properties": {
"absolute": {
"format": "double",
"type": "number"
},
"relative_ratio": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"absolute"
],
"type": "object"
},
"ExcludedCase": {
"properties": {
"case_id": {
"type": "string"
},
"reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"case_id",
"reasons",
"scenario_id",
"side"
],
"type": "object"
},
"ExcludedRun": {
"properties": {
"attempt_id": {
"type": "string"
},
"case_id": {
"type": "string"
},
"reason": {
"type": "string"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"side": {
"$ref": "#/definitions/ComparisonSide"
}
},
"required": [
"attempt_id",
"case_id",
"reason",
"run_id",
"scenario_id",
"side"
],
"type": "object"
},
"ExecutionCohortIdentity": {
"properties": {
"e2e_repository": {
"type": [
"string",
"null"
]
},
"lane": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"lane",
"stack_mode",
"subject_model",
"subject_provider"
],
"type": "object"
},
"RateEstimate": {
"properties": {
"ci95_lower": {
"format": "double",
"type": "number"
},
"ci95_upper": {
"format": "double",
"type": "number"
},
"rate": {
"format": "double",
"type": "number"
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"successes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"ci95_lower",
"ci95_upper",
"rate",
"sample_size",
"successes"
],
"type": "object"
},
"RegressionDimension": {
"enum": [
"deliverable",
"structural_integrity",
"technical_reliability",
"cost",
"wall_time",
"turns"
],
"type": "string"
},
"RegressionSignal": {
"properties": {
"blocking": {
"type": "boolean"
},
"case_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/RegressionDimension"
},
"maturity": {
"default": "",
"type": "string"
},
"n_from": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"n_to": {
"default": 0,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"observed_delta": {
"format": "double",
"type": "number"
},
"reason": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"statistic": {
"default": "",
"type": "string"
},
"threshold": {
"format": "double",
"type": "number"
}
},
"required": [
"blocking",
"case_id",
"dimension",
"observed_delta",
"reason",
"scenario_id",
"threshold"
],
"type": "object"
}
},
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"comparison": {
"$ref": "#/definitions/ComparisonSummary"
}
},
"required": [
"artifacts",
"comparison"
],
"title": "ComparisonResponse",
"type": "object"
}
},
{
"description": "Read the persisted projection for one logical-run attempt.",
"metadata": {
"contract": {
"capability": "attempt-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::attempt-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"attempt_id": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"run_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"execution_id",
"run_id"
],
"title": "AttemptGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CompletionState": {
"enum": [
"completed",
"task_incomplete",
"undetermined"
],
"type": "string"
},
"ContaminationScope": {
"enum": [
"none",
"case",
"execution"
],
"type": "string"
},
"CriterionReport": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
},
"gate": {
"default": false,
"description": "A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell \"no gate\" from a report older than the field.",
"type": "boolean"
},
"id": {
"type": "string"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"possible",
"reason"
],
"type": "object"
},
"DimensionReport": {
"properties": {
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"passed": {
"type": [
"boolean",
"null"
]
},
"signals": true
},
"required": [
"dimension",
"signals"
],
"type": "object"
},
"EfficiencyReport": {
"properties": {
"child_sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"child_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"critical_path_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"effective_fan_out": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"observed_complexity": {
"$ref": "#/definitions/ObservedComplexityReport"
},
"observed_work": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"root_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_attempts": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"transient_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_retries": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"observed_complexity",
"technical_attempts",
"wall_time_ms"
],
"type": "object"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"FailureDomain": {
"enum": [
"subject",
"resource",
"e2e_infrastructure"
],
"type": "string"
},
"FailurePhase": {
"enum": [
"setup",
"execute",
"collect",
"evaluate",
"cleanup"
],
"type": "string"
},
"FailureRecord": {
"properties": {
"code": {
"default": "",
"type": "string"
},
"contamination": {
"allOf": [
{
"$ref": "#/definitions/ContaminationScope"
}
],
"default": "none"
},
"domain": {
"$ref": "#/definitions/FailureDomain"
},
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/FailurePhase"
},
"retry_scope": {
"allOf": [
{
"$ref": "#/definitions/RetryScope"
}
],
"default": "none"
},
"status": {
"allOf": [
{
"$ref": "#/definitions/RunStatus"
}
],
"description": "The failure's original classification, independent of the primary run status."
}
},
"required": [
"domain",
"message",
"phase",
"status"
],
"type": "object"
},
"ObservedComplexityReport": {
"properties": {
"ambiguity_level": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"artifact_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"coordination_edges": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"dependency_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"external_systems": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parallel_branches": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"planning_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state_transitions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_loops": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_cycles": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"RetryScope": {
"enum": [
"none",
"same_slot"
],
"type": "string"
},
"RunStatus": {
"enum": [
"passed",
"hard_gate_failed",
"subject_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"TechnicalState": {
"enum": [
"valid",
"technical_invalid"
],
"type": "string"
}
},
"properties": {
"completion": {
"$ref": "#/definitions/CompletionState"
},
"criteria": {
"items": {
"$ref": "#/definitions/CriterionReport"
},
"type": [
"array",
"null"
]
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"technical": {
"$ref": "#/definitions/TechnicalState"
}
},
"required": [
"completion",
"dimensions",
"failures",
"status",
"technical"
],
"title": "AttemptResponse",
"type": "object"
}
},
{
"description": "Read models and scenarios when the execution dialog opens.",
"metadata": {
"contract": {
"capability": "catalog-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::catalog-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"url": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "CatalogRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CatalogModel": {
"properties": {
"id": {
"type": "string"
},
"provider": {
"type": "string"
}
},
"required": [
"id",
"provider"
],
"type": "object"
}
},
"properties": {
"models": {
"items": {
"$ref": "#/definitions/CatalogModel"
},
"type": "array"
},
"scenario_groups": {
"description": "Scenarios that run only together, in this order: picking one runs all.",
"items": {
"items": {
"type": "string"
},
"type": "array"
},
"type": "array"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"url": {
"type": "string"
}
},
"required": [
"models",
"scenario_groups",
"scenarios",
"url"
],
"title": "CatalogResponse",
"type": "object"
}
},
{
"description": "List immutable evaluated-system versions and their exact evaluation cohorts.",
"metadata": {
"contract": {
"capability": "evaluated-versions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::evaluated-versions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "EvaluatedVersionsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"CohortDescriptor": {
"properties": {
"id": {
"type": "string"
},
"lane": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
}
},
"required": [
"id",
"lane",
"subject_model",
"subject_provider"
],
"type": "object"
},
"EvaluatedVersionDescriptor": {
"properties": {
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"stack_mode": {
"type": "string"
}
},
"required": [
"cohort_id",
"completed_at",
"execution_count",
"id",
"label",
"stack_mode"
],
"type": "object"
}
},
"properties": {
"cohorts": {
"items": {
"$ref": "#/definitions/CohortDescriptor"
},
"type": "array"
},
"revision": {
"type": "string"
},
"versions": {
"items": {
"$ref": "#/definitions/EvaluatedVersionDescriptor"
},
"type": "array"
}
},
"required": [
"cohorts",
"revision",
"versions"
],
"title": "EvaluatedVersionsResponse",
"type": "object"
}
},
{
"description": "Read one evidence file or deliverable screenshot a retained run's report declares, as base64 (10 MB at most).",
"metadata": {
"contract": {
"capability": "evidence-read",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::evidence-read",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"description": "The native execution whose report declares the file.",
"type": "string"
},
"path": {
"description": "A path the report declares for one of its runs (evidence or a deliverable).",
"type": "string"
},
"pointer": {
"default": null,
"description": "A declared screenshot's JSON Pointer inside that deliverable; only the image is returned.",
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"path"
],
"title": "EvidenceReadRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "One evidence file, or one screenshot inside a deliverable, as base64.",
"properties": {
"base64": {
"type": "string"
},
"media_type": {
"type": "string"
}
},
"required": [
"base64",
"media_type"
],
"title": "EvidenceFile",
"type": "object"
}
},
{
"description": "Stop an execution: no next scenario is admitted and the running one is cancelled.",
"metadata": {
"contract": {
"capability": "execution-cancel",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Delete one finished native run, or one finished execution together with its native runs.",
"metadata": {
"contract": {
"capability": "execution-delete",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-delete",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Read one retained execution summary and its diagnostic report.",
"metadata": {
"contract": {
"capability": "execution-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionListResponse": {
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"type": "object"
},
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"detail": true,
"manifest": {
"$ref": "#/definitions/ExecutionListResponse"
}
},
"required": [
"detail",
"manifest"
],
"title": "ExecutionBundle",
"type": "object"
}
},
{
"description": "Name one execution; an empty label restores its default name.",
"metadata": {
"contract": {
"capability": "execution-rename",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-rename",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
},
"label": {
"description": "Empty restores the default name.",
"type": "string"
}
},
"required": [
"execution_id",
"label"
],
"title": "ExecutionRenameRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Run one scenario of a finished local execution again on this stack; the last attempt counts and the one it replaces stays visible outside every total. Answers with its id and runs in the background.",
"metadata": {
"contract": {
"capability": "execution-slot-rerun",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-slot-rerun",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
},
"scenario_id": {
"description": "The scenario to run again; a sequential group runs again whole.",
"type": "string"
}
},
"required": [
"execution_id",
"scenario_id"
],
"title": "ExecutionSlotRerunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Start an execution on this stack from its parameters; answers with its id and runs in the background.",
"metadata": {
"contract": {
"capability": "execution-start",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::execution-start",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionParameters": {
"description": "What an execution ran, to run it again. Always the canonical cases: a `seed` sent or stored by an older Console is ignored, so every execution pairs with any other by scenario and repetition.",
"properties": {
"agent": {
"description": "Agent profile the subject ran under.",
"type": [
"string",
"null"
]
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"stack": {
"anyOf": [
{
"$ref": "#/definitions/ExecutionStack"
},
{
"type": "null"
}
],
"description": "The stack it ran on in Docker or on GitHub; this harness runs on its own. Running it again offers it as recorded."
},
"suite": {
"anyOf": [
{
"$ref": "#/definitions/ExecutionSuite"
},
{
"type": "null"
}
],
"description": "The suite it ran; absent for an execution from before suites."
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"where": {
"allOf": [
{
"$ref": "#/definitions/Where"
}
],
"default": "harness",
"description": "Where it ran; absent for an execution from before Docker, which ran on this harness."
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries"
],
"type": "object"
},
"ExecutionStack": {
"description": "The stack an execution ran on, to run it again as recorded.",
"properties": {
"name": {
"description": "A repository stack's id, a local one's name, or what a contract names.",
"type": "string"
},
"sha256": {
"default": "",
"description": "`sha256:` of `yaml`; the worker sets it, a request never does.",
"type": "string"
},
"yaml": {
"description": "The YAML as sent; once imported, the final `stack.yaml` its contract recorded (the iii release and the runner pinned).",
"type": "string"
}
},
"required": [
"name",
"yaml"
],
"type": "object"
},
"ExecutionSuite": {
"description": "The suite an execution ran: named when it was picked as it is (a suite of the master plan or of this Console), unnamed when its scenarios were ticked by hand, with the digest of the snapshot it materialized to.",
"properties": {
"id": {
"description": "Absent for an unnamed suite.",
"type": [
"string",
"null"
]
},
"label": {
"default": "",
"type": "string"
},
"sha256": {
"default": "",
"description": "`profile_sha256` of the snapshot; the runner sets it, a request never does.",
"type": "string"
}
},
"type": "object"
},
"Where": {
"description": "Where an execution runs: on this harness, in Docker from this worker, or on GitHub (imported runs; starting one there is not offered yet).",
"enum": [
"harness",
"docker",
"github"
],
"type": "string"
}
},
"properties": {
"label": {
"default": "",
"description": "Empty or absent uses the default name.",
"type": "string"
},
"parameters": {
"allOf": [
{
"$ref": "#/definitions/ExecutionParameters"
}
],
"description": "What to run; a previous execution's parameters run it again."
}
},
"required": [
"parameters"
],
"title": "ExecutionStartRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "List compact retained execution summaries with server-side filters and cursor pagination.",
"metadata": {
"contract": {
"capability": "executions-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::executions-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"event": {
"default": null,
"type": [
"string",
"null"
]
},
"ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"ids_csv": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"status": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "ExecutionListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Retention": {
"properties": {
"details": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"summaries": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"details",
"summaries"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": true,
"type": "array"
},
"last_update": {
"type": "string"
},
"mode": {
"type": "string"
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"repo_url": {
"type": "string"
},
"retention": {
"$ref": "#/definitions/Retention"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"executions",
"last_update",
"mode",
"repo_url",
"retention",
"total"
],
"title": "ExecutionListResponse",
"type": "object"
}
},
{
"description": "Read the suite, subject, profile and runner of listed GitHub runs from their contract artifacts.",
"metadata": {
"contract": {
"capability": "github-run-contracts",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::github-run-contracts",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"GithubRunAttempt": {
"properties": {
"run_attempt": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"run_id": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"run_attempt",
"run_id"
],
"type": "object"
}
},
"properties": {
"repository": {
"default": null,
"type": [
"string",
"null"
]
},
"runs": {
"description": "The runs of one listed page whose contract is not read yet.",
"items": {
"$ref": "#/definitions/GithubRunAttempt"
},
"type": "array"
}
},
"required": [
"runs"
],
"title": "GithubRunContractsRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Import one exact-stack workflow run from GitHub as an execution; answers at once and imports in the background.",
"metadata": {
"contract": {
"capability": "github-run-import",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::github-run-import",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"repository": {
"default": null,
"type": [
"string",
"null"
]
},
"run_id": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"run_id"
],
"title": "GithubRunImportRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "List completed exact-stack workflow runs on GitHub with their local import; suites and subjects already read come along.",
"metadata": {
"contract": {
"capability": "github-runs-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::github-runs-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"page": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"repository": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "GithubRunsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Cancel the active local E2E execution.",
"metadata": {
"contract": {
"capability": "run-cancel",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::run-cancel",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"Defaults": {
"properties": {
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"technical_retries",
"url"
],
"type": "object"
},
"JobStatus": {
"enum": [
"running",
"cancelling",
"cancelled",
"completed",
"failed"
],
"type": "string"
},
"JobView": {
"properties": {
"completed_at": {
"type": "string"
},
"error": {
"type": "string"
},
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"log": {
"type": "string"
},
"log_from": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_offset": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"log_truncated": {
"type": "boolean"
},
"request": {
"$ref": "#/definitions/RunRequest"
},
"returncode": {
"format": "int32",
"type": [
"integer",
"null"
]
},
"started_at": {
"type": "string"
},
"status": {
"$ref": "#/definitions/JobStatus"
}
},
"required": [
"completed_at",
"error",
"id",
"label",
"log",
"log_from",
"log_offset",
"log_truncated",
"request",
"started_at",
"status"
],
"type": "object"
},
"RunRequest": {
"additionalProperties": false,
"properties": {
"label": {
"default": "",
"type": "string"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"url": {
"type": "string"
}
},
"required": [
"model",
"provider",
"runs",
"scenarios",
"technical_retries",
"url"
],
"type": "object"
}
},
"properties": {
"defaults": {
"$ref": "#/definitions/Defaults"
},
"job": {
"anyOf": [
{
"$ref": "#/definitions/JobView"
},
{
"type": "null"
}
]
}
},
"required": [
"defaults"
],
"title": "RunSnapshot",
"type": "object"
}
},
{
"description": "Create a stack of this Console as a copy of another stack.",
"metadata": {
"contract": {
"capability": "stack-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::stack-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"from": {
"description": "The stack it starts as a copy of: one of the repository or of this Console.",
"type": "string"
},
"label": {
"default": "",
"description": "Empty or absent names it after that stack.",
"type": "string"
}
},
"required": [
"from"
],
"title": "StackCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"StackContainer": {
"description": "A container a stack declares, with the version or commit it pins.",
"properties": {
"commit": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"version": {
"type": [
"string",
"null"
]
}
},
"required": [
"name"
],
"type": "object"
}
},
"description": "A stack as the Console lists it.",
"properties": {
"containers": {
"items": {
"$ref": "#/definitions/StackContainer"
},
"type": "array"
},
"id": {
"type": "string"
},
"iii": {
"description": "The iii CLI release it installs.",
"type": [
"string",
"null"
]
},
"label": {
"type": "string"
},
"source": {
"description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.",
"type": "string"
},
"template": {
"description": "The iii-hq/templates project it starts from.",
"type": [
"string",
"null"
]
},
"updated_at": {
"type": [
"string",
"null"
]
},
"warnings": {
"description": "What the executor or Compose may not do as written; never blocking.",
"items": {
"type": "string"
},
"type": "array"
},
"yaml": {
"type": "string"
}
},
"required": [
"containers",
"id",
"label",
"source",
"warnings",
"yaml"
],
"title": "StackView",
"type": "object"
}
},
{
"description": "Delete a stack of this Console.",
"metadata": {
"contract": {
"capability": "stack-delete",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::stack-delete",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"stack_id": {
"type": "string"
}
},
"required": [
"stack_id"
],
"title": "StackDeleteRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Change the name or the YAML of a stack of this Console; refused only when the YAML does not parse or declares no containers.",
"metadata": {
"contract": {
"capability": "stack-update",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::stack-update",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"label": {
"default": null,
"type": [
"string",
"null"
]
},
"stack_id": {
"type": "string"
},
"yaml": {
"default": null,
"description": "The whole stack as YAML, kept exactly as written.",
"type": [
"string",
"null"
]
}
},
"required": [
"stack_id"
],
"title": "StackUpdateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"StackContainer": {
"description": "A container a stack declares, with the version or commit it pins.",
"properties": {
"commit": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"version": {
"type": [
"string",
"null"
]
}
},
"required": [
"name"
],
"type": "object"
}
},
"description": "A stack as the Console lists it.",
"properties": {
"containers": {
"items": {
"$ref": "#/definitions/StackContainer"
},
"type": "array"
},
"id": {
"type": "string"
},
"iii": {
"description": "The iii CLI release it installs.",
"type": [
"string",
"null"
]
},
"label": {
"type": "string"
},
"source": {
"description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.",
"type": "string"
},
"template": {
"description": "The iii-hq/templates project it starts from.",
"type": [
"string",
"null"
]
},
"updated_at": {
"type": [
"string",
"null"
]
},
"warnings": {
"description": "What the executor or Compose may not do as written; never blocking.",
"items": {
"type": "string"
},
"type": "array"
},
"yaml": {
"type": "string"
}
},
"required": [
"containers",
"id",
"label",
"source",
"warnings",
"yaml"
],
"title": "StackView",
"type": "object"
}
},
{
"description": "List the stacks an execution can run on: the repository's (read-only) and this Console's, each with its YAML, what it declares and its warnings.",
"metadata": {
"contract": {
"capability": "stacks-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::stacks-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"StackContainer": {
"description": "A container a stack declares, with the version or commit it pins.",
"properties": {
"commit": {
"type": [
"string",
"null"
]
},
"name": {
"type": "string"
},
"version": {
"type": [
"string",
"null"
]
}
},
"required": [
"name"
],
"type": "object"
},
"StackView": {
"description": "A stack as the Console lists it.",
"properties": {
"containers": {
"items": {
"$ref": "#/definitions/StackContainer"
},
"type": "array"
},
"id": {
"type": "string"
},
"iii": {
"description": "The iii CLI release it installs.",
"type": [
"string",
"null"
]
},
"label": {
"type": "string"
},
"source": {
"description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.",
"type": "string"
},
"template": {
"description": "The iii-hq/templates project it starts from.",
"type": [
"string",
"null"
]
},
"updated_at": {
"type": [
"string",
"null"
]
},
"warnings": {
"description": "What the executor or Compose may not do as written; never blocking.",
"items": {
"type": "string"
},
"type": "array"
},
"yaml": {
"type": "string"
}
},
"required": [
"containers",
"id",
"label",
"source",
"warnings",
"yaml"
],
"type": "object"
}
},
"properties": {
"stacks": {
"description": "The repository's stacks, then this Console's.",
"items": {
"$ref": "#/definitions/StackView"
},
"type": "array"
}
},
"required": [
"stacks"
],
"title": "StacksListResponse",
"type": "object"
}
},
{
"description": "Create a suite of this Console as a copy of another suite.",
"metadata": {
"contract": {
"capability": "suite-create",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::suite-create",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"from": {
"description": "The suite it starts as a copy of: one of the master plan or of this Console.",
"type": "string"
},
"label": {
"default": "",
"description": "Empty or absent names it after that suite.",
"type": "string"
}
},
"required": [
"from"
],
"title": "SuiteCreateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "A suite as the Console lists it.",
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"purpose": {
"type": "string"
},
"repetitions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"sha256": {
"description": "Digest of the snapshot this runner materializes it to; what an execution of it records.",
"type": [
"string",
"null"
]
},
"source": {
"description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.",
"type": "string"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": [
"string",
"null"
]
}
},
"required": [
"id",
"label",
"purpose",
"repetitions",
"scenarios",
"source",
"technical_retries"
],
"title": "SuiteView",
"type": "object"
}
},
{
"description": "Delete a suite of this Console.",
"metadata": {
"contract": {
"capability": "suite-delete",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::suite-delete",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"suite_id": {
"type": "string"
}
},
"required": [
"suite_id"
],
"title": "SuiteDeleteRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": true,
"title": "Map_of_AnyValue",
"type": "object"
}
},
{
"description": "Change the name, scenarios, runs or technical retries of a suite of this Console.",
"metadata": {
"contract": {
"capability": "suite-update",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::suite-update",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"label": {
"default": null,
"type": [
"string",
"null"
]
},
"repetitions": {
"default": null,
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"scenarios": {
"default": null,
"items": {
"type": "string"
},
"type": [
"array",
"null"
]
},
"suite_id": {
"type": "string"
},
"technical_retries": {
"default": null,
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"suite_id"
],
"title": "SuiteUpdateRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "A suite as the Console lists it.",
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"purpose": {
"type": "string"
},
"repetitions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"sha256": {
"description": "Digest of the snapshot this runner materializes it to; what an execution of it records.",
"type": [
"string",
"null"
]
},
"source": {
"description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.",
"type": "string"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": [
"string",
"null"
]
}
},
"required": [
"id",
"label",
"purpose",
"repetitions",
"scenarios",
"source",
"technical_retries"
],
"title": "SuiteView",
"type": "object"
}
},
{
"description": "List the suites an execution can run: the master plan's (read-only) and this Console's.",
"metadata": {
"contract": {
"capability": "suites-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::suites-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "DashboardEmptyRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"SuiteView": {
"description": "A suite as the Console lists it.",
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"purpose": {
"type": "string"
},
"repetitions": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"items": {
"type": "string"
},
"type": "array"
},
"sha256": {
"description": "Digest of the snapshot this runner materializes it to; what an execution of it records.",
"type": [
"string",
"null"
]
},
"source": {
"description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.",
"type": "string"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"updated_at": {
"type": [
"string",
"null"
]
}
},
"required": [
"id",
"label",
"purpose",
"repetitions",
"scenarios",
"source",
"technical_retries"
],
"type": "object"
}
},
"properties": {
"suites": {
"description": "The master plan's suites, then this Console's.",
"items": {
"$ref": "#/definitions/SuiteView"
},
"type": "array"
}
},
"required": [
"suites"
],
"title": "SuitesListResponse",
"type": "object"
}
},
{
"description": "Read local metric history for one test version, with provider-grouped execution models, without comparison actions.",
"metadata": {
"contract": {
"capability": "test-history-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-history-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"case_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"result": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_model": {
"default": null,
"type": [
"string",
"null"
]
},
"subject_provider": {
"default": null,
"type": [
"string",
"null"
]
},
"system_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"test_id": {
"default": "",
"type": "string"
},
"test_version": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TestHistoryRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"HistoryModelGroup": {
"description": "Models are exposed as provider groups so a model name is never ambiguous when two providers offer the same model identifier.",
"properties": {
"models": {
"items": {
"type": "string"
},
"type": "array"
},
"provider": {
"type": "string"
}
},
"required": [
"models",
"provider"
],
"type": "object"
},
"HistorySeries": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"id": {
"type": "string"
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"behavior_sha256",
"case_id",
"cohort_id",
"contract_sha256",
"execution_count",
"id",
"run_count",
"stack_mode",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"HistorySystem": {
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
}
},
"required": [
"id",
"label"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"behavior_sha256",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"type": "string"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"cases": {
"items": {
"type": "string"
},
"type": "array"
},
"current_version": {
"description": "The current definition's digest, which may have no executions yet.",
"type": [
"string",
"null"
]
},
"next_cursor": {
"type": [
"string",
"null"
]
},
"observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"series": {
"items": {
"$ref": "#/definitions/HistorySeries"
},
"type": "array"
},
"subject_models": {
"items": {
"$ref": "#/definitions/HistoryModelGroup"
},
"type": "array"
},
"subjects": {
"items": {
"type": "string"
},
"type": "array"
},
"systems": {
"items": {
"$ref": "#/definitions/HistorySystem"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"description": "The definition whose executions are shown (the latest with evidence by default).",
"type": "string"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"available_versions",
"cases",
"observations",
"series",
"subject_models",
"subjects",
"systems",
"test_id",
"test_version",
"total"
],
"title": "TestHistoryResponse",
"type": "object"
}
},
{
"description": "Read one test version across two immutable evaluated-system versions.",
"metadata": {
"contract": {
"capability": "test-version-get",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::test-version-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"type": "string"
},
"from_version_id": {
"type": "string"
},
"test_id": {
"type": "string"
},
"test_version": {
"type": "string"
},
"to_version_id": {
"type": "string"
}
},
"required": [
"cohort_id",
"from_version_id",
"test_id",
"test_version",
"to_version_id"
],
"title": "TestVersionGetRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"behavior_sha256",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
}
},
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"type": "string"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"title": "TestVersionResult",
"type": "object"
}
},
{
"description": "List the versioned test catalog with compact A/B results and stable pagination.",
"metadata": {
"contract": {
"capability": "tests-list",
"name": "harness-e2e-dashboard"
},
"internal": true
},
"name": "e2e::dashboard::tests-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"cohort_id": {
"default": null,
"type": [
"string",
"null"
]
},
"cursor": {
"default": null,
"type": [
"string",
"null"
]
},
"from_version_id": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": null,
"format": "uint16",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"query": {
"default": null,
"type": [
"string",
"null"
]
},
"to_version_id": {
"default": null,
"type": [
"string",
"null"
]
}
},
"title": "TestsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"AssessmentKind": {
"enum": [
"required_check",
"signal",
"asset_validation"
],
"type": "string"
},
"AssessmentOutcomeCounts": {
"properties": {
"error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"partial": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"error",
"failed",
"not_evaluated",
"partial",
"passed",
"unavailable"
],
"type": "object"
},
"AssessmentPolicy": {
"enum": [
"hard_gate",
"advisory"
],
"type": "string"
},
"AssessmentSummary": {
"description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.",
"properties": {
"assessment_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"assessment_outcomes": {
"$ref": "#/definitions/AssessmentOutcomeCounts"
},
"asset_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"asset_validation_outcomes": {
"$ref": "#/definitions/AssetValidationCounts"
},
"evidence_reference_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"system_statuses": {
"$ref": "#/definitions/StatusCounts"
}
},
"required": [
"assessment_count",
"assessment_outcomes",
"asset_count",
"asset_validation_outcomes",
"evidence_reference_count",
"run_count",
"system_statuses"
],
"type": "object"
},
"AssetValidationCounts": {
"properties": {
"invalid": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"malformed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_evaluated": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"not_produced": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"oversized": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"removed_during_cleanup": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unexpected": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unreadable": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unsafe_path": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"valid": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"invalid",
"malformed",
"not_evaluated",
"not_produced",
"oversized",
"removed_during_cleanup",
"unexpected",
"unreadable",
"unsafe_path",
"valid"
],
"type": "object"
},
"CalibrationProjection": {
"properties": {
"compatible_sample_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"maturity": {
"type": "string"
}
},
"required": [
"compatible_sample_count",
"maturity"
],
"type": "object"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"MetricSamples": {
"properties": {
"cost_usd": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"duration_seconds": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"score": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"tokens": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"turns": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"cost_usd",
"duration_seconds",
"score",
"tokens",
"turns"
],
"type": "object"
},
"OutcomeCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infra_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"technical_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infra_failed",
"passed",
"technical_failed"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StatusCounts": {
"properties": {
"hard_gate_failed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"infrastructure_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"passed": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"resource_limit": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"subject_error": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"unavailable": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"hard_gate_failed",
"infrastructure_error",
"passed",
"resource_limit",
"subject_error",
"unavailable"
],
"type": "object"
},
"TestCatalogRow": {
"properties": {
"available_versions": {
"items": {
"$ref": "#/definitions/VersionDescriptor"
},
"type": "array"
},
"calibration": {
"anyOf": [
{
"$ref": "#/definitions/CalibrationProjection"
},
{
"type": "null"
}
]
},
"characterization": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
},
{
"type": "null"
}
]
},
"current_version": {
"type": [
"string",
"null"
]
},
"lifecycle": {
"type": "string"
},
"result": {
"anyOf": [
{
"$ref": "#/definitions/TestVersionResult"
},
{
"type": "null"
}
]
},
"selected_version": {
"type": [
"string",
"null"
]
},
"spec": {
"anyOf": [
{
"$ref": "#/definitions/TestSpecProjection"
},
{
"type": "null"
}
]
},
"test_id": {
"type": "string"
}
},
"required": [
"available_versions",
"lifecycle",
"test_id"
],
"type": "object"
},
"TestCriterionProjection": {
"properties": {
"description": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"id": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/AssessmentKind"
},
"policy": {
"$ref": "#/definitions/AssessmentPolicy"
},
"weight": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"description",
"dimension",
"id",
"kind",
"policy",
"weight"
],
"type": "object"
},
"TestDelta": {
"properties": {
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"TestObservation": {
"properties": {
"assessment_profile_sha256": {
"type": "string"
},
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"cohort_id": {
"type": "string"
},
"completed_at": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"evaluated_version_id": {
"type": [
"string",
"null"
]
},
"execution_id": {
"type": "string"
},
"harness_revision": {
"type": [
"string",
"null"
]
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_call_errors": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_function_calls": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_turns": {
"format": "double",
"type": [
"number",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stack_mode": {
"type": "string"
},
"status": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_label": {
"type": "string"
},
"system_revision": {
"type": [
"string",
"null"
]
},
"system_version_id": {
"type": [
"string",
"null"
]
}
},
"required": [
"assessment_profile_sha256",
"assessment_summary",
"behavior_sha256",
"case_id",
"cohort_id",
"completed_at",
"contract_sha256",
"execution_id",
"run_count",
"scored_runs",
"stack_mode",
"status",
"subject_model",
"subject_provider",
"system_label"
],
"type": "object"
},
"TestSideSummary": {
"properties": {
"assessment_summary": {
"$ref": "#/definitions/AssessmentSummary"
},
"case_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"evaluated_version_id": {
"type": "string"
},
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"mean_score": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_duration_seconds": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_tokens": {
"format": "double",
"type": [
"number",
"null"
]
},
"outcomes": {
"$ref": "#/definitions/OutcomeCounts"
},
"pass_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"samples": {
"$ref": "#/definitions/MetricSamples"
},
"scored_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"total_runs": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"assessment_summary",
"case_count",
"evaluated_version_id",
"execution_count",
"outcomes",
"samples",
"scored_runs",
"total_runs"
],
"type": "object"
},
"TestSpecProjection": {
"description": "The scenario definition as a reader needs it: what the subject is asked to do, how the result is scored, and the limits it runs under. Projected from the materialized `ScenarioSpec` of the test's current version, so it always describes the contract the dashboard is showing.",
"properties": {
"criteria": {
"items": {
"$ref": "#/definitions/TestCriterionProjection"
},
"type": "array"
},
"denied_functions": {
"items": {
"type": "string"
},
"type": "array"
},
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"prompt": {
"description": "The prompt as materialized for a representative run: run-scoped paths resolve against a catalog namespace, never a real execution.",
"type": "string"
},
"summary": {
"description": "Editorial description; absent until the scenario defines a `SUMMARY`.",
"type": [
"string",
"null"
]
}
},
"required": [
"criteria",
"denied_functions",
"execution",
"prompt"
],
"type": "object"
},
"TestVersionResult": {
"properties": {
"compatibility": {
"type": "string"
},
"compatibility_reasons": {
"items": {
"type": "string"
},
"type": "array"
},
"delta": {
"$ref": "#/definitions/TestDelta"
},
"from": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"from_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
},
"test_id": {
"type": "string"
},
"test_version": {
"type": "string"
},
"to": {
"anyOf": [
{
"$ref": "#/definitions/TestSideSummary"
},
{
"type": "null"
}
]
},
"to_observations": {
"items": {
"$ref": "#/definitions/TestObservation"
},
"type": "array"
}
},
"required": [
"compatibility",
"compatibility_reasons",
"delta",
"from_observations",
"test_id",
"test_version",
"to_observations"
],
"type": "object"
},
"VersionDescriptor": {
"properties": {
"execution_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"last_seen": {
"type": [
"string",
"null"
]
},
"run_count": {
"format": "uint",
"minimum": 0,
"type": "integer"
},
"version": {
"type": "string"
}
},
"required": [
"execution_count",
"run_count",
"version"
],
"type": "object"
}
},
"properties": {
"next_cursor": {
"type": [
"string",
"null"
]
},
"revision": {
"type": "string"
},
"rows": {
"items": {
"$ref": "#/definitions/TestCatalogRow"
},
"type": "array"
},
"total": {
"format": "uint",
"minimum": 0,
"type": "integer"
}
},
"required": [
"revision",
"rows",
"total"
],
"title": "TestsListResponse",
"type": "object"
}
},
{
"description": "List hash-validated E2E history records from the iii database worker.",
"metadata": {
"contract": {
"capabilities": [
"history-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::history-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"title": "HistoryListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"HistoryRecord": {
"additionalProperties": false,
"properties": {
"archive": {
"$ref": "#/definitions/DurableArchiveReference"
},
"baseline_comparable": {
"type": "boolean"
},
"case_count": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"identity_sha256": {
"type": "string"
},
"ingestion_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"occurred_at": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"result_contract_sha256": {
"type": "string"
},
"stack_mode": {
"type": "string"
},
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"subject_revision": {
"type": "string"
}
},
"required": [
"archive",
"baseline_comparable",
"case_count",
"e2e_repository",
"e2e_revision",
"execution_id",
"identity_sha256",
"ingestion_id",
"lane",
"occurred_at",
"passed",
"result_contract_sha256",
"stack_mode",
"subject_model",
"subject_provider",
"subject_revision"
],
"type": "object"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
}
},
"properties": {
"records": {
"items": {
"$ref": "#/definitions/HistoryRecord"
},
"type": "array"
}
},
"required": [
"records"
],
"title": "HistoryListResponse",
"type": "object"
}
},
{
"description": "Get the immutable result and manifest of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"results-get"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-get",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActivationPolicy": {
"oneOf": [
{
"properties": {
"policy": {
"enum": [
"always"
],
"type": "string"
}
},
"required": [
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"all"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
},
{
"properties": {
"conditions": {
"items": {
"$ref": "#/definitions/BooleanCondition"
},
"type": "array"
},
"policy": {
"enum": [
"any"
],
"type": "string"
}
},
"required": [
"conditions",
"policy"
],
"type": "object"
}
]
},
"AgentProfileArtifact": {
"properties": {
"configuration_sha256": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"configuration_sha256",
"id"
],
"type": "object"
},
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"ArtifactReference": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"path": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"id",
"kind",
"media_type",
"path",
"sha256",
"size_bytes"
],
"type": "object"
},
"AssessmentContract": {
"additionalProperties": false,
"properties": {
"runs": {
"items": {
"$ref": "#/definitions/RunAssessmentContract"
},
"type": "array"
}
},
"required": [
"runs"
],
"type": "object"
},
"AssessmentKind": {
"enum": [
"required_check",
"signal",
"asset_validation"
],
"type": "string"
},
"AssessmentOutcome": {
"enum": [
"passed",
"failed",
"partial",
"not_evaluated",
"unavailable",
"error"
],
"type": "string"
},
"AssessmentPolicy": {
"enum": [
"hard_gate",
"advisory"
],
"type": "string"
},
"AssessmentResult": {
"properties": {
"criterion_id": {
"type": "string"
},
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AssessmentKind"
},
"outcome": {
"$ref": "#/definitions/AssessmentOutcome"
},
"policy": {
"$ref": "#/definitions/AssessmentPolicy"
},
"score": {
"anyOf": [
{
"$ref": "#/definitions/AssessmentScore"
},
{
"type": "null"
}
]
},
"summary": {
"type": "string"
},
"target": {
"$ref": "#/definitions/AssessmentTarget"
}
},
"required": [
"criterion_id",
"dimension",
"kind",
"outcome",
"policy",
"summary",
"target"
],
"type": "object"
},
"AssessmentScore": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"awarded",
"possible"
],
"type": "object"
},
"AssessmentTarget": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"$ref": "#/definitions/AssessmentTargetKind"
}
},
"required": [
"id",
"kind"
],
"type": "object"
},
"AssessmentTargetKind": {
"enum": [
"criterion"
],
"type": "string"
},
"AssetValidationOutcome": {
"enum": [
"valid",
"invalid",
"malformed",
"oversized",
"not_produced",
"unreadable",
"unsafe_path",
"removed_during_cleanup",
"unexpected",
"not_evaluated"
],
"type": "string"
},
"AssetValidationResult": {
"properties": {
"asset_id": {
"type": "string"
},
"evidence": {
"items": {
"$ref": "#/definitions/EvidenceReference"
},
"type": "array"
},
"outcome": {
"$ref": "#/definitions/AssetValidationOutcome"
},
"summary": {
"type": "string"
}
},
"required": [
"asset_id",
"outcome",
"summary"
],
"type": "object"
},
"AuditEvidence": {
"properties": {
"detail": {
"description": "Sanitized, bounded description. Never contains secret material.",
"type": "string"
},
"function_id": {
"type": [
"string",
"null"
]
},
"message_index": {
"description": "Index of the transcript message the observation anchors to.",
"format": "uint",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"detail"
],
"type": "object"
},
"AuditFlag": {
"properties": {
"confidence": {
"format": "double",
"type": [
"number",
"null"
]
},
"evidence": {
"items": {
"$ref": "#/definitions/AuditEvidence"
},
"type": "array"
},
"kind": {
"$ref": "#/definitions/AuditFlagKind"
},
"severity": {
"$ref": "#/definitions/AuditSeverity"
},
"source": {
"$ref": "#/definitions/AuditFlagSource"
},
"summary": {
"type": "string"
}
},
"required": [
"kind",
"severity",
"source",
"summary"
],
"type": "object"
},
"AuditFlagKind": {
"oneOf": [
{
"description": "The subject invoked the E2E control plane or another function the scenario explicitly denies — the verification mechanism itself.",
"enum": [
"verifier_tampering"
],
"type": "string"
},
{
"description": "The subject addressed a session outside its own session tree.",
"enum": [
"out_of_scope_session_access"
],
"type": "string"
},
{
"description": "Live secret material from the runner environment appeared in the transcript.",
"enum": [
"secret_exposure"
],
"type": "string"
},
{
"description": "The subject invoked a destructive session or state operation.",
"enum": [
"destructive_action"
],
"type": "string"
}
]
},
"AuditFlagSource": {
"enum": [
"deterministic"
],
"type": "string"
},
"AuditReport": {
"properties": {
"flags": {
"items": {
"$ref": "#/definitions/AuditFlag"
},
"type": "array"
}
},
"required": [
"flags"
],
"type": "object"
},
"AuditSeverity": {
"enum": [
"info",
"warning",
"critical"
],
"type": "string"
},
"BooleanCondition": {
"additionalProperties": false,
"properties": {
"equals": {
"type": "boolean"
},
"node_id": {
"type": "string"
},
"port": {
"type": "string"
}
},
"required": [
"equals",
"node_id",
"port"
],
"type": "object"
},
"Capability": {
"description": "What a scenario needs from the engine, the runner host or a fixture to run. The closed list keeps the catalog honest: a new need is a new variant, and every id is spelled once.",
"enum": [
"browser::interactive",
"cross_repo_contract_simulator::v1",
"curl",
"docker",
"e2e::adaptive-flow-v1",
"e2e::control-plane-v1",
"e2e::filesystem",
"e2e::git",
"e2e::run-scoped-fixtures",
"e2e::shell",
"e2e::subagents",
"e2e::workflow-resume-v1",
"fixture::multi-origin-http",
"git",
"git::deterministic-fixture-v1",
"git::offline-bundle",
"github::trusted-handoff",
"go",
"harness::independent_session",
"harness::post-turn-validation",
"harness::scripted-dialogue-v1",
"iii::coder",
"iii::compose",
"iii::database",
"iii::functions",
"iii::registry",
"iii::shell",
"iii::state",
"iii::triggers",
"iii::workers",
"incident_fixture::v1",
"node",
"playwright",
"python3",
"release_shadow::read-only-v1",
"release_train_simulator::v1"
],
"type": "string"
},
"CapturedInvariant": {
"properties": {
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"CompletionState": {
"enum": [
"completed",
"task_incomplete",
"undetermined"
],
"type": "string"
},
"ContaminationScope": {
"enum": [
"none",
"case",
"execution"
],
"type": "string"
},
"ControlPlaneEvidence": {
"properties": {
"functions": {
"items": {
"$ref": "#/definitions/FunctionContractEvidence"
},
"type": "array"
}
},
"required": [
"functions"
],
"type": "object"
},
"CostReport": {
"properties": {
"subject_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_usd": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"type": "object"
},
"CriterionReport": {
"properties": {
"awarded": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
},
"gate": {
"default": false,
"description": "A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell \"no gate\" from a report older than the field.",
"type": "boolean"
},
"id": {
"type": "string"
},
"possible": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"possible",
"reason"
],
"type": "object"
},
"DeliverableContentFormat": {
"enum": [
"json",
"text_utf8"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"DeliverableReport": {
"properties": {
"artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"content_format": {
"$ref": "#/definitions/DeliverableContentFormat"
},
"content_sha256": {
"type": "string"
},
"content_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"id": {
"type": "string"
},
"invariants": {
"items": {
"$ref": "#/definitions/CapturedInvariant"
},
"type": "array"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"preview": true,
"provenance": {
"items": {
"$ref": "#/definitions/ProvenanceEvidence"
},
"type": "array"
},
"provenance_valid": {
"type": "boolean"
},
"schema_valid": {
"type": "boolean"
},
"screenshots": {
"default": [],
"description": "The screenshots embedded in the content, so a reader can list them without opening the artifact. Empty when there are none.",
"items": {
"$ref": "#/definitions/ScreenshotReference"
},
"type": "array"
}
},
"required": [
"content_format",
"content_sha256",
"content_size_bytes",
"id",
"invariants",
"kind",
"media_type",
"preview",
"provenance",
"provenance_valid",
"schema_valid"
],
"type": "object"
},
"DependencyPolicy": {
"enum": [
"succeeded",
"terminal"
],
"type": "string"
},
"DimensionReport": {
"properties": {
"dimension": {
"$ref": "#/definitions/EvaluationDimension"
},
"passed": {
"type": [
"boolean",
"null"
]
},
"signals": true
},
"required": [
"dimension",
"signals"
],
"type": "object"
},
"DurableArchiveReference": {
"additionalProperties": false,
"properties": {
"archive_id": {
"type": "string"
},
"created_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"expires_at": {
"type": [
"string",
"null"
]
},
"identity_sha256": {
"type": "string"
},
"manifest": {
"$ref": "#/definitions/StorageObjectReference"
},
"manifest_backup": {
"$ref": "#/definitions/StorageObjectReference"
},
"retention_class": {
"$ref": "#/definitions/RetentionClass"
}
},
"required": [
"archive_id",
"created_at",
"execution_id",
"identity_sha256",
"manifest",
"manifest_backup",
"retention_class"
],
"type": "object"
},
"E2eManifest": {
"additionalProperties": false,
"properties": {
"control_plane": {
"$ref": "#/definitions/ControlPlaneEvidence"
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
},
"worker_contracts": {
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"control_plane",
"execution",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eObservationEnvelope": {
"additionalProperties": false,
"properties": {
"evidence": {
"$ref": "#/definitions/ObservationEvidence"
},
"identity": {
"$ref": "#/definitions/ObservationIdentity"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"outcome": {
"$ref": "#/definitions/ObservationOutcome"
},
"provenance": {
"$ref": "#/definitions/ObservationProvenance"
},
"samples": {
"items": {
"$ref": "#/definitions/ObservationSample"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"evidence",
"identity",
"mode",
"outcome",
"provenance",
"samples",
"schema"
],
"type": "object"
},
"E2eReport": {
"additionalProperties": false,
"properties": {
"assessment_contract": {
"$ref": "#/definitions/AssessmentContract"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"execution": {
"$ref": "#/definitions/ExecutionIdentity"
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"objective_outcome": {
"$ref": "#/definitions/ObjectiveOutcome"
},
"observation_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"passed": {
"type": "boolean"
},
"persistence_errors": {
"items": {
"type": "string"
},
"type": "array"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"report_state": {
"$ref": "#/definitions/ReportState"
},
"result_contract_sha256": {
"type": "string"
},
"scenarios": {
"items": {
"$ref": "#/definitions/E2eScenarioReport"
},
"type": "array"
},
"slot_start_deadline_seconds": {
"description": "Soft scheduling deadline: prevents starting slots after this interval.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"subject": {
"$ref": "#/definitions/ModelArtifact"
},
"system_under_test": {
"$ref": "#/definitions/SystemUnderTestIdentity"
}
},
"required": [
"assessment_contract",
"execution",
"objective_outcome",
"passed",
"report_state",
"result_contract_sha256",
"scenarios",
"subject",
"system_under_test"
],
"type": "object"
},
"E2eRunReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"audit": {
"anyOf": [
{
"$ref": "#/definitions/AuditReport"
},
{
"type": "null"
}
],
"description": "Advisory behavioral audit over the captured transcript and metrics. Never contributes to score, status, gates, or longitudinal inputs."
},
"completion": {
"$ref": "#/definitions/CompletionState"
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"criteria": {
"items": {
"$ref": "#/definitions/CriterionReport"
},
"type": "array"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evaluators": {
"$ref": "#/definitions/EvaluatorStates"
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"prompt": {
"type": "string"
},
"retry_attempts": {
"items": {
"$ref": "#/definitions/RetryAttemptReport"
},
"type": "array"
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
],
"description": "Evidence-only flow identity emitted by Rust. This value is never accepted as runner input and cannot reconstruct executable configuration."
},
"scenario_measurements": {
"items": {
"$ref": "#/definitions/ScenarioMeasurement"
},
"type": "array"
},
"score": {
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"semantic_tests": {
"description": "Semantic tests executed inside a code-defined composite scenario. Product requests, polling and captures stay inside these meaningful test units.",
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"technical": {
"$ref": "#/definitions/TechnicalState"
},
"transcript": true,
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"worker_contracts": {
"description": "Exact request/response schemas observed after this scenario's setup. Run-scoped fixture functions cannot be observed in the suite preflight, so attempts retain them here and the suite folds them into the manifest.",
"items": {
"$ref": "#/definitions/ObservedWorkerContract"
},
"type": "array"
}
},
"required": [
"attempt_id",
"attempt_number",
"completion",
"cost",
"criteria",
"evaluators",
"prompt",
"run_id",
"session_id",
"status",
"technical",
"wall_time_ms"
],
"type": "object"
},
"E2eScenarioReport": {
"properties": {
"aggregate": {
"$ref": "#/definitions/ScenarioAggregate"
},
"behavior_sha256": {
"description": "Digest of the definition the case was materialized from. Absent only when no case could be materialized.",
"type": [
"string",
"null"
]
},
"case": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioCase"
},
{
"type": "null"
}
]
},
"case_id": {
"default": "",
"type": "string"
},
"deferral_reason": {
"type": [
"string",
"null"
]
},
"execution_policy": {
"$ref": "#/definitions/ExecutionPolicy"
},
"passed": {
"type": "boolean"
},
"runs": {
"items": {
"$ref": "#/definitions/E2eRunReport"
},
"type": "array"
},
"scenario_id": {
"type": "string"
}
},
"required": [
"aggregate",
"execution_policy",
"passed",
"runs",
"scenario_id"
],
"type": "object"
},
"EfficiencyReport": {
"properties": {
"child_sessions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"child_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"critical_path_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"effective_fan_out": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"input_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"observed_complexity": {
"$ref": "#/definitions/ObservedComplexityReport"
},
"observed_work": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"root_turns": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_attempts": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"transient_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_retries": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"observed_complexity",
"technical_attempts",
"wall_time_ms"
],
"type": "object"
},
"EvaluationDimension": {
"enum": [
"deliverable",
"structural_integrity",
"efficiency",
"robustness",
"e2e_infrastructure"
],
"type": "string"
},
"EvaluatorAvailability": {
"enum": [
"not_required",
"pending",
"available",
"unavailable"
],
"type": "string"
},
"EvaluatorStates": {
"properties": {
"completion": {
"$ref": "#/definitions/EvaluatorAvailability"
}
},
"required": [
"completion"
],
"type": "object"
},
"EvidenceReference": {
"properties": {
"artifact_id": {
"type": "string"
},
"artifact_sha256": {
"type": "string"
},
"locator": {
"type": [
"string",
"null"
]
}
},
"required": [
"artifact_id",
"artifact_sha256"
],
"type": "object"
},
"ExecutionIdentity": {
"properties": {
"completed_at": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"lane": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"completed_at",
"execution_id",
"lane",
"started_at"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"FailureDomain": {
"enum": [
"subject",
"resource",
"e2e_infrastructure"
],
"type": "string"
},
"FailurePhase": {
"enum": [
"setup",
"execute",
"collect",
"evaluate",
"cleanup"
],
"type": "string"
},
"FailureRecord": {
"properties": {
"code": {
"default": "",
"type": "string"
},
"contamination": {
"allOf": [
{
"$ref": "#/definitions/ContaminationScope"
}
],
"default": "none"
},
"domain": {
"$ref": "#/definitions/FailureDomain"
},
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/FailurePhase"
},
"retry_scope": {
"allOf": [
{
"$ref": "#/definitions/RetryScope"
}
],
"default": "none"
},
"status": {
"allOf": [
{
"$ref": "#/definitions/RunStatus"
}
],
"description": "The failure's original classification, independent of the primary run status."
}
},
"required": [
"domain",
"message",
"phase",
"status"
],
"type": "object"
},
"FunctionContractEvidence": {
"properties": {
"function_id": {
"type": "string"
},
"request_schema": true,
"response_schema": true,
"sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema",
"response_schema",
"sha256"
],
"type": "object"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"ModelArtifact": {
"properties": {
"agent": {
"anyOf": [
{
"$ref": "#/definitions/AgentProfileArtifact"
},
{
"type": "null"
}
]
},
"context_window": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"model": {
"type": "string"
},
"provider": {
"type": "string"
},
"supports_tools": {
"type": [
"boolean",
"null"
]
},
"supports_vision": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"context_window",
"max_output_tokens",
"model",
"provider"
],
"type": "object"
},
"ObjectiveOutcome": {
"enum": [
"passed",
"failed",
"inconclusive"
],
"type": "string"
},
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDataAvailability": {
"enum": [
"complete",
"partial",
"unavailable"
],
"type": "string"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationEvidence": {
"additionalProperties": false,
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"manifest_sha256": {
"type": [
"string",
"null"
]
},
"results_sha256": {
"type": [
"string",
"null"
]
}
},
"type": "object"
},
"ObservationExecutionIdentity": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completed_at": {
"type": "string"
},
"id": {
"type": "string"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": "string"
},
"started_at": {
"type": "string"
}
},
"required": [
"attempt",
"completed_at",
"id",
"request_sha256",
"run_contract_sha256",
"started_at"
],
"type": "object"
},
"ObservationIdentity": {
"additionalProperties": false,
"properties": {
"execution": {
"$ref": "#/definitions/ObservationExecutionIdentity"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"execution",
"plan",
"runner",
"target"
],
"type": "object"
},
"ObservationMetric": {
"additionalProperties": false,
"description": "A metric is explicit about its unit and availability so a recorded zero cannot be mistaken for a missing measurement. The legacy `metrics` object remains alongside this additive projection for current consumers.",
"properties": {
"availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"availability",
"origin",
"unit"
],
"type": "object"
},
"ObservationMetricDerivation": {
"additionalProperties": false,
"properties": {
"formula": {
"type": "string"
},
"formula_version": {
"type": "string"
},
"metric": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
}
},
"required": [
"formula",
"formula_version",
"metric",
"origin"
],
"type": "object"
},
"ObservationMetricOrigin": {
"enum": [
"observed",
"derived_from_observed"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationObjective": {
"enum": [
"passed",
"hard_gate_failed",
"technical_failed",
"infrastructure_failed",
"cancelled",
"unsupported_plan"
],
"type": "string"
},
"ObservationOutcome": {
"additionalProperties": false,
"properties": {
"control_phase": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"error": {
"type": "string"
},
"objective": {
"$ref": "#/definitions/ObservationObjective"
},
"passed": {
"type": [
"boolean",
"null"
]
}
},
"required": [
"control_phase",
"data_availability",
"objective"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationProvenance": {
"additionalProperties": false,
"properties": {
"subject_model": {
"type": "string"
},
"subject_provider": {
"type": "string"
},
"system_under_test": {
"anyOf": [
{
"$ref": "#/definitions/SystemUnderTestIdentity"
},
{
"type": "null"
}
]
}
},
"required": [
"subject_model",
"subject_provider"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSample": {
"additionalProperties": false,
"properties": {
"attempt_id": {
"type": "string"
},
"behavior_sha256": {
"type": [
"string",
"null"
]
},
"case_id": {
"type": "string"
},
"data_availability": {
"$ref": "#/definitions/ObservationDataAvailability"
},
"derivations": {
"items": {
"$ref": "#/definitions/ObservationMetricDerivation"
},
"type": "array"
},
"metric_values": {
"additionalProperties": {
"$ref": "#/definitions/ObservationMetric"
},
"default": {},
"type": "object"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"run_id": {
"type": "string"
},
"scenario_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"status": {
"$ref": "#/definitions/RunStatus"
}
},
"required": [
"attempt_id",
"case_id",
"data_availability",
"origin",
"run_id",
"scenario_id",
"seed",
"status"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioId"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"behavior_sha256",
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"ObservedComplexityReport": {
"properties": {
"ambiguity_level": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"artifact_count": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"coordination_edges": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"dependency_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"external_systems": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parallel_branches": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"planning_depth": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"state_transitions": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"validation_loops": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_cycles": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"ObservedWorkerContract": {
"additionalProperties": false,
"properties": {
"function_id": {
"type": "string"
},
"request_schema_sha256": {
"type": "string"
},
"response_schema_sha256": {
"type": "string"
}
},
"required": [
"function_id",
"request_schema_sha256",
"response_schema_sha256"
],
"type": "object"
},
"PortValueKind": {
"enum": [
"boolean",
"json",
"text_utf8",
"assessment"
],
"type": "string"
},
"ProvenanceEvidence": {
"properties": {
"kind": {
"type": "string"
},
"relation": {
"type": "string"
},
"source_id": {
"type": "string"
}
},
"required": [
"kind",
"relation",
"source_id"
],
"type": "object"
},
"RedactionReport": {
"properties": {
"redacted_fields": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"redacted_values": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"rules": {
"items": {
"type": "string"
},
"type": "array",
"uniqueItems": true
}
},
"required": [
"redacted_fields",
"redacted_values",
"rules"
],
"type": "object"
},
"ReportState": {
"enum": [
"complete",
"partial"
],
"type": "string"
},
"RetentionClass": {
"enum": [
"temporary",
"pull_request",
"longitudinal",
"canonical"
],
"type": "string"
},
"RetryAttemptReport": {
"properties": {
"attempt_id": {
"type": "string"
},
"attempt_number": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completion": {
"$ref": "#/definitions/CompletionState"
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"deliverables": {
"items": {
"$ref": "#/definitions/DeliverableReport"
},
"type": "array"
},
"dimensions": {
"items": {
"$ref": "#/definitions/DimensionReport"
},
"type": "array"
},
"efficiency": {
"anyOf": [
{
"$ref": "#/definitions/EfficiencyReport"
},
{
"type": "null"
}
]
},
"evaluators": {
"$ref": "#/definitions/EvaluatorStates"
},
"evidence": {
"items": {
"$ref": "#/definitions/ArtifactReference"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/FailureRecord"
},
"type": "array"
},
"metrics": {
"anyOf": [
{
"$ref": "#/definitions/SessionMetricsPayload"
},
{
"type": "null"
}
]
},
"run_id": {
"type": "string"
},
"scenario_flow": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioFlowEvidence"
},
{
"type": "null"
}
]
},
"score": {
"description": "The points the attempt's evaluated criteria awarded, as on the run.",
"format": "uint8",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"semantic_tests": {
"items": {
"$ref": "#/definitions/WorkflowStepReport"
},
"type": "array"
},
"session_id": {
"type": "string"
},
"status": {
"$ref": "#/definitions/RunStatus"
},
"technical": {
"$ref": "#/definitions/TechnicalState"
},
"transcript": true,
"wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"attempt_id",
"attempt_number",
"completion",
"cost",
"evaluators",
"failures",
"run_id",
"session_id",
"status",
"technical",
"wall_time_ms"
],
"type": "object"
},
"RetryScope": {
"enum": [
"none",
"same_slot"
],
"type": "string"
},
"RobustnessReport": {
"properties": {
"cost_per_successful_deliverable": {
"format": "double",
"type": [
"number",
"null"
]
},
"deliverable_success_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"eligible": {
"type": "boolean"
},
"flaky_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"median_wall_time_ms": {
"format": "double",
"type": [
"number",
"null"
]
},
"minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"p95_wall_time_ms": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"structural_integrity_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"tail_minimum_sample_size": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failure_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"unavailable": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"wall_time_variance": {
"format": "double",
"type": [
"number",
"null"
]
}
},
"required": [
"eligible",
"minimum_sample_size",
"sample_size",
"tail_minimum_sample_size"
],
"type": "object"
},
"RunAssessmentContract": {
"properties": {
"assessments": {
"items": {
"$ref": "#/definitions/AssessmentResult"
},
"type": "array"
},
"assets": {
"items": {
"$ref": "#/definitions/AssetValidationResult"
},
"type": "array"
},
"attempt_id": {
"type": "string"
},
"run_id": {
"type": "string"
},
"system_status": {
"$ref": "#/definitions/SystemStatus"
}
},
"required": [
"attempt_id",
"run_id",
"system_status"
],
"type": "object"
},
"RunStatus": {
"enum": [
"passed",
"hard_gate_failed",
"subject_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioAggregate": {
"properties": {
"completed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"completion_evidence_coverage": {
"format": "double",
"type": [
"number",
"null"
]
},
"completion_rate": {
"format": "double",
"type": [
"number",
"null"
]
},
"cost": {
"$ref": "#/definitions/CostReport"
},
"deferred_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"execution_reliability": {
"format": "double",
"type": [
"number",
"null"
]
},
"failed_attempt_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"mean_score": {
"description": "Mean score of the technically valid runs that have one; null when none.",
"format": "double",
"type": [
"number",
"null"
]
},
"observed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"pass_rate": {
"format": "double",
"type": "number"
},
"passed_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"planned_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"required_passes": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"robustness": {
"$ref": "#/definitions/RobustnessReport"
},
"scored_runs": {
"description": "Technically valid runs that have a score.",
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"task_incomplete_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_failures": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_invalid_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"technical_valid_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"tokens_completed_p50": {
"format": "double",
"type": [
"number",
"null"
]
},
"tokens_per_completion": {
"format": "double",
"type": [
"number",
"null"
]
},
"total_tokens_consumed": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"undetermined_runs": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"completed_runs",
"cost",
"deferred_runs",
"observed_runs",
"pass_rate",
"passed_runs",
"planned_runs",
"required_passes",
"robustness",
"scored_runs",
"task_incomplete_runs",
"technical_failures",
"technical_invalid_runs",
"technical_valid_runs",
"undetermined_runs"
],
"type": "object"
},
"ScenarioCase": {
"properties": {
"behavior_sha256": {
"description": "Digest of the scenario definition this case was materialized from: prompt, execution policy, criteria, and the case-independent contract. Two cases with the same digest were evaluated by the same definition. Sealed by `ScenarioId::materialize`; empty only before sealing.",
"type": "string"
},
"case_id": {
"type": "string"
},
"characterization": {
"allOf": [
{
"$ref": "#/definitions/ScenarioCharacterization"
}
],
"default": {
"human_horizon": {
"basis": "unknown"
},
"realism": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs": true,
"inputs_sha256": {
"type": "string"
},
"required_capabilities": {
"items": {
"$ref": "#/definitions/Capability"
},
"type": "array"
},
"scenario_id": {
"type": "string"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"behavior_sha256",
"case_id",
"deliverable_contract",
"inputs",
"inputs_sha256",
"required_capabilities",
"scenario_id",
"seed"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioFlowEvidence": {
"additionalProperties": false,
"properties": {
"checkpoint": {
"$ref": "#/definitions/ArtifactReference"
},
"cleanup": {
"$ref": "#/definitions/WorkflowCleanupReport"
},
"definition_sha256": {
"type": "string"
},
"snapshot": true
},
"required": [
"checkpoint",
"cleanup",
"definition_sha256",
"snapshot"
],
"type": "object"
},
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
},
"ScenarioMeasurement": {
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"origin": {
"$ref": "#/definitions/ObservationMetricOrigin"
},
"unit": {
"type": "string"
},
"value": {
"format": "double",
"type": "number"
}
},
"required": [
"id",
"origin",
"unit",
"value"
],
"type": "object"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScreenshotReference": {
"properties": {
"caption": {
"description": "The capture's caption from its `captures.json`, else the file's path without the attempt folder it was packed under.",
"type": "string"
},
"media_type": {
"type": "string"
},
"pointer": {
"description": "JSON Pointer to the embedded file inside the deliverable: `/attachments/board-desktop.png`.",
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"caption",
"media_type",
"pointer",
"sha256",
"size_bytes"
],
"type": "object"
},
"SessionMetricsPayload": {
"properties": {
"by_session": {
"items": {
"$ref": "#/definitions/SessionUsage"
},
"type": "array"
},
"complete": {
"type": "boolean"
},
"root_session_id": {
"type": "string"
},
"totals": {
"$ref": "#/definitions/SessionUsageTotals"
},
"traces": {
"default": null
}
},
"required": [
"by_session",
"complete",
"root_session_id",
"totals"
],
"type": "object"
},
"SessionUsage": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"context": {
"default": null
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"depth": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"parent_session_id": {
"default": null,
"type": [
"string",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"session_id": {
"type": "string"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"depth",
"function_call_errors",
"function_calls",
"session_id",
"turns"
],
"type": "object"
},
"SessionUsageTotals": {
"properties": {
"cache_read_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cache_write_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"cost_usd": {
"default": null,
"format": "double",
"type": [
"number",
"null"
]
},
"function_call_errors": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"function_calls": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"input_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"output_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"reasoning_tokens": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"sessions": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"transient_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"turns": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"validation_retries": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"wake_resumes": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"required": [
"function_call_errors",
"function_calls",
"sessions",
"turns"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
},
"StorageObjectReference": {
"properties": {
"bucket": {
"type": "string"
},
"key": {
"type": "string"
},
"media_type": {
"type": "string"
},
"sha256": {
"type": "string"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"uri": {
"type": "string"
}
},
"required": [
"bucket",
"key",
"media_type",
"sha256",
"size_bytes",
"uri"
],
"type": "object"
},
"SystemStatus": {
"enum": [
"unavailable",
"passed",
"hard_gate_failed",
"subject_error",
"resource_limit",
"infrastructure_error"
],
"type": "string"
},
"SystemUnderTestIdentity": {
"properties": {
"contract_hashes": {
"additionalProperties": {
"type": "string"
},
"type": "object"
},
"e2e_repository": {
"type": "string"
},
"e2e_revision": {
"type": "string"
},
"engine_revision": {
"type": [
"string",
"null"
]
},
"engine_version": {
"type": "string"
},
"harness_version": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
}
},
"required": [
"contract_hashes",
"e2e_repository",
"e2e_revision",
"engine_version",
"harness_version",
"stack"
],
"type": "object"
},
"TechnicalState": {
"enum": [
"valid",
"technical_invalid"
],
"type": "string"
},
"TypedPortValue": {
"additionalProperties": false,
"properties": {
"kind": {
"$ref": "#/definitions/PortValueKind"
},
"value": true
},
"required": [
"kind",
"value"
],
"type": "object"
},
"WorkflowAssetReport": {
"additionalProperties": false,
"properties": {
"artifact": {
"$ref": "#/definitions/ArtifactReference"
},
"content_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"media_type": {
"type": "string"
},
"namespaced_id": {
"type": "string"
},
"preview": true,
"preview_truncated": {
"type": "boolean"
},
"size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"artifact",
"content_sha256",
"id",
"kind",
"media_type",
"namespaced_id",
"preview",
"preview_truncated",
"size_bytes"
],
"type": "object"
},
"WorkflowCleanupReport": {
"additionalProperties": false,
"properties": {
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"failure": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowCleanupStatus"
}
},
"required": [
"duration_ms",
"status"
],
"type": "object"
},
"WorkflowCleanupStatus": {
"enum": [
"succeeded",
"failed"
],
"type": "string"
},
"WorkflowEvaluationOutcome": {
"enum": [
"passed",
"failed",
"advisory",
"not_evaluated"
],
"type": "string"
},
"WorkflowEvaluationResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"outcome": {
"$ref": "#/definitions/WorkflowEvaluationOutcome"
},
"score": {
"format": "double",
"type": [
"number",
"null"
]
},
"summary": {
"type": "string"
}
},
"required": [
"id",
"outcome",
"summary"
],
"type": "object"
},
"WorkflowFailurePhase": {
"enum": [
"preflight",
"execute",
"capture",
"persist",
"evaluate",
"cleanup",
"cancel"
],
"type": "string"
},
"WorkflowGateResult": {
"additionalProperties": false,
"properties": {
"evidence_ids": {
"default": [],
"items": {
"type": "string"
},
"type": "array"
},
"id": {
"type": "string"
},
"passed": {
"type": "boolean"
},
"reason": {
"type": "string"
}
},
"required": [
"id",
"passed",
"reason"
],
"type": "object"
},
"WorkflowStepFailure": {
"additionalProperties": false,
"properties": {
"message": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/WorkflowFailurePhase"
},
"technical": {
"type": "boolean"
}
},
"required": [
"message",
"phase",
"technical"
],
"type": "object"
},
"WorkflowStepReport": {
"additionalProperties": false,
"properties": {
"activation": {
"$ref": "#/definitions/ActivationPolicy"
},
"assets": {
"items": {
"$ref": "#/definitions/WorkflowAssetReport"
},
"type": "array"
},
"completed_at": {
"type": [
"string",
"null"
]
},
"cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"dependencies": {
"items": {
"type": "string"
},
"type": "array"
},
"dependency_policy": {
"$ref": "#/definitions/DependencyPolicy"
},
"duration_ms": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"evaluations": {
"items": {
"$ref": "#/definitions/WorkflowEvaluationResult"
},
"type": "array"
},
"failures": {
"items": {
"$ref": "#/definitions/WorkflowStepFailure"
},
"type": "array"
},
"hard_gates": {
"items": {
"$ref": "#/definitions/WorkflowGateResult"
},
"type": "array"
},
"harness_session_id": {
"type": [
"string",
"null"
]
},
"metrics": true,
"node_id": {
"type": "string"
},
"outputs": {
"additionalProperties": {
"$ref": "#/definitions/TypedPortValue"
},
"type": "object"
},
"redaction": {
"allOf": [
{
"$ref": "#/definitions/RedactionReport"
}
],
"default": {
"redacted_fields": 0,
"redacted_values": 0,
"rules": []
}
},
"required": {
"type": "boolean"
},
"skip_reason": {
"type": [
"string",
"null"
]
},
"started_at": {
"type": [
"string",
"null"
]
},
"status": {
"$ref": "#/definitions/WorkflowStepStatus"
},
"step_type": {
"type": "string"
},
"transcript": true
},
"required": [
"activation",
"dependencies",
"dependency_policy",
"duration_ms",
"node_id",
"required",
"status",
"step_type"
],
"type": "object"
},
"WorkflowStepStatus": {
"enum": [
"pending",
"running",
"succeeded",
"hard_gate_failed",
"failed",
"skipped",
"cancelled"
],
"type": "string"
}
},
"properties": {
"archive": {
"anyOf": [
{
"$ref": "#/definitions/DurableArchiveReference"
},
{
"type": "null"
}
]
},
"execution_id": {
"type": "string"
},
"manifest": {
"anyOf": [
{
"$ref": "#/definitions/E2eManifest"
},
{
"type": "null"
}
]
},
"observation": {
"anyOf": [
{
"$ref": "#/definitions/E2eObservationEnvelope"
},
{
"type": "null"
}
]
},
"observation_artifact": {
"anyOf": [
{
"$ref": "#/definitions/ArtifactReference"
},
{
"type": "null"
}
]
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"report": {
"anyOf": [
{
"$ref": "#/definitions/E2eReport"
},
{
"type": "null"
}
]
},
"request_sha256": {
"type": "string"
},
"result_path": {
"type": [
"string",
"null"
]
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"execution_id",
"phase",
"request_sha256"
],
"title": "ResultsGetResponse",
"type": "object"
}
},
{
"description": "List durable E2E executions using comparable identity filters.",
"metadata": {
"contract": {
"capabilities": [
"results-list"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::results-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
}
},
"properties": {
"lane": {
"default": null,
"type": [
"string",
"null"
]
},
"limit": {
"default": 50,
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"scenario_id": {
"anyOf": [
{
"$ref": "#/definitions/ScenarioId"
},
{
"type": "null"
}
],
"default": null
}
},
"title": "ResultsListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioId"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"JournalProgress": {
"properties": {
"active_attempt_id": {
"type": [
"string",
"null"
]
},
"attempts_finished": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"attempts_started": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"committed_events": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_event_sha256": {
"type": [
"string",
"null"
]
},
"phase": {
"type": [
"string",
"null"
]
},
"planned_slots": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"runs_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"slots_deferred": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"subject_observations_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"terminal": {
"type": "boolean"
},
"terminal_reason": {
"type": [
"string",
"null"
]
},
"terminal_state": {
"anyOf": [
{
"$ref": "#/definitions/JournalTerminalState"
},
{
"type": "null"
}
]
}
},
"required": [
"attempts_finished",
"attempts_started",
"committed_events",
"planned_slots",
"runs_committed",
"slots_deferred",
"subject_observations_committed",
"terminal"
],
"type": "object"
},
"JournalTerminalState": {
"enum": [
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
},
"StatusResponse": {
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"journal_progress": {
"$ref": "#/definitions/JournalProgress"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"journal_progress",
"phase",
"terminal",
"transitions",
"updated_at"
],
"type": "object"
}
},
"properties": {
"executions": {
"items": {
"$ref": "#/definitions/StatusResponse"
},
"type": "array"
}
},
"required": [
"executions"
],
"title": "ResultsListResponse",
"type": "object"
}
},
{
"description": "Delete expired E2E objects and tombstone their history records.",
"metadata": {
"contract": {
"capabilities": [
"retention-sweep"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::retention-sweep",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"before": {
"type": "string"
},
"dry_run": {
"default": false,
"type": "boolean"
},
"limit": {
"default": 100,
"format": "uint16",
"minimum": 0,
"type": "integer"
}
},
"required": [
"before"
],
"title": "RetentionSweepRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"deleted_archives": {
"items": {
"type": "string"
},
"type": "array"
},
"dry_run": {
"type": "boolean"
},
"inspected": {
"format": "uint32",
"minimum": 0,
"type": "integer"
}
},
"required": [
"deleted_archives",
"dry_run",
"inspected"
],
"title": "RetentionSweepResponse",
"type": "object"
}
},
{
"description": "Admit an asynchronous, idempotent E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"run"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::run",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"definitions": {
"ObservationCorrelation": {
"additionalProperties": false,
"properties": {
"deployment_id": {
"type": "string"
},
"operation_id": {
"type": "string"
},
"system": {
"type": "string"
}
},
"required": [
"deployment_id",
"operation_id",
"system"
],
"type": "object"
},
"ObservationDecision": {
"enum": [
"observe_only"
],
"type": "string"
},
"ObservationEnvironment": {
"enum": [
"demonstration"
],
"type": "string"
},
"ObservationMode": {
"additionalProperties": false,
"properties": {
"decision": {
"$ref": "#/definitions/ObservationDecision"
},
"environment": {
"$ref": "#/definitions/ObservationEnvironment"
}
},
"required": [
"decision",
"environment"
],
"type": "object"
},
"ObservationPlanIdentity": {
"additionalProperties": false,
"properties": {
"catalog_sha256": {
"type": "string"
},
"id": {
"type": "string"
},
"revision": {
"type": "string"
},
"sha256": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"id",
"revision",
"sha256"
],
"type": "object"
},
"ObservationRunContract": {
"additionalProperties": false,
"properties": {
"attempt": {
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"correlation": {
"$ref": "#/definitions/ObservationCorrelation"
},
"mode": {
"$ref": "#/definitions/ObservationMode"
},
"plan": {
"$ref": "#/definitions/ObservationPlanIdentity"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"selected_cases": {
"items": {
"$ref": "#/definitions/ObservationSelectedCase"
},
"type": "array"
},
"target": {
"$ref": "#/definitions/ObservationTargetIdentity"
}
},
"required": [
"attempt",
"correlation",
"mode",
"plan",
"runner",
"selected_cases",
"target"
],
"type": "object"
},
"ObservationSelectedCase": {
"additionalProperties": false,
"properties": {
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"contract_sha256": {
"type": "string"
},
"inputs_sha256": {
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioId"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"behavior_sha256",
"case_id",
"contract_sha256",
"inputs_sha256",
"scenario_id",
"seed"
],
"type": "object"
},
"ObservationTargetIdentity": {
"additionalProperties": false,
"properties": {
"application": {
"type": "string"
},
"stack": {
"$ref": "#/definitions/StackIdentity"
},
"version": {
"type": "string"
}
},
"required": [
"application",
"stack",
"version"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
},
"StackIdentity": {
"oneOf": [
{
"properties": {
"mode": {
"enum": [
"source"
],
"type": "string"
},
"workers_repository": {
"type": "string"
},
"workers_revision": {
"type": "string"
}
},
"required": [
"mode",
"workers_repository",
"workers_revision"
],
"type": "object"
},
{
"properties": {
"mode": {
"enum": [
"registry"
],
"type": "string"
},
"stack_lock_digest": {
"type": "string"
},
"stack_versions": {
"additionalProperties": {
"type": "string"
},
"type": "object"
}
},
"required": [
"mode",
"stack_lock_digest",
"stack_versions"
],
"type": "object"
}
]
}
},
"properties": {
"agent": {
"type": [
"string",
"null"
]
},
"idempotency_key": {
"type": "string"
},
"label": {
"default": "",
"type": "string"
},
"lane": {
"default": "local",
"type": "string"
},
"model": {
"type": "string"
},
"progress_interval_seconds": {
"default": 15,
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"provider": {
"type": "string"
},
"rotating_seeds": {
"default": [],
"items": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"type": "array"
},
"run_contract": {
"anyOf": [
{
"$ref": "#/definitions/ObservationRunContract"
},
{
"type": "null"
}
]
},
"runs": {
"default": 1,
"format": "uint32",
"minimum": 0,
"type": "integer"
},
"scenarios": {
"default": [],
"items": {
"$ref": "#/definitions/ScenarioId"
},
"type": "array"
},
"seed": {
"default": null,
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"slot_start_deadline_seconds": {
"description": "Soft scheduling budget, resolved before request hashing/admission.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"technical_retries": {
"default": 1,
"format": "uint8",
"minimum": 0,
"type": "integer"
}
},
"required": [
"idempotency_key",
"model",
"provider"
],
"title": "RunRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
}
},
"properties": {
"duplicate": {
"type": "boolean"
},
"execution_id": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"request_sha256": {
"type": "string"
},
"run_contract_sha256": {
"type": [
"string",
"null"
]
}
},
"required": [
"duplicate",
"execution_id",
"phase",
"request_sha256"
],
"title": "RunAccepted",
"type": "object"
}
},
{
"description": "List every built-in E2E test with its immutable contract, capabilities, complexity, and materialized case.",
"metadata": {
"agent": {
"audience": "harness",
"category": "e2e_local_test_authoring",
"operation": "list_tests",
"starts_execution": false,
"writes_local_state": false
},
"contract": {
"capabilities": [
"scenarios-list"
],
"name": "e2e-control-plane"
},
"internal": false
},
"name": "e2e::scenarios-list",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"seed": {
"default": null,
"description": "Optional deterministic seed used to materialize each listed test case. Omit it to use each scenario's canonical seed.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"title": "ScenariosListRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ArtifactExpectation": {
"properties": {
"id": {
"type": "string"
},
"kind": {
"type": "string"
},
"max_size_bytes": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"media_type": {
"type": "string"
},
"schema": true
},
"required": [
"id",
"kind",
"max_size_bytes",
"media_type",
"schema"
],
"type": "object"
},
"Capability": {
"description": "What a scenario needs from the engine, the runner host or a fixture to run. The closed list keeps the catalog honest: a new need is a new variant, and every id is spelled once.",
"enum": [
"browser::interactive",
"cross_repo_contract_simulator::v1",
"curl",
"docker",
"e2e::adaptive-flow-v1",
"e2e::control-plane-v1",
"e2e::filesystem",
"e2e::git",
"e2e::run-scoped-fixtures",
"e2e::shell",
"e2e::subagents",
"e2e::workflow-resume-v1",
"fixture::multi-origin-http",
"git",
"git::deterministic-fixture-v1",
"git::offline-bundle",
"github::trusted-handoff",
"go",
"harness::independent_session",
"harness::post-turn-validation",
"harness::scripted-dialogue-v1",
"iii::coder",
"iii::compose",
"iii::database",
"iii::functions",
"iii::registry",
"iii::shell",
"iii::state",
"iii::triggers",
"iii::workers",
"incident_fixture::v1",
"node",
"playwright",
"python3",
"release_shadow::read-only-v1",
"release_train_simulator::v1"
],
"type": "string"
},
"DeliverableContract": {
"properties": {
"artifacts": {
"items": {
"$ref": "#/definitions/ArtifactExpectation"
},
"type": "array"
},
"capture_before_cleanup": {
"type": "boolean"
},
"invariants": {
"items": {
"$ref": "#/definitions/InvariantSpec"
},
"type": "array"
},
"provenance_required": {
"type": "boolean"
}
},
"required": [
"artifacts",
"capture_before_cleanup",
"invariants",
"provenance_required"
],
"type": "object"
},
"ExecutionPolicy": {
"properties": {
"max_output_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_total_tokens": {
"description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.",
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_turns": {
"description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"max_validation_retries": {
"description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.",
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"stuck_timeout_seconds": {
"description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.",
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"stuck_timeout_seconds"
],
"type": "object"
},
"ExecutionRealism": {
"enum": [
"synthetic",
"realistic_simulator",
"frozen_real_artifact"
],
"type": "string"
},
"HumanHorizon": {
"properties": {
"basis": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizonBasis"
}
],
"default": "unknown"
},
"max_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"min_minutes": {
"format": "uint32",
"minimum": 0,
"type": [
"integer",
"null"
]
}
},
"type": "object"
},
"HumanHorizonBasis": {
"enum": [
"unknown",
"author_estimate",
"measured"
],
"type": "string"
},
"InvariantSpec": {
"properties": {
"description": {
"type": "string"
},
"id": {
"type": "string"
}
},
"required": [
"description",
"id"
],
"type": "object"
},
"RunnerIdentity": {
"additionalProperties": false,
"properties": {
"name": {
"type": "string"
},
"revision": {
"type": "string"
},
"version": {
"type": "string"
}
},
"required": [
"name",
"revision",
"version"
],
"type": "object"
},
"ScenarioCharacterization": {
"properties": {
"human_horizon": {
"allOf": [
{
"$ref": "#/definitions/HumanHorizon"
}
],
"default": {
"basis": "unknown"
}
},
"realism": {
"allOf": [
{
"$ref": "#/definitions/ScenarioRealism"
}
],
"default": {
"execution": "synthetic",
"shadow": "none"
}
}
},
"type": "object"
},
"ScenarioDescriptor": {
"properties": {
"behavior_sha256": {
"type": "string"
},
"case_id": {
"type": "string"
},
"characterization": {
"$ref": "#/definitions/ScenarioCharacterization"
},
"contract_sha256": {
"type": "string"
},
"deliverable_contract": {
"$ref": "#/definitions/DeliverableContract"
},
"inputs_sha256": {
"type": "string"
},
"required_capabilities": {
"items": {
"$ref": "#/definitions/Capability"
},
"type": "array"
},
"resource_envelope": {
"$ref": "#/definitions/ScenarioResourceEnvelope"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioId"
},
"seed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"behavior_sha256",
"case_id",
"characterization",
"contract_sha256",
"deliverable_contract",
"inputs_sha256",
"required_capabilities",
"resource_envelope",
"scenario_id",
"seed"
],
"type": "object"
},
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
},
"ScenarioRealism": {
"properties": {
"execution": {
"allOf": [
{
"$ref": "#/definitions/ExecutionRealism"
}
],
"default": "synthetic"
},
"shadow": {
"allOf": [
{
"$ref": "#/definitions/ShadowMode"
}
],
"default": "none"
}
},
"type": "object"
},
"ScenarioResourceEnvelope": {
"properties": {
"execution": {
"$ref": "#/definitions/ExecutionPolicy"
},
"workflow": {
"anyOf": [
{
"$ref": "#/definitions/WorkflowResourceEnvelope"
},
{
"type": "null"
}
]
}
},
"required": [
"execution"
],
"type": "object"
},
"ShadowMode": {
"enum": [
"none",
"read_only"
],
"type": "string"
},
"WorkflowResourceEnvelope": {
"properties": {
"max_cost_usd": {
"format": "double",
"type": [
"number",
"null"
]
},
"max_nodes": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_parallel": {
"format": "uint16",
"minimum": 0,
"type": "integer"
},
"max_total_tokens": {
"format": "uint64",
"minimum": 0,
"type": [
"integer",
"null"
]
},
"step_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"technical_retries": {
"format": "uint8",
"minimum": 0,
"type": "integer"
},
"workflow_timeout_seconds": {
"format": "uint64",
"minimum": 0,
"type": "integer"
}
},
"required": [
"max_nodes",
"max_parallel",
"step_timeout_seconds",
"technical_retries",
"workflow_timeout_seconds"
],
"type": "object"
}
},
"properties": {
"catalog_sha256": {
"type": "string"
},
"runner": {
"$ref": "#/definitions/RunnerIdentity"
},
"scenarios": {
"items": {
"$ref": "#/definitions/ScenarioDescriptor"
},
"type": "array"
},
"schema": {
"type": "string"
}
},
"required": [
"catalog_sha256",
"runner",
"scenarios",
"schema"
],
"title": "ScenariosListResponse",
"type": "object"
}
},
{
"description": "Read the durable phase and progress of an E2E execution.",
"metadata": {
"contract": {
"capabilities": [
"status"
],
"name": "e2e-control-plane"
},
"internal": true
},
"name": "e2e::status",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"execution_id": {
"type": "string"
}
},
"required": [
"execution_id"
],
"title": "ExecutionRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"ActiveAttempt": {
"properties": {
"attempt_id": {
"type": "string"
},
"resume_state_path": {
"type": [
"string",
"null"
]
},
"resume_state_sha256": {
"type": [
"string",
"null"
]
},
"run_id": {
"default": "",
"type": "string"
},
"scenario_id": {
"$ref": "#/definitions/ScenarioId"
},
"session_id": {
"type": "string"
}
},
"required": [
"attempt_id",
"scenario_id",
"session_id"
],
"type": "object"
},
"ExecutionPhase": {
"enum": [
"requested",
"admitted",
"preflighting",
"materializing",
"setting_up",
"executing",
"collecting",
"evaluating",
"persisting",
"cleaning_up",
"finalizing",
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"JournalProgress": {
"properties": {
"active_attempt_id": {
"type": [
"string",
"null"
]
},
"attempts_finished": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"attempts_started": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"committed_events": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"last_event_sha256": {
"type": [
"string",
"null"
]
},
"phase": {
"type": [
"string",
"null"
]
},
"planned_slots": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"runs_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"slots_deferred": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"subject_observations_committed": {
"format": "uint64",
"minimum": 0,
"type": "integer"
},
"terminal": {
"type": "boolean"
},
"terminal_reason": {
"type": [
"string",
"null"
]
},
"terminal_state": {
"anyOf": [
{
"$ref": "#/definitions/JournalTerminalState"
},
{
"type": "null"
}
]
}
},
"required": [
"attempts_finished",
"attempts_started",
"committed_events",
"planned_slots",
"runs_committed",
"slots_deferred",
"subject_observations_committed",
"terminal"
],
"type": "object"
},
"JournalTerminalState": {
"enum": [
"completed",
"failed",
"cancelled",
"needs_reconciliation",
"unsupported"
],
"type": "string"
},
"PhaseTransition": {
"properties": {
"at": {
"type": "string"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"reason": {
"type": "string"
}
},
"required": [
"at",
"phase",
"reason"
],
"type": "object"
},
"ScenarioId": {
"enum": [
"registry_planning",
"registry_implementation",
"registry_environment",
"registry_verification",
"kanban_c1_foundation",
"kanban_c2_persistence",
"kanban_c3_board",
"kanban_c4_ticket_flow",
"kanban_c5_edit_move",
"kanban_c6_discussion",
"kanban_c7_live",
"linkly_tutorial",
"context_pressure",
"minimal_path",
"persistent_state",
"insert_record",
"sequential_pipeline",
"database_migration_recovery",
"shell_coder_sandbox",
"research_pipeline",
"fanout_ladder",
"incident_response",
"todo_worker_simple",
"todo_worker_planned",
"engineering_endurance_ladder",
"git_regression_forensics",
"alertmanager_route_match",
"mechanical_reaction",
"timer_wake",
"receiving_operation",
"validation_loop",
"subagent_validation",
"subagent_validation_failure",
"validation_self_repair",
"validation_scope_enforcement",
"validation_chain",
"secret_hygiene",
"prompt_injection_resilience",
"moving_target",
"poison_message",
"cleanup_under_failure",
"depth_ladder",
"quorum_fan_in",
"contention_ledger",
"wake_chain_soak",
"chess_engine_build",
"form_flow_build",
"state_machine_canvas_build",
"chess_play_ladder",
"trend_blog",
"trending_topics_build",
"typescript_chat_service",
"tool_contract_recovery",
"policy_bound_action",
"cross_app_transaction",
"performance_regression",
"browser_cross_site",
"release_train_recovery",
"cross_repo_contract_migration"
],
"type": "string"
}
},
"properties": {
"active_attempt": {
"anyOf": [
{
"$ref": "#/definitions/ActiveAttempt"
},
{
"type": "null"
}
]
},
"cancel_requested": {
"type": "boolean"
},
"error": {
"type": "string"
},
"execution_id": {
"type": "string"
},
"journal_progress": {
"$ref": "#/definitions/JournalProgress"
},
"phase": {
"$ref": "#/definitions/ExecutionPhase"
},
"terminal": {
"type": "boolean"
},
"transitions": {
"items": {
"$ref": "#/definitions/PhaseTransition"
},
"type": "array"
},
"updated_at": {
"type": "string"
}
},
"required": [
"cancel_requested",
"execution_id",
"journal_progress",
"phase",
"terminal",
"transitions",
"updated_at"
],
"title": "StatusResponse",
"type": "object"
}
},
{
"description": "Serve the Harness E2E injectable Console assets.",
"metadata": {
"internal": true
},
"name": "e2e::ui-content",
"request_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"path": {
"type": "string"
}
},
"required": [
"path"
],
"title": "UiContentRequest",
"type": "object"
},
"response_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"properties": {
"content": {
"type": "string"
},
"content_type": {
"type": "string"
}
},
"required": [
"content",
"content_type"
],
"title": "UiContentResponse",
"type": "object"
}
}
],
"triggers": [
{
"description": "A local E2E execution started, produced progress, or reached a terminal state.",
"invocation_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"additionalProperties": false,
"title": "EventBindingConfig",
"type": "object"
},
"metadata": {},
"name": "e2e::dashboard::changed",
"return_schema": {}
},
{
"description": "State trigger",
"invocation_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"description": "Trigger config schema — field-parity with the engine's StateTriggerConfig (engine/src/workers/state/trigger.rs:18-23).",
"properties": {
"condition_function_id": {
"description": "Optional function evaluated per event; only an explicit `false` skips.",
"type": [
"string",
"null"
]
},
"key": {
"description": "Only fire for writes to this key (any key when unset).",
"type": [
"string",
"null"
]
},
"scope": {
"description": "Only fire for writes in this scope (any scope when unset).",
"type": [
"string",
"null"
]
}
},
"title": "StateTriggerSpec",
"type": "object"
},
"metadata": {},
"name": "state",
"return_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"definitions": {
"StateEventType": {
"enum": [
"state:created",
"state:updated",
"state:deleted"
],
"type": "string"
}
},
"properties": {
"event_type": {
"allOf": [
{
"$ref": "#/definitions/StateEventType"
}
],
"description": "Type of state change"
},
"key": {
"description": "State key",
"type": "string"
},
"new_value": {
"description": "New value"
},
"old_value": {
"description": "Previous value (null for created events)"
},
"scope": {
"description": "State scope",
"type": "string"
},
"type": {
"description": "Always \"state\"",
"type": "string"
}
},
"required": [
"event_type",
"key",
"new_value",
"scope",
"type"
],
"title": "StateCallRequest",
"type": "object"
}
},
{
"description": "Fires when an object is written to a configured bucket whose notifications source is wired up.",
"invocation_schema": {},
"metadata": {},
"name": "storage::object-created",
"return_schema": {}
},
{
"description": "Fires when an object is removed from a configured bucket whose notifications source is wired up.",
"invocation_schema": {},
"metadata": {},
"name": "storage::object-deleted",
"return_schema": {}
}
]
}