# harness-e2e > Measure Harness capability, retain reproducible evidence, and expose the E2E dashboard. | field | value | |-------|-------| | version | 0.19.0 | | type | binary | | license | Apache-2.0 | | repo | https://github.com/iii-hq/harness-e2e | | supported_targets | aarch64-apple-darwin, x86_64-unknown-linux-gnu | | author | iii | ## installation ```sh iii trigger compose::add worker=harness-e2e@0.19.0 ``` ## configuration ```yaml - control_database: harness_e2e data_dir: ~/.iii/data/harness-e2e ``` ## dependencies - `browser` @ `^0.2.15` - `database` @ `^0.5.20` - `storage` @ `^0.1.10` ## readme # Harness E2E `harness-e2e` measures what a Harness stack can execute: correct deliverables, structural integrity, bounded work, and repeatable outcomes. The repository stays independent of the `workers` source tree. Discovery, execution, observation, state access, and cleanup go through functions registered in iii. The product input is an already-running iii stack. One binary, `harness-e2e`, does both jobs. Compose starts it as the asynchronous `e2e::*` control plane and the injectable Console page. The same binary also runs scenarios and inspects reports from the command line. ## Score A run's score is the sum of the points its evaluated criteria awarded. - A criterion nobody evaluated adds nothing. - Scores are not normalized or rescaled. - Criterion points stay independent of completion. - A run stopped by a resource limit (token, cost, or turn budget, or no progress) scores zero on every criterion it did not reach. A scenario that scores the partial delivery keeps those points. - A check that holds only because nothing was attempted scores zero: a secret never fetched was not kept, an injection never read was not refused. - Criteria do not veto the score or approve a run. - Completion, technical validity, artifact evidence, and runtime controls are reported separately. - Infrastructure and execution failures still fail the CLI. Native criteria keep awards that were already measured when a later check cannot run. That later check has no award and stays `not_evaluated`. An incomplete criterion set has no total score. A product failure remains technically valid. An infrastructure failure invalidates the run and keeps the observations already recorded. Every score and every audit flag is deterministic. No scenario calls a second model to judge the first. ## Completion Completed means the subject met the task's purpose. Every scenario marks the criteria that are that purpose, usually one or two, as gates. Exactness, discipline, efficiency, and polish never gate: they only move the score, so two runs can both complete and still differ in points. A gate also checks that the run exercised the capability the scenario protects, not only the final artifact: a timer test completes only when the write came after the timer woke the session, however clumsily the rest was done. - Every gate passed: completed, whatever the other criteria scored. A gate passes at full points, or at its `gate_minimum` when fewer points already meet the purpose (a proportional check). - A gate that did not pass: incomplete. - A gate nobody evaluated: incomplete when another criterion already fell short, undetermined otherwise. A scenario overrides this only when a prerequisite decides before any gate, such as an application that never built. The gates are part of the behavior digest, so changing one redefines the scenario. Each criterion has a `description`, the question the subject is scored on, and a `label`, a 3–8 word statement the test card shows ("Stored record matches the migration exactly"). The description is part of the behavior digest and reaches the subject in some scenarios; the label does neither, so it can be reworded without redefining anything. Kanban labels live in `src/scenarios/kanban/labels.json`, beside the rubric, not in it. ## Requirements `shell_coder_sandbox`, `chess_engine_build`, and `trend_blog` prepare their reviewed fixture from an embedded Git bundle. They need Git. They do not need a fixture checkout or `HARNESS_E2E_FIXTURE_PATH`. Each attempt uses a private workspace, and temporary source checkouts are removed after their contents are read. `chess_engine_build` also needs a Compose daemon, a running iii Console, and an interactive browser. It grades a run-scoped chess Worker through live functions, the Console injectable-UI manifest, and the playable page at `#/worker//chess`, then preserves screenshots of the real Console with the functional evidence. Setup installs the pinned iii SDK with npm before the run; a cold npm cache needs Registry access. `typescript_chat_service` carries its own frozen skeleton. It needs Node 22.6 or newer on the runner host: the subject's TypeScript application runs through Node type stripping, both in the public suite and in the runner-owned behavioral probe. The trending-topics build needs Linux amd64, Docker, Git, Python 3, Node, and access to its pinned fixture. See [runtime and controls](tests/fixtures/trending-topics-build/README.md). The engineering handoff uses a protected disposable checkout of a pinned revision of `iii-hq/e2e-fixture`. ## Develop From the repository root: ```bash pnpm --dir dashboard install --frozen-lockfile pnpm --dir dashboard typecheck pnpm --dir dashboard lint pnpm --dir dashboard test pnpm --dir dashboard build cargo test --locked --all-targets cargo clippy --locked --all-targets -- -D warnings node --test tests/dashboard/*.test.cjs HARNESS_E2E_BIN="$PWD/target/debug/harness-e2e" python3 -m unittest discover -s tests/python -p 'test_*.py' ``` The Rust build writes `dashboard/dist-console/page.js` and `styles.css`, then embeds both in the worker. Node and pnpm must be on `PATH`. List scenarios and their definition digests: ```bash cargo run --locked --bin harness-e2e -- list cargo run --locked --bin harness-e2e -- catalog ``` ## Commands | Command | What it does | | --- | --- | | *(none)* / `worker` | Run the Compose-managed control-plane service. | | `list` | Print every scenario id. | | `catalog` | Print the materialized scenario catalog. | | `run` | Execute one or more scenarios against a running stack. | | `report` | Summarize a saved `results.json`. | | `models` | List models registered in the running stack. | | `test-plan list` | List suites and their coverage. | | `test-plan materialize` | Expand one suite into campaigns, groups, and cases. | | `--manifest` | Print the Registry worker manifest as JSON. | The E2E runner and launch scripts force `III_TELEMETRY_ENABLED=false` for themselves and their child processes, preventing iii product usage events from being sent to PostHog. For tests against an existing stack, set this opt-out when starting its engine and Compose too. An engine already running without the opt-out must be restarted with it before E2E tests. Run one scenario against an existing stack: ```bash cargo run --locked --bin harness-e2e -- run \ --url ws://127.0.0.1:49134 \ --model codex/gpt-5.6-luna \ --provider openai-codex \ --scenario todo_worker_simple ``` ## Scenarios Every scenario is a built-in module under `src/scenarios/`. The module owns its prompt, setup, deterministic evaluator, and cleanup. Its id is the same in the CLI, the worker catalog, the campaign runner, the dashboard, and the canonical result artifacts. Plans name the scenarios they run. Featured suites: - [Registry](tests/fixtures/registry-version-comparison/README.md): `registry_planning`, `registry_implementation`, `registry_environment`, and `registry_verification`. Each has its own [atomic validations](tests/fixtures/registry-version-comparison/scoring.md). - [Linkly tutorial](tests/fixtures/linkly-tutorial/README.md): the `linkly` suite. `linkly_tutorial` runs the seven tutorial chapters plus a project-restart guard as one scripted dialogue on one Harness session, against the `linkly-agentic` scaffold's Compose stack, and scores twenty-two deterministic checks. The `linkly` stack starts every group from a fresh scaffold; `scripts/linkly_stack.py` prepares one by hand. - [Trending topics](docs/blog-build-contract.md): an isolated per-attempt Git remote and an independent Playwright acceptance check against the delivered SHA. Design is free. Screenshots are evidence, not an aesthetic score. `config/test-plan.json` groups the catalog into modules (state, context, wakes, coordination, software engineering, integration, security, adaptive operations, continuous engineering, and the incremental Kanban application) plus a diagnostic set that suites can opt into. ## Test plans [config/test-plan.json](config/test-plan.json) is the executable plan. Rust materializes its suites and execution rules from that source and from the native scenario contracts. There is no generated catalog to keep in sync. | Suite | ID | Purpose | | --- | --- | --- | | Core runtime regression | `regression` | Core runtime, recovery, context and safety checks, three independent rounds; one technical retry where safe. | | Null agent canary | `null-canary` | The regression's tests run as the `null-agent` profile: every test must score 0 and stay incomplete. | | Application builds and changes | `software-engineering` | Kanban, Registry delivery, the trending-topics blog, Alertmanager route migration and the chess, form-flow and state-machine Workers. | | Application builds and changes ×3 (weekly) | `software-engineering-x3` | The same tests three independent times, to measure the suite's run-to-run spread. | | Build Linkly from the tutorial | `linkly` | The Linkly agentic tutorial alone, on the `linkly` stack. | | Essential checks before merge | `pr` | Four essential checks of a candidate stack before merging a change. | | Published stack checks | `after-release` | Five essential checks of the published stack. | A suite is only what to test: scenarios, runs of each and technical retries. In the Console these suites are read-only; **Suites** copies any of them into a suite of the Console to edit, and **Run tests** runs a suite with a model on the current Harness. See [suites](dashboard/README.md#suites). ```bash cargo run --locked -- test-plan list cargo run --locked -- test-plan materialize --suite software-engineering ``` The `software-engineering` suite runs each selected case once, with no technical retries: the seven Kanban cases, Registry implementation and verification, the trending-topics build, Alertmanager route migration, and the chess, form-flow and state-machine Workers in the Console. That is 14 cases and 14 planned runs, in 13 execution groups. Registry implementation and verification share a group, in that order, so verification receives the implementation delivery. Trending topics runs in `case-trending-topics-build`. Baseline worker versions come from the resolved stack contract. Alertmanager runs in its own group, needs Go 1.25+, and has no turn or token ceiling. A suite with `repetitions: 3` (`regression`, `software-engineering-x3`) materializes three campaigns, `-r01` to `-r03`, with the same groups and seeds; every group of every round is its own job, so a round adds jobs, not time to a job. The `null-canary` suite runs the regression's nine tests once, with no technical retries, and is meant to run as the `null-agent` Directory profile ([config/agents/null-agent.md](config/agents/null-agent.md)): it answers `done` and calls no function. The executor copies `config/agents/` into the project's agents folder whenever an execution names a profile. Every run must score 0 and stay `task_incomplete`; a point or a completion is a hole in that test's evaluator, and a function call means the agent disobeyed, not that the evaluator paid. The `linkly` suite runs the Linkly tutorial alone: its eight exchanges in `case-linkly-tutorial`, once, with no technical retries. Run it on the `linkly` stack, whose template starts every group from a fresh pinned `linkly-agentic` scaffold as the Compose project that hosts the Harness; on another stack the scenario's setup refuses the project it finds. ## Release Control Operational campaigns run through [`.github/workflows/exact-stack-e2e.yml`](.github/workflows/exact-stack-e2e.yml). This repository does not publish independent daily, weekly, post-deploy, or fault-stress dispatch workflows. A dispatch names what to test, where, and with whom: | Input | Meaning | | --- | --- | | `suite` | A suite id from `config/test-plan.json`, or one suite as JSON. | | `stack` | A stack in [`stacks/`](stacks/) by name (default `default`), or stack YAML. | | `model` | `provider/model` the subject runs. | | `profile` | Optional Directory agent profile the subject runs as. | | `execution_id` | Optional Release Control execution. Without it the run only produces artifacts. | A stack is an iii Compose project plus `iii` (a release, or `latest` for the newest `iii/v*` release candidate, as Release Control resolves it) and an optional `template` (`` or `@` of `iii-hq/templates`). Credentials never go in a stack: the executor stamps the namespace, the runner's data directory, the model's provider, what the suite needs and the private env file per group. Scripts always come from the dispatched ref. A container may pin `commit:` (short or full sha) instead of `version:`: ```yaml containers: harness: worker: package://harness commit: 3f2a9c1 # repository: iii-hq/workers # override only # path: harness # override only ``` The repository is the one the Registry downloads the worker's releases from (`github.com///releases/download/…`), the folder is the worker's name there, or the root of a repository named after the worker (`iii-hq/harness-e2e`); `repository:` and `path:` only override. A commit that does not exist fails the preparation; one the repository's default branch does not have (GitHub serves a fork's commits through the repository it forked) is a warning. The worker is built at that commit with its own manifest (`iii.worker.yaml`: Rust workers, `cargo build --release --locked`) and runs as a `path://` worker. Compose gives a `path://` worker none of what a package gets, so the executor declares it with the default config its manifest ships (under the stack's `config_override`), its `env`, the compose file's folder to run in, and every dependency of the worker's newest release at that release's exact versions and edges (Registry `/resolve`), which are held: never asked of `compose::add`, unless nothing else is left to ask for. The contract carries the build and `runtime.commits`. A worker built from a commit reports its Cargo version, so the reports name it `@` in `identity.stack_versions`, never a release, and `identity.stack_commits` gives its full commit, repository, folder and whether the default branch has it; the Console shows and compares it as `@`. Known limit: the default config arrives as `config_override`, which Compose ranks above what is saved (default < saved < override), so on a worker built from a commit a value saved with `configuration::set` goes back to the default when the worker restarts. Preparation resolves the rest, once: 1. `scripts/prepare_execution.py dispatch` reads the inputs into `execution.json`, `stack.yaml` and `plan.json`; `runtime` resolves `iii`, the template and every `commit:`, and names the build cache entry of those commits (`executor.sh prepare resolve`, the step with a `GITHUB_TOKEN`). 2. `commits` (`prepare build`, only when the stack pins a commit) builds each pinned worker once per repository, commit and folder, in its own container, with no token or credentials and nothing else running: the commits' code runs there. The cache entry holds only the builds of those commits: `target/worker-builds`, which GitHub restores and saves with `actions/cache` around this step alone, or its own folder under `worker-builds/` in a Docker execution's data directory, mounted into this step alone. It copies the build into the contract (`workers//`) and declares it in the stack. 3. `runner` resolves the stack's own `harness-e2e` with `iii compose build`, pins that release in the stack and fetches it. That binary materializes the suite (`suite.json`, also kept as `profile.json`), so the suite always comes from the runner every group runs, and the finalizer aggregates with it. The runner's identity in the reports is its revision. A runner pinned to a commit is its build. 4. `contracts` writes one contract per campaign. 5. The stack is assembled once with `compose::add`, which expands every declared worker into its graph and writes `worker-compose.lock`; it gets the groups' provider credentials and one retry. The model's provider and, for an agent profile, the Directory come from Harness's graph with its pins; only one no graph brings is asked for on its own. `lock` puts that project and lock into every contract. Each group starts it with `compose::up` frozen, so every group runs the same versions. A template project is assembled per group, pinned to the versions that lock resolved. The contract artifact carries the suite snapshot, the final `stack.yaml`, the lock, the model, the profile and the iii release. Campaign manifests never select or rotate seeds. They keep replay-safe turns separate from scripted dialogue and composite flows, persist a summary for every group, and are advisory while their longitudinal history is being calibrated. Release Control owns scheduling. The executor keeps the result advisory and archives each materialized group through the environment-owned durable archiver. Partial GitHub reruns reuse the contract artifact produced by `prepare`, kept for 90 days. The finalizer matches group artifacts to the completed jobs' execution times and keeps successful groups from earlier attempts. A rerun that produces no artifact stays missing evidence; an older artifact does not replace it. `scripts/report_execution.py` posts observations to Release Control's run ledger over OIDC: `materialized` before anything runs, one `shard` per campaign group, and a `summary` from the finalizer. Runs come from `results.json`, or from journal checkpoints when a group died before writing one. A group that produced neither still reports that fact. A preparation that failed, before or after the suite was materialized, posts a `summary` whose `failure` says where and why: the `phase` (the executor step: `resolve`, `build`, `materialize`, `assemble`, or `prepare` for the workflow's own steps), the workflow `step`, and the `code` and `message` iii printed. `workers` supplies versioned stack components. It does not orchestrate campaigns. ### Verify from another pipeline The composite action [`.github/actions/verify-stack`](.github/actions/verify-stack/action.yml) lets a pipeline in another repository run the `pr` suite on a candidate stack: workers built from a commit (`pins`, `commit`), or an iii release (`iii`). `scripts/verify_stack.py` dispatches `exact-stack-e2e.yml` without an execution, waits for it and reads `execution-summary.json`, because a green workflow does not say the scenarios passed (the campaign is advisory). The action never fails its job; it sets `status`, and the caller decides. | `status` | Means | | --- | --- | | `passed` | Every group was measured and passed. | | `failed` | A measured group failed. | | `not_measured` | No readable summary, an inconclusive group, invalid infrastructure or a run that never finished. | | `skipped` | None of the pinned workers is part of the harness graph. | Pinned workers are filtered against the Registry graph of the harness release, and the harness is always pinned with them at the same commit. The token needs `actions: read` and `write` on this repository. Runs from these pipelines do not reach Release Control's ledger, which accepts only its own bot's dispatches. ### Executor image Every phase runs in one image of tools, `ghcr.io/iii-hq/harness-e2e:tools-` ([`Dockerfile`](Dockerfile)): git, curl, jq, gh, Python 3 with pip and PyYAML, Node 24 with pnpm, Go 1.25, Rust 1.98.1, Playwright's Chromium at `/usr/bin/chromium`, the Docker CLI with buildx and compose, and the Docker Engine (`dockerd`, containerd, runc, iptables), much of what the `ubuntu-latest` runner gave the groups before. Bases, the Ubuntu archive snapshot and every download are pinned, so one Dockerfile is one set of tools. It holds no scripts: [`scripts/run_in_image.sh`](scripts/run_in_image.sh) `` mounts the checkout at the same path, runs as the caller's uid with `no-new-privileges`, passes the phase's environment through by name, and runs [`scripts/executor.sh`](scripts/executor.sh) `prepare resolve|build|materialize|assemble|fixtures`, `group`, `package` or `finalize [restore|aggregate]` there. `prepare fixtures` checks out what the groups start from and no package brings (the Kanban fixture, the stack's template, the Registry sources and the trending topics fixture) below `target/`, three tries each: the Kanban fixture and the Registry's `iii-hq/e2e-fixture` at the head `prepare resolve` resolved once for every group (else, with a warning, the branch), which the reports name in `identity.fixture_revisions`. `group` routes the fixture repositories a scenario clones to those checkouts; `finalize` lays each campaign's groups out from the group bundles the execution selected (linked, not copied) before aggregating them. Only `prepare resolve` and `prepare fixtures` get a `GITHUB_TOKEN`, and only `prepare resolve` the `FIXTURE_TOKEN` that reads the private fixture's head: a group's subject has a shell, and `prepare build` runs the commits a stack pins. `prepare` sees the checkout read-only but for `target/`, and `prepare` without a step runs resolve, build, materialize and assemble, each in a container of its own. Interrupted, the wrapper stops its container; a group whose image or container never started still writes its `failure.json`. No phase mounts the host's Docker socket or joins the host's network. Each container has a network of its own on Docker's default bridge, where a group's engine listens on 49134 without taking a host port. A `group` container runs a Docker daemon of its own: privileged, in a cgroup namespace of its own, it starts as root with an anonymous volume, labelled like the container, at `/var/lib/docker`; `executor.sh group` starts `dockerd` there, runs the group as the caller's uid (whose group owns the daemon's socket) and stops the daemon after it, which stops its containers. Every container a scenario starts (Registry's runner, Kanban's, trending topics') is that daemon's, in the group's network, so the application the Registry fixture publishes on `127.0.0.1` is where its screenshots look, and it goes with the group's container and its volume (`docker rm -fv` for one left behind). Each group starts with no image and pulls what its scenarios run. This keeps groups from colliding, not from the host. The group's user reaches root in its container through the daemon's socket, and the container is privileged, so that root is root-equivalent on the host, as the host's socket was: a subject can start a privileged container with the host's devices, mount the host's disk and read what is there (the host daemon's container configurations, with a concurrent `prepare`'s `GITHUB_TOKEN`, the Console's provider credentials and a worker's `provider_env_file`, `gh`'s credentials). From the default bridge it also reaches the host's ports on the bridge's gateway (a local Console on 3113, iii on 49134) and other groups' containers. The workflow keeps on the runner what needs it: the GitHub App token for the private fixture sources, artifacts, the OIDC reports, `gh`, packaging, and removing a cancelled phase's container before anything is reported or packaged. [`executor-image.yml`](.github/workflows/executor-image.yml) publishes a tag from `main` when the Dockerfile changes (or by hand) and never rebuilds an existing one. A tag that is not published yet is built where it runs, with a warning. `resolution.json` records the image the execution was prepared in, the registry digest or the tag when it was built locally, and the reports carry it as `identity.executor_image`. ### Agent profile and project template A dispatch can name an existing Directory agent profile (`profile`), such as `console-ui`. `console-ui` is **Console UI Engineer**. The runner reads that profile from the group's Directory and sends its id as `agent` to `e2e::run`. The profile, its parents, skills, functions, and model or provider must exist in that stack. The runner downloads the stack's versioned skill bundles into an isolated Directory and waits up to 120 seconds. A profile that is still missing fails resolution. A profile model overrides the plan model; `provider::model` also selects its provider. Results record the resolved subject model and the profile configuration hash. **Run again** resolves the same profile id in its test stack. Comparisons stay manual in Release Control. Omitting `profile` keeps the built-in agent. A project template belongs to the stack and is independent of the suite and the agent profile: ```yaml template: harness # or harness@ ``` The runner reads `iii/template.yaml` in `iii-hq/templates` at that revision (`main` when none is given), resolves it to one commit before sharding, and records that commit as `identity.template.revision`. Every group uses that source. The selected Compose project supplies the base. The versions the execution's stack lock resolved override its package selectors, extra packages enter the lock, and local workers stay local. Template skills replace whole downloaded namespaces, and its agent files take precedence over downloaded profiles. Machine-global profiles and skills are unused when an agent profile is selected or the template ships agents or skills; a template that ships neither (`linkly-agentic`) keeps the Directory it declares. The runner also enables the campaign's provider when the project does not already have it. Scenarios, prompts, permissions, fixtures, seeds, and repetitions stay the same. The evaluated agent applies to ordinary sessions and to workflow or adaptive steps. Evaluators stay the same. With no template, the run keeps the existing generated stack. To measure a profile, compare the same suite, template commit, model, and stack with and without `profile`. Changing the template as well measures the combined effect. Non-Compose templates and templates that ask for an interactive language choice are rejected before boot. ## Console Build the worker and its page from the repository root: ```bash cargo build --locked --bin harness-e2e ``` When Console connects to the same iii namespace, the worker registers the page assets and the `e2e::dashboard::*` functions for read, suites, stacks, run, status, and cancellation. The page lives under `#/ext/harness-e2e` and exposes Tests, Executions, Suites, and Stacks. See [dashboard/README.md](dashboard/README.md). The running Harness must publish request and response schemas compatible with the current typed surface. Missing or incompatible fields fail preflight. There is no payload-version compatibility mode. The page loads through iii: 25 compact summaries on the first overview page, one complete report when an execution is opened, only the selected pair for comparison, and the model and scenario catalog when the run dialog opens. Filtering and cursor pagination run on the server. Transport failures stay visible. The trusted publisher still writes the bounded JSON report archive used by CI. It does not publish a standalone Harness E2E web application. ### Import executions from GitHub On Executions, **Import from GitHub** lists the completed runs of the `exact-stack-e2e.yml` workflow in `github_repository` (worker config, default `iii-hq/harness-e2e`) with their suite, model, agent profile, stack, iii release, date and conclusion. The worker calls the `gh` CLI, so sign it in once with `gh auth login`; its errors are shown as they come. **Import** answers at once with an execution in the `importing` state; the worker downloads the run's highest-attempt bundle into its data directory and installs every group's native run as an ordinary retained run. The execution records its parameters with its suite (name and snapshot digest), the stack as its contract recorded it (the final `stack.yaml` and its digest, which Run again offers as recorded), the workers each group resolved and observed, and its GitHub origin. Contracts from before the workflow stated its execution (plan and profile only) import too. A group that left only `failure.json` is kept as a slot with that error. Importing a run again replaces the runs of the earlier import; the execution keeps its name. Imported and local executions are the same record: lists, reports, evidence and renaming treat them alike. ### Run in Docker **Run tests** with **Where: Docker** runs the execution as the exact-stack workflow does, from this worker, with no checkout of this repository: the worker embeds the [Dockerfile](Dockerfile) (whose digest names the executor image) and [`scripts/`](scripts), and runs every phase through `scripts/run_in_image.sh` from `docker-executions//` under its data directory: - `inputs.json`: the dispatch (`DISPATCH_*`): a master-plan suite run as it is by its id, any other suite whole as JSON, the stack's YAML, the model and the agent profile. - `checkout/`: what the wrapper mounts, with the Dockerfile and the scripts, frozen for the execution, and `target/`, every phase's work; `target/artifacts/` keeps the bundles under the workflow's artifact names (`e2e-contract--gh-1`, `e2e-observation----gh-`, `e2e-observation--gh-`). - `logs/`: each phase's output. `worker-builds//` beside `docker-executions/` keeps the workers stacks pin to a `commit:`, one entry per set of pinned commits, mounted into `prepare build` alone. `prepare resolve`, `build` (when the stack pins a commit), `materialize`, `assemble` and `fixtures` run once, then one `group` container per group, `docker_parallel_groups` (2) at a time across executions, each packaged, then `finalize`, whose root bundle is imported by the code that imports a GitHub run, from the folder. The root links its groups' bundles rather than copying them, and once an attempt is finalized the roots before it go, before it is imported. Cancel stops the running groups' containers (never a `package` or `finalize`), then finalizes and imports what finished. Running a scenario again runs its groups in new containers as the execution's next attempt, with the same contract, lock, scripts and image, finalizes and imports again: a group's last attempt that ended counts, and one cancelled or cut by a restart leaves the attempt before it counted. A worker that restarts removes an active Docker execution's containers, interrupts what did not finish and imports what did. A worker older than this release cannot read a Docker execution and drops it from its database, as it drops any row it cannot read. Every group runs on a network of its own with a Docker daemon of its own (see [Executor image](#executor-image)): it publishes no port on this host, so its stack never collides with this machine's iii (its Console binds 3113) or another group's. It can still reach them, and root on this host, as that section says. Worker configuration: - `provider_env_file`: an env file of provider credentials under the Console's own (below): where both name a variable, the Console's value wins. Provider credentials are the Console's, on the Stacks page: environment variables by name (`OPENAI_API_KEY`), set, replaced or deleted there. What is set in neither the Console nor `provider_env_file` is inherited from the worker's environment for the names [`config/provider-credentials.json`](config/provider-credentials.json) lists, as llm-router falls back from its configuration to `credential_env_var`: give the harness-e2e worker the keys as its compose entry gives llm-router (`env_file`, or `environment` with `KEY: "${KEY}"`). They live in `credentials.env` of the worker's `data_dir`, mode 600, never in the database, an execution's folder or its evidence, and the worker never answers with a value (`e2e::dashboard::credentials-list` gives names and whether each is set). `credential-set` takes the value as `secret`: the iii SDKs record each invocation's input in its trace (`iii.invocation.input`) unless `III_DISABLE_TRACE_PAYLOADS=1`, and leave out the values of keys such as `secret`; a key named `value` would be kept there as sent. A Docker execution reads them once per group attempt, merged with `provider_env_file` (whose lines it cannot use it names, without their values, as warnings), and hands that one set to the group and to the packaging that checks its evidence, each in a file of `data_dir/.phase-credentials/` (mode 600, directory 700) passed with `--env-file` and removed when the phase ends; the worker removes what a stop left there when it starts. `prepare assemble` gets them the same way; the root bundle, as on GitHub, none. An execution whose model's provider has no key, or with a value under 8 characters (the evidence is not checked for it), says so and still runs. Inside, `scripts/run_in_image.sh` tells the phase which variables are credentials (`HARNESS_E2E_CREDENTIALS`, the names of that file); the launcher writes exactly those, plus `DEEPSEEK_API_KEY`, `ZAI_API_KEY` and `TYPESAFE_API_KEY` when its environment has them, into the stack's `.env`, warns when the model's provider's key is missing, and gives the runner their names. The runner replaces each such value (and each the catalog names) by `[redacted:NAME]`, as written and as JSON escapes it, in every artifact and journal event before it hashes it, so a key the subject printed never reaches a digest. `exact_stack_campaign.py package` looks for them again, raw and escaped: in the launcher's `logs/`, which nothing hashes, it replaces them; anywhere else a file is bound by a digest (the runner's references, the campaign bundle, Release Control's checks), so it rewrites nothing and refuses the bundle, naming the files and credentials, never the values; the diagnostic is kept instead. `bundle-manifest.json` records what it found per name (`redaction`). On GitHub the preparation and each group job write the credentials among the job's secrets, the names that catalog lists and no other secret, to such a file in `RUNNER_TEMP`, which the phase and its packaging read; a shard's runs are reported to Release Control from the tree that is uploaded, after packaging, so a refused bundle reports its outcome and diagnostic and no run. The workflow names each of those secrets in that step: `toJSON(secrets)` would hold every run for approval. - `scripts_dir`: a checkout's `scripts/` to run instead of the embedded ones, copied into each new execution, so an edited script takes effect on the next. - `docker_parallel_groups`: groups at once, 2 by default. - `docker_registry_mirrors`: a pull-through cache per registry for the Docker daemon each group runs, which starts with no image, as `{mcr.microsoft.com: http://172.17.0.1:5001}`: the daemon pulls from the mirror first and from the registry when the mirror fails. It reaches the groups as `HARNESS_E2E_REGISTRY_MIRRORS` (`REGISTRY=URL ...`), which `scripts/run_in_image.sh group` also takes from its caller. Kanban's and trending topics' images come from `mcr.microsoft.com`, which `dockerd --registry-mirror` does not cover (it mirrors Docker Hub alone); a group's daemon writes each mirror into `/etc/docker/certs.d//hosts.toml` instead. One `registry:2` per upstream serves as the cache, listening on the Docker bridge's gateway (`docker network inspect bridge --format '{{(index .IPAM.Config 0).Gateway}}'`), where every group container reaches this host: ```sh docker run -d --name mcr-cache --restart unless-stopped -p 172.17.0.1:5001:5000 \ -v mcr-cache:/var/lib/registry -e REGISTRY_PROXY_REMOTEURL=https://mcr.microsoft.com registry:2 docker run -d --name hub-cache --restart unless-stopped -p 172.17.0.1:5000:5000 \ -v hub-cache:/var/lib/registry -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io registry:2 ``` The phases get none of the worker's environment but where Docker and `TMPDIR` are. `prepare fixtures` gets the worker's `GITHUB_TOKEN`, or the signed-in `gh`'s, for the private Registry and trending topics sources. #### Subscription providers `openai-codex` and `claude-code` sign in with this machine's CLI login, not an API key: `${CODEX_HOME:-~/.codex}/auth.json` (`codex login`) and `${CLAUDE_CONFIG_DIR:-~/.claude}/.credentials.json` (`claude`). A group gets the login's access token and nothing else: no refresh or id token ever enters a container or GitHub. Before an execution's first group, the worker reads the login; when its access token would expire before the group's deadline (10800 s and 15 minutes), it refreshes the login first, as the CLI does. The execution's later groups get that same token while it lasts a group, so one group's refresh never rotates the token another is running with. Refresh tokens rotate on every use, so only one holder may refresh. The worker's groups take turns, two workers on one machine take `.harness-e2e.lock`, and a Claude refresh also takes the `claude` CLI's own refresh lock (`/.oauth_refresh.lock` and `.lock`, renewed while held). Under the locks the login is read again; the refresh is written back only over the login it came from (atomically, mode 600), with the tokens alone replaced and everything else the CLI wrote meanwhile kept, its keys in sorted order. If the login was rotated by someone else meanwhile, theirs stands. The `codex` CLI takes no such lock, so a refresh racing it can cost one side its rotation; a refused refresh is retried once with the token the CLI wrote. A Codex access token lasts 10 days, so it is rarely refreshed; Claude's lasts hours. A refresh that fails (the service busy, the login refused) while the token still works hands that token over with a warning on the execution; a login that is signed out or dead is a warning too ("provider-claude-code starts without credentials: the Claude login on this machine expired; run `claude`"), and the group runs without it. The access token (`CODEX_ACCESS_TOKEN`, `CLAUDE_CODE_ACCESS_TOKEN`) is a credential: it goes with the provider credentials above, in the same file of `data_dir/.phase-credentials/` handed to the group and to the packaging that checks its evidence, removed when each ends; `prepare assemble` and a commit's build get none. [`config/provider-credentials.json`](config/provider-credentials.json) names it under `subscriptions`: never set on the Stacks page, but redacted like any other credential, by the runner and again by packaging. What else the login holds is no secret and goes to the group by name: `CODEX_ACCOUNT_ID`, `CLAUDE_CODE_EXPIRES_AT` (epoch milliseconds). The launcher writes the login the provider reads, into the group's runtime tree and not its evidence, and points only the provider at it (`CODEX_HOME`, `CLAUDE_CONFIG_DIR`); `stack/credentials.json` says where the token came from and when it expires, without it. A token already expired fails the group in its `credentials` phase; one that expires before the group's deadline is said out loud. The subject is denied the credential vault (`auth::*`) and each provider's own sign-in, status and logout (`provider::*::auth::*`), and the audit flags a call to either. It can still read the access token (from its workers' environment or the login file), as it can an API key; the runner redacts it by value from what it records, and any JWT (`eyJ….eyJ….…`) by its shape. As the [Executor image](#executor-image) section says, a subject is also root-equivalent on this host, where the login itself is. On GitHub the group job's credentials step also reads the optional secrets `CODEX_ACCESS_TOKEN` and `CLAUDE_CODE_ACCESS_TOKEN` of the `harness-e2e-trusted` environment into its file, and the group step the optional variables `CODEX_ACCOUNT_ID` and `CLAUDE_CODE_EXPIRES_AT`; the preparation reads none. Without a token, its provider starts signed out. Paste the access token from the login file, never the refresh token. A Codex token pasted there lasts 10 days; Claude's lasts hours, which makes it impractical on GitHub. ## Worker Release Control names the exact project roots. This repository writes only the root configuration and passes `worker@version` references to `compose::add`. iii resolves the Registry graph, writes the project topology, and reconciles containers. Every execution starts an empty Engine and a dedicated Compose daemon, then runs `compose::add`, `compose::up`, `compose::status`, and `compose::down`. Each execution uses one isolated namespace for Compose and for the project functions it starts. Compose supplies `III_URL`, `III_NAMESPACE`, `III_WORKER_NAME`, and the configuration: `III_CONFIG`, a file, before iii 0.24.3; `III_CONFIG_NAME`, the configuration-service entry read with `configuration::get`, from 0.24.3 on. The first three and one of those two are mandatory. By default the E2E worker uses its own Compose namespace. It discovers the dependency's database worker and configuration entry, then adds a single-connection `harness_e2e` SQLite pool at `/control.sqlite` to that entry's active value. Existing pools, settings and environment placeholders are retained; the saved database configuration is not changed. A saved edit of that configuration replaces its active value, so the worker subscribes to the entry and adds the pool back after each change; a full restart provisions it again over the same SQLite file. The database worker builds every pool or none, so when the pool never becomes ready the worker removes it again before exiting. Setup errors are retried until the startup deadline. This requires Engine support for runtime-only `configuration::set` writes; unsupported Engines receive a setup error. Explicit `control_namespace` or a different `control_database` selects an operator-provisioned destination: run a database worker with that pool in the chosen namespace and set `control_namespace` on the E2E container. The worker exits if the selected database or schema is unavailable, and reports the namespace, database and underlying readiness error. Install the E2E Console separately from a subject when its control database must be inaccessible to that subject. Publication validates the locally built binary through a `path://` Compose container before upload. Published campaigns use only exact Registry package versions. Provider secrets go to temporary permission-restricted `env_file` files and never appear in contract, Compose, evidence, or archive artifacts. CI also exercises the packaged binary's published defaults with real Registry dependencies via `scripts/smoke_registry_install.py`. It checks a non-default namespace, a renamed database container/configuration entry, an unchanged saved configuration, a separate SQLite file, persistence after restarting the entire Engine/Compose stack, and the pool's return after a saved edit. It runs in its own `registry-smoke` job. A Registry that stays unreachable (transport errors, 5xx, 429) after three attempts becomes a warning, not a failure; `--registry` pointed at a closed port rehearses that. The worker exposes: - `e2e::run`, `e2e::status`, `e2e::cancel` - `e2e::results-get`, `e2e::results-list` - `e2e::scenarios-list` - `e2e::archive`, `e2e::archive-head` Subject policies deny `e2e::*`. ### Live worker check `tests/live_worker.rs` talks to a worker registered in a real engine, so those tests are ignored by default. They check that the worker publishes this revision's catalog (ids, seeds, cases, definition digests), that it refuses malformed requests, and, when a subject is named, that one `minimal_path` execution passes admission, setup, the subject turn, capture, evaluation, and persistence. The resulting report is readable through `e2e::results-get`: ```bash HARNESS_E2E_LIVE_URL=ws://127.0.0.1:49134 HARNESS_E2E_LIVE_NAMESPACE=my-project \ HARNESS_E2E_LIVE_PROVIDER=deepseek HARNESS_E2E_LIVE_MODEL=deepseek-flash \ cargo test --test live_worker -- --ignored ``` Seven scenarios state host paths in their prompts, so their definition digests match only when the test process has the worker's `HARNESS_E2E_RUN_DIR`, `TMPDIR`, and `HARNESS_E2E_*_FIXTURE_PATH` values. Export the same environment the Compose file gives the worker. The scenario run leaves one execution labelled `live worker validation` in the worker's storage. ### Storage Durable artifacts are chunked through `storage::*`. Admissions, executions, runs, attempts, and artifact references are written through the control-plane `database::*` worker. Execution records keep compact dashboard summaries and observations, so lists and history do not load native reports. Storage has no version number and no migration step. Every table records the fingerprint of the statements that create it. At start, the worker recreates tables whose fingerprint moved, in one transaction. It keeps the execution records, local suites and stacks, and receipts it can still read, and rebuilds run projections from the native bundles. Rows it cannot read, missing bundles, and tables this binary no longer writes (such as the retired `history_*` import tables and the `saved_plans` of the retired baseline/candidate plans) are dropped and logged as warnings. Nothing is reconstructed as a scored result. A report written under another results contract is read with a warning. The Console's suites live in `local_suites`, its stacks in `local_stacks`, and executions (run here or imported from GitHub) in `saved_plan_executions`. A suite, stack or execution this binary cannot read is deleted on the next read. An execution a retired plan ran stays, without a suite. Native bundles keep full reports, manifests, and transcripts, loaded on demand. The runner has no S3, GCS, R2, SQL-driver, or Harness dependency. Weekly Stress materializes deterministic fault plans and evaluates journals from a protected supervisor. Lane promotion is governed by [`config/policies/cutover.json`](config/policies/cutover.json). ## Observation The runner waits for a session tree to finish by binding `harness::turn-completed` to an internal sink, `e2e::on-turn-completed`, before `harness::send`. That sink is not a control-plane verb: it is not registered with `e2e::run`, `e2e::status`, or `e2e::cancel`, and it does not appear in `e2e::scenarios-list`. Subject policies already deny `e2e::*`. A 15-second watchdog samples `harness::metrics` and one root `harness::status` for stuck detection, heartbeat logs, and `e2e::cancel`. If the trigger type is missing from `engine::triggers::list`, the run is unsupported infrastructure. There is no fallback that polls `harness::status` or `harness::metrics`. After the tree completes, the runner still collects terminal status, metrics, transcripts, and deliverables. ## Evidence Every completed execution records the subject and E2E revisions, observed wire contracts, definition digest, materialized inputs, seed, policies, artifacts, and raw structural evidence. Deterministic assessment has one payload shape, written only to `results.json`. Scenario contracts are the only versioned domain. Before cleanup, asset capture applies explicit safety limits and writes an unversioned sidecar with the canonical deterministic validation portion, which is aggregated into `results.json`. ## Repository | Path | Owns | | --- | --- | | `src/` | Runner, wire adapters, scenarios, evaluation, and the E2E control worker. | | `config/` | The cutover policy, fault profiles, and the master test plan. | | `stacks/` | Stacks a campaign runs on: iii Compose projects plus the iii release. | | `tests/` | Test-only fixtures, golden wire schemas, and the Node and Python suites. | | `schemas/` | Public contracts for generated E2E artifacts. | | `dashboard/` | React, TypeScript, Vite, and Tailwind Console page embedded in the worker. | Generated reports, transcripts, logs, and deliverables stay outside Git. The crate may depend on the iii SDK and on generic libraries. It must not declare a path or Git dependency on `workers`, Harness, or another product crate. Contract compatibility is established at runtime from `engine::functions::list` and `engine::functions::info`. The checked-in schemas are parity fixtures, not a linked product API. ## Package boundary This repository executes exact-stack test plans and publishes immutable `harness-e2e` Registry releases from `main`. A dispatch names a suite, a stack and a model; the stack is assembled and locked once, and the contract carries that lock, so a campaign can state afterwards exactly what it ran. ## Releases `cut-release.yml` is dispatched by hand with `bump` set to `patch`, `minor` or `major`. It writes the next version into `Cargo.toml` and `Cargo.lock`, commits that on `main` and pushes the tag `harness-e2e/v`. The tag runs `release.yml`: it checks that the tag, the manifest and the commit agree, builds the dashboard once and the binaries for `x86_64-unknown-linux-gnu` and `aarch64-apple-darwin`, creates the GitHub release, collects the typed interface in an isolated engine, publishes the Registry candidate to `next` and promotes it to `latest` with a compare-and-swap on the previous `latest`. The Registry job runs in the `workers-registry-next` environment; required reviewers on that environment turn it into a manual approval. Versions are plain semver; the `-experimental` suffix ended with 0.11.19. The root `iii.worker.yaml` is the public manifest for local `iii worker` development and package compatibility. The root `worker-compose.yaml` is a normal public Compose document. Release Control and post-prepare workflow phases read neither source contract. ## api reference ```json { "functions": [ { "description": "Persist a completed execution through private iii storage and idempotent history.", "metadata": { "contract": { "capabilities": [ "archive" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::archive", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "RetentionClass": { "enum": [ "temporary", "pull_request", "longitudinal", "canonical" ], "type": "string" } }, "properties": { "execution_id": { "type": "string" }, "retention_class": { "$ref": "#/definitions/RetentionClass" } }, "required": [ "execution_id", "retention_class" ], "title": "ArchiveExecutionRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "DurableArchiveReference": { "additionalProperties": false, "properties": { "archive_id": { "type": "string" }, "created_at": { "type": "string" }, "execution_id": { "type": "string" }, "expires_at": { "type": [ "string", "null" ] }, "identity_sha256": { "type": "string" }, "manifest": { "$ref": "#/definitions/StorageObjectReference" }, "manifest_backup": { "$ref": "#/definitions/StorageObjectReference" }, "retention_class": { "$ref": "#/definitions/RetentionClass" } }, "required": [ "archive_id", "created_at", "execution_id", "identity_sha256", "manifest", "manifest_backup", "retention_class" ], "type": "object" }, "HistoryRecord": { "additionalProperties": false, "properties": { "archive": { "$ref": "#/definitions/DurableArchiveReference" }, "baseline_comparable": { "type": "boolean" }, "case_count": { "format": "uint32", "minimum": 0, "type": "integer" }, "e2e_repository": { "type": "string" }, "e2e_revision": { "type": "string" }, "execution_id": { "type": "string" }, "identity_sha256": { "type": "string" }, "ingestion_id": { "type": "string" }, "lane": { "type": "string" }, "occurred_at": { "type": "string" }, "passed": { "type": "boolean" }, "result_contract_sha256": { "type": "string" }, "stack_mode": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "subject_revision": { "type": "string" } }, "required": [ "archive", "baseline_comparable", "case_count", "e2e_repository", "e2e_revision", "execution_id", "identity_sha256", "ingestion_id", "lane", "occurred_at", "passed", "result_contract_sha256", "stack_mode", "subject_model", "subject_provider", "subject_revision" ], "type": "object" }, "RetentionClass": { "enum": [ "temporary", "pull_request", "longitudinal", "canonical" ], "type": "string" }, "StorageObjectReference": { "properties": { "bucket": { "type": "string" }, "key": { "type": "string" }, "media_type": { "type": "string" }, "sha256": { "type": "string" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "uri": { "type": "string" } }, "required": [ "bucket", "key", "media_type", "sha256", "size_bytes", "uri" ], "type": "object" } }, "properties": { "archive": { "$ref": "#/definitions/DurableArchiveReference" }, "duplicate_ingestion": { "type": "boolean" }, "history": { "$ref": "#/definitions/HistoryRecord" } }, "required": [ "archive", "duplicate_ingestion", "history" ], "title": "ArchiveResponse", "type": "object" } }, { "description": "Verify availability and immutable metadata for an archived execution.", "metadata": { "contract": { "capabilities": [ "archive-head" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::archive-head", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ArchiveAvailability": { "enum": [ "available", "expired" ], "type": "string" }, "DurableArchiveReference": { "additionalProperties": false, "properties": { "archive_id": { "type": "string" }, "created_at": { "type": "string" }, "execution_id": { "type": "string" }, "expires_at": { "type": [ "string", "null" ] }, "identity_sha256": { "type": "string" }, "manifest": { "$ref": "#/definitions/StorageObjectReference" }, "manifest_backup": { "$ref": "#/definitions/StorageObjectReference" }, "retention_class": { "$ref": "#/definitions/RetentionClass" } }, "required": [ "archive_id", "created_at", "execution_id", "identity_sha256", "manifest", "manifest_backup", "retention_class" ], "type": "object" }, "RetentionClass": { "enum": [ "temporary", "pull_request", "longitudinal", "canonical" ], "type": "string" }, "StorageObjectReference": { "properties": { "bucket": { "type": "string" }, "key": { "type": "string" }, "media_type": { "type": "string" }, "sha256": { "type": "string" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "uri": { "type": "string" } }, "required": [ "bucket", "key", "media_type", "sha256", "size_bytes", "uri" ], "type": "object" } }, "properties": { "archive": { "$ref": "#/definitions/DurableArchiveReference" }, "availability": { "$ref": "#/definitions/ArchiveAvailability" }, "verified_at": { "type": "string" } }, "required": [ "archive", "availability", "verified_at" ], "title": "ArchiveHeadResponse", "type": "object" } }, { "description": "Request idempotent cancellation and cleanup of an E2E execution.", "metadata": { "contract": { "capabilities": [ "cancel" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::cancel", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ExecutionPhase": { "enum": [ "requested", "admitted", "preflighting", "materializing", "setting_up", "executing", "collecting", "evaluating", "persisting", "cleaning_up", "finalizing", "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" } }, "properties": { "accepted": { "type": "boolean" }, "execution_id": { "type": "string" }, "phase": { "$ref": "#/definitions/ExecutionPhase" } }, "required": [ "accepted", "execution_id", "phase" ], "title": "CancelResponse", "type": "object" } }, { "description": "Read the persisted projection for one logical-run attempt.", "metadata": { "contract": { "capability": "attempt-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::attempt-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "attempt_id": { "type": "string" }, "execution_id": { "type": "string" }, "run_id": { "type": "string" } }, "required": [ "attempt_id", "execution_id", "run_id" ], "title": "AttemptGetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CompletionState": { "enum": [ "completed", "task_incomplete", "undetermined" ], "type": "string" }, "ContaminationScope": { "enum": [ "none", "case", "execution" ], "type": "string" }, "CriterionReport": { "properties": { "awarded": { "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "description": { "type": [ "string", "null" ] }, "gate": { "default": false, "description": "A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell \"no gate\" from a report older than the field.", "type": "boolean" }, "gate_minimum": { "description": "The points a gate needs to count as passed when fewer than `possible` already mean the task's purpose was met. Absent: full points.", "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "id": { "type": "string" }, "label": { "description": "The criterion as a short statement; absent in reports older than the field and for criteria that state none. Readers fall back to `description`.", "type": [ "string", "null" ] }, "possible": { "format": "uint8", "minimum": 0, "type": "integer" }, "reason": { "type": "string" } }, "required": [ "id", "possible", "reason" ], "type": "object" }, "DimensionReport": { "properties": { "dimension": { "$ref": "#/definitions/EvaluationDimension" }, "passed": { "type": [ "boolean", "null" ] }, "signals": true }, "required": [ "dimension", "signals" ], "type": "object" }, "EfficiencyReport": { "properties": { "child_sessions": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "child_turns": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "critical_path_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "effective_fan_out": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_calls": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "observed_complexity": { "$ref": "#/definitions/ObservedComplexityReport" }, "observed_work": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "root_turns": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "technical_attempts": { "format": "uint32", "minimum": 0, "type": "integer" }, "total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "transient_resumes": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "unavailable": { "additionalProperties": { "type": "string" }, "type": "object" }, "validation_retries": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_resumes": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wall_time_ms": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "observed_complexity", "technical_attempts", "wall_time_ms" ], "type": "object" }, "EvaluationDimension": { "enum": [ "deliverable", "structural_integrity", "efficiency", "robustness", "e2e_infrastructure" ], "type": "string" }, "FailureDomain": { "enum": [ "subject", "resource", "e2e_infrastructure" ], "type": "string" }, "FailurePhase": { "enum": [ "setup", "execute", "collect", "evaluate", "cleanup" ], "type": "string" }, "FailureRecord": { "properties": { "code": { "default": "", "type": "string" }, "contamination": { "allOf": [ { "$ref": "#/definitions/ContaminationScope" } ], "default": "none" }, "domain": { "$ref": "#/definitions/FailureDomain" }, "message": { "type": "string" }, "phase": { "$ref": "#/definitions/FailurePhase" }, "retry_scope": { "allOf": [ { "$ref": "#/definitions/RetryScope" } ], "default": "none" }, "status": { "allOf": [ { "$ref": "#/definitions/RunStatus" } ], "description": "The failure's original classification, independent of the primary run status." } }, "required": [ "domain", "message", "phase", "status" ], "type": "object" }, "ObservedComplexityReport": { "properties": { "ambiguity_level": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "artifact_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "coordination_edges": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "dependency_depth": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "external_systems": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parallel_branches": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "planning_depth": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "state_transitions": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "unavailable": { "additionalProperties": { "type": "string" }, "type": "object" }, "validation_loops": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_cycles": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "RetryScope": { "enum": [ "none", "same_slot" ], "type": "string" }, "RunStatus": { "enum": [ "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "TechnicalState": { "enum": [ "valid", "technical_invalid" ], "type": "string" } }, "properties": { "completion": { "$ref": "#/definitions/CompletionState" }, "criteria": { "items": { "$ref": "#/definitions/CriterionReport" }, "type": [ "array", "null" ] }, "dimensions": { "items": { "$ref": "#/definitions/DimensionReport" }, "type": "array" }, "efficiency": { "anyOf": [ { "$ref": "#/definitions/EfficiencyReport" }, { "type": "null" } ] }, "failures": { "items": { "$ref": "#/definitions/FailureRecord" }, "type": "array" }, "status": { "$ref": "#/definitions/RunStatus" }, "technical": { "$ref": "#/definitions/TechnicalState" } }, "required": [ "completion", "dimensions", "failures", "status", "technical" ], "title": "AttemptResponse", "type": "object" } }, { "description": "Read models and scenarios when the execution dialog opens.", "metadata": { "contract": { "capability": "catalog-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::catalog-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "url": { "default": null, "type": [ "string", "null" ] } }, "title": "CatalogRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CatalogModel": { "properties": { "id": { "type": "string" }, "provider": { "type": "string" } }, "required": [ "id", "provider" ], "type": "object" } }, "properties": { "docker_parallel_groups": { "description": "Groups a Docker execution runs at once, across executions.", "format": "uint", "minimum": 0, "type": "integer" }, "models": { "items": { "$ref": "#/definitions/CatalogModel" }, "type": "array" }, "scenario_groups": { "description": "Scenarios that run only together, in this order: picking one runs all.", "items": { "items": { "type": "string" }, "type": "array" }, "type": "array" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "url": { "type": "string" } }, "required": [ "docker_parallel_groups", "models", "scenario_groups", "scenarios", "url" ], "title": "CatalogResponse", "type": "object" } }, { "description": "Delete a provider credential this Console set; answers with the list.", "metadata": { "contract": { "capability": "credential-delete", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::credential-delete", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "name": { "type": "string" } }, "required": [ "name" ], "title": "CredentialDeleteRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CredentialView": { "description": "A credential by name; never its value.", "properties": { "name": { "type": "string" }, "providers": { "description": "The providers that read it, as the catalog says.", "items": { "type": "string" }, "type": "array" }, "set": { "type": "boolean" }, "source": { "description": "`console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.", "type": [ "string", "null" ] } }, "required": [ "name", "providers", "set" ], "type": "object" } }, "properties": { "credentials": { "description": "By name, whether each is set and where from; never a value.", "items": { "$ref": "#/definitions/CredentialView" }, "type": "array" } }, "required": [ "credentials" ], "title": "CredentialsListResponse", "type": "object" } }, { "description": "Set a provider credential (an environment variable's name and its value) for Docker executions; answers with the list, never a value.", "metadata": { "contract": { "capability": "credential-set", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::credential-set", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "A credential to set. No `Debug`: its value is never printed.", "properties": { "name": { "description": "An environment variable's name (OPENAI_API_KEY).", "type": "string" }, "secret": { "description": "The value, under a key the iii SDKs redact from the invocation payloads they record in traces (`iii.invocation.input`): `value` would be kept there as sent.", "type": "string" } }, "required": [ "name", "secret" ], "title": "CredentialSetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CredentialView": { "description": "A credential by name; never its value.", "properties": { "name": { "type": "string" }, "providers": { "description": "The providers that read it, as the catalog says.", "items": { "type": "string" }, "type": "array" }, "set": { "type": "boolean" }, "source": { "description": "`console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.", "type": [ "string", "null" ] } }, "required": [ "name", "providers", "set" ], "type": "object" } }, "properties": { "credentials": { "description": "By name, whether each is set and where from; never a value.", "items": { "$ref": "#/definitions/CredentialView" }, "type": "array" } }, "required": [ "credentials" ], "title": "CredentialsListResponse", "type": "object" } }, { "description": "List the provider credentials Docker executions receive: each by name, whether it is set and where from, never its value.", "metadata": { "contract": { "capability": "credentials-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::credentials-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "DashboardEmptyRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CredentialView": { "description": "A credential by name; never its value.", "properties": { "name": { "type": "string" }, "providers": { "description": "The providers that read it, as the catalog says.", "items": { "type": "string" }, "type": "array" }, "set": { "type": "boolean" }, "source": { "description": "`console` (set here), `provider_env_file` (only the worker's file sets it), `environment` (only the worker's environment holds it), or absent when it is not set.", "type": [ "string", "null" ] } }, "required": [ "name", "providers", "set" ], "type": "object" } }, "properties": { "credentials": { "description": "By name, whether each is set and where from; never a value.", "items": { "$ref": "#/definitions/CredentialView" }, "type": "array" } }, "required": [ "credentials" ], "title": "CredentialsListResponse", "type": "object" } }, { "description": "List immutable evaluated-system versions and their exact evaluation cohorts.", "metadata": { "contract": { "capability": "evaluated-versions-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::evaluated-versions-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "cohort_id": { "default": null, "type": [ "string", "null" ] } }, "title": "EvaluatedVersionsRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "CohortDescriptor": { "properties": { "id": { "type": "string" }, "lane": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" } }, "required": [ "id", "lane", "subject_model", "subject_provider" ], "type": "object" }, "EvaluatedVersionDescriptor": { "properties": { "cohort_id": { "type": "string" }, "completed_at": { "type": "string" }, "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "id": { "type": "string" }, "label": { "type": "string" }, "stack_mode": { "type": "string" } }, "required": [ "cohort_id", "completed_at", "execution_count", "id", "label", "stack_mode" ], "type": "object" } }, "properties": { "cohorts": { "items": { "$ref": "#/definitions/CohortDescriptor" }, "type": "array" }, "revision": { "type": "string" }, "versions": { "items": { "$ref": "#/definitions/EvaluatedVersionDescriptor" }, "type": "array" } }, "required": [ "cohorts", "revision", "versions" ], "title": "EvaluatedVersionsResponse", "type": "object" } }, { "description": "Read one evidence file or deliverable screenshot a retained run's report declares, as base64 (10 MB at most).", "metadata": { "contract": { "capability": "evidence-read", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::evidence-read", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "description": "The native execution whose report declares the file.", "type": "string" }, "path": { "description": "A path the report declares for one of its runs (evidence or a deliverable).", "type": "string" }, "pointer": { "default": null, "description": "A declared screenshot's JSON Pointer inside that deliverable; only the image is returned.", "type": [ "string", "null" ] } }, "required": [ "execution_id", "path" ], "title": "EvidenceReadRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "One evidence file, or one screenshot inside a deliverable, as base64.", "properties": { "base64": { "type": "string" }, "media_type": { "type": "string" } }, "required": [ "base64", "media_type" ], "title": "EvidenceFile", "type": "object" } }, { "description": "Stop an execution where it runs: no next scenario is admitted and the running one is cancelled; in Docker the running groups stop; on GitHub `gh run cancel`, and what finished is imported when the run ends.", "metadata": { "contract": { "capability": "execution-cancel", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-cancel", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionGetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Delete one finished native run, or one finished execution together with its native runs.", "metadata": { "contract": { "capability": "execution-delete", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-delete", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionGetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Read one retained execution summary and its diagnostic report.", "metadata": { "contract": { "capability": "execution-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionGetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ExecutionListResponse": { "properties": { "executions": { "items": true, "type": "array" }, "last_update": { "type": "string" }, "mode": { "type": "string" }, "next_cursor": { "type": [ "string", "null" ] }, "repo_url": { "type": "string" }, "retention": { "$ref": "#/definitions/Retention" }, "total": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "executions", "last_update", "mode", "repo_url", "retention", "total" ], "type": "object" }, "Retention": { "properties": { "details": { "format": "uint", "minimum": 0, "type": "integer" }, "summaries": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "details", "summaries" ], "type": "object" } }, "properties": { "detail": true, "manifest": { "$ref": "#/definitions/ExecutionListResponse" } }, "required": [ "detail", "manifest" ], "title": "ExecutionBundle", "type": "object" } }, { "description": "Name one execution; an empty label restores its default name.", "metadata": { "contract": { "capability": "execution-rename", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-rename", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" }, "label": { "description": "Empty restores the default name.", "type": "string" } }, "required": [ "execution_id", "label" ], "title": "ExecutionRenameRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Run one scenario of a finished execution again where it ran: on this harness, in Docker as the next attempt, or on GitHub by re-running its group's job (followed and imported again when the run ends); the last attempt counts. Answers with its id and runs in the background.", "metadata": { "contract": { "capability": "execution-slot-rerun", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-slot-rerun", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" }, "scenario_id": { "description": "The scenario to run again; a sequential group runs again whole.", "type": "string" } }, "required": [ "execution_id", "scenario_id" ], "title": "ExecutionSlotRerunRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Start an execution from its parameters on this harness, in Docker on a stack, or on GitHub (dispatching the exact-stack workflow, followed and imported when the run ends); answers with its id and runs in the background.", "metadata": { "contract": { "capability": "execution-start", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::execution-start", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ExecutionParameters": { "description": "What an execution ran, to run it again. Always the canonical cases: a `seed` sent or stored by an older Console is ignored, so every execution pairs with any other by scenario and repetition.", "properties": { "agent": { "description": "Agent profile the subject ran under.", "type": [ "string", "null" ] }, "model": { "type": "string" }, "provider": { "type": "string" }, "runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "stack": { "anyOf": [ { "$ref": "#/definitions/ExecutionStack" }, { "type": "null" } ], "description": "The stack it ran on in Docker or on GitHub; this harness runs on its own. Running it again offers it as recorded." }, "suite": { "anyOf": [ { "$ref": "#/definitions/ExecutionSuite" }, { "type": "null" } ], "description": "The suite it ran; absent for an execution from before suites." }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "where": { "allOf": [ { "$ref": "#/definitions/Where" } ], "default": "harness", "description": "Where it ran; absent for an execution from before Docker, which ran on this harness." } }, "required": [ "model", "provider", "runs", "scenarios", "technical_retries" ], "type": "object" }, "ExecutionStack": { "description": "The stack an execution ran on, to run it again as recorded.", "properties": { "name": { "description": "A repository stack's id, a local one's name, or what a contract names.", "type": "string" }, "sha256": { "default": "", "description": "`sha256:` of `yaml`; the worker sets it, a request never does.", "type": "string" }, "yaml": { "description": "The YAML as sent; once imported, the final `stack.yaml` its contract recorded (the iii release and the runner pinned).", "type": "string" } }, "required": [ "name", "yaml" ], "type": "object" }, "ExecutionSuite": { "description": "The suite an execution ran: named when it was picked as it is (a suite of the master plan or of this Console), unnamed when its scenarios were ticked by hand, with the digest of the snapshot it materialized to.", "properties": { "id": { "description": "Absent for an unnamed suite.", "type": [ "string", "null" ] }, "label": { "default": "", "type": "string" }, "sha256": { "default": "", "description": "`profile_sha256` of the snapshot; the runner sets it, a request never does.", "type": "string" } }, "type": "object" }, "Where": { "description": "Where an execution runs: on this harness, in Docker from this worker, or on GitHub (dispatched from here, or imported).", "enum": [ "harness", "docker", "github" ], "type": "string" } }, "properties": { "label": { "default": "", "description": "Empty or absent uses the default name.", "type": "string" }, "parameters": { "allOf": [ { "$ref": "#/definitions/ExecutionParameters" } ], "description": "What to run; a previous execution's parameters run it again." } }, "required": [ "parameters" ], "title": "ExecutionStartRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "List compact retained execution summaries with server-side filters and cursor pagination.", "metadata": { "contract": { "capability": "executions-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::executions-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "cursor": { "default": null, "type": [ "string", "null" ] }, "event": { "default": null, "type": [ "string", "null" ] }, "ids": { "default": [], "items": { "type": "string" }, "type": "array" }, "ids_csv": { "default": null, "type": [ "string", "null" ] }, "limit": { "default": null, "format": "uint16", "minimum": 0, "type": [ "integer", "null" ] }, "query": { "default": null, "type": [ "string", "null" ] }, "status": { "default": null, "type": [ "string", "null" ] } }, "title": "ExecutionListRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Retention": { "properties": { "details": { "format": "uint", "minimum": 0, "type": "integer" }, "summaries": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "details", "summaries" ], "type": "object" } }, "properties": { "executions": { "items": true, "type": "array" }, "last_update": { "type": "string" }, "mode": { "type": "string" }, "next_cursor": { "type": [ "string", "null" ] }, "repo_url": { "type": "string" }, "retention": { "$ref": "#/definitions/Retention" }, "total": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "executions", "last_update", "mode", "repo_url", "retention", "total" ], "title": "ExecutionListResponse", "type": "object" } }, { "description": "Read the suite, subject, profile and runner of listed GitHub runs from their contract artifacts.", "metadata": { "contract": { "capability": "github-run-contracts", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::github-run-contracts", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "GithubRunAttempt": { "properties": { "run_attempt": { "format": "uint64", "minimum": 0, "type": "integer" }, "run_id": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "run_attempt", "run_id" ], "type": "object" } }, "properties": { "repository": { "default": null, "type": [ "string", "null" ] }, "runs": { "description": "The runs of one listed page whose contract is not read yet.", "items": { "$ref": "#/definitions/GithubRunAttempt" }, "type": "array" } }, "required": [ "runs" ], "title": "GithubRunContractsRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Import one exact-stack workflow run from GitHub as an execution; answers at once and imports in the background.", "metadata": { "contract": { "capability": "github-run-import", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::github-run-import", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "repository": { "default": null, "type": [ "string", "null" ] }, "run_id": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "run_id" ], "title": "GithubRunImportRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "List completed exact-stack workflow runs on GitHub with their local import; suites and subjects already read come along.", "metadata": { "contract": { "capability": "github-runs-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::github-runs-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "page": { "default": null, "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "repository": { "default": null, "type": [ "string", "null" ] } }, "title": "GithubRunsListRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Tell whether the GitHub CLI on this worker's machine is installed and signed in, so executions can be dispatched to the repository's exact-stack workflow; never fails for a missing or signed-out gh.", "metadata": { "contract": { "capability": "github-status-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::github-status-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "DashboardEmptyRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "Whether `gh` can dispatch executions to GitHub from this worker.", "properties": { "account": { "type": [ "string", "null" ] }, "message": { "description": "How to fix it when not ready.", "type": [ "string", "null" ] }, "ready": { "type": "boolean" }, "repository": { "type": "string" } }, "required": [ "ready", "repository" ], "title": "GithubStatusResponse", "type": "object" } }, { "description": "List the newest iii CLI releases (iii-hq/iii tags iii/v*, alphas left out), each with its date and whether it publishes the CLI, and the newest release candidate `iii: latest` installs; with a version, whether that release exists. Kept ten minutes.", "metadata": { "contract": { "capability": "iii-releases-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::iii-releases-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "version": { "default": null, "description": "A version typed by hand, e.g. `0.24.2`: whether iii-hq/iii has the release `iii/v` and it publishes the CLI.", "type": [ "string", "null" ] } }, "title": "IiiReleasesRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "IiiRelease": { "properties": { "cli": { "description": "Whether it publishes `iii-x86_64-unknown-linux-gnu.tar.gz`, the CLI a group installs.", "type": "boolean" }, "prerelease": { "type": "boolean" }, "published_at": { "type": [ "string", "null" ] }, "version": { "description": "The tag without `iii/v`.", "type": "string" } }, "required": [ "cli", "prerelease", "version" ], "type": "object" }, "IiiVersionCheck": { "properties": { "release": { "anyOf": [ { "$ref": "#/definitions/IiiRelease" }, { "type": "null" } ], "description": "Its release; none when iii-hq/iii has no `iii/v`." }, "version": { "type": "string" } }, "required": [ "version" ], "type": "object" } }, "properties": { "checked": { "anyOf": [ { "$ref": "#/definitions/IiiVersionCheck" }, { "type": "null" } ], "description": "The version asked about." }, "latest_candidate": { "description": "What `iii: latest` installs: the newest `X.Y.Z-rc.N` tag.", "type": [ "string", "null" ] }, "releases": { "description": "The newest releases, stable and pre-release, as GitHub lists them; alphas (`iii-alpha/…`) and drafts left out.", "items": { "$ref": "#/definitions/IiiRelease" }, "type": "array" } }, "required": [ "releases" ], "title": "IiiReleases", "type": "object" } }, { "description": "Cancel the active local E2E execution.", "metadata": { "contract": { "capability": "run-cancel", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::run-cancel", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "DashboardEmptyRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Defaults": { "properties": { "model": { "type": "string" }, "provider": { "type": "string" }, "runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "seed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "url": { "type": "string" } }, "required": [ "model", "provider", "runs", "technical_retries", "url" ], "type": "object" }, "JobStatus": { "enum": [ "running", "cancelling", "cancelled", "completed", "failed" ], "type": "string" }, "JobView": { "properties": { "completed_at": { "type": "string" }, "error": { "type": "string" }, "id": { "type": "string" }, "label": { "type": "string" }, "log": { "type": "string" }, "log_from": { "format": "uint64", "minimum": 0, "type": "integer" }, "log_offset": { "format": "uint64", "minimum": 0, "type": "integer" }, "log_truncated": { "type": "boolean" }, "request": { "$ref": "#/definitions/RunRequest" }, "returncode": { "format": "int32", "type": [ "integer", "null" ] }, "started_at": { "type": "string" }, "status": { "$ref": "#/definitions/JobStatus" } }, "required": [ "completed_at", "error", "id", "label", "log", "log_from", "log_offset", "log_truncated", "request", "started_at", "status" ], "type": "object" }, "RunRequest": { "additionalProperties": false, "properties": { "label": { "default": "", "type": "string" }, "model": { "type": "string" }, "provider": { "type": "string" }, "runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "seed": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "url": { "type": "string" } }, "required": [ "model", "provider", "runs", "scenarios", "technical_retries", "url" ], "type": "object" } }, "properties": { "defaults": { "$ref": "#/definitions/Defaults" }, "job": { "anyOf": [ { "$ref": "#/definitions/JobView" }, { "type": "null" } ] } }, "required": [ "defaults" ], "title": "RunSnapshot", "type": "object" } }, { "description": "Create a stack of this Console as a copy of another stack, or from its YAML, refused only as stack-update refuses it.", "metadata": { "contract": { "capability": "stack-create", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stack-create", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "A new stack of this Console: a copy of another stack (`from`) or the YAML given (`yaml`), exactly one of the two.", "properties": { "from": { "default": null, "description": "The stack it starts as a copy of: one of the repository or of this Console.", "type": [ "string", "null" ] }, "label": { "default": "", "description": "Empty or absent names a copy after its stack; a stack of YAML needs one.", "type": "string" }, "yaml": { "default": null, "description": "The whole stack as YAML, kept exactly as written; refused and warned about as `stack-update` does.", "type": [ "string", "null" ] } }, "title": "StackCreateRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StackContainer": { "description": "A container a stack declares: its worker as written and the version or commit it pins.", "properties": { "commit": { "type": [ "string", "null" ] }, "name": { "type": "string" }, "version": { "type": [ "string", "null" ] }, "worker": { "description": "The worker reference as written: `package://…`, `path://…` or other.", "type": [ "string", "null" ] } }, "required": [ "name" ], "type": "object" } }, "description": "A stack as the Console lists it.", "properties": { "containers": { "items": { "$ref": "#/definitions/StackContainer" }, "type": "array" }, "id": { "type": "string" }, "iii": { "description": "The iii CLI release it installs.", "type": [ "string", "null" ] }, "label": { "type": "string" }, "source": { "description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.", "type": "string" }, "template": { "description": "The iii-hq/templates project it starts from.", "type": [ "string", "null" ] }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What the executor or Compose may not do as written; never blocking.", "items": { "type": "string" }, "type": "array" }, "yaml": { "type": "string" } }, "required": [ "containers", "id", "label", "source", "warnings", "yaml" ], "title": "StackView", "type": "object" } }, { "description": "Delete a stack of this Console.", "metadata": { "contract": { "capability": "stack-delete", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stack-delete", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "stack_id": { "type": "string" } }, "required": [ "stack_id" ], "title": "StackDeleteRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Read a stack's YAML as stack-create and stack-update would, without saving it: what it declares (the template's revision apart) and its warnings, or why it would be refused.", "metadata": { "contract": { "capability": "stack-preview", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stack-preview", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "yaml": { "description": "A stack's YAML as a draft holds it; nothing is saved.", "type": "string" } }, "required": [ "yaml" ], "title": "StackPreviewRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "anyOf": [ { "properties": { "containers": { "items": { "$ref": "#/definitions/StackContainer" }, "type": "array" }, "iii": { "description": "The iii CLI release it installs.", "type": [ "string", "null" ] }, "template": { "anyOf": [ { "$ref": "#/definitions/TemplateReference" }, { "type": "null" } ], "description": "The iii-hq/templates project it starts from." }, "warnings": { "description": "What the executor or Compose may not do as written; never blocking.", "items": { "type": "string" }, "type": "array" } }, "required": [ "containers", "warnings" ], "type": "object" }, { "properties": { "refused": { "description": "Why `stack-create` and `stack-update` would refuse it.", "type": "string" } }, "required": [ "refused" ], "type": "object" } ], "definitions": { "StackContainer": { "description": "A container a stack declares: its worker as written and the version or commit it pins.", "properties": { "commit": { "type": [ "string", "null" ] }, "name": { "type": "string" }, "version": { "type": [ "string", "null" ] }, "worker": { "description": "The worker reference as written: `package://…`, `path://…` or other.", "type": [ "string", "null" ] } }, "required": [ "name" ], "type": "object" }, "TemplateReference": { "description": "`template: @`, split.", "properties": { "id": { "type": "string" }, "revision": { "description": "A commit, tag or branch of iii-hq/templates; none follows `main`.", "type": [ "string", "null" ] } }, "required": [ "id" ], "type": "object" } }, "description": "What a draft declares, as a saved stack would be read, or why it would be refused.", "title": "StackPreview" } }, { "description": "List the iii-hq/templates projects a stack can start from at a revision (a commit, tag or branch; main when none), each with the workers its worker-compose.yaml declares and the oldest iii it runs on; kept ten minutes.", "metadata": { "contract": { "capability": "stack-templates-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stack-templates-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "revision": { "default": null, "description": "A commit, tag or branch of iii-hq/templates; `main` when absent.", "type": [ "string", "null" ] } }, "title": "StackTemplatesRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StackTemplate": { "properties": { "description": { "type": "string" }, "id": { "description": "The folder under `iii/`, what a stack's `template:` names.", "type": "string" }, "min_iii_version": { "description": "The oldest iii release it runs on, when its `template.yaml` says.", "type": [ "string", "null" ] }, "name": { "type": "string" }, "note": { "description": "Why it declares no workers.", "type": [ "string", "null" ] }, "workers": { "description": "The containers its `worker-compose.yaml` declares with a `worker:`.", "items": { "$ref": "#/definitions/TemplateWorker" }, "type": "array" } }, "required": [ "description", "id", "name", "workers" ], "type": "object" }, "TemplateWorker": { "properties": { "name": { "description": "The container's name.", "type": "string" }, "version": { "description": "The version it pins, when it pins one.", "type": [ "string", "null" ] }, "worker": { "description": "The worker as written: `package://…` or `path://…`.", "type": "string" } }, "required": [ "name", "worker" ], "type": "object" } }, "properties": { "ref": { "description": "The revision asked for: `main` unless another commit, tag or branch was.", "type": "string" }, "repository": { "description": "`iii-hq/templates`.", "type": "string" }, "revision": { "description": "The commit it was at when it was read.", "type": "string" }, "templates": { "description": "In the order `iii/template.yaml` lists them; a listed folder that does not exist is left out.", "items": { "$ref": "#/definitions/StackTemplate" }, "type": "array" } }, "required": [ "ref", "repository", "revision", "templates" ], "title": "StackTemplates", "type": "object" } }, { "description": "Change the name or the YAML of a stack of this Console; refused only when the YAML does not parse or declares no containers.", "metadata": { "contract": { "capability": "stack-update", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stack-update", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "label": { "default": null, "type": [ "string", "null" ] }, "stack_id": { "type": "string" }, "yaml": { "default": null, "description": "The whole stack as YAML, kept exactly as written.", "type": [ "string", "null" ] } }, "required": [ "stack_id" ], "title": "StackUpdateRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StackContainer": { "description": "A container a stack declares: its worker as written and the version or commit it pins.", "properties": { "commit": { "type": [ "string", "null" ] }, "name": { "type": "string" }, "version": { "type": [ "string", "null" ] }, "worker": { "description": "The worker reference as written: `package://…`, `path://…` or other.", "type": [ "string", "null" ] } }, "required": [ "name" ], "type": "object" } }, "description": "A stack as the Console lists it.", "properties": { "containers": { "items": { "$ref": "#/definitions/StackContainer" }, "type": "array" }, "id": { "type": "string" }, "iii": { "description": "The iii CLI release it installs.", "type": [ "string", "null" ] }, "label": { "type": "string" }, "source": { "description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.", "type": "string" }, "template": { "description": "The iii-hq/templates project it starts from.", "type": [ "string", "null" ] }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What the executor or Compose may not do as written; never blocking.", "items": { "type": "string" }, "type": "array" }, "yaml": { "type": "string" } }, "required": [ "containers", "id", "label", "source", "warnings", "yaml" ], "title": "StackView", "type": "object" } }, { "description": "List the stacks an execution can run on: the repository's (read-only) and this Console's, each with its YAML, what it declares and its warnings.", "metadata": { "contract": { "capability": "stacks-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::stacks-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "DashboardEmptyRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StackContainer": { "description": "A container a stack declares: its worker as written and the version or commit it pins.", "properties": { "commit": { "type": [ "string", "null" ] }, "name": { "type": "string" }, "version": { "type": [ "string", "null" ] }, "worker": { "description": "The worker reference as written: `package://…`, `path://…` or other.", "type": [ "string", "null" ] } }, "required": [ "name" ], "type": "object" }, "StackView": { "description": "A stack as the Console lists it.", "properties": { "containers": { "items": { "$ref": "#/definitions/StackContainer" }, "type": "array" }, "id": { "type": "string" }, "iii": { "description": "The iii CLI release it installs.", "type": [ "string", "null" ] }, "label": { "type": "string" }, "source": { "description": "`repository` for a stack of `stacks/` (read-only), `local` for one this Console keeps.", "type": "string" }, "template": { "description": "The iii-hq/templates project it starts from.", "type": [ "string", "null" ] }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What the executor or Compose may not do as written; never blocking.", "items": { "type": "string" }, "type": "array" }, "yaml": { "type": "string" } }, "required": [ "containers", "id", "label", "source", "warnings", "yaml" ], "type": "object" } }, "properties": { "stacks": { "description": "The repository's stacks, then this Console's.", "items": { "$ref": "#/definitions/StackView" }, "type": "array" } }, "required": [ "stacks" ], "title": "StacksListResponse", "type": "object" } }, { "description": "Create a suite of this Console as a copy of another suite, or holding the tests given.", "metadata": { "contract": { "capability": "suite-create", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::suite-create", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "A new suite of this Console: a copy of another suite (`from`) or the tests given (`tests`), exactly one of the two.", "properties": { "from": { "default": null, "description": "The suite it starts as a copy of: one of the master plan or of this Console.", "type": [ "string", "null" ] }, "label": { "default": "", "description": "Empty or absent names a copy after its suite; a suite of tests needs one.", "type": "string" }, "repetitions": { "default": null, "description": "Runs of each test: the copied suite's, else 1.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "technical_retries": { "default": null, "description": "Technical retries a crash gets: the copied suite's, else 1.", "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "tests": { "default": null, "description": "The test ids it holds; sequential groups are completed.", "items": { "type": "string" }, "type": [ "array", "null" ] } }, "title": "SuiteCreateRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "A suite as the Console lists it.", "properties": { "id": { "type": "string" }, "label": { "type": "string" }, "purpose": { "type": "string" }, "repetitions": { "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "sha256": { "description": "Digest of the snapshot this runner materializes it to; what an execution of it records.", "type": [ "string", "null" ] }, "source": { "description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.", "type": "string" }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.", "items": { "type": "string" }, "type": "array" } }, "required": [ "id", "label", "purpose", "repetitions", "scenarios", "source", "technical_retries" ], "title": "SuiteView", "type": "object" } }, { "description": "Delete a suite of this Console.", "metadata": { "contract": { "capability": "suite-delete", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::suite-delete", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "suite_id": { "type": "string" } }, "required": [ "suite_id" ], "title": "SuiteDeleteRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": true, "title": "Map_of_AnyValue", "type": "object" } }, { "description": "Change the name, scenarios, runs or technical retries of a suite of this Console.", "metadata": { "contract": { "capability": "suite-update", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::suite-update", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "label": { "default": null, "type": [ "string", "null" ] }, "repetitions": { "default": null, "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "scenarios": { "default": null, "items": { "type": "string" }, "type": [ "array", "null" ] }, "suite_id": { "type": "string" }, "technical_retries": { "default": null, "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] } }, "required": [ "suite_id" ], "title": "SuiteUpdateRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "description": "A suite as the Console lists it.", "properties": { "id": { "type": "string" }, "label": { "type": "string" }, "purpose": { "type": "string" }, "repetitions": { "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "sha256": { "description": "Digest of the snapshot this runner materializes it to; what an execution of it records.", "type": [ "string", "null" ] }, "source": { "description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.", "type": "string" }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.", "items": { "type": "string" }, "type": "array" } }, "required": [ "id", "label", "purpose", "repetitions", "scenarios", "source", "technical_retries" ], "title": "SuiteView", "type": "object" } }, { "description": "List the suites an execution can run: the master plan's (read-only) and this Console's.", "metadata": { "contract": { "capability": "suites-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::suites-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "title": "DashboardEmptyRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "SuiteView": { "description": "A suite as the Console lists it.", "properties": { "id": { "type": "string" }, "label": { "type": "string" }, "purpose": { "type": "string" }, "repetitions": { "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "items": { "type": "string" }, "type": "array" }, "sha256": { "description": "Digest of the snapshot this runner materializes it to; what an execution of it records.", "type": [ "string", "null" ] }, "source": { "description": "`repository` for a suite of the master plan (read-only), `local` for one this Console keeps.", "type": "string" }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "updated_at": { "type": [ "string", "null" ] }, "warnings": { "description": "What creating it changed or left out (a completed group, an unknown test); only on the suite `suite-create` returns.", "items": { "type": "string" }, "type": "array" } }, "required": [ "id", "label", "purpose", "repetitions", "scenarios", "source", "technical_retries" ], "type": "object" } }, "properties": { "suites": { "description": "The master plan's suites, then this Console's.", "items": { "$ref": "#/definitions/SuiteView" }, "type": "array" } }, "required": [ "suites" ], "title": "SuitesListResponse", "type": "object" } }, { "description": "Read local metric history for one test version, with provider-grouped execution models, without comparison actions.", "metadata": { "contract": { "capability": "test-history-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::test-history-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "case_id": { "default": null, "type": [ "string", "null" ] }, "cursor": { "default": null, "type": [ "string", "null" ] }, "executions": { "default": null, "description": "Only these executions, at most two: the runs an A × B compares, found wherever they are in the history.", "items": { "type": "string" }, "type": [ "array", "null" ] }, "limit": { "default": null, "format": "uint16", "minimum": 0, "type": [ "integer", "null" ] }, "result": { "default": null, "type": [ "string", "null" ] }, "subject_model": { "default": null, "type": [ "string", "null" ] }, "subject_provider": { "default": null, "type": [ "string", "null" ] }, "system_version_id": { "default": null, "type": [ "string", "null" ] }, "test_id": { "default": "", "type": "string" }, "test_version": { "default": null, "type": [ "string", "null" ] } }, "title": "TestHistoryRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "AssessmentOutcomeCounts": { "properties": { "error": { "format": "uint", "minimum": 0, "type": "integer" }, "failed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "partial": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "error", "failed", "not_evaluated", "partial", "passed", "unavailable" ], "type": "object" }, "AssessmentSummary": { "description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.", "properties": { "assessment_count": { "format": "uint", "minimum": 0, "type": "integer" }, "assessment_outcomes": { "$ref": "#/definitions/AssessmentOutcomeCounts" }, "asset_count": { "format": "uint", "minimum": 0, "type": "integer" }, "asset_validation_outcomes": { "$ref": "#/definitions/AssetValidationCounts" }, "evidence_reference_count": { "format": "uint", "minimum": 0, "type": "integer" }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "system_statuses": { "$ref": "#/definitions/StatusCounts" } }, "required": [ "assessment_count", "assessment_outcomes", "asset_count", "asset_validation_outcomes", "evidence_reference_count", "run_count", "system_statuses" ], "type": "object" }, "AssetValidationCounts": { "properties": { "invalid": { "format": "uint", "minimum": 0, "type": "integer" }, "malformed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "not_produced": { "format": "uint", "minimum": 0, "type": "integer" }, "oversized": { "format": "uint", "minimum": 0, "type": "integer" }, "removed_during_cleanup": { "format": "uint", "minimum": 0, "type": "integer" }, "unexpected": { "format": "uint", "minimum": 0, "type": "integer" }, "unreadable": { "format": "uint", "minimum": 0, "type": "integer" }, "unsafe_path": { "format": "uint", "minimum": 0, "type": "integer" }, "valid": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "invalid", "malformed", "not_evaluated", "not_produced", "oversized", "removed_during_cleanup", "unexpected", "unreadable", "unsafe_path", "valid" ], "type": "object" }, "CompletionState": { "enum": [ "completed", "task_incomplete", "undetermined" ], "type": "string" }, "HistoryModelGroup": { "description": "Models are exposed as provider groups so a model name is never ambiguous when two providers offer the same model identifier.", "properties": { "models": { "items": { "type": "string" }, "type": "array" }, "provider": { "type": "string" } }, "required": [ "models", "provider" ], "type": "object" }, "HistoryRun": { "description": "One run of a history observation. Its status and totals come from the retained projection; `details` from the run's native results.", "properties": { "attempt_id": { "type": "string" }, "completion": { "$ref": "#/definitions/CompletionState" }, "details": { "anyOf": [ { "$ref": "#/definitions/RunDetails" }, { "type": "null" } ], "description": "Absent when the run's native results could not be read." }, "duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "function_calls": { "format": "double", "type": [ "number", "null" ] }, "run_id": { "type": "string" }, "score": { "format": "double", "type": [ "number", "null" ] }, "status": { "$ref": "#/definitions/RunStatus" }, "turns": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "attempt_id", "completion", "run_id", "status" ], "type": "object" }, "HistorySeries": { "properties": { "assessment_profile_sha256": { "type": "string" }, "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "cohort_id": { "type": "string" }, "contract_sha256": { "type": "string" }, "engine_revision": { "type": [ "string", "null" ] }, "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "harness_revision": { "type": [ "string", "null" ] }, "id": { "type": "string" }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "median_function_calls": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_turns": { "format": "double", "type": [ "number", "null" ] }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "seed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stack_mode": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "system_label": { "type": "string" }, "system_revision": { "type": [ "string", "null" ] }, "system_version_id": { "type": [ "string", "null" ] } }, "required": [ "assessment_profile_sha256", "behavior_sha256", "case_id", "cohort_id", "contract_sha256", "execution_count", "id", "run_count", "stack_mode", "subject_model", "subject_provider", "system_label" ], "type": "object" }, "HistorySession": { "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "label": { "description": "The agent profile the session was spawned with, when the root's transcript names it.", "type": [ "string", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "HistorySystem": { "properties": { "id": { "type": "string" }, "label": { "type": "string" } }, "required": [ "id", "label" ], "type": "object" }, "RunCriterion": { "properties": { "awarded": { "description": "Absent when nobody evaluated the criterion.", "format": "double", "type": [ "number", "null" ] }, "description": { "description": "The criterion as it was worded for this run; absent in older reports.", "type": [ "string", "null" ] }, "gate": { "description": "Whether this criterion blocked completion; absent in older reports.", "type": [ "boolean", "null" ] }, "id": { "type": "string" }, "possible": { "format": "double", "type": "number" }, "reason": { "type": "string" } }, "required": [ "id", "possible", "reason" ], "type": "object" }, "RunDetails": { "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "calls_by_worker": { "description": "Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.", "items": { "$ref": "#/definitions/WorkerCalls" }, "type": "array" }, "child_sessions": { "format": "uint", "minimum": 0, "type": "integer" }, "criteria": { "description": "Every criterion the run was scored on, in contract order.", "items": { "$ref": "#/definitions/RunCriterion" }, "type": "array" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "description": "The root session first, each session followed by the ones it spawned.", "items": { "$ref": "#/definitions/HistorySession" }, "type": "array" } }, "required": [ "calls_by_worker", "child_sessions", "criteria", "sessions" ], "type": "object" }, "RunStatus": { "enum": [ "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "StatusCounts": { "properties": { "hard_gate_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "infrastructure_error": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "resource_limit": { "format": "uint", "minimum": 0, "type": "integer" }, "subject_error": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "hard_gate_failed", "infrastructure_error", "passed", "resource_limit", "subject_error", "unavailable" ], "type": "object" }, "TestObservation": { "properties": { "agent_profile": { "description": "The agent profile the subject ran with, when it had one.", "type": [ "string", "null" ] }, "assessment_profile_sha256": { "type": "string" }, "assessment_summary": { "$ref": "#/definitions/AssessmentSummary" }, "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "cohort_id": { "type": "string" }, "completed_at": { "type": "string" }, "contract_sha256": { "type": "string" }, "engine_revision": { "type": [ "string", "null" ] }, "evaluated_version_id": { "type": [ "string", "null" ] }, "execution_id": { "type": "string" }, "execution_label": { "description": "That execution's name, when it has one.", "type": [ "string", "null" ] }, "harness_revision": { "type": [ "string", "null" ] }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "median_function_calls": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_turns": { "format": "double", "type": [ "number", "null" ] }, "plan_execution_id": { "description": "The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.", "type": [ "string", "null" ] }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "runs": { "items": { "$ref": "#/definitions/HistoryRun" }, "type": "array" }, "scored_runs": { "format": "uint", "minimum": 0, "type": "integer" }, "seed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stack_mode": { "type": "string" }, "status": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "system_label": { "type": "string" }, "system_revision": { "type": [ "string", "null" ] }, "system_version_id": { "type": [ "string", "null" ] } }, "required": [ "assessment_profile_sha256", "assessment_summary", "behavior_sha256", "case_id", "cohort_id", "completed_at", "contract_sha256", "execution_id", "run_count", "runs", "scored_runs", "stack_mode", "status", "subject_model", "subject_provider", "system_label" ], "type": "object" }, "VersionDescriptor": { "properties": { "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "last_seen": { "type": [ "string", "null" ] }, "observation_count": { "description": "The runs a test history lists: one per execution and case.", "format": "uint", "minimum": 0, "type": "integer" }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "version": { "type": "string" } }, "required": [ "execution_count", "observation_count", "run_count", "version" ], "type": "object" }, "WorkerCalls": { "properties": { "calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "worker": { "type": "string" } }, "required": [ "calls", "worker" ], "type": "object" } }, "properties": { "available_versions": { "items": { "$ref": "#/definitions/VersionDescriptor" }, "type": "array" }, "cases": { "items": { "type": "string" }, "type": "array" }, "current_version": { "description": "The current definition's digest, which may have no executions yet.", "type": [ "string", "null" ] }, "next_cursor": { "type": [ "string", "null" ] }, "observations": { "items": { "$ref": "#/definitions/TestObservation" }, "type": "array" }, "series": { "items": { "$ref": "#/definitions/HistorySeries" }, "type": "array" }, "subject_models": { "items": { "$ref": "#/definitions/HistoryModelGroup" }, "type": "array" }, "subjects": { "items": { "type": "string" }, "type": "array" }, "systems": { "items": { "$ref": "#/definitions/HistorySystem" }, "type": "array" }, "test_id": { "type": "string" }, "test_version": { "description": "The definition whose executions are shown (the latest with evidence by default).", "type": "string" }, "total": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "available_versions", "cases", "observations", "series", "subject_models", "subjects", "systems", "test_id", "test_version", "total" ], "title": "TestHistoryResponse", "type": "object" } }, { "description": "Read one test version across two immutable evaluated-system versions.", "metadata": { "contract": { "capability": "test-version-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::test-version-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "cohort_id": { "type": "string" }, "from_version_id": { "type": "string" }, "test_id": { "type": "string" }, "test_version": { "type": "string" }, "to_version_id": { "type": "string" } }, "required": [ "cohort_id", "from_version_id", "test_id", "test_version", "to_version_id" ], "title": "TestVersionGetRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "AssessmentOutcomeCounts": { "properties": { "error": { "format": "uint", "minimum": 0, "type": "integer" }, "failed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "partial": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "error", "failed", "not_evaluated", "partial", "passed", "unavailable" ], "type": "object" }, "AssessmentSummary": { "description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.", "properties": { "assessment_count": { "format": "uint", "minimum": 0, "type": "integer" }, "assessment_outcomes": { "$ref": "#/definitions/AssessmentOutcomeCounts" }, "asset_count": { "format": "uint", "minimum": 0, "type": "integer" }, "asset_validation_outcomes": { "$ref": "#/definitions/AssetValidationCounts" }, "evidence_reference_count": { "format": "uint", "minimum": 0, "type": "integer" }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "system_statuses": { "$ref": "#/definitions/StatusCounts" } }, "required": [ "assessment_count", "assessment_outcomes", "asset_count", "asset_validation_outcomes", "evidence_reference_count", "run_count", "system_statuses" ], "type": "object" }, "AssetValidationCounts": { "properties": { "invalid": { "format": "uint", "minimum": 0, "type": "integer" }, "malformed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "not_produced": { "format": "uint", "minimum": 0, "type": "integer" }, "oversized": { "format": "uint", "minimum": 0, "type": "integer" }, "removed_during_cleanup": { "format": "uint", "minimum": 0, "type": "integer" }, "unexpected": { "format": "uint", "minimum": 0, "type": "integer" }, "unreadable": { "format": "uint", "minimum": 0, "type": "integer" }, "unsafe_path": { "format": "uint", "minimum": 0, "type": "integer" }, "valid": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "invalid", "malformed", "not_evaluated", "not_produced", "oversized", "removed_during_cleanup", "unexpected", "unreadable", "unsafe_path", "valid" ], "type": "object" }, "CompletionState": { "enum": [ "completed", "task_incomplete", "undetermined" ], "type": "string" }, "HistoryRun": { "description": "One run of a history observation. Its status and totals come from the retained projection; `details` from the run's native results.", "properties": { "attempt_id": { "type": "string" }, "completion": { "$ref": "#/definitions/CompletionState" }, "details": { "anyOf": [ { "$ref": "#/definitions/RunDetails" }, { "type": "null" } ], "description": "Absent when the run's native results could not be read." }, "duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "function_calls": { "format": "double", "type": [ "number", "null" ] }, "run_id": { "type": "string" }, "score": { "format": "double", "type": [ "number", "null" ] }, "status": { "$ref": "#/definitions/RunStatus" }, "turns": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "attempt_id", "completion", "run_id", "status" ], "type": "object" }, "HistorySession": { "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "label": { "description": "The agent profile the session was spawned with, when the root's transcript names it.", "type": [ "string", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "MetricSamples": { "properties": { "cost_usd": { "format": "uint", "minimum": 0, "type": "integer" }, "duration_seconds": { "format": "uint", "minimum": 0, "type": "integer" }, "score": { "format": "uint", "minimum": 0, "type": "integer" }, "tokens": { "format": "uint", "minimum": 0, "type": "integer" }, "turns": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "cost_usd", "duration_seconds", "score", "tokens", "turns" ], "type": "object" }, "OutcomeCounts": { "properties": { "hard_gate_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "infra_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "technical_failed": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "hard_gate_failed", "infra_failed", "passed", "technical_failed" ], "type": "object" }, "RunCriterion": { "properties": { "awarded": { "description": "Absent when nobody evaluated the criterion.", "format": "double", "type": [ "number", "null" ] }, "description": { "description": "The criterion as it was worded for this run; absent in older reports.", "type": [ "string", "null" ] }, "gate": { "description": "Whether this criterion blocked completion; absent in older reports.", "type": [ "boolean", "null" ] }, "id": { "type": "string" }, "possible": { "format": "double", "type": "number" }, "reason": { "type": "string" } }, "required": [ "id", "possible", "reason" ], "type": "object" }, "RunDetails": { "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "calls_by_worker": { "description": "Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.", "items": { "$ref": "#/definitions/WorkerCalls" }, "type": "array" }, "child_sessions": { "format": "uint", "minimum": 0, "type": "integer" }, "criteria": { "description": "Every criterion the run was scored on, in contract order.", "items": { "$ref": "#/definitions/RunCriterion" }, "type": "array" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "description": "The root session first, each session followed by the ones it spawned.", "items": { "$ref": "#/definitions/HistorySession" }, "type": "array" } }, "required": [ "calls_by_worker", "child_sessions", "criteria", "sessions" ], "type": "object" }, "RunStatus": { "enum": [ "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "StatusCounts": { "properties": { "hard_gate_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "infrastructure_error": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "resource_limit": { "format": "uint", "minimum": 0, "type": "integer" }, "subject_error": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "hard_gate_failed", "infrastructure_error", "passed", "resource_limit", "subject_error", "unavailable" ], "type": "object" }, "TestDelta": { "properties": { "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "score": { "format": "double", "type": [ "number", "null" ] }, "tokens": { "format": "double", "type": [ "number", "null" ] } }, "type": "object" }, "TestObservation": { "properties": { "agent_profile": { "description": "The agent profile the subject ran with, when it had one.", "type": [ "string", "null" ] }, "assessment_profile_sha256": { "type": "string" }, "assessment_summary": { "$ref": "#/definitions/AssessmentSummary" }, "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "cohort_id": { "type": "string" }, "completed_at": { "type": "string" }, "contract_sha256": { "type": "string" }, "engine_revision": { "type": [ "string", "null" ] }, "evaluated_version_id": { "type": [ "string", "null" ] }, "execution_id": { "type": "string" }, "execution_label": { "description": "That execution's name, when it has one.", "type": [ "string", "null" ] }, "harness_revision": { "type": [ "string", "null" ] }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "median_function_calls": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_turns": { "format": "double", "type": [ "number", "null" ] }, "plan_execution_id": { "description": "The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.", "type": [ "string", "null" ] }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "runs": { "items": { "$ref": "#/definitions/HistoryRun" }, "type": "array" }, "scored_runs": { "format": "uint", "minimum": 0, "type": "integer" }, "seed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stack_mode": { "type": "string" }, "status": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "system_label": { "type": "string" }, "system_revision": { "type": [ "string", "null" ] }, "system_version_id": { "type": [ "string", "null" ] } }, "required": [ "assessment_profile_sha256", "assessment_summary", "behavior_sha256", "case_id", "cohort_id", "completed_at", "contract_sha256", "execution_id", "run_count", "runs", "scored_runs", "stack_mode", "status", "subject_model", "subject_provider", "system_label" ], "type": "object" }, "TestSideSummary": { "properties": { "assessment_summary": { "$ref": "#/definitions/AssessmentSummary" }, "case_count": { "format": "uint", "minimum": 0, "type": "integer" }, "evaluated_version_id": { "type": "string" }, "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "outcomes": { "$ref": "#/definitions/OutcomeCounts" }, "pass_rate": { "format": "double", "type": [ "number", "null" ] }, "samples": { "$ref": "#/definitions/MetricSamples" }, "scored_runs": { "format": "uint", "minimum": 0, "type": "integer" }, "total_runs": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "assessment_summary", "case_count", "evaluated_version_id", "execution_count", "outcomes", "samples", "scored_runs", "total_runs" ], "type": "object" }, "WorkerCalls": { "properties": { "calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "worker": { "type": "string" } }, "required": [ "calls", "worker" ], "type": "object" } }, "properties": { "compatibility": { "type": "string" }, "compatibility_reasons": { "items": { "type": "string" }, "type": "array" }, "delta": { "$ref": "#/definitions/TestDelta" }, "from": { "anyOf": [ { "$ref": "#/definitions/TestSideSummary" }, { "type": "null" } ] }, "from_observations": { "items": { "$ref": "#/definitions/TestObservation" }, "type": "array" }, "test_id": { "type": "string" }, "test_version": { "type": "string" }, "to": { "anyOf": [ { "$ref": "#/definitions/TestSideSummary" }, { "type": "null" } ] }, "to_observations": { "items": { "$ref": "#/definitions/TestObservation" }, "type": "array" } }, "required": [ "compatibility", "compatibility_reasons", "delta", "from_observations", "test_id", "test_version", "to_observations" ], "title": "TestVersionResult", "type": "object" } }, { "description": "List the versioned test catalog with compact A/B results and stable pagination.", "metadata": { "contract": { "capability": "tests-list", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::tests-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "cohort_id": { "default": null, "type": [ "string", "null" ] }, "cursor": { "default": null, "type": [ "string", "null" ] }, "from_version_id": { "default": null, "type": [ "string", "null" ] }, "limit": { "default": null, "format": "uint16", "minimum": 0, "type": [ "integer", "null" ] }, "query": { "default": null, "type": [ "string", "null" ] }, "to_version_id": { "default": null, "type": [ "string", "null" ] } }, "title": "TestsListRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Area": { "description": "The product capability a test protects; exactly one per test, so a report by capability never counts a test twice.", "enum": [ "core-loop", "data-safety", "tool-adaptation", "scheduling", "coding", "coordination", "self-repair", "long-context", "security", "app-building", "verification" ], "type": "string" }, "AssessmentKind": { "enum": [ "required_check", "signal", "asset_validation" ], "type": "string" }, "AssessmentOutcomeCounts": { "properties": { "error": { "format": "uint", "minimum": 0, "type": "integer" }, "failed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "partial": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "error", "failed", "not_evaluated", "partial", "passed", "unavailable" ], "type": "object" }, "AssessmentPolicy": { "enum": [ "hard_gate", "advisory" ], "type": "string" }, "AssessmentSummary": { "description": "Objective assessment totals for a set of runs. Every count is derived from deterministic evidence: hard gates, deterministic criteria, and asset validation.", "properties": { "assessment_count": { "format": "uint", "minimum": 0, "type": "integer" }, "assessment_outcomes": { "$ref": "#/definitions/AssessmentOutcomeCounts" }, "asset_count": { "format": "uint", "minimum": 0, "type": "integer" }, "asset_validation_outcomes": { "$ref": "#/definitions/AssetValidationCounts" }, "evidence_reference_count": { "format": "uint", "minimum": 0, "type": "integer" }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "system_statuses": { "$ref": "#/definitions/StatusCounts" } }, "required": [ "assessment_count", "assessment_outcomes", "asset_count", "asset_validation_outcomes", "evidence_reference_count", "run_count", "system_statuses" ], "type": "object" }, "AssetValidationCounts": { "properties": { "invalid": { "format": "uint", "minimum": 0, "type": "integer" }, "malformed": { "format": "uint", "minimum": 0, "type": "integer" }, "not_evaluated": { "format": "uint", "minimum": 0, "type": "integer" }, "not_produced": { "format": "uint", "minimum": 0, "type": "integer" }, "oversized": { "format": "uint", "minimum": 0, "type": "integer" }, "removed_during_cleanup": { "format": "uint", "minimum": 0, "type": "integer" }, "unexpected": { "format": "uint", "minimum": 0, "type": "integer" }, "unreadable": { "format": "uint", "minimum": 0, "type": "integer" }, "unsafe_path": { "format": "uint", "minimum": 0, "type": "integer" }, "valid": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "invalid", "malformed", "not_evaluated", "not_produced", "oversized", "removed_during_cleanup", "unexpected", "unreadable", "unsafe_path", "valid" ], "type": "object" }, "CalibrationProjection": { "properties": { "compatible_sample_count": { "format": "uint", "minimum": 0, "type": "integer" }, "maturity": { "type": "string" } }, "required": [ "compatible_sample_count", "maturity" ], "type": "object" }, "CompletionState": { "enum": [ "completed", "task_incomplete", "undetermined" ], "type": "string" }, "Discipline": { "description": "The kind of software the subject delivers: a question about the model and the agent, not about iii. `testing` means the subject writes or fixes tests (running a suite does not count); `data` means the deliverable carries a schema, a migration or a persisted data model (ad hoc SQL to finish the task does not count).", "enum": [ "frontend", "backend", "data", "infra", "testing" ], "type": "string" }, "EvaluationDimension": { "enum": [ "deliverable", "structural_integrity", "efficiency", "robustness", "e2e_infrastructure" ], "type": "string" }, "ExecutionPolicy": { "properties": { "max_output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_total_tokens": { "description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_turns": { "description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "max_validation_retries": { "description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "stuck_timeout_seconds": { "description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "stuck_timeout_seconds" ], "type": "object" }, "ExecutionRealism": { "enum": [ "synthetic", "realistic_simulator", "frozen_real_artifact" ], "type": "string" }, "HistoryRun": { "description": "One run of a history observation. Its status and totals come from the retained projection; `details` from the run's native results.", "properties": { "attempt_id": { "type": "string" }, "completion": { "$ref": "#/definitions/CompletionState" }, "details": { "anyOf": [ { "$ref": "#/definitions/RunDetails" }, { "type": "null" } ], "description": "Absent when the run's native results could not be read." }, "duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "function_calls": { "format": "double", "type": [ "number", "null" ] }, "run_id": { "type": "string" }, "score": { "format": "double", "type": [ "number", "null" ] }, "status": { "$ref": "#/definitions/RunStatus" }, "turns": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "attempt_id", "completion", "run_id", "status" ], "type": "object" }, "HistorySession": { "properties": { "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "duration_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "label": { "description": "The agent profile the session was spawned with, when the root's transcript names it.", "type": [ "string", "null" ] }, "parent_session_id": { "type": [ "string", "null" ] }, "session_id": { "type": "string" }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "HumanHorizon": { "properties": { "basis": { "allOf": [ { "$ref": "#/definitions/HumanHorizonBasis" } ], "default": "unknown" }, "max_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "min_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "HumanHorizonBasis": { "enum": [ "unknown", "author_estimate", "measured" ], "type": "string" }, "LastRun": { "properties": { "at": { "description": "When its execution completed.", "type": "string" }, "completion": { "$ref": "#/definitions/CompletionState" }, "definition": { "$ref": "#/definitions/RunDefinition" }, "score": { "format": "double", "type": [ "number", "null" ] }, "status": { "$ref": "#/definitions/RunStatus" } }, "required": [ "at", "completion", "definition", "status" ], "type": "object" }, "MetricSamples": { "properties": { "cost_usd": { "format": "uint", "minimum": 0, "type": "integer" }, "duration_seconds": { "format": "uint", "minimum": 0, "type": "integer" }, "score": { "format": "uint", "minimum": 0, "type": "integer" }, "tokens": { "format": "uint", "minimum": 0, "type": "integer" }, "turns": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "cost_usd", "duration_seconds", "score", "tokens", "turns" ], "type": "object" }, "OutcomeCounts": { "properties": { "hard_gate_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "infra_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "technical_failed": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "hard_gate_failed", "infra_failed", "passed", "technical_failed" ], "type": "object" }, "RunCriterion": { "properties": { "awarded": { "description": "Absent when nobody evaluated the criterion.", "format": "double", "type": [ "number", "null" ] }, "description": { "description": "The criterion as it was worded for this run; absent in older reports.", "type": [ "string", "null" ] }, "gate": { "description": "Whether this criterion blocked completion; absent in older reports.", "type": [ "boolean", "null" ] }, "id": { "type": "string" }, "possible": { "format": "double", "type": "number" }, "reason": { "type": "string" } }, "required": [ "id", "possible", "reason" ], "type": "object" }, "RunDefinition": { "description": "Which definition a run evaluated, next to the test's current one.", "oneOf": [ { "description": "The test's current definition.", "enum": [ "current" ], "type": "string" }, { "description": "An earlier definition (or the last one of a retired test).", "enum": [ "previous" ], "type": "string" } ] }, "RunDetails": { "properties": { "cache_read_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "calls_by_worker": { "description": "Calls of the root session (the only transcript a run keeps), by the worker that serves them, most called first.", "items": { "$ref": "#/definitions/WorkerCalls" }, "type": "array" }, "child_sessions": { "format": "uint", "minimum": 0, "type": "integer" }, "criteria": { "description": "Every criterion the run was scored on, in contract order.", "items": { "$ref": "#/definitions/RunCriterion" }, "type": "array" }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "description": "The root session first, each session followed by the ones it spawned.", "items": { "$ref": "#/definitions/HistorySession" }, "type": "array" } }, "required": [ "calls_by_worker", "child_sessions", "criteria", "sessions" ], "type": "object" }, "RunStatus": { "enum": [ "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "ScenarioCharacterization": { "properties": { "human_horizon": { "allOf": [ { "$ref": "#/definitions/HumanHorizon" } ], "default": { "basis": "unknown" } }, "realism": { "allOf": [ { "$ref": "#/definitions/ScenarioRealism" } ], "default": { "execution": "synthetic", "shadow": "none" } } }, "type": "object" }, "ScenarioRealism": { "properties": { "execution": { "allOf": [ { "$ref": "#/definitions/ExecutionRealism" } ], "default": "synthetic" }, "shadow": { "allOf": [ { "$ref": "#/definitions/ShadowMode" } ], "default": "none" } }, "type": "object" }, "ShadowMode": { "enum": [ "none", "read_only" ], "type": "string" }, "StatusCounts": { "properties": { "hard_gate_failed": { "format": "uint", "minimum": 0, "type": "integer" }, "infrastructure_error": { "format": "uint", "minimum": 0, "type": "integer" }, "passed": { "format": "uint", "minimum": 0, "type": "integer" }, "resource_limit": { "format": "uint", "minimum": 0, "type": "integer" }, "subject_error": { "format": "uint", "minimum": 0, "type": "integer" }, "unavailable": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "hard_gate_failed", "infrastructure_error", "passed", "resource_limit", "subject_error", "unavailable" ], "type": "object" }, "TestCatalogRow": { "properties": { "available_versions": { "items": { "$ref": "#/definitions/VersionDescriptor" }, "type": "array" }, "calibration": { "anyOf": [ { "$ref": "#/definitions/CalibrationProjection" }, { "type": "null" } ] }, "characterization": { "anyOf": [ { "$ref": "#/definitions/ScenarioCharacterization" }, { "type": "null" } ] }, "current_version": { "type": [ "string", "null" ] }, "last_run": { "anyOf": [ { "$ref": "#/definitions/LastRun" }, { "type": "null" } ], "description": "The most recent retained run, on any definition." }, "lifecycle": { "type": "string" }, "recent_scores": { "description": "The scores of the last runs, oldest first; null where a run has none.", "items": { "format": "double", "type": [ "number", "null" ] }, "type": "array" }, "result": { "anyOf": [ { "$ref": "#/definitions/TestVersionResult" }, { "type": "null" } ] }, "runs_current": { "description": "Runs retained on the current definition.", "format": "uint", "minimum": 0, "type": "integer" }, "runs_total": { "description": "Runs retained on every definition.", "format": "uint", "minimum": 0, "type": "integer" }, "selected_version": { "type": [ "string", "null" ] }, "spec": { "anyOf": [ { "$ref": "#/definitions/TestSpecProjection" }, { "type": "null" } ] }, "test_id": { "type": "string" } }, "required": [ "available_versions", "lifecycle", "recent_scores", "runs_current", "runs_total", "test_id" ], "type": "object" }, "TestCriterionProjection": { "properties": { "description": { "type": "string" }, "dimension": { "$ref": "#/definitions/EvaluationDimension" }, "gate": { "type": "boolean" }, "id": { "type": "string" }, "kind": { "$ref": "#/definitions/AssessmentKind" }, "label": { "type": [ "string", "null" ] }, "policy": { "$ref": "#/definitions/AssessmentPolicy" }, "weight": { "format": "uint8", "minimum": 0, "type": "integer" } }, "required": [ "description", "dimension", "gate", "id", "kind", "policy", "weight" ], "type": "object" }, "TestDelta": { "properties": { "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "score": { "format": "double", "type": [ "number", "null" ] }, "tokens": { "format": "double", "type": [ "number", "null" ] } }, "type": "object" }, "TestObservation": { "properties": { "agent_profile": { "description": "The agent profile the subject ran with, when it had one.", "type": [ "string", "null" ] }, "assessment_profile_sha256": { "type": "string" }, "assessment_summary": { "$ref": "#/definitions/AssessmentSummary" }, "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "cohort_id": { "type": "string" }, "completed_at": { "type": "string" }, "contract_sha256": { "type": "string" }, "engine_revision": { "type": [ "string", "null" ] }, "evaluated_version_id": { "type": [ "string", "null" ] }, "execution_id": { "type": "string" }, "execution_label": { "description": "That execution's name, when it has one.", "type": [ "string", "null" ] }, "harness_revision": { "type": [ "string", "null" ] }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "median_function_calls": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "median_turns": { "format": "double", "type": [ "number", "null" ] }, "plan_execution_id": { "description": "The execution as the Executions list shows it: the plan execution the run belongs to, when it has one. Only a history page carries it.", "type": [ "string", "null" ] }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "runs": { "items": { "$ref": "#/definitions/HistoryRun" }, "type": "array" }, "scored_runs": { "format": "uint", "minimum": 0, "type": "integer" }, "seed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "stack_mode": { "type": "string" }, "status": { "type": "string" }, "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "system_label": { "type": "string" }, "system_revision": { "type": [ "string", "null" ] }, "system_version_id": { "type": [ "string", "null" ] } }, "required": [ "assessment_profile_sha256", "assessment_summary", "behavior_sha256", "case_id", "cohort_id", "completed_at", "contract_sha256", "execution_id", "run_count", "runs", "scored_runs", "stack_mode", "status", "subject_model", "subject_provider", "system_label" ], "type": "object" }, "TestSideSummary": { "properties": { "assessment_summary": { "$ref": "#/definitions/AssessmentSummary" }, "case_count": { "format": "uint", "minimum": 0, "type": "integer" }, "evaluated_version_id": { "type": "string" }, "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "mean_score": { "format": "double", "type": [ "number", "null" ] }, "median_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "median_duration_seconds": { "format": "double", "type": [ "number", "null" ] }, "median_tokens": { "format": "double", "type": [ "number", "null" ] }, "outcomes": { "$ref": "#/definitions/OutcomeCounts" }, "pass_rate": { "format": "double", "type": [ "number", "null" ] }, "samples": { "$ref": "#/definitions/MetricSamples" }, "scored_runs": { "format": "uint", "minimum": 0, "type": "integer" }, "total_runs": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "assessment_summary", "case_count", "evaluated_version_id", "execution_count", "outcomes", "samples", "scored_runs", "total_runs" ], "type": "object" }, "TestSpecProjection": { "description": "The scenario definition as a reader needs it: what the subject is asked to do, how the result is scored, and the limits it runs under. Projected from the materialized `ScenarioSpec` of the test's current version, so it always describes the contract the dashboard is showing.", "properties": { "area": { "anyOf": [ { "$ref": "#/definitions/Area" }, { "type": "null" } ] }, "criteria": { "items": { "$ref": "#/definitions/TestCriterionProjection" }, "type": "array" }, "denied_functions": { "items": { "type": "string" }, "type": "array" }, "disciplines": { "items": { "$ref": "#/definitions/Discipline" }, "type": "array" }, "execution": { "$ref": "#/definitions/ExecutionPolicy" }, "prompt": { "description": "The prompt as materialized for a representative run: run-scoped paths resolve against a catalog namespace, never a real execution.", "type": "string" }, "protects": { "description": "What the test protects and what breaks when it fails.", "type": [ "string", "null" ] }, "summary": { "description": "The task in plain words; absent when the scenario states none.", "type": [ "string", "null" ] }, "title": { "description": "Human-readable test name; absent on responses from older runners.", "type": [ "string", "null" ] }, "workers": { "description": "The iii workers the test protects; empty when it protects the harness alone.", "items": { "$ref": "#/definitions/Worker" }, "type": "array" } }, "required": [ "criteria", "denied_functions", "execution", "prompt" ], "type": "object" }, "TestVersionResult": { "properties": { "compatibility": { "type": "string" }, "compatibility_reasons": { "items": { "type": "string" }, "type": "array" }, "delta": { "$ref": "#/definitions/TestDelta" }, "from": { "anyOf": [ { "$ref": "#/definitions/TestSideSummary" }, { "type": "null" } ] }, "from_observations": { "items": { "$ref": "#/definitions/TestObservation" }, "type": "array" }, "test_id": { "type": "string" }, "test_version": { "type": "string" }, "to": { "anyOf": [ { "$ref": "#/definitions/TestSideSummary" }, { "type": "null" } ] }, "to_observations": { "items": { "$ref": "#/definitions/TestObservation" }, "type": "array" } }, "required": [ "compatibility", "compatibility_reasons", "delta", "from_observations", "test_id", "test_version", "to_observations" ], "type": "object" }, "VersionDescriptor": { "properties": { "execution_count": { "format": "uint", "minimum": 0, "type": "integer" }, "last_seen": { "type": [ "string", "null" ] }, "observation_count": { "description": "The runs a test history lists: one per execution and case.", "format": "uint", "minimum": 0, "type": "integer" }, "run_count": { "format": "uint", "minimum": 0, "type": "integer" }, "version": { "type": "string" } }, "required": [ "execution_count", "observation_count", "run_count", "version" ], "type": "object" }, "Worker": { "description": "A product worker of the stack, named as in the Registry. `engine` is the exception: it groups the engine's built-in functions and follows iii releases. harness, harness-e2e, fp and provider-* are left out.", "enum": [ "ade", "auth", "browser", "canvas", "compose", "context-manager", "cron", "database", "engine", "http", "ide", "iii-directory", "judge", "llm-router", "pubsub", "queue", "rbac-proxy", "secrets", "session-manager", "state", "storage", "stream" ], "type": "string" }, "WorkerCalls": { "properties": { "calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "worker": { "type": "string" } }, "required": [ "calls", "worker" ], "type": "object" } }, "properties": { "next_cursor": { "type": [ "string", "null" ] }, "revision": { "type": "string" }, "rows": { "items": { "$ref": "#/definitions/TestCatalogRow" }, "type": "array" }, "total": { "format": "uint", "minimum": 0, "type": "integer" } }, "required": [ "revision", "rows", "total" ], "title": "TestsListResponse", "type": "object" } }, { "description": "Read how one suite moves over time on one provider, model and profile (or `any` model or profile, on one line): every series, the stacks of the selected one, and a point per execution measured over its technically valid runs.", "metadata": { "contract": { "capability": "trends-get", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::trends-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "model": { "default": null, "description": "A model, or `any`: the suite's executions on every model, one line.", "type": [ "string", "null" ] }, "profile": { "default": null, "description": "Null or empty: no profile; `any`: every profile, one line; absent: the latest series that fits the rest.", "type": [ "string", "null" ] }, "provider": { "default": null, "description": "Left open (with the provider) when the model is `any`.", "type": [ "string", "null" ] }, "since": { "description": "Only executions started at or after this RFC 3339 instant; the series and the stacks' workers are still read over all time.", "type": [ "string", "null" ] }, "stack": { "default": null, "description": "A stack name, `not_recorded` or `any`; absent, or one the series never ran on: `any`.", "type": [ "string", "null" ] }, "suite": { "default": null, "description": "A suite id, or the sha256 of an unsaved suite.", "type": [ "string", "null" ] }, "until": { "description": "Only executions started at or before this RFC 3339 instant.", "type": [ "string", "null" ] } }, "title": "TrendsRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "StackCount": { "properties": { "executions": { "format": "uint", "minimum": 0, "type": "integer" }, "name": { "type": "string" } }, "required": [ "executions", "name" ], "type": "object" }, "TrendMeasures": { "properties": { "completed": { "format": "uint", "minimum": 0, "type": "integer" }, "duration_ms_mean": { "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "double", "type": [ "number", "null" ] }, "function_calls": { "format": "double", "type": [ "number", "null" ] }, "function_calls_mean": { "format": "double", "type": [ "number", "null" ] }, "input_tokens_mean": { "format": "double", "type": [ "number", "null" ] }, "planned": { "description": "The runs planned (tests × runs per test), else the runs observed: what `completed` counts.", "format": "uint", "minimum": 0, "type": "integer" }, "score_mean": { "format": "double", "type": [ "number", "null" ] }, "turns_mean": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "completed", "planned" ], "type": "object" }, "TrendPoint": { "properties": { "counted": { "description": "Technically valid runs.", "format": "uint", "minimum": 0, "type": "integer" }, "engine": { "type": [ "string", "null" ] }, "execution_id": { "type": "string" }, "label": { "type": [ "string", "null" ] }, "measures": { "anyOf": [ { "$ref": "#/definitions/TrendMeasures" }, { "type": "null" } ], "description": "Null when nothing counted." }, "model": { "type": "string" }, "planned": { "items": { "type": "string" }, "type": [ "array", "null" ] }, "profile": { "type": [ "string", "null" ] }, "provider": { "description": "Its own model and profile: under `any` they differ between points.", "type": "string" }, "reason": { "description": "When nothing counted: the execution's error, else why.", "type": [ "string", "null" ] }, "runs": { "description": "Runs observed.", "format": "uint", "minimum": 0, "type": "integer" }, "source": { "$ref": "#/definitions/TrendSource" }, "stack": { "$ref": "#/definitions/TrendStack" }, "started_at": { "type": "string" }, "tests": { "description": "Planned tests first, in plan order, then any other that ran.", "items": { "$ref": "#/definitions/TrendTest" }, "type": "array" }, "workers": { "additionalProperties": { "type": "string" }, "description": "Identity per worker, runner and engine left out; null: not recorded.", "type": [ "object", "null" ] } }, "required": [ "counted", "execution_id", "model", "provider", "runs", "source", "stack", "started_at", "tests" ], "type": "object" }, "TrendSeries": { "properties": { "executions": { "format": "uint", "minimum": 0, "type": "integer" }, "latest_at": { "type": "string" }, "model": { "type": "string" }, "profile": { "type": [ "string", "null" ] }, "provider": { "type": "string" }, "suite": { "type": "string" }, "suite_label": { "type": "string" }, "where": { "items": { "$ref": "#/definitions/Where" }, "type": "array" } }, "required": [ "executions", "latest_at", "model", "provider", "suite", "suite_label", "where" ], "type": "object" }, "TrendSeriesKey": { "properties": { "model": { "type": "string" }, "profile": { "type": [ "string", "null" ] }, "provider": { "type": "string" }, "suite": { "type": "string" } }, "required": [ "model", "provider", "suite" ], "type": "object" }, "TrendSource": { "properties": { "kind": { "description": "`local`, `docker` or `github`.", "type": "string" }, "release_control_execution_id": { "type": [ "string", "null" ] }, "run_attempt": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "run_id": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "url": { "type": [ "string", "null" ] } }, "required": [ "kind" ], "type": "object" }, "TrendStack": { "properties": { "matched_by_workers": { "description": "Not recorded, joined by its workers' names.", "type": "boolean" }, "name": { "type": [ "string", "null" ] } }, "required": [ "matched_by_workers" ], "type": "object" }, "TrendTest": { "properties": { "behavior_sha256": { "type": [ "string", "null" ] }, "id": { "type": "string" }, "score": { "description": "Mean of the counted runs' scores.", "format": "double", "type": [ "number", "null" ] }, "state": { "description": "`scored`, `no_score`, `technical_invalid` or `not_run`.", "type": "string" } }, "required": [ "id", "state" ], "type": "object" }, "Where": { "description": "Where an execution runs: on this harness, in Docker from this worker, or on GitHub (dispatched from here, or imported).", "enum": [ "harness", "docker", "github" ], "type": "string" } }, "properties": { "points": { "description": "Oldest first.", "items": { "$ref": "#/definitions/TrendPoint" }, "type": "array" }, "selected": { "anyOf": [ { "$ref": "#/definitions/TrendSeriesKey" }, { "type": "null" } ], "description": "The request's series, else the one with the latest execution; its provider and model, or its profile, are `any` when the request left them open." }, "series": { "description": "Every series, latest execution first.", "items": { "$ref": "#/definitions/TrendSeries" }, "type": "array" }, "stack": { "description": "The stack filter applied.", "type": "string" }, "stacks": { "description": "Of the selected series, then `not_recorded` (if any) and `any`.", "items": { "$ref": "#/definitions/StackCount" }, "type": "array" } }, "required": [ "points", "series", "stack", "stacks" ], "title": "TrendsResponse", "type": "object" } }, { "description": "Link to what changed between two builds of iii, the runner or a worker (release tags or commits) on GitHub, with how many commits apart they are when gh can tell.", "metadata": { "contract": { "capability": "version-compare", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::version-compare", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "base": { "description": "A version, or a `@sha7` identity (`*` marks uncommitted changes).", "type": "string" }, "head": { "type": "string" }, "name": { "description": "`iii`, `harness-e2e` or a worker of iii-hq/workers.", "type": "string" } }, "required": [ "base", "head", "name" ], "title": "VersionCompareRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "total_commits": { "description": "Null when `gh` is missing or GitHub answers an error.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "url": { "type": "string" } }, "required": [ "url" ], "title": "VersionCompareResponse", "type": "object" } }, { "description": "What the iii registry resolves a worker to: its newest release and the workers it brings, or the registry's error; kept ten minutes.", "metadata": { "contract": { "capability": "worker-resolve", "name": "harness-e2e-dashboard" }, "internal": true }, "name": "e2e::dashboard::worker-resolve", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "worker": { "description": "A worker's name in the iii registry, e.g. `harness`.", "type": "string" } }, "required": [ "worker" ], "title": "WorkerResolveRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "anyOf": [ { "properties": { "dependencies": { "description": "Every worker its graph brings, engine built-ins left out.", "items": { "type": "string" }, "type": "array" }, "name": { "type": "string" }, "version": { "description": "Its newest release.", "type": "string" } }, "required": [ "dependencies", "name", "version" ], "type": "object" }, { "properties": { "error": { "$ref": "#/definitions/RegistryRefusal" } }, "required": [ "error" ], "type": "object" } ], "definitions": { "RegistryRefusal": { "properties": { "code": { "description": "The registry's code, e.g. `worker_not_found`.", "type": "string" }, "message": { "type": "string" } }, "required": [ "code", "message" ], "type": "object" } }, "description": "A worker's newest release and what it brings, or why the registry has none.", "title": "WorkerResolution" } }, { "description": "Get the immutable result and manifest of an E2E execution.", "metadata": { "contract": { "capabilities": [ "results-get" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::results-get", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ActivationPolicy": { "oneOf": [ { "properties": { "policy": { "enum": [ "always" ], "type": "string" } }, "required": [ "policy" ], "type": "object" }, { "properties": { "conditions": { "items": { "$ref": "#/definitions/BooleanCondition" }, "type": "array" }, "policy": { "enum": [ "all" ], "type": "string" } }, "required": [ "conditions", "policy" ], "type": "object" }, { "properties": { "conditions": { "items": { "$ref": "#/definitions/BooleanCondition" }, "type": "array" }, "policy": { "enum": [ "any" ], "type": "string" } }, "required": [ "conditions", "policy" ], "type": "object" } ] }, "AgentProfileArtifact": { "properties": { "configuration_sha256": { "type": "string" }, "id": { "type": "string" } }, "required": [ "configuration_sha256", "id" ], "type": "object" }, "Area": { "description": "The product capability a test protects; exactly one per test, so a report by capability never counts a test twice.", "enum": [ "core-loop", "data-safety", "tool-adaptation", "scheduling", "coding", "coordination", "self-repair", "long-context", "security", "app-building", "verification" ], "type": "string" }, "ArtifactExpectation": { "properties": { "id": { "type": "string" }, "kind": { "type": "string" }, "max_size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "media_type": { "type": "string" }, "schema": true }, "required": [ "id", "kind", "max_size_bytes", "media_type", "schema" ], "type": "object" }, "ArtifactReference": { "properties": { "id": { "type": "string" }, "kind": { "type": "string" }, "media_type": { "type": "string" }, "path": { "type": "string" }, "sha256": { "type": "string" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "id", "kind", "media_type", "path", "sha256", "size_bytes" ], "type": "object" }, "AssessmentContract": { "additionalProperties": false, "properties": { "runs": { "items": { "$ref": "#/definitions/RunAssessmentContract" }, "type": "array" } }, "required": [ "runs" ], "type": "object" }, "AssessmentKind": { "enum": [ "required_check", "signal", "asset_validation" ], "type": "string" }, "AssessmentOutcome": { "enum": [ "passed", "failed", "partial", "not_evaluated", "unavailable", "error" ], "type": "string" }, "AssessmentPolicy": { "enum": [ "hard_gate", "advisory" ], "type": "string" }, "AssessmentResult": { "properties": { "criterion_id": { "type": "string" }, "dimension": { "$ref": "#/definitions/EvaluationDimension" }, "evidence": { "items": { "$ref": "#/definitions/EvidenceReference" }, "type": "array" }, "kind": { "$ref": "#/definitions/AssessmentKind" }, "outcome": { "$ref": "#/definitions/AssessmentOutcome" }, "policy": { "$ref": "#/definitions/AssessmentPolicy" }, "score": { "anyOf": [ { "$ref": "#/definitions/AssessmentScore" }, { "type": "null" } ] }, "summary": { "type": "string" }, "target": { "$ref": "#/definitions/AssessmentTarget" } }, "required": [ "criterion_id", "dimension", "kind", "outcome", "policy", "summary", "target" ], "type": "object" }, "AssessmentScore": { "properties": { "awarded": { "format": "uint8", "minimum": 0, "type": "integer" }, "possible": { "format": "uint8", "minimum": 0, "type": "integer" } }, "required": [ "awarded", "possible" ], "type": "object" }, "AssessmentTarget": { "properties": { "id": { "type": "string" }, "kind": { "$ref": "#/definitions/AssessmentTargetKind" } }, "required": [ "id", "kind" ], "type": "object" }, "AssessmentTargetKind": { "enum": [ "criterion" ], "type": "string" }, "AssetValidationOutcome": { "enum": [ "valid", "invalid", "malformed", "oversized", "not_produced", "unreadable", "unsafe_path", "removed_during_cleanup", "unexpected", "not_evaluated" ], "type": "string" }, "AssetValidationResult": { "properties": { "asset_id": { "type": "string" }, "evidence": { "items": { "$ref": "#/definitions/EvidenceReference" }, "type": "array" }, "outcome": { "$ref": "#/definitions/AssetValidationOutcome" }, "summary": { "type": "string" } }, "required": [ "asset_id", "outcome", "summary" ], "type": "object" }, "AuditEvidence": { "properties": { "detail": { "description": "Sanitized, bounded description. Never contains secret material.", "type": "string" }, "function_id": { "type": [ "string", "null" ] }, "message_index": { "description": "Index of the transcript message the observation anchors to.", "format": "uint", "minimum": 0, "type": [ "integer", "null" ] } }, "required": [ "detail" ], "type": "object" }, "AuditFlag": { "properties": { "confidence": { "format": "double", "type": [ "number", "null" ] }, "evidence": { "items": { "$ref": "#/definitions/AuditEvidence" }, "type": "array" }, "kind": { "$ref": "#/definitions/AuditFlagKind" }, "severity": { "$ref": "#/definitions/AuditSeverity" }, "source": { "$ref": "#/definitions/AuditFlagSource" }, "summary": { "type": "string" } }, "required": [ "kind", "severity", "source", "summary" ], "type": "object" }, "AuditFlagKind": { "oneOf": [ { "description": "The subject invoked the E2E control plane or another function the scenario explicitly denies — the verification mechanism itself.", "enum": [ "verifier_tampering" ], "type": "string" }, { "description": "The subject addressed a session outside its own session tree.", "enum": [ "out_of_scope_session_access" ], "type": "string" }, { "description": "Live secret material from the runner environment appeared in the transcript.", "enum": [ "secret_exposure" ], "type": "string" }, { "description": "The subject invoked a destructive session or state operation.", "enum": [ "destructive_action" ], "type": "string" } ] }, "AuditFlagSource": { "enum": [ "deterministic" ], "type": "string" }, "AuditReport": { "properties": { "flags": { "items": { "$ref": "#/definitions/AuditFlag" }, "type": "array" } }, "required": [ "flags" ], "type": "object" }, "AuditSeverity": { "enum": [ "info", "warning", "critical" ], "type": "string" }, "BooleanCondition": { "additionalProperties": false, "properties": { "equals": { "type": "boolean" }, "node_id": { "type": "string" }, "port": { "type": "string" } }, "required": [ "equals", "node_id", "port" ], "type": "object" }, "Capability": { "description": "What a scenario needs from the engine, the runner host or a fixture to run. The closed list keeps the catalog honest: a new need is a new variant, and every id is spelled once.", "enum": [ "browser::interactive", "cross_repo_contract_simulator::v1", "curl", "docker", "e2e::adaptive-flow-v1", "e2e::control-plane-v1", "e2e::filesystem", "e2e::git", "e2e::run-scoped-fixtures", "e2e::shell", "e2e::subagents", "e2e::workflow-resume-v1", "fixture::multi-origin-http", "git", "git::deterministic-fixture-v1", "git::offline-bundle", "github::trusted-handoff", "go", "harness::independent_session", "harness::post-turn-validation", "harness::scripted-dialogue-v1", "iii::coder", "iii::compose", "iii::database", "iii::functions", "iii::registry", "iii::shell", "iii::state", "iii::triggers", "iii::workers", "incident_fixture::v1", "node", "playwright", "python3", "release_shadow::read-only-v1", "release_train_simulator::v1" ], "type": "string" }, "CapturedInvariant": { "properties": { "id": { "type": "string" }, "passed": { "type": "boolean" }, "reason": { "type": "string" } }, "required": [ "id", "passed", "reason" ], "type": "object" }, "CompletionState": { "enum": [ "completed", "task_incomplete", "undetermined" ], "type": "string" }, "ContaminationScope": { "enum": [ "none", "case", "execution" ], "type": "string" }, "ControlPlaneEvidence": { "properties": { "functions": { "items": { "$ref": "#/definitions/FunctionContractEvidence" }, "type": "array" } }, "required": [ "functions" ], "type": "object" }, "CostReport": { "properties": { "subject_usd": { "format": "double", "type": [ "number", "null" ] }, "total_usd": { "format": "double", "type": [ "number", "null" ] } }, "type": "object" }, "CriterionReport": { "properties": { "awarded": { "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "description": { "type": [ "string", "null" ] }, "gate": { "default": false, "description": "A gate criterion decides completion: the task counts as completed only when every gate passed. Always written, so a reader can tell \"no gate\" from a report older than the field.", "type": "boolean" }, "gate_minimum": { "description": "The points a gate needs to count as passed when fewer than `possible` already mean the task's purpose was met. Absent: full points.", "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "id": { "type": "string" }, "label": { "description": "The criterion as a short statement; absent in reports older than the field and for criteria that state none. Readers fall back to `description`.", "type": [ "string", "null" ] }, "possible": { "format": "uint8", "minimum": 0, "type": "integer" }, "reason": { "type": "string" } }, "required": [ "id", "possible", "reason" ], "type": "object" }, "DeliverableContentFormat": { "enum": [ "json", "text_utf8" ], "type": "string" }, "DeliverableContract": { "properties": { "artifacts": { "items": { "$ref": "#/definitions/ArtifactExpectation" }, "type": "array" }, "capture_before_cleanup": { "type": "boolean" }, "invariants": { "items": { "$ref": "#/definitions/InvariantSpec" }, "type": "array" }, "provenance_required": { "type": "boolean" } }, "required": [ "artifacts", "capture_before_cleanup", "invariants", "provenance_required" ], "type": "object" }, "DeliverableReport": { "properties": { "artifact": { "anyOf": [ { "$ref": "#/definitions/ArtifactReference" }, { "type": "null" } ] }, "content_format": { "$ref": "#/definitions/DeliverableContentFormat" }, "content_sha256": { "type": "string" }, "content_size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "id": { "type": "string" }, "invariants": { "items": { "$ref": "#/definitions/CapturedInvariant" }, "type": "array" }, "kind": { "type": "string" }, "media_type": { "type": "string" }, "preview": true, "provenance": { "items": { "$ref": "#/definitions/ProvenanceEvidence" }, "type": "array" }, "provenance_valid": { "type": "boolean" }, "schema_valid": { "type": "boolean" }, "screenshots": { "default": [], "description": "The screenshots embedded in the content, so a reader can list them without opening the artifact. Empty when there are none.", "items": { "$ref": "#/definitions/ScreenshotReference" }, "type": "array" } }, "required": [ "content_format", "content_sha256", "content_size_bytes", "id", "invariants", "kind", "media_type", "preview", "provenance", "provenance_valid", "schema_valid" ], "type": "object" }, "DependencyPolicy": { "enum": [ "succeeded", "terminal" ], "type": "string" }, "DimensionReport": { "properties": { "dimension": { "$ref": "#/definitions/EvaluationDimension" }, "passed": { "type": [ "boolean", "null" ] }, "signals": true }, "required": [ "dimension", "signals" ], "type": "object" }, "Discipline": { "description": "The kind of software the subject delivers: a question about the model and the agent, not about iii. `testing` means the subject writes or fixes tests (running a suite does not count); `data` means the deliverable carries a schema, a migration or a persisted data model (ad hoc SQL to finish the task does not count).", "enum": [ "frontend", "backend", "data", "infra", "testing" ], "type": "string" }, "DurableArchiveReference": { "additionalProperties": false, "properties": { "archive_id": { "type": "string" }, "created_at": { "type": "string" }, "execution_id": { "type": "string" }, "expires_at": { "type": [ "string", "null" ] }, "identity_sha256": { "type": "string" }, "manifest": { "$ref": "#/definitions/StorageObjectReference" }, "manifest_backup": { "$ref": "#/definitions/StorageObjectReference" }, "retention_class": { "$ref": "#/definitions/RetentionClass" } }, "required": [ "archive_id", "created_at", "execution_id", "identity_sha256", "manifest", "manifest_backup", "retention_class" ], "type": "object" }, "E2eManifest": { "additionalProperties": false, "properties": { "control_plane": { "$ref": "#/definitions/ControlPlaneEvidence" }, "execution": { "$ref": "#/definitions/ExecutionIdentity" }, "observation_contract": { "anyOf": [ { "$ref": "#/definitions/ObservationRunContract" }, { "type": "null" } ] }, "subject": { "$ref": "#/definitions/ModelArtifact" }, "system_under_test": { "$ref": "#/definitions/SystemUnderTestIdentity" }, "worker_contracts": { "items": { "$ref": "#/definitions/ObservedWorkerContract" }, "type": "array" } }, "required": [ "control_plane", "execution", "subject", "system_under_test" ], "type": "object" }, "E2eObservationEnvelope": { "additionalProperties": false, "properties": { "evidence": { "$ref": "#/definitions/ObservationEvidence" }, "identity": { "$ref": "#/definitions/ObservationIdentity" }, "mode": { "$ref": "#/definitions/ObservationMode" }, "outcome": { "$ref": "#/definitions/ObservationOutcome" }, "provenance": { "$ref": "#/definitions/ObservationProvenance" }, "samples": { "items": { "$ref": "#/definitions/ObservationSample" }, "type": "array" }, "schema": { "type": "string" } }, "required": [ "evidence", "identity", "mode", "outcome", "provenance", "samples", "schema" ], "type": "object" }, "E2eReport": { "additionalProperties": false, "properties": { "assessment_contract": { "$ref": "#/definitions/AssessmentContract" }, "engine_revision": { "type": [ "string", "null" ] }, "execution": { "$ref": "#/definitions/ExecutionIdentity" }, "manifest": { "anyOf": [ { "$ref": "#/definitions/ArtifactReference" }, { "type": "null" } ] }, "objective_outcome": { "$ref": "#/definitions/ObjectiveOutcome" }, "observation_contract": { "anyOf": [ { "$ref": "#/definitions/ObservationRunContract" }, { "type": "null" } ] }, "passed": { "type": "boolean" }, "persistence_errors": { "items": { "type": "string" }, "type": "array" }, "redaction": { "allOf": [ { "$ref": "#/definitions/RedactionReport" } ], "default": { "redacted_fields": 0, "redacted_values": 0, "rules": [] } }, "report_state": { "$ref": "#/definitions/ReportState" }, "result_contract_sha256": { "type": "string" }, "scenarios": { "items": { "$ref": "#/definitions/E2eScenarioReport" }, "type": "array" }, "slot_start_deadline_seconds": { "description": "Soft scheduling deadline: prevents starting slots after this interval.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "subject": { "$ref": "#/definitions/ModelArtifact" }, "system_under_test": { "$ref": "#/definitions/SystemUnderTestIdentity" } }, "required": [ "assessment_contract", "execution", "objective_outcome", "passed", "report_state", "result_contract_sha256", "scenarios", "subject", "system_under_test" ], "type": "object" }, "E2eRunReport": { "properties": { "attempt_id": { "type": "string" }, "attempt_number": { "format": "uint32", "minimum": 0, "type": "integer" }, "audit": { "anyOf": [ { "$ref": "#/definitions/AuditReport" }, { "type": "null" } ], "description": "Advisory behavioral audit over the captured transcript and metrics. Never contributes to score, status, gates, or longitudinal inputs." }, "completion": { "$ref": "#/definitions/CompletionState" }, "cost": { "$ref": "#/definitions/CostReport" }, "criteria": { "items": { "$ref": "#/definitions/CriterionReport" }, "type": "array" }, "deliverables": { "items": { "$ref": "#/definitions/DeliverableReport" }, "type": "array" }, "dimensions": { "items": { "$ref": "#/definitions/DimensionReport" }, "type": "array" }, "efficiency": { "anyOf": [ { "$ref": "#/definitions/EfficiencyReport" }, { "type": "null" } ] }, "evaluators": { "$ref": "#/definitions/EvaluatorStates" }, "evidence": { "items": { "$ref": "#/definitions/ArtifactReference" }, "type": "array" }, "failures": { "items": { "$ref": "#/definitions/FailureRecord" }, "type": "array" }, "metrics": { "anyOf": [ { "$ref": "#/definitions/SessionMetricsPayload" }, { "type": "null" } ] }, "prompt": { "type": "string" }, "retry_attempts": { "items": { "$ref": "#/definitions/RetryAttemptReport" }, "type": "array" }, "run_id": { "type": "string" }, "scenario_flow": { "anyOf": [ { "$ref": "#/definitions/ScenarioFlowEvidence" }, { "type": "null" } ], "description": "Evidence-only flow identity emitted by Rust. This value is never accepted as runner input and cannot reconstruct executable configuration." }, "scenario_measurements": { "items": { "$ref": "#/definitions/ScenarioMeasurement" }, "type": "array" }, "score": { "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "semantic_tests": { "description": "Semantic tests executed inside a code-defined composite scenario. Product requests, polling and captures stay inside these meaningful test units.", "items": { "$ref": "#/definitions/WorkflowStepReport" }, "type": "array" }, "session_id": { "type": "string" }, "status": { "$ref": "#/definitions/RunStatus" }, "technical": { "$ref": "#/definitions/TechnicalState" }, "transcript": true, "wall_time_ms": { "format": "uint64", "minimum": 0, "type": "integer" }, "worker_contracts": { "description": "Exact request/response schemas observed after this scenario's setup. Run-scoped fixture functions cannot be observed in the suite preflight, so attempts retain them here and the suite folds them into the manifest.", "items": { "$ref": "#/definitions/ObservedWorkerContract" }, "type": "array" } }, "required": [ "attempt_id", "attempt_number", "completion", "cost", "criteria", "evaluators", "prompt", "run_id", "session_id", "status", "technical", "wall_time_ms" ], "type": "object" }, "E2eScenarioReport": { "properties": { "aggregate": { "$ref": "#/definitions/ScenarioAggregate" }, "area": { "anyOf": [ { "$ref": "#/definitions/Area" }, { "type": "null" } ] }, "behavior_sha256": { "description": "Digest of the definition the case was materialized from. Absent only when no case could be materialized.", "type": [ "string", "null" ] }, "case": { "anyOf": [ { "$ref": "#/definitions/ScenarioCase" }, { "type": "null" } ] }, "case_id": { "default": "", "type": "string" }, "deferral_reason": { "type": [ "string", "null" ] }, "disciplines": { "items": { "$ref": "#/definitions/Discipline" }, "type": "array" }, "execution_policy": { "$ref": "#/definitions/ExecutionPolicy" }, "passed": { "type": "boolean" }, "protects": { "description": "What the test protects and what breaks when it fails.", "type": [ "string", "null" ] }, "runs": { "items": { "$ref": "#/definitions/E2eRunReport" }, "type": "array" }, "scenario_id": { "type": "string" }, "summary": { "type": [ "string", "null" ] }, "title": { "description": "Presentation captured when this result was produced; absent in older results.", "type": [ "string", "null" ] }, "workers": { "description": "The iii workers the test protects; empty when it protects the harness alone.", "items": { "$ref": "#/definitions/Worker" }, "type": "array" } }, "required": [ "aggregate", "execution_policy", "passed", "runs", "scenario_id" ], "type": "object" }, "EfficiencyReport": { "properties": { "child_sessions": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "child_turns": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "critical_path_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "effective_fan_out": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "function_calls": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "input_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "observed_complexity": { "$ref": "#/definitions/ObservedComplexityReport" }, "observed_work": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "root_turns": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "technical_attempts": { "format": "uint32", "minimum": 0, "type": "integer" }, "total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "transient_resumes": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "unavailable": { "additionalProperties": { "type": "string" }, "type": "object" }, "validation_retries": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_resumes": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wall_time_ms": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "observed_complexity", "technical_attempts", "wall_time_ms" ], "type": "object" }, "EvaluationDimension": { "enum": [ "deliverable", "structural_integrity", "efficiency", "robustness", "e2e_infrastructure" ], "type": "string" }, "EvaluatorAvailability": { "enum": [ "not_required", "pending", "available", "unavailable" ], "type": "string" }, "EvaluatorStates": { "properties": { "completion": { "$ref": "#/definitions/EvaluatorAvailability" } }, "required": [ "completion" ], "type": "object" }, "EvidenceReference": { "properties": { "artifact_id": { "type": "string" }, "artifact_sha256": { "type": "string" }, "locator": { "type": [ "string", "null" ] } }, "required": [ "artifact_id", "artifact_sha256" ], "type": "object" }, "ExecutionIdentity": { "properties": { "completed_at": { "type": "string" }, "execution_id": { "type": "string" }, "lane": { "type": "string" }, "started_at": { "type": "string" } }, "required": [ "completed_at", "execution_id", "lane", "started_at" ], "type": "object" }, "ExecutionPhase": { "enum": [ "requested", "admitted", "preflighting", "materializing", "setting_up", "executing", "collecting", "evaluating", "persisting", "cleaning_up", "finalizing", "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" }, "ExecutionPolicy": { "properties": { "max_output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_total_tokens": { "description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_turns": { "description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "max_validation_retries": { "description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "stuck_timeout_seconds": { "description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "stuck_timeout_seconds" ], "type": "object" }, "ExecutionRealism": { "enum": [ "synthetic", "realistic_simulator", "frozen_real_artifact" ], "type": "string" }, "FailureDomain": { "enum": [ "subject", "resource", "e2e_infrastructure" ], "type": "string" }, "FailurePhase": { "enum": [ "setup", "execute", "collect", "evaluate", "cleanup" ], "type": "string" }, "FailureRecord": { "properties": { "code": { "default": "", "type": "string" }, "contamination": { "allOf": [ { "$ref": "#/definitions/ContaminationScope" } ], "default": "none" }, "domain": { "$ref": "#/definitions/FailureDomain" }, "message": { "type": "string" }, "phase": { "$ref": "#/definitions/FailurePhase" }, "retry_scope": { "allOf": [ { "$ref": "#/definitions/RetryScope" } ], "default": "none" }, "status": { "allOf": [ { "$ref": "#/definitions/RunStatus" } ], "description": "The failure's original classification, independent of the primary run status." } }, "required": [ "domain", "message", "phase", "status" ], "type": "object" }, "FunctionContractEvidence": { "properties": { "function_id": { "type": "string" }, "request_schema": true, "response_schema": true, "sha256": { "type": "string" } }, "required": [ "function_id", "request_schema", "response_schema", "sha256" ], "type": "object" }, "HumanHorizon": { "properties": { "basis": { "allOf": [ { "$ref": "#/definitions/HumanHorizonBasis" } ], "default": "unknown" }, "max_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "min_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "HumanHorizonBasis": { "enum": [ "unknown", "author_estimate", "measured" ], "type": "string" }, "InvariantSpec": { "properties": { "description": { "type": "string" }, "id": { "type": "string" } }, "required": [ "description", "id" ], "type": "object" }, "ModelArtifact": { "properties": { "agent": { "anyOf": [ { "$ref": "#/definitions/AgentProfileArtifact" }, { "type": "null" } ] }, "context_window": { "format": "uint64", "minimum": 0, "type": "integer" }, "max_output_tokens": { "format": "uint64", "minimum": 0, "type": "integer" }, "model": { "type": "string" }, "provider": { "type": "string" }, "supports_tools": { "type": [ "boolean", "null" ] }, "supports_vision": { "type": [ "boolean", "null" ] } }, "required": [ "context_window", "max_output_tokens", "model", "provider" ], "type": "object" }, "ObjectiveOutcome": { "enum": [ "passed", "failed", "inconclusive" ], "type": "string" }, "ObservationCorrelation": { "additionalProperties": false, "properties": { "deployment_id": { "type": "string" }, "operation_id": { "type": "string" }, "system": { "type": "string" } }, "required": [ "deployment_id", "operation_id", "system" ], "type": "object" }, "ObservationDataAvailability": { "enum": [ "complete", "partial", "unavailable" ], "type": "string" }, "ObservationDecision": { "enum": [ "observe_only" ], "type": "string" }, "ObservationEnvironment": { "enum": [ "demonstration" ], "type": "string" }, "ObservationEvidence": { "additionalProperties": false, "properties": { "artifacts": { "items": { "$ref": "#/definitions/ArtifactReference" }, "type": "array" }, "manifest_sha256": { "type": [ "string", "null" ] }, "results_sha256": { "type": [ "string", "null" ] } }, "type": "object" }, "ObservationExecutionIdentity": { "additionalProperties": false, "properties": { "attempt": { "format": "uint32", "minimum": 0, "type": "integer" }, "completed_at": { "type": "string" }, "id": { "type": "string" }, "request_sha256": { "type": "string" }, "run_contract_sha256": { "type": "string" }, "started_at": { "type": "string" } }, "required": [ "attempt", "completed_at", "id", "request_sha256", "run_contract_sha256", "started_at" ], "type": "object" }, "ObservationIdentity": { "additionalProperties": false, "properties": { "execution": { "$ref": "#/definitions/ObservationExecutionIdentity" }, "plan": { "$ref": "#/definitions/ObservationPlanIdentity" }, "runner": { "$ref": "#/definitions/RunnerIdentity" }, "target": { "$ref": "#/definitions/ObservationTargetIdentity" } }, "required": [ "execution", "plan", "runner", "target" ], "type": "object" }, "ObservationMetric": { "additionalProperties": false, "description": "A metric is explicit about its unit and availability so a recorded zero cannot be mistaken for a missing measurement. The legacy `metrics` object remains alongside this additive projection for current consumers.", "properties": { "availability": { "$ref": "#/definitions/ObservationDataAvailability" }, "origin": { "$ref": "#/definitions/ObservationMetricOrigin" }, "unit": { "type": "string" }, "value": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "availability", "origin", "unit" ], "type": "object" }, "ObservationMetricDerivation": { "additionalProperties": false, "properties": { "formula": { "type": "string" }, "formula_version": { "type": "string" }, "metric": { "type": "string" }, "origin": { "$ref": "#/definitions/ObservationMetricOrigin" } }, "required": [ "formula", "formula_version", "metric", "origin" ], "type": "object" }, "ObservationMetricOrigin": { "enum": [ "observed", "derived_from_observed" ], "type": "string" }, "ObservationMode": { "additionalProperties": false, "properties": { "decision": { "$ref": "#/definitions/ObservationDecision" }, "environment": { "$ref": "#/definitions/ObservationEnvironment" } }, "required": [ "decision", "environment" ], "type": "object" }, "ObservationObjective": { "enum": [ "passed", "hard_gate_failed", "technical_failed", "infrastructure_failed", "cancelled", "unsupported_plan" ], "type": "string" }, "ObservationOutcome": { "additionalProperties": false, "properties": { "control_phase": { "type": "string" }, "data_availability": { "$ref": "#/definitions/ObservationDataAvailability" }, "error": { "type": "string" }, "objective": { "$ref": "#/definitions/ObservationObjective" }, "passed": { "type": [ "boolean", "null" ] } }, "required": [ "control_phase", "data_availability", "objective" ], "type": "object" }, "ObservationPlanIdentity": { "additionalProperties": false, "properties": { "catalog_sha256": { "type": "string" }, "id": { "type": "string" }, "revision": { "type": "string" }, "sha256": { "type": "string" } }, "required": [ "catalog_sha256", "id", "revision", "sha256" ], "type": "object" }, "ObservationProvenance": { "additionalProperties": false, "properties": { "subject_model": { "type": "string" }, "subject_provider": { "type": "string" }, "system_under_test": { "anyOf": [ { "$ref": "#/definitions/SystemUnderTestIdentity" }, { "type": "null" } ] } }, "required": [ "subject_model", "subject_provider" ], "type": "object" }, "ObservationRunContract": { "additionalProperties": false, "properties": { "attempt": { "format": "uint32", "minimum": 0, "type": "integer" }, "correlation": { "$ref": "#/definitions/ObservationCorrelation" }, "mode": { "$ref": "#/definitions/ObservationMode" }, "plan": { "$ref": "#/definitions/ObservationPlanIdentity" }, "runner": { "$ref": "#/definitions/RunnerIdentity" }, "selected_cases": { "items": { "$ref": "#/definitions/ObservationSelectedCase" }, "type": "array" }, "target": { "$ref": "#/definitions/ObservationTargetIdentity" } }, "required": [ "attempt", "correlation", "mode", "plan", "runner", "selected_cases", "target" ], "type": "object" }, "ObservationSample": { "additionalProperties": false, "properties": { "attempt_id": { "type": "string" }, "behavior_sha256": { "type": [ "string", "null" ] }, "case_id": { "type": "string" }, "data_availability": { "$ref": "#/definitions/ObservationDataAvailability" }, "derivations": { "items": { "$ref": "#/definitions/ObservationMetricDerivation" }, "type": "array" }, "metric_values": { "additionalProperties": { "$ref": "#/definitions/ObservationMetric" }, "default": {}, "type": "object" }, "metrics": { "anyOf": [ { "$ref": "#/definitions/EfficiencyReport" }, { "type": "null" } ] }, "origin": { "$ref": "#/definitions/ObservationMetricOrigin" }, "run_id": { "type": "string" }, "scenario_id": { "type": "string" }, "seed": { "format": "uint64", "minimum": 0, "type": "integer" }, "status": { "$ref": "#/definitions/RunStatus" } }, "required": [ "attempt_id", "case_id", "data_availability", "origin", "run_id", "scenario_id", "seed", "status" ], "type": "object" }, "ObservationSelectedCase": { "additionalProperties": false, "properties": { "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "contract_sha256": { "type": "string" }, "inputs_sha256": { "type": "string" }, "scenario_id": { "$ref": "#/definitions/ScenarioId" }, "seed": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "behavior_sha256", "case_id", "contract_sha256", "inputs_sha256", "scenario_id", "seed" ], "type": "object" }, "ObservationTargetIdentity": { "additionalProperties": false, "properties": { "application": { "type": "string" }, "stack": { "$ref": "#/definitions/StackIdentity" }, "version": { "type": "string" } }, "required": [ "application", "stack", "version" ], "type": "object" }, "ObservedComplexityReport": { "properties": { "ambiguity_level": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "artifact_count": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "coordination_edges": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "dependency_depth": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "external_systems": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parallel_branches": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "planning_depth": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "state_transitions": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "unavailable": { "additionalProperties": { "type": "string" }, "type": "object" }, "validation_loops": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_cycles": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "ObservedWorkerContract": { "additionalProperties": false, "properties": { "function_id": { "type": "string" }, "request_schema_sha256": { "type": "string" }, "response_schema_sha256": { "type": "string" } }, "required": [ "function_id", "request_schema_sha256", "response_schema_sha256" ], "type": "object" }, "PortValueKind": { "enum": [ "boolean", "json", "text_utf8", "assessment" ], "type": "string" }, "ProvenanceEvidence": { "properties": { "kind": { "type": "string" }, "relation": { "type": "string" }, "source_id": { "type": "string" } }, "required": [ "kind", "relation", "source_id" ], "type": "object" }, "RedactionReport": { "properties": { "redacted_fields": { "format": "uint32", "minimum": 0, "type": "integer" }, "redacted_values": { "format": "uint32", "minimum": 0, "type": "integer" }, "rules": { "items": { "type": "string" }, "type": "array", "uniqueItems": true } }, "required": [ "redacted_fields", "redacted_values", "rules" ], "type": "object" }, "ReportState": { "enum": [ "complete", "partial" ], "type": "string" }, "RetentionClass": { "enum": [ "temporary", "pull_request", "longitudinal", "canonical" ], "type": "string" }, "RetryAttemptReport": { "properties": { "attempt_id": { "type": "string" }, "attempt_number": { "format": "uint32", "minimum": 0, "type": "integer" }, "completion": { "$ref": "#/definitions/CompletionState" }, "cost": { "$ref": "#/definitions/CostReport" }, "deliverables": { "items": { "$ref": "#/definitions/DeliverableReport" }, "type": "array" }, "dimensions": { "items": { "$ref": "#/definitions/DimensionReport" }, "type": "array" }, "efficiency": { "anyOf": [ { "$ref": "#/definitions/EfficiencyReport" }, { "type": "null" } ] }, "evaluators": { "$ref": "#/definitions/EvaluatorStates" }, "evidence": { "items": { "$ref": "#/definitions/ArtifactReference" }, "type": "array" }, "failures": { "items": { "$ref": "#/definitions/FailureRecord" }, "type": "array" }, "metrics": { "anyOf": [ { "$ref": "#/definitions/SessionMetricsPayload" }, { "type": "null" } ] }, "run_id": { "type": "string" }, "scenario_flow": { "anyOf": [ { "$ref": "#/definitions/ScenarioFlowEvidence" }, { "type": "null" } ] }, "score": { "description": "The points the attempt's evaluated criteria awarded, as on the run.", "format": "uint8", "minimum": 0, "type": [ "integer", "null" ] }, "semantic_tests": { "items": { "$ref": "#/definitions/WorkflowStepReport" }, "type": "array" }, "session_id": { "type": "string" }, "status": { "$ref": "#/definitions/RunStatus" }, "technical": { "$ref": "#/definitions/TechnicalState" }, "transcript": true, "wall_time_ms": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "attempt_id", "attempt_number", "completion", "cost", "evaluators", "failures", "run_id", "session_id", "status", "technical", "wall_time_ms" ], "type": "object" }, "RetryScope": { "enum": [ "none", "same_slot" ], "type": "string" }, "RobustnessReport": { "properties": { "cost_per_successful_deliverable": { "format": "double", "type": [ "number", "null" ] }, "deliverable_success_rate": { "format": "double", "type": [ "number", "null" ] }, "eligible": { "type": "boolean" }, "flaky_rate": { "format": "double", "type": [ "number", "null" ] }, "median_wall_time_ms": { "format": "double", "type": [ "number", "null" ] }, "minimum_sample_size": { "format": "uint32", "minimum": 0, "type": "integer" }, "p95_wall_time_ms": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sample_size": { "format": "uint32", "minimum": 0, "type": "integer" }, "structural_integrity_rate": { "format": "double", "type": [ "number", "null" ] }, "tail_minimum_sample_size": { "format": "uint32", "minimum": 0, "type": "integer" }, "technical_failure_rate": { "format": "double", "type": [ "number", "null" ] }, "unavailable": { "additionalProperties": { "type": "string" }, "type": "object" }, "wall_time_variance": { "format": "double", "type": [ "number", "null" ] } }, "required": [ "eligible", "minimum_sample_size", "sample_size", "tail_minimum_sample_size" ], "type": "object" }, "RunAssessmentContract": { "properties": { "assessments": { "items": { "$ref": "#/definitions/AssessmentResult" }, "type": "array" }, "assets": { "items": { "$ref": "#/definitions/AssetValidationResult" }, "type": "array" }, "attempt_id": { "type": "string" }, "run_id": { "type": "string" }, "system_status": { "$ref": "#/definitions/SystemStatus" } }, "required": [ "attempt_id", "run_id", "system_status" ], "type": "object" }, "RunStatus": { "enum": [ "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "RunnerIdentity": { "additionalProperties": false, "properties": { "name": { "type": "string" }, "revision": { "type": "string" }, "version": { "type": "string" } }, "required": [ "name", "revision", "version" ], "type": "object" }, "ScenarioAggregate": { "properties": { "completed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "completion_evidence_coverage": { "format": "double", "type": [ "number", "null" ] }, "completion_rate": { "format": "double", "type": [ "number", "null" ] }, "cost": { "$ref": "#/definitions/CostReport" }, "deferred_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "execution_reliability": { "format": "double", "type": [ "number", "null" ] }, "failed_attempt_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "mean_score": { "description": "Mean score of the technically valid runs that have one; null when none.", "format": "double", "type": [ "number", "null" ] }, "observed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "pass_rate": { "format": "double", "type": "number" }, "passed_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "planned_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "required_passes": { "format": "uint32", "minimum": 0, "type": "integer" }, "robustness": { "$ref": "#/definitions/RobustnessReport" }, "scored_runs": { "description": "Technically valid runs that have a score.", "format": "uint32", "minimum": 0, "type": "integer" }, "task_incomplete_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "technical_failures": { "format": "uint32", "minimum": 0, "type": "integer" }, "technical_invalid_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "technical_valid_runs": { "format": "uint32", "minimum": 0, "type": "integer" }, "tokens_completed_p50": { "format": "double", "type": [ "number", "null" ] }, "tokens_per_completion": { "format": "double", "type": [ "number", "null" ] }, "total_tokens_consumed": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "undetermined_runs": { "format": "uint32", "minimum": 0, "type": "integer" } }, "required": [ "completed_runs", "cost", "deferred_runs", "observed_runs", "pass_rate", "passed_runs", "planned_runs", "required_passes", "robustness", "scored_runs", "task_incomplete_runs", "technical_failures", "technical_invalid_runs", "technical_valid_runs", "undetermined_runs" ], "type": "object" }, "ScenarioCase": { "properties": { "behavior_sha256": { "description": "Digest of the scenario definition this case was materialized from: prompt, execution policy, criteria, and the case-independent contract. Two cases with the same digest were evaluated by the same definition. Sealed by `ScenarioId::materialize`; empty only before sealing.", "type": "string" }, "case_id": { "type": "string" }, "characterization": { "allOf": [ { "$ref": "#/definitions/ScenarioCharacterization" } ], "default": { "human_horizon": { "basis": "unknown" }, "realism": { "execution": "synthetic", "shadow": "none" } } }, "deliverable_contract": { "$ref": "#/definitions/DeliverableContract" }, "inputs": true, "inputs_sha256": { "type": "string" }, "required_capabilities": { "items": { "$ref": "#/definitions/Capability" }, "type": "array" }, "scenario_id": { "type": "string" }, "seed": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "behavior_sha256", "case_id", "deliverable_contract", "inputs", "inputs_sha256", "required_capabilities", "scenario_id", "seed" ], "type": "object" }, "ScenarioCharacterization": { "properties": { "human_horizon": { "allOf": [ { "$ref": "#/definitions/HumanHorizon" } ], "default": { "basis": "unknown" } }, "realism": { "allOf": [ { "$ref": "#/definitions/ScenarioRealism" } ], "default": { "execution": "synthetic", "shadow": "none" } } }, "type": "object" }, "ScenarioFlowEvidence": { "additionalProperties": false, "properties": { "checkpoint": { "$ref": "#/definitions/ArtifactReference" }, "cleanup": { "$ref": "#/definitions/WorkflowCleanupReport" }, "definition_sha256": { "type": "string" }, "snapshot": true }, "required": [ "checkpoint", "cleanup", "definition_sha256", "snapshot" ], "type": "object" }, "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" }, "ScenarioMeasurement": { "additionalProperties": false, "properties": { "id": { "type": "string" }, "origin": { "$ref": "#/definitions/ObservationMetricOrigin" }, "unit": { "type": "string" }, "value": { "format": "double", "type": "number" } }, "required": [ "id", "origin", "unit", "value" ], "type": "object" }, "ScenarioRealism": { "properties": { "execution": { "allOf": [ { "$ref": "#/definitions/ExecutionRealism" } ], "default": "synthetic" }, "shadow": { "allOf": [ { "$ref": "#/definitions/ShadowMode" } ], "default": "none" } }, "type": "object" }, "ScreenshotReference": { "properties": { "caption": { "description": "The capture's caption from its `captures.json`, else the file's path without the attempt folder it was packed under.", "type": "string" }, "media_type": { "type": "string" }, "pointer": { "description": "JSON Pointer to the embedded file inside the deliverable: `/attachments/board-desktop.png`.", "type": "string" }, "sha256": { "type": "string" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "caption", "media_type", "pointer", "sha256", "size_bytes" ], "type": "object" }, "SessionMetricsPayload": { "properties": { "by_session": { "items": { "$ref": "#/definitions/SessionUsage" }, "type": "array" }, "complete": { "type": "boolean" }, "root_session_id": { "type": "string" }, "totals": { "$ref": "#/definitions/SessionUsageTotals" }, "traces": { "default": null } }, "required": [ "by_session", "complete", "root_session_id", "totals" ], "type": "object" }, "SessionUsage": { "properties": { "cache_read_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "context": { "default": null }, "cost_usd": { "default": null, "format": "double", "type": [ "number", "null" ] }, "depth": { "format": "uint32", "minimum": 0, "type": "integer" }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "parent_session_id": { "default": null, "type": [ "string", "null" ] }, "reasoning_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "session_id": { "type": "string" }, "transient_resumes": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" }, "validation_retries": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_resumes": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "required": [ "depth", "function_call_errors", "function_calls", "session_id", "turns" ], "type": "object" }, "SessionUsageTotals": { "properties": { "cache_read_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cache_write_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "cost_usd": { "default": null, "format": "double", "type": [ "number", "null" ] }, "function_call_errors": { "format": "uint64", "minimum": 0, "type": "integer" }, "function_calls": { "format": "uint64", "minimum": 0, "type": "integer" }, "input_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "output_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "reasoning_tokens": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "sessions": { "format": "uint64", "minimum": 0, "type": "integer" }, "transient_resumes": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "turns": { "format": "uint64", "minimum": 0, "type": "integer" }, "validation_retries": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "wake_resumes": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "required": [ "function_call_errors", "function_calls", "sessions", "turns" ], "type": "object" }, "ShadowMode": { "enum": [ "none", "read_only" ], "type": "string" }, "StackIdentity": { "oneOf": [ { "properties": { "mode": { "enum": [ "source" ], "type": "string" }, "workers_repository": { "type": "string" }, "workers_revision": { "type": "string" } }, "required": [ "mode", "workers_repository", "workers_revision" ], "type": "object" }, { "properties": { "mode": { "enum": [ "registry" ], "type": "string" }, "stack_lock_digest": { "type": "string" }, "stack_versions": { "additionalProperties": { "type": "string" }, "type": "object" } }, "required": [ "mode", "stack_lock_digest", "stack_versions" ], "type": "object" } ] }, "StorageObjectReference": { "properties": { "bucket": { "type": "string" }, "key": { "type": "string" }, "media_type": { "type": "string" }, "sha256": { "type": "string" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "uri": { "type": "string" } }, "required": [ "bucket", "key", "media_type", "sha256", "size_bytes", "uri" ], "type": "object" }, "SystemStatus": { "enum": [ "unavailable", "passed", "hard_gate_failed", "subject_error", "resource_limit", "infrastructure_error" ], "type": "string" }, "SystemUnderTestIdentity": { "properties": { "contract_hashes": { "additionalProperties": { "type": "string" }, "type": "object" }, "e2e_repository": { "type": "string" }, "e2e_revision": { "type": "string" }, "engine_revision": { "type": [ "string", "null" ] }, "engine_version": { "type": "string" }, "harness_version": { "type": "string" }, "stack": { "$ref": "#/definitions/StackIdentity" } }, "required": [ "contract_hashes", "e2e_repository", "e2e_revision", "engine_version", "harness_version", "stack" ], "type": "object" }, "TechnicalState": { "enum": [ "valid", "technical_invalid" ], "type": "string" }, "TypedPortValue": { "additionalProperties": false, "properties": { "kind": { "$ref": "#/definitions/PortValueKind" }, "value": true }, "required": [ "kind", "value" ], "type": "object" }, "Worker": { "description": "A product worker of the stack, named as in the Registry. `engine` is the exception: it groups the engine's built-in functions and follows iii releases. harness, harness-e2e, fp and provider-* are left out.", "enum": [ "ade", "auth", "browser", "canvas", "compose", "context-manager", "cron", "database", "engine", "http", "ide", "iii-directory", "judge", "llm-router", "pubsub", "queue", "rbac-proxy", "secrets", "session-manager", "state", "storage", "stream" ], "type": "string" }, "WorkflowAssetReport": { "additionalProperties": false, "properties": { "artifact": { "$ref": "#/definitions/ArtifactReference" }, "content_sha256": { "type": "string" }, "id": { "type": "string" }, "kind": { "type": "string" }, "media_type": { "type": "string" }, "namespaced_id": { "type": "string" }, "preview": true, "preview_truncated": { "type": "boolean" }, "size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "artifact", "content_sha256", "id", "kind", "media_type", "namespaced_id", "preview", "preview_truncated", "size_bytes" ], "type": "object" }, "WorkflowCleanupReport": { "additionalProperties": false, "properties": { "duration_ms": { "format": "uint64", "minimum": 0, "type": "integer" }, "failure": { "type": [ "string", "null" ] }, "status": { "$ref": "#/definitions/WorkflowCleanupStatus" } }, "required": [ "duration_ms", "status" ], "type": "object" }, "WorkflowCleanupStatus": { "enum": [ "succeeded", "failed" ], "type": "string" }, "WorkflowEvaluationOutcome": { "enum": [ "passed", "failed", "advisory", "not_evaluated" ], "type": "string" }, "WorkflowEvaluationResult": { "additionalProperties": false, "properties": { "evidence_ids": { "default": [], "items": { "type": "string" }, "type": "array" }, "id": { "type": "string" }, "outcome": { "$ref": "#/definitions/WorkflowEvaluationOutcome" }, "score": { "format": "double", "type": [ "number", "null" ] }, "summary": { "type": "string" } }, "required": [ "id", "outcome", "summary" ], "type": "object" }, "WorkflowFailurePhase": { "enum": [ "preflight", "execute", "capture", "persist", "evaluate", "cleanup", "cancel" ], "type": "string" }, "WorkflowGateResult": { "additionalProperties": false, "properties": { "evidence_ids": { "default": [], "items": { "type": "string" }, "type": "array" }, "id": { "type": "string" }, "passed": { "type": "boolean" }, "reason": { "type": "string" } }, "required": [ "id", "passed", "reason" ], "type": "object" }, "WorkflowStepFailure": { "additionalProperties": false, "properties": { "message": { "type": "string" }, "phase": { "$ref": "#/definitions/WorkflowFailurePhase" }, "technical": { "type": "boolean" } }, "required": [ "message", "phase", "technical" ], "type": "object" }, "WorkflowStepReport": { "additionalProperties": false, "properties": { "activation": { "$ref": "#/definitions/ActivationPolicy" }, "assets": { "items": { "$ref": "#/definitions/WorkflowAssetReport" }, "type": "array" }, "completed_at": { "type": [ "string", "null" ] }, "cost_usd": { "format": "double", "type": [ "number", "null" ] }, "dependencies": { "items": { "type": "string" }, "type": "array" }, "dependency_policy": { "$ref": "#/definitions/DependencyPolicy" }, "duration_ms": { "format": "uint64", "minimum": 0, "type": "integer" }, "evaluations": { "items": { "$ref": "#/definitions/WorkflowEvaluationResult" }, "type": "array" }, "failures": { "items": { "$ref": "#/definitions/WorkflowStepFailure" }, "type": "array" }, "hard_gates": { "items": { "$ref": "#/definitions/WorkflowGateResult" }, "type": "array" }, "harness_session_id": { "type": [ "string", "null" ] }, "metrics": true, "node_id": { "type": "string" }, "outputs": { "additionalProperties": { "$ref": "#/definitions/TypedPortValue" }, "type": "object" }, "redaction": { "allOf": [ { "$ref": "#/definitions/RedactionReport" } ], "default": { "redacted_fields": 0, "redacted_values": 0, "rules": [] } }, "required": { "type": "boolean" }, "skip_reason": { "type": [ "string", "null" ] }, "started_at": { "type": [ "string", "null" ] }, "status": { "$ref": "#/definitions/WorkflowStepStatus" }, "step_type": { "type": "string" }, "transcript": true }, "required": [ "activation", "dependencies", "dependency_policy", "duration_ms", "node_id", "required", "status", "step_type" ], "type": "object" }, "WorkflowStepStatus": { "enum": [ "pending", "running", "succeeded", "hard_gate_failed", "failed", "skipped", "cancelled" ], "type": "string" } }, "properties": { "archive": { "anyOf": [ { "$ref": "#/definitions/DurableArchiveReference" }, { "type": "null" } ] }, "execution_id": { "type": "string" }, "manifest": { "anyOf": [ { "$ref": "#/definitions/E2eManifest" }, { "type": "null" } ] }, "observation": { "anyOf": [ { "$ref": "#/definitions/E2eObservationEnvelope" }, { "type": "null" } ] }, "observation_artifact": { "anyOf": [ { "$ref": "#/definitions/ArtifactReference" }, { "type": "null" } ] }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "report": { "anyOf": [ { "$ref": "#/definitions/E2eReport" }, { "type": "null" } ] }, "request_sha256": { "type": "string" }, "result_path": { "type": [ "string", "null" ] }, "run_contract_sha256": { "type": [ "string", "null" ] } }, "required": [ "execution_id", "phase", "request_sha256" ], "title": "ResultsGetResponse", "type": "object" } }, { "description": "List durable E2E executions using comparable identity filters.", "metadata": { "contract": { "capabilities": [ "results-list" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::results-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" } }, "properties": { "lane": { "default": null, "type": [ "string", "null" ] }, "limit": { "default": 50, "format": "uint16", "minimum": 0, "type": "integer" }, "scenario_id": { "anyOf": [ { "$ref": "#/definitions/ScenarioId" }, { "type": "null" } ], "default": null } }, "title": "ResultsListRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ActiveAttempt": { "properties": { "attempt_id": { "type": "string" }, "run_id": { "default": "", "type": "string" }, "scenario_id": { "$ref": "#/definitions/ScenarioId" }, "session_id": { "type": "string" } }, "required": [ "attempt_id", "scenario_id", "session_id" ], "type": "object" }, "ExecutionPhase": { "enum": [ "requested", "admitted", "preflighting", "materializing", "setting_up", "executing", "collecting", "evaluating", "persisting", "cleaning_up", "finalizing", "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" }, "JournalProgress": { "properties": { "active_attempt_id": { "type": [ "string", "null" ] }, "attempts_finished": { "format": "uint64", "minimum": 0, "type": "integer" }, "attempts_started": { "format": "uint64", "minimum": 0, "type": "integer" }, "committed_events": { "format": "uint64", "minimum": 0, "type": "integer" }, "last_event_sha256": { "type": [ "string", "null" ] }, "phase": { "type": [ "string", "null" ] }, "planned_slots": { "format": "uint64", "minimum": 0, "type": "integer" }, "runs_committed": { "format": "uint64", "minimum": 0, "type": "integer" }, "slots_deferred": { "format": "uint64", "minimum": 0, "type": "integer" }, "subject_observations_committed": { "format": "uint64", "minimum": 0, "type": "integer" }, "terminal": { "type": "boolean" }, "terminal_reason": { "type": [ "string", "null" ] }, "terminal_state": { "anyOf": [ { "$ref": "#/definitions/JournalTerminalState" }, { "type": "null" } ] } }, "required": [ "attempts_finished", "attempts_started", "committed_events", "planned_slots", "runs_committed", "slots_deferred", "subject_observations_committed", "terminal" ], "type": "object" }, "JournalTerminalState": { "enum": [ "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" }, "PhaseTransition": { "properties": { "at": { "type": "string" }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "reason": { "type": "string" } }, "required": [ "at", "phase", "reason" ], "type": "object" }, "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" }, "StatusResponse": { "properties": { "active_attempt": { "anyOf": [ { "$ref": "#/definitions/ActiveAttempt" }, { "type": "null" } ] }, "cancel_requested": { "type": "boolean" }, "error": { "type": "string" }, "execution_id": { "type": "string" }, "journal_progress": { "$ref": "#/definitions/JournalProgress" }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "terminal": { "type": "boolean" }, "transitions": { "items": { "$ref": "#/definitions/PhaseTransition" }, "type": "array" }, "updated_at": { "type": "string" } }, "required": [ "cancel_requested", "execution_id", "journal_progress", "phase", "terminal", "transitions", "updated_at" ], "type": "object" } }, "properties": { "executions": { "items": { "$ref": "#/definitions/StatusResponse" }, "type": "array" } }, "required": [ "executions" ], "title": "ResultsListResponse", "type": "object" } }, { "description": "Admit an asynchronous, idempotent E2E execution.", "metadata": { "contract": { "capabilities": [ "run" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::run", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "definitions": { "ObservationCorrelation": { "additionalProperties": false, "properties": { "deployment_id": { "type": "string" }, "operation_id": { "type": "string" }, "system": { "type": "string" } }, "required": [ "deployment_id", "operation_id", "system" ], "type": "object" }, "ObservationDecision": { "enum": [ "observe_only" ], "type": "string" }, "ObservationEnvironment": { "enum": [ "demonstration" ], "type": "string" }, "ObservationMode": { "additionalProperties": false, "properties": { "decision": { "$ref": "#/definitions/ObservationDecision" }, "environment": { "$ref": "#/definitions/ObservationEnvironment" } }, "required": [ "decision", "environment" ], "type": "object" }, "ObservationPlanIdentity": { "additionalProperties": false, "properties": { "catalog_sha256": { "type": "string" }, "id": { "type": "string" }, "revision": { "type": "string" }, "sha256": { "type": "string" } }, "required": [ "catalog_sha256", "id", "revision", "sha256" ], "type": "object" }, "ObservationRunContract": { "additionalProperties": false, "properties": { "attempt": { "format": "uint32", "minimum": 0, "type": "integer" }, "correlation": { "$ref": "#/definitions/ObservationCorrelation" }, "mode": { "$ref": "#/definitions/ObservationMode" }, "plan": { "$ref": "#/definitions/ObservationPlanIdentity" }, "runner": { "$ref": "#/definitions/RunnerIdentity" }, "selected_cases": { "items": { "$ref": "#/definitions/ObservationSelectedCase" }, "type": "array" }, "target": { "$ref": "#/definitions/ObservationTargetIdentity" } }, "required": [ "attempt", "correlation", "mode", "plan", "runner", "selected_cases", "target" ], "type": "object" }, "ObservationSelectedCase": { "additionalProperties": false, "properties": { "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "contract_sha256": { "type": "string" }, "inputs_sha256": { "type": "string" }, "scenario_id": { "$ref": "#/definitions/ScenarioId" }, "seed": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "behavior_sha256", "case_id", "contract_sha256", "inputs_sha256", "scenario_id", "seed" ], "type": "object" }, "ObservationTargetIdentity": { "additionalProperties": false, "properties": { "application": { "type": "string" }, "stack": { "$ref": "#/definitions/StackIdentity" }, "version": { "type": "string" } }, "required": [ "application", "stack", "version" ], "type": "object" }, "RunnerIdentity": { "additionalProperties": false, "properties": { "name": { "type": "string" }, "revision": { "type": "string" }, "version": { "type": "string" } }, "required": [ "name", "revision", "version" ], "type": "object" }, "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" }, "StackIdentity": { "oneOf": [ { "properties": { "mode": { "enum": [ "source" ], "type": "string" }, "workers_repository": { "type": "string" }, "workers_revision": { "type": "string" } }, "required": [ "mode", "workers_repository", "workers_revision" ], "type": "object" }, { "properties": { "mode": { "enum": [ "registry" ], "type": "string" }, "stack_lock_digest": { "type": "string" }, "stack_versions": { "additionalProperties": { "type": "string" }, "type": "object" } }, "required": [ "mode", "stack_lock_digest", "stack_versions" ], "type": "object" } ] } }, "properties": { "agent": { "type": [ "string", "null" ] }, "idempotency_key": { "type": "string" }, "label": { "default": "", "type": "string" }, "lane": { "default": "local", "type": "string" }, "model": { "type": "string" }, "progress_interval_seconds": { "default": 15, "format": "uint64", "minimum": 0, "type": "integer" }, "provider": { "type": "string" }, "rotating_seeds": { "default": [], "items": { "format": "uint64", "minimum": 0, "type": "integer" }, "type": "array" }, "run_contract": { "anyOf": [ { "$ref": "#/definitions/ObservationRunContract" }, { "type": "null" } ] }, "runs": { "default": 1, "format": "uint32", "minimum": 0, "type": "integer" }, "scenarios": { "default": [], "items": { "$ref": "#/definitions/ScenarioId" }, "type": "array" }, "seed": { "default": null, "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "slot_start_deadline_seconds": { "description": "Soft scheduling budget, resolved before request hashing/admission.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "technical_retries": { "default": 1, "format": "uint8", "minimum": 0, "type": "integer" } }, "required": [ "idempotency_key", "model", "provider" ], "title": "RunRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ExecutionPhase": { "enum": [ "requested", "admitted", "preflighting", "materializing", "setting_up", "executing", "collecting", "evaluating", "persisting", "cleaning_up", "finalizing", "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" } }, "properties": { "duplicate": { "type": "boolean" }, "execution_id": { "type": "string" }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "request_sha256": { "type": "string" }, "run_contract_sha256": { "type": [ "string", "null" ] } }, "required": [ "duplicate", "execution_id", "phase", "request_sha256" ], "title": "RunAccepted", "type": "object" } }, { "description": "List every built-in E2E test with its immutable contract, capabilities, complexity, and materialized case.", "metadata": { "agent": { "audience": "harness", "category": "e2e_local_test_authoring", "operation": "list_tests", "starts_execution": false, "writes_local_state": false }, "contract": { "capabilities": [ "scenarios-list" ], "name": "e2e-control-plane" }, "internal": false }, "name": "e2e::scenarios-list", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "seed": { "default": null, "description": "Optional deterministic seed used to materialize each listed test case. Omit it to use each scenario's canonical seed.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] } }, "title": "ScenariosListRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Area": { "description": "The product capability a test protects; exactly one per test, so a report by capability never counts a test twice.", "enum": [ "core-loop", "data-safety", "tool-adaptation", "scheduling", "coding", "coordination", "self-repair", "long-context", "security", "app-building", "verification" ], "type": "string" }, "ArtifactExpectation": { "properties": { "id": { "type": "string" }, "kind": { "type": "string" }, "max_size_bytes": { "format": "uint64", "minimum": 0, "type": "integer" }, "media_type": { "type": "string" }, "schema": true }, "required": [ "id", "kind", "max_size_bytes", "media_type", "schema" ], "type": "object" }, "Capability": { "description": "What a scenario needs from the engine, the runner host or a fixture to run. The closed list keeps the catalog honest: a new need is a new variant, and every id is spelled once.", "enum": [ "browser::interactive", "cross_repo_contract_simulator::v1", "curl", "docker", "e2e::adaptive-flow-v1", "e2e::control-plane-v1", "e2e::filesystem", "e2e::git", "e2e::run-scoped-fixtures", "e2e::shell", "e2e::subagents", "e2e::workflow-resume-v1", "fixture::multi-origin-http", "git", "git::deterministic-fixture-v1", "git::offline-bundle", "github::trusted-handoff", "go", "harness::independent_session", "harness::post-turn-validation", "harness::scripted-dialogue-v1", "iii::coder", "iii::compose", "iii::database", "iii::functions", "iii::registry", "iii::shell", "iii::state", "iii::triggers", "iii::workers", "incident_fixture::v1", "node", "playwright", "python3", "release_shadow::read-only-v1", "release_train_simulator::v1" ], "type": "string" }, "DeliverableContract": { "properties": { "artifacts": { "items": { "$ref": "#/definitions/ArtifactExpectation" }, "type": "array" }, "capture_before_cleanup": { "type": "boolean" }, "invariants": { "items": { "$ref": "#/definitions/InvariantSpec" }, "type": "array" }, "provenance_required": { "type": "boolean" } }, "required": [ "artifacts", "capture_before_cleanup", "invariants", "provenance_required" ], "type": "object" }, "Discipline": { "description": "The kind of software the subject delivers: a question about the model and the agent, not about iii. `testing` means the subject writes or fixes tests (running a suite does not count); `data` means the deliverable carries a schema, a migration or a persisted data model (ad hoc SQL to finish the task does not count).", "enum": [ "frontend", "backend", "data", "infra", "testing" ], "type": "string" }, "ExecutionPolicy": { "properties": { "max_output_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_total_tokens": { "description": "Shared Harness token budget. `None` leaves the Harness budget unbounded for this scenario.", "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "max_turns": { "description": "Subject turn ceiling. `None` leaves the run without one: only `stuck_timeout_seconds` stops it, and lane admission does not count it.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "max_validation_retries": { "description": "Optional per-session cap for post-turn validation denials. `None` preserves the Harness-configured default.", "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "stuck_timeout_seconds": { "description": "Stop only after this many seconds without observable useful progress. Large scenarios have no fixed wall-clock deadline.", "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "stuck_timeout_seconds" ], "type": "object" }, "ExecutionRealism": { "enum": [ "synthetic", "realistic_simulator", "frozen_real_artifact" ], "type": "string" }, "HumanHorizon": { "properties": { "basis": { "allOf": [ { "$ref": "#/definitions/HumanHorizonBasis" } ], "default": "unknown" }, "max_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] }, "min_minutes": { "format": "uint32", "minimum": 0, "type": [ "integer", "null" ] } }, "type": "object" }, "HumanHorizonBasis": { "enum": [ "unknown", "author_estimate", "measured" ], "type": "string" }, "InvariantSpec": { "properties": { "description": { "type": "string" }, "id": { "type": "string" } }, "required": [ "description", "id" ], "type": "object" }, "RunnerIdentity": { "additionalProperties": false, "properties": { "name": { "type": "string" }, "revision": { "type": "string" }, "version": { "type": "string" } }, "required": [ "name", "revision", "version" ], "type": "object" }, "ScenarioCharacterization": { "properties": { "human_horizon": { "allOf": [ { "$ref": "#/definitions/HumanHorizon" } ], "default": { "basis": "unknown" } }, "realism": { "allOf": [ { "$ref": "#/definitions/ScenarioRealism" } ], "default": { "execution": "synthetic", "shadow": "none" } } }, "type": "object" }, "ScenarioDescriptor": { "properties": { "area": { "anyOf": [ { "$ref": "#/definitions/Area" }, { "type": "null" } ] }, "behavior_sha256": { "type": "string" }, "case_id": { "type": "string" }, "characterization": { "$ref": "#/definitions/ScenarioCharacterization" }, "contract_sha256": { "type": "string" }, "deliverable_contract": { "$ref": "#/definitions/DeliverableContract" }, "disciplines": { "items": { "$ref": "#/definitions/Discipline" }, "type": "array" }, "inputs_sha256": { "type": "string" }, "protects": { "type": [ "string", "null" ] }, "required_capabilities": { "items": { "$ref": "#/definitions/Capability" }, "type": "array" }, "resource_envelope": { "$ref": "#/definitions/ScenarioResourceEnvelope" }, "scenario_id": { "$ref": "#/definitions/ScenarioId" }, "seed": { "format": "uint64", "minimum": 0, "type": "integer" }, "summary": { "type": [ "string", "null" ] }, "title": { "type": [ "string", "null" ] }, "workers": { "description": "The iii workers the test protects; empty when it protects the harness alone.", "items": { "$ref": "#/definitions/Worker" }, "type": "array" } }, "required": [ "behavior_sha256", "case_id", "characterization", "contract_sha256", "deliverable_contract", "inputs_sha256", "required_capabilities", "resource_envelope", "scenario_id", "seed" ], "type": "object" }, "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" }, "ScenarioRealism": { "properties": { "execution": { "allOf": [ { "$ref": "#/definitions/ExecutionRealism" } ], "default": "synthetic" }, "shadow": { "allOf": [ { "$ref": "#/definitions/ShadowMode" } ], "default": "none" } }, "type": "object" }, "ScenarioResourceEnvelope": { "properties": { "execution": { "$ref": "#/definitions/ExecutionPolicy" }, "workflow": { "anyOf": [ { "$ref": "#/definitions/WorkflowResourceEnvelope" }, { "type": "null" } ] } }, "required": [ "execution" ], "type": "object" }, "ShadowMode": { "enum": [ "none", "read_only" ], "type": "string" }, "Worker": { "description": "A product worker of the stack, named as in the Registry. `engine` is the exception: it groups the engine's built-in functions and follows iii releases. harness, harness-e2e, fp and provider-* are left out.", "enum": [ "ade", "auth", "browser", "canvas", "compose", "context-manager", "cron", "database", "engine", "http", "ide", "iii-directory", "judge", "llm-router", "pubsub", "queue", "rbac-proxy", "secrets", "session-manager", "state", "storage", "stream" ], "type": "string" }, "WorkflowResourceEnvelope": { "properties": { "max_cost_usd": { "format": "double", "type": [ "number", "null" ] }, "max_nodes": { "format": "uint16", "minimum": 0, "type": "integer" }, "max_parallel": { "format": "uint16", "minimum": 0, "type": "integer" }, "max_total_tokens": { "format": "uint64", "minimum": 0, "type": [ "integer", "null" ] }, "step_timeout_seconds": { "format": "uint64", "minimum": 0, "type": "integer" }, "technical_retries": { "format": "uint8", "minimum": 0, "type": "integer" }, "workflow_timeout_seconds": { "format": "uint64", "minimum": 0, "type": "integer" } }, "required": [ "max_nodes", "max_parallel", "step_timeout_seconds", "technical_retries", "workflow_timeout_seconds" ], "type": "object" } }, "properties": { "catalog_sha256": { "type": "string" }, "runner": { "$ref": "#/definitions/RunnerIdentity" }, "scenarios": { "items": { "$ref": "#/definitions/ScenarioDescriptor" }, "type": "array" }, "schema": { "type": "string" } }, "required": [ "catalog_sha256", "runner", "scenarios", "schema" ], "title": "ScenariosListResponse", "type": "object" } }, { "description": "Read the durable phase and progress of an E2E execution.", "metadata": { "contract": { "capabilities": [ "status" ], "name": "e2e-control-plane" }, "internal": true }, "name": "e2e::status", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "execution_id": { "type": "string" } }, "required": [ "execution_id" ], "title": "ExecutionRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "ActiveAttempt": { "properties": { "attempt_id": { "type": "string" }, "run_id": { "default": "", "type": "string" }, "scenario_id": { "$ref": "#/definitions/ScenarioId" }, "session_id": { "type": "string" } }, "required": [ "attempt_id", "scenario_id", "session_id" ], "type": "object" }, "ExecutionPhase": { "enum": [ "requested", "admitted", "preflighting", "materializing", "setting_up", "executing", "collecting", "evaluating", "persisting", "cleaning_up", "finalizing", "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" }, "JournalProgress": { "properties": { "active_attempt_id": { "type": [ "string", "null" ] }, "attempts_finished": { "format": "uint64", "minimum": 0, "type": "integer" }, "attempts_started": { "format": "uint64", "minimum": 0, "type": "integer" }, "committed_events": { "format": "uint64", "minimum": 0, "type": "integer" }, "last_event_sha256": { "type": [ "string", "null" ] }, "phase": { "type": [ "string", "null" ] }, "planned_slots": { "format": "uint64", "minimum": 0, "type": "integer" }, "runs_committed": { "format": "uint64", "minimum": 0, "type": "integer" }, "slots_deferred": { "format": "uint64", "minimum": 0, "type": "integer" }, "subject_observations_committed": { "format": "uint64", "minimum": 0, "type": "integer" }, "terminal": { "type": "boolean" }, "terminal_reason": { "type": [ "string", "null" ] }, "terminal_state": { "anyOf": [ { "$ref": "#/definitions/JournalTerminalState" }, { "type": "null" } ] } }, "required": [ "attempts_finished", "attempts_started", "committed_events", "planned_slots", "runs_committed", "slots_deferred", "subject_observations_committed", "terminal" ], "type": "object" }, "JournalTerminalState": { "enum": [ "completed", "failed", "cancelled", "needs_reconciliation", "unsupported" ], "type": "string" }, "PhaseTransition": { "properties": { "at": { "type": "string" }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "reason": { "type": "string" } }, "required": [ "at", "phase", "reason" ], "type": "object" }, "ScenarioId": { "enum": [ "registry_planning", "registry_implementation", "registry_environment", "registry_verification", "kanban_c1_foundation", "kanban_c2_persistence", "kanban_c3_board", "kanban_c4_ticket_flow", "kanban_c5_edit_move", "kanban_c6_discussion", "kanban_c7_live", "linkly_tutorial", "context_pressure", "minimal_path", "persistent_state", "insert_record", "sequential_pipeline", "database_migration_recovery", "shell_coder_sandbox", "research_pipeline", "fanout_ladder", "incident_response", "todo_worker_simple", "todo_worker_planned", "engineering_endurance_ladder", "git_regression_forensics", "alertmanager_route_match", "mechanical_reaction", "timer_wake", "receiving_operation", "validation_loop", "subagent_validation", "subagent_validation_failure", "validation_self_repair", "validation_scope_enforcement", "validation_chain", "secret_hygiene", "prompt_injection_resilience", "moving_target", "poison_message", "cleanup_under_failure", "depth_ladder", "quorum_fan_in", "contention_ledger", "wake_chain_soak", "chess_engine_build", "form_flow_build", "state_machine_canvas_build", "chess_play_ladder", "trend_blog", "trending_topics_build", "typescript_chat_service", "tool_contract_recovery", "policy_bound_action", "cross_app_transaction", "performance_regression", "browser_cross_site", "release_train_recovery", "cross_repo_contract_migration" ], "type": "string" } }, "properties": { "active_attempt": { "anyOf": [ { "$ref": "#/definitions/ActiveAttempt" }, { "type": "null" } ] }, "cancel_requested": { "type": "boolean" }, "error": { "type": "string" }, "execution_id": { "type": "string" }, "journal_progress": { "$ref": "#/definitions/JournalProgress" }, "phase": { "$ref": "#/definitions/ExecutionPhase" }, "terminal": { "type": "boolean" }, "transitions": { "items": { "$ref": "#/definitions/PhaseTransition" }, "type": "array" }, "updated_at": { "type": "string" } }, "required": [ "cancel_requested", "execution_id", "journal_progress", "phase", "terminal", "transitions", "updated_at" ], "title": "StatusResponse", "type": "object" } }, { "description": "Serve the Harness E2E injectable Console assets.", "metadata": { "internal": true }, "name": "e2e::ui-content", "request_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "path": { "type": "string" } }, "required": [ "path" ], "title": "UiContentRequest", "type": "object" }, "response_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "properties": { "content": { "type": "string" }, "content_type": { "type": "string" } }, "required": [ "content", "content_type" ], "title": "UiContentResponse", "type": "object" } } ], "triggers": [ { "description": "A local E2E execution started, produced progress, or reached a terminal state.", "invocation_schema": { "$schema": "http://json-schema.org/draft-07/schema#", "additionalProperties": false, "title": "EventBindingConfig", "type": "object" }, "metadata": {}, "name": "e2e::dashboard::changed", "return_schema": {} } ] } ```