sentinel

v0.1.1

Error monitor for the iii engine: groups failures from trace and log telemetry by deterministic fingerprint, freezes the evidence before the in-memory store discards it, and opens assisted harness investigations that record a structured diagnosis.

license: Apache-2.0Published 3d agodiagnostics errors incidents monitoring observability

Install

bash

iii trigger compose::add worker=sentinel

Run in your project with the iii engine running. Exact versions are immutable and digest-pinned.

Functions25

sentinel::diagnoses::list

Function

List every diagnosis recorded against a group, newest first. Each recording is a version and none are overwritten, so the same failure diagnosed twice can be compared.

Request

group_idstringrequired
limitinteger· uint32min 0
offsetinteger· uint32min 0

Response

diagnosesobject[]required
Newest first: the head is the one in force.
- created_msinteger· int64required
- diagnosisany of
  any of (2)
  Variant 1
  categoryone ofrequired
  Show 3 nestedHide nested
  one of (3)
  Variant 1
  valuestringenum bug, configuration, dependency, unknown
  Variant 2
  valuestringenum transient
  Variant 3
  valuestringenum expected
  confidencestringrequiredenum high, medium, low
  missing_evidencestring[]
  What would have raised the confidence. An honest `low` with this filled in is a better answer than a guess.
  proposed_fixany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  descriptionstringrequired
  filesstring[]required
  riskstringrequiredenum low, medium, high
  stepsstring[]required
  Variant 2
  valuenull
  related_groupsstring[]
  Groups the agent believes share this cause. A suggestion the console shows; nothing is merged on the strength of it.
  reproductionstring
  root_causeobjectrequired
  Show 2 nestedHide nested
  descriptionstringrequired
  evidenceobject[]required
  Show 6 nestedHide nested
  excerptstringrequired
  At most 400 characters of the thing itself.
  kindone ofrequired
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum trace, log
  Variant 2
  valuestringenum code
  lineinteger· uint32min 0
  pathstring
  Relative to the repository root, for `code`.
  span_idstring
  The span this points at, for `trace`.
  whystringrequired
  summarystringrequired
  One to three sentences, written for a person.
  version_notestring
  Drift between the checkout that was read and the version that failed.
  Variant 2
  valuenull
- group_idstringrequired
- idstringrequired
- investigation_idstringrequired
- modelstringrequired
- raw_resultstring
- session_idstringrequired
- sourceone ofrequired
  one of (2)
  Variant 1
  valuestringenum first_pass
  Variant 2
  valuestringenum conversation
- turn_idstring
- validbooleanrequired
  False when the stored payload no longer parses as a `DiagnosisV1` — the raw text is kept so nothing is lost silently.
totalinteger· uint64requiredmin 0

sentinel::diagnosis::record

Function

Record what an investigation concluded about a group. Called by the agent from inside its own investigation session; the session's identity comes from the invocation, never from the payload. Each call is a new version and the most recent one stands.

Request

diagnosisobjectrequired
What an investigation concluded. Written by the agent, never by this worker, and kept verbatim: a later version supersedes it without erasing it, so the same failure diagnosed twice can be compared.
- categoryone ofrequired
  one of (3)
  Variant 1
  valuestringenum bug, configuration, dependency, unknown
  Variant 2
  valuestringenum transient
  Variant 3
  valuestringenum expected
- confidencestringrequiredenum high, medium, low
- missing_evidencestring[]
  What would have raised the confidence. An honest `low` with this filled in is a better answer than a guess.
- proposed_fixany of
  any of (2)
  Variant 1
  descriptionstringrequired
  filesstring[]required
  riskstringrequiredenum low, medium, high
  stepsstring[]required
  Variant 2
  valuenull
- related_groupsstring[]
  Groups the agent believes share this cause. A suggestion the console shows; nothing is merged on the strength of it.
- reproductionstring
- root_causeobjectrequired
  descriptionstringrequired
  evidenceobject[]required
  Show 6 nestedHide nested
  excerptstringrequired
  At most 400 characters of the thing itself.
  kindone ofrequired
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum trace, log
  Variant 2
  valuestringenum code
  lineinteger· uint32min 0
  pathstring
  Relative to the repository root, for `code`.
  span_idstring
  The span this points at, for `trace`.
  whystringrequired
- summarystringrequired
  One to three sentences, written for a person.
- version_notestring
  Drift between the checkout that was read and the version that failed.
group_idstringrequired

Response

diagnosis_idstringrequired
group_statusall ofrequired
Where the group was left. A resolved, ignored or regressed group keeps its state and takes the diagnosis anyway.
all of (1)
Variant 1
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed
versioninteger· uint64requiredmin 0
1 on this investigation's first recording, and up from there.

sentinel::evidence::get

Function

Read the frozen evidence of one occurrence: the span tree, the logs of that trace and the session tags, captured before the engine's ring discarded them. Reports pruned when retention has removed it.

Request

occurrence_idstringrequired

Response

evidenceany of
Absent when retention has pruned it: the row stays, the bundle goes.
any of (2)
Variant 1
captured_at_msinteger· int64required
logsobject[]required
Show 5 nestedHide nested
attributesobjectrequired
bodystringrequired
severity_textstringrequired
span_idstring
timestamp_unix_nanointeger· uint64requiredmin 0
origin_span_idstringrequired
The leaf error span this bundle is about.
propagated_throughstring[]required
Ancestors that carry the same failure upward. They are context, not groups of their own.
settledbooleanrequired
True once the settle pass re-read the trace: ancestors that were still open at capture are closed here.
spansobject[]required
Show 12 nestedHide nested
attributesobjectrequired
depthinteger· uint32requiredmin 0
end_time_unix_nanointeger· uint64requiredmin 0
Zero while the span is still open — the engine's own convention for a pending span, kept rather than normalized so a half-finished trace reads as one.
eventsobject[]required
Show 3 nestedHide nested
attributesobjectrequired
namestringrequired
timestamp_unix_nanointeger· uint64requiredmin 0
function_idstring
namestringrequired
parent_span_idstring
service_namestringrequired
span_idstringrequired
start_time_unix_nanointeger· uint64requiredmin 0
statusstringrequired
status_descriptionstring
trace_idstringrequired
trace_tagsobjectrequired
truncatedobjectrequired
Show 3 nestedHide nested
attributesinteger· uint32requiredmin 0
logsinteger· uint32requiredmin 0
spansinteger· uint32requiredmin 0
versioninteger· uint32requiredmin 0
workerobjectrequired
Show 2 nestedHide nested
service_namestringrequired
versionstring
Variant 2
valuenull
prunedbooleanrequired

sentinel::groups::get

Function

Read one error group with its latest occurrence, and whether that occurrence's trace is still readable in the engine or only as the frozen snapshot.

Request

group_idstringrequired

Response

active_investigationany of
The first pass running right now, if one is.
any of (2)
Variant 1
checkout_refstring
The commit the mapped checkout was on when the pass started.
cost_usdnumber· double
created_msinteger· int64required
duration_msinteger· int64
errorstring
finished_msinteger· int64
first_pass_turn_idstring
group_idstringrequired
idstringrequired
investigated_versionstring
The worker version the investigated occurrence ran on.
modeone ofrequired
How an investigation was opened.
Show 2 nestedHide nested
one of (2)
Variant 1
valuestringenum assisted
Variant 2
valuestringenum chat
modelstringrequired
occurrence_idstringrequired
providerstring
repository_idstring
session_idstringrequired
The harness session. The console opens this beside the page.
statusone ofrequired
Show 4 nestedHide nested
one of (4)
Variant 1
valuestringenum failed, cancelled
Variant 2
valuestringenum running
Variant 3
valuestringenum completed
Variant 4
valuestringenum open
turnsinteger· uint64min 0
Observed after the fact, and informative only: this worker imposes no ceiling on an investigation.
Variant 2
valuenull
diagnosisany of
The diagnosis in force, when an investigation recorded one.
any of (2)
Variant 1
created_msinteger· int64required
diagnosisany of
Show 2 nestedHide nested
any of (2)
Variant 1
categoryone ofrequired
Show 3 nestedHide nested
one of (3)
Variant 1
valuestringenum bug, configuration, dependency, unknown
Variant 2
valuestringenum transient
Variant 3
valuestringenum expected
confidencestringrequiredenum high, medium, low
missing_evidencestring[]
What would have raised the confidence. An honest `low` with this filled in is a better answer than a guess.
proposed_fixany of
Show 2 nestedHide nested
any of (2)
Variant 1
descriptionstringrequired
filesstring[]required
riskstringrequiredenum low, medium, high
stepsstring[]required
Variant 2
valuenull
related_groupsstring[]
Groups the agent believes share this cause. A suggestion the console shows; nothing is merged on the strength of it.
reproductionstring
root_causeobjectrequired
Show 2 nestedHide nested
descriptionstringrequired
evidenceobject[]required
Show 6 nestedHide nested
excerptstringrequired
At most 400 characters of the thing itself.
kindone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
valuestringenum trace, log
Variant 2
valuestringenum code
lineinteger· uint32min 0
pathstring
Relative to the repository root, for `code`.
span_idstring
The span this points at, for `trace`.
whystringrequired
summarystringrequired
One to three sentences, written for a person.
version_notestring
Drift between the checkout that was read and the version that failed.
Variant 2
valuenull
group_idstringrequired
idstringrequired
investigation_idstringrequired
modelstringrequired
raw_resultstring
session_idstringrequired
sourceone ofrequired
Show 2 nestedHide nested
one of (2)
Variant 1
valuestringenum first_pass
Variant 2
valuestringenum conversation
turn_idstring
validbooleanrequired
False when the stored payload no longer parses as a `DiagnosisV1` — the raw text is kept so nothing is lost silently.
Variant 2
valuenull
groupobjectrequired
A group as the list shows it.
- exception_typestring
- fingerprintstringrequired
- first_seen_msinteger· int64required
- first_versionstring
- function_idstring
- has_diagnosisbooleanrequired
- idstringrequired
- ignore_ruleany of
  any of (2)
  Variant 1
  one of (3)
  Variant 1
  kindstringrequiredenum forever
  Variant 2
  countinteger· uint64requiredmin 0
  kindstringrequiredenum occurrences
  Variant 3
  kindstringrequiredenum version_change
  Variant 2
  valuenull
- last_seen_msinteger· int64required
- last_versionstring
- namespacestringrequired
- occurrence_countinteger· uint64requiredmin 0
- regressed_at_msinteger· int64
- relevantbooleanrequired
  Whether the default list shows it; see [`RelevanceV1::Relevant`].
- resolve_until_version_changeboolean
- resolved_at_msinteger· int64
  When the group was last resolved and on which version — what a regression is measured against, so the page can say what came back.
- resolved_versionstring
- service_namestringrequired
  The worker that owns the failing function.
- sessions_affectedinteger· uint64requiredmin 0
  Distinct sessions this failure touched. Counted from its own table, so it stays true after occurrence rows are pruned.
- sourceone ofrequired
  Where an error event came from. Part of the fingerprint: the same text seen as a span and as a log is two groups, because one has a trace to investigate and the other does not.
  one of (4)
  Variant 1
  valuestringenum report
  Variant 2
  valuestringenum trace
  Variant 3
  valuestringenum log
  Variant 4
  valuestringenum harness-turn
- sparklineinteger[]· uint64requiredmin 0
  Occurrences per hour for the last day, oldest first.
- statusone ofrequired
  The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
  one of (3)
  Variant 1
  valuestringenum new, diagnosed, resolved, ignored
  Variant 2
  valuestringenum investigating
  Variant 3
  valuestringenum regressed
- titlestringrequired
- triageany of
  Absent until the group has been triaged.
  any of (2)
  Variant 1
  at_msinteger· int64required
  confidencenumber· doublerequired
  Between 0 and 1, as the judge reports it; 1 for a rule.
  kindone ofrequired
  What kind of failure a group is, as triage reads it.
  Show 5 nestedHide nested
  one of (5)
  Variant 1
  valuestringenum defect
  Variant 2
  valuestringenum caller_error
  Variant 3
  valuestringenum transient
  Variant 4
  valuestringenum environment
  Variant 5
  valuestringenum test_traffic
  modelstring
  The judge model that answered.
  sourceone ofrequired
  Who decided a triage.
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum rule
  Variant 2
  valuestringenum judge
  Variant 2
  valuenull
latest_investigationany of
The most recent investigation of any state — what "Continue in chat" reopens.
any of (2)
Variant 1
checkout_refstring
The commit the mapped checkout was on when the pass started.
cost_usdnumber· double
created_msinteger· int64required
duration_msinteger· int64
errorstring
finished_msinteger· int64
first_pass_turn_idstring
group_idstringrequired
idstringrequired
investigated_versionstring
The worker version the investigated occurrence ran on.
modeone ofrequired
How an investigation was opened.
Show 2 nestedHide nested
one of (2)
Variant 1
valuestringenum assisted
Variant 2
valuestringenum chat
modelstringrequired
occurrence_idstringrequired
providerstring
repository_idstring
session_idstringrequired
The harness session. The console opens this beside the page.
statusone ofrequired
Show 4 nestedHide nested
one of (4)
Variant 1
valuestringenum failed, cancelled
Variant 2
valuestringenum running
Variant 3
valuestringenum completed
Variant 4
valuestringenum open
turnsinteger· uint64min 0
Observed after the fact, and informative only: this worker imposes no ceiling on an investigation.
Variant 2
valuenull
latest_occurrenceany of
any of (2)
Variant 1
at_msinteger· int64required
has_evidencebooleanrequired
Whether the frozen bundle is still kept for this one.
idstringrequired
messagestringrequired
namespace_ambiguousbooleanrequired
The owner could not be pinned to one namespace.
session_idstring
settledbooleanrequired
Whether the trace was re-read after the ancestors closed.
sourceone ofrequired
Where an error event came from. Part of the fingerprint: the same text seen as a span and as a log is two groups, because one has a trace to investigate and the other does not.
Show 4 nestedHide nested
one of (4)
Variant 1
valuestringenum report
Variant 2
valuestringenum trace
Variant 3
valuestringenum log
Variant 4
valuestringenum harness-turn
span_idstring
trace_idstring
turn_idstring
worker_versionstring
Variant 2
valuenull
message_samplestringrequired
The message exactly as it was captured, after redaction.
trace_availablebooleanrequired
Whether the trace behind the latest occurrence is still in the engine's ring. False means the frozen snapshot is all there is.

sentinel::groups::history

Function

Every move a group has made, newest first: what changed it, why, and whether it came from the pipeline, from an agent or from a person.

Request

group_idstringrequired
limitinteger· uint32min 0
offsetinteger· uint32min 0

Response

totalinteger· uint64requiredmin 0
transitionsobject[]required
Newest first.
- actorstringrequired
  A role rather than a person: `ingest`, `agent`, `investigation` or `console`. The console hands this worker no user identity, and a worker uuid here would read as one.
- at_msinteger· int64required
- from_statusany of
  Absent on the row that records the group's own beginning.
  any of (2)
  Variant 1
  one of (3)
  Variant 1
  valuestringenum new, diagnosed, resolved, ignored
  Variant 2
  valuestringenum investigating
  Variant 3
  valuestringenum regressed
  Variant 2
  valuenull
- idstringrequired
- reasonany of
  any of (2)
  Variant 1
  valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
  Variant 2
  valuenull
- to_statusone ofrequired
  The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
  one of (3)
  Variant 1
  valuestringenum new, diagnosed, resolved, ignored
  Variant 2
  valuestringenum investigating
  Variant 3
  valuestringenum regressed

sentinel::groups::ignore

Function

Ignore a group forever, for a number of further occurrences, or until the worker version changes. Occurrences keep counting either way.

Request

group_idstringrequired
ruleone ofrequired
How long an ignore lasts. Every rule keeps counting occurrences; what changes is when the group comes back to the open list.
one of (3)
Variant 1
kindstringrequiredenum forever
Variant 2
countinteger· uint64requiredmin 0
kindstringrequiredenum occurrences
Variant 3
kindstringrequiredenum version_change

Response

group_idstringrequired
reasonany of
any of (2)
Variant 1
valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
Variant 2
valuenull
statusone ofrequired
The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

sentinel::groups::list

Function

List error groups, regressions first. Filters by state, worker, time window and a search over title, message and function id; defaults to the open states.

Request

limitinteger· uint32min 0
offsetinteger· uint32min 0
relevanceany of
Absent lists both sides.
any of (2)
Variant 1
one of (2)
Variant 1
valuestringenum relevant
Variant 2
valuestringenum noise
Variant 2
valuenull
searchstring
Matches the title, the message sample and the function id.
service_namestring
since_msinteger· int64
Only groups seen since this moment.
statusone of[]
Defaults to the open states.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

Response

groupsobject[]required
- exception_typestring
- fingerprintstringrequired
- first_seen_msinteger· int64required
- first_versionstring
- function_idstring
- has_diagnosisbooleanrequired
- idstringrequired
- ignore_ruleany of
  any of (2)
  Variant 1
  one of (3)
  Variant 1
  kindstringrequiredenum forever
  Variant 2
  countinteger· uint64requiredmin 0
  kindstringrequiredenum occurrences
  Variant 3
  kindstringrequiredenum version_change
  Variant 2
  valuenull
- last_seen_msinteger· int64required
- last_versionstring
- namespacestringrequired
- occurrence_countinteger· uint64requiredmin 0
- regressed_at_msinteger· int64
- relevantbooleanrequired
  Whether the default list shows it; see [`RelevanceV1::Relevant`].
- resolve_until_version_changeboolean
- resolved_at_msinteger· int64
  When the group was last resolved and on which version — what a regression is measured against, so the page can say what came back.
- resolved_versionstring
- service_namestringrequired
  The worker that owns the failing function.
- sessions_affectedinteger· uint64requiredmin 0
  Distinct sessions this failure touched. Counted from its own table, so it stays true after occurrence rows are pruned.
- sourceone ofrequired
  Where an error event came from. Part of the fingerprint: the same text seen as a span and as a log is two groups, because one has a trace to investigate and the other does not.
  one of (4)
  Variant 1
  valuestringenum report
  Variant 2
  valuestringenum trace
  Variant 3
  valuestringenum log
  Variant 4
  valuestringenum harness-turn
- sparklineinteger[]· uint64requiredmin 0
  Occurrences per hour for the last day, oldest first.
- statusone ofrequired
  The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
  one of (3)
  Variant 1
  valuestringenum new, diagnosed, resolved, ignored
  Variant 2
  valuestringenum investigating
  Variant 3
  valuestringenum regressed
- titlestringrequired
- triageany of
  Absent until the group has been triaged.
  any of (2)
  Variant 1
  at_msinteger· int64required
  confidencenumber· doublerequired
  Between 0 and 1, as the judge reports it; 1 for a rule.
  kindone ofrequired
  What kind of failure a group is, as triage reads it.
  Show 5 nestedHide nested
  one of (5)
  Variant 1
  valuestringenum defect
  Variant 2
  valuestringenum caller_error
  Variant 3
  valuestringenum transient
  Variant 4
  valuestringenum environment
  Variant 5
  valuestringenum test_traffic
  modelstring
  The judge model that answered.
  sourceone ofrequired
  Who decided a triage.
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum rule
  Variant 2
  valuestringenum judge
  Variant 2
  valuenull
noise_totalinteger· uint64requiredmin 0
relevant_totalinteger· uint64requiredmin 0
Both sides under the same filters, whatever `relevance` asked for.
totalinteger· uint64requiredmin 0

sentinel::groups::reopen

Function

Reopen a resolved or ignored group, clearing the rule that closed it.

Request

group_idstringrequired

Response

group_idstringrequired
reasonany of
any of (2)
Variant 1
valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
Variant 2
valuenull
statusone ofrequired
The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

sentinel::groups::resolve

Function

Mark a group resolved. With until_version_change, occurrences on the resolved version keep counting without reopening it — the fix is not deployed there yet.

Request

group_idstringrequired
until_version_changeboolean
Occurrences on the version that was resolved keep counting without reopening the group — the fix is not deployed here yet.

Response

group_idstringrequired
reasonany of
any of (2)
Variant 1
valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
Variant 2
valuenull
statusone ofrequired
The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

sentinel::groups::unignore

Function

Take a group off the ignore list and back into the open list.

Request

group_idstringrequired

Response

group_idstringrequired
reasonany of
any of (2)
Variant 1
valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
Variant 2
valuenull
statusone ofrequired
The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

sentinel::ingest

Function

Internal durable queue step: reads one trace or log and records what failed.

Request

one of (4)

Variant 1

kindstringrequiredenum trace
trace_idstringrequired

Variant 2

kindstringrequiredenum log
logunknown
trace_idstringrequired

Variant 3

kindstringrequiredenum settle
trace_idstringrequired

Variant 4

kindstringrequiredenum promote
occurrence_idstringrequired
trace_idstringrequired

Response

dedupedinteger· uint64requiredmin 0
recordedinteger· uint64requiredmin 0

sentinel::investigate

Function

Open an investigation on a group: a harness session beside the page that starts from the frozen evidence. Mode assisted runs a first pass immediately; mode chat only seats the evidence and waits for a person. A first pass already running is returned rather than duplicated.

Request

group_idstringrequired
modeany of
Defaults to `assisted`.
any of (2)
Variant 1
one of (2)
Variant 1
valuestringenum assisted
Variant 2
valuestringenum chat
Variant 2
valuenull
modelstring
Catalog id, or a raw model id alongside `provider`. Falls back to the configured one.
occurrence_idstring
Defaults to the most recent occurrence that still has evidence.
providerstring

Response

existingbooleanrequired
A first pass was already running: this is that one, not a second.
first_pass_turn_idstring
investigation_idstringrequired
session_idstringrequired
What the console opens beside the page.

sentinel::investigations::cancel

Function

Stop an investigation's first pass. The group returns to the state it was in unless a diagnosis was already recorded.

Request

investigation_idstringrequired

Response

checkout_refstring
The commit the mapped checkout was on when the pass started.
cost_usdnumber· double
created_msinteger· int64required
duration_msinteger· int64
errorstring
finished_msinteger· int64
first_pass_turn_idstring
group_idstringrequired
idstringrequired
investigated_versionstring
The worker version the investigated occurrence ran on.
modeone ofrequired
How an investigation was opened.
one of (2)
Variant 1
valuestringenum assisted
Variant 2
valuestringenum chat
modelstringrequired
occurrence_idstringrequired
providerstring
repository_idstring
session_idstringrequired
The harness session. The console opens this beside the page.
statusone ofrequired
one of (4)
Variant 1
valuestringenum failed, cancelled
Variant 2
valuestringenum running
Variant 3
valuestringenum completed
Variant 4
valuestringenum open
turnsinteger· uint64min 0
Observed after the fact, and informative only: this worker imposes no ceiling on an investigation.

sentinel::investigations::get

Function

Read one investigation and every diagnosis it recorded, newest first.

Request

investigation_idstringrequired

Response

diagnosesobject[]required
Newest first.
- created_msinteger· int64required
- diagnosisany of
  any of (2)
  Variant 1
  categoryone ofrequired
  Show 3 nestedHide nested
  one of (3)
  Variant 1
  valuestringenum bug, configuration, dependency, unknown
  Variant 2
  valuestringenum transient
  Variant 3
  valuestringenum expected
  confidencestringrequiredenum high, medium, low
  missing_evidencestring[]
  What would have raised the confidence. An honest `low` with this filled in is a better answer than a guess.
  proposed_fixany of
  Show 2 nestedHide nested
  any of (2)
  Variant 1
  descriptionstringrequired
  filesstring[]required
  riskstringrequiredenum low, medium, high
  stepsstring[]required
  Variant 2
  valuenull
  related_groupsstring[]
  Groups the agent believes share this cause. A suggestion the console shows; nothing is merged on the strength of it.
  reproductionstring
  root_causeobjectrequired
  Show 2 nestedHide nested
  descriptionstringrequired
  evidenceobject[]required
  Show 6 nestedHide nested
  excerptstringrequired
  At most 400 characters of the thing itself.
  kindone ofrequired
  Show 2 nestedHide nested
  one of (2)
  Variant 1
  valuestringenum trace, log
  Variant 2
  valuestringenum code
  lineinteger· uint32min 0
  pathstring
  Relative to the repository root, for `code`.
  span_idstring
  The span this points at, for `trace`.
  whystringrequired
  summarystringrequired
  One to three sentences, written for a person.
  version_notestring
  Drift between the checkout that was read and the version that failed.
  Variant 2
  valuenull
- group_idstringrequired
- idstringrequired
- investigation_idstringrequired
- modelstringrequired
- raw_resultstring
- session_idstringrequired
- sourceone ofrequired
  one of (2)
  Variant 1
  valuestringenum first_pass
  Variant 2
  valuestringenum conversation
- turn_idstring
- validbooleanrequired
  False when the stored payload no longer parses as a `DiagnosisV1` — the raw text is kept so nothing is lost silently.
investigationobjectrequired
- checkout_refstring
  The commit the mapped checkout was on when the pass started.
- cost_usdnumber· double
- created_msinteger· int64required
- duration_msinteger· int64
- errorstring
- finished_msinteger· int64
- first_pass_turn_idstring
- group_idstringrequired
- idstringrequired
- investigated_versionstring
  The worker version the investigated occurrence ran on.
- modeone ofrequired
  How an investigation was opened.
  one of (2)
  Variant 1
  valuestringenum assisted
  Variant 2
  valuestringenum chat
- modelstringrequired
- occurrence_idstringrequired
- providerstring
- repository_idstring
- session_idstringrequired
  The harness session. The console opens this beside the page.
- statusone ofrequired
  one of (4)
  Variant 1
  valuestringenum failed, cancelled
  Variant 2
  valuestringenum running
  Variant 3
  valuestringenum completed
  Variant 4
  valuestringenum open
- turnsinteger· uint64min 0
  Observed after the fact, and informative only: this worker imposes no ceiling on an investigation.

sentinel::investigations::list

Function

List investigations, newest first, optionally narrowed to one group or a set of states.

Request

group_idstring
limitinteger· uint32min 0
offsetinteger· uint32min 0
statusone of[]
one of (4)
Variant 1
valuestringenum failed, cancelled
Variant 2
valuestringenum running
Variant 3
valuestringenum completed
Variant 4
valuestringenum open

Response

investigationsobject[]required
- checkout_refstring
  The commit the mapped checkout was on when the pass started.
- cost_usdnumber· double
- created_msinteger· int64required
- duration_msinteger· int64
- errorstring
- finished_msinteger· int64
- first_pass_turn_idstring
- group_idstringrequired
- idstringrequired
- investigated_versionstring
  The worker version the investigated occurrence ran on.
- modeone ofrequired
  How an investigation was opened.
  one of (2)
  Variant 1
  valuestringenum assisted
  Variant 2
  valuestringenum chat
- modelstringrequired
- occurrence_idstringrequired
- providerstring
- repository_idstring
- session_idstringrequired
  The harness session. The console opens this beside the page.
- statusone ofrequired
  one of (4)
  Variant 1
  valuestringenum failed, cancelled
  Variant 2
  valuestringenum running
  Variant 3
  valuestringenum completed
  Variant 4
  valuestringenum open
- turnsinteger· uint64min 0
  Observed after the fact, and informative only: this worker imposes no ceiling on an investigation.
totalinteger· uint64requiredmin 0

sentinel::logs::list

Function

Read logs from the live engine through the redactor, for an investigation.

Request

levelstring
`error`, `warn`, `info`, … Matched by the engine.
limitinteger· uint32min 0
service_namestring
since_msinteger· int64
trace_idstring

Response

logsunknown[]required
redactionsinteger· uint64requiredmin 0

sentinel::occurrences::list

Function

List the recorded occurrences of one group, newest first, each saying whether its frozen evidence is still kept.

Request

group_idstringrequired
limitinteger· uint32min 0
offsetinteger· uint32min 0

Response

occurrencesobject[]required
- at_msinteger· int64required
- has_evidencebooleanrequired
  Whether the frozen bundle is still kept for this one.
- idstringrequired
- messagestringrequired
- namespace_ambiguousbooleanrequired
  The owner could not be pinned to one namespace.
- session_idstring
- settledbooleanrequired
  Whether the trace was re-read after the ancestors closed.
- sourceone ofrequired
  Where an error event came from. Part of the fingerprint: the same text seen as a span and as a log is two groups, because one has a trace to investigate and the other does not.
  one of (4)
  Variant 1
  valuestringenum report
  Variant 2
  valuestringenum trace
  Variant 3
  valuestringenum log
  Variant 4
  valuestringenum harness-turn
- span_idstring
- trace_idstring
- turn_idstring
- worker_versionstring
totalinteger· uint64requiredmin 0

sentinel::on-config-change

Function

Internal configuration doorbell: re-reads the authoritative sentinel entry and swaps the live settings.

Request

idstring
Configuration id that changed.

Response

okbooleanrequired

sentinel::on-log

Function

Internal handler for the engine's ERROR-log trigger.

Request

attributesobject
bodystring
instrumentation_scope_namestring
instrumentation_scope_versionstring
observed_timestamp_unix_nanointeger· uint64min 0
resourceobject
service_namestring
severity_numberinteger· int32
severity_textstring
span_idstring
timestamp_unix_nanointeger· uint64min 0
trace_idstring

Response

phantom_droppedinteger· uint64requiredmin 0
Ticks for this worker's own traces, dropped before the queue.
queuedinteger· uint64requiredmin 0

sentinel::on-schedule

Function

Internal daily prune of buckets and archived groups.

Request

Empty object

Response

buckets_removedinteger· uint64requiredmin 0
evidence_prunedinteger· uint64requiredmin 0
groups_archivedinteger· uint64requiredmin 0

sentinel::on-trace-activity

Function

Internal handler for the engine's error-span trigger; queues the traces worth reading.

Request

trace_idsstring[]

Response

phantom_droppedinteger· uint64requiredmin 0
Ticks for this worker's own traces, dropped before the queue.
queuedinteger· uint64requiredmin 0

sentinel::on-turn-completed

Function

Internal doorbell for a finished investigation turn; the status is re-read rather than trusted.

Request

reasonstring
session_idstring
statusstring
terminalboolean
turn_idstring

Response

handledbooleanrequired
Whether this doorbell named an investigation of this worker's.

sentinel::status

Function

Health and counters: whether ingest is enabled, the state of the engine's telemetry stores, per-source ingest counters (including values redacted on capture and traces lost before capture), group and investigation counts, and which mapped repositories are present on this machine.

Request

Empty object

Response

config_errorstring
Why the stored configuration was refused, when it was. The worker runs on shipped defaults and stays disabled until a valid value arrives, so a typo reads as a named field here instead of a silent idle worker.
enabledbooleanrequired
True once the configuration says so *and* the durable dependencies are claimed. False means nothing is being ingested.
engineobjectrequired
- logsbooleanrequired
- trace_storeone ofrequired
  State of the engine's in-memory telemetry stores, as the worker last observed them.
  one of (3)
  Variant 1
  valuestringenum memory
  Variant 2
  valuestringenum disabled
  Variant 3
  valuestringenum unknown
groupsobjectrequired
- ignoredinteger· uint64requiredmin 0
- last_seen_msinteger· int64
  The newest occurrence across every group: when ingest last recorded something.
- openinteger· uint64requiredmin 0
- regressedinteger· uint64requiredmin 0
- resolvedinteger· uint64requiredmin 0
ingestobjectrequired
- dedupedinteger· uint64requiredmin 0
- dropped_failedinteger· uint64requiredmin 0
  Jobs that exhausted their retries.
- dropped_ignored_serviceinteger· uint64requiredmin 0
- dropped_investigationinteger· uint64requiredmin 0
  Traces tagged with an investigation session.
- dropped_not_readyinteger· uint64requiredmin 0
  Deliveries refused because the durable dependencies were not claimed yet. They are redelivered, not lost.
- dropped_own_traceinteger· uint64requiredmin 0
  Traces whose root is this worker — the cut that stops the monitor from monitoring itself.
- logs_pending_joininteger· uint64requiredmin 0
  ERROR logs currently waiting for the error span of their trace.
- logs_unattributedinteger· uint64requiredmin 0
  Logs promoted without resolving their trace: a bounded, visible gap.
- lost_before_captureinteger· uint64requiredmin 0
  Traces that had already left the engine's ring when the job ran.
- paused_untilinteger· int64
  Set while the ingest breaker is open.
- phantom_ticks_droppedinteger· uint64requiredmin 0
  Ticks for the ingest's own traces, filtered before the queue.
- processed_totalinteger· uint64requiredmin 0
- queuedinteger· uint64requiredmin 0
  Jobs handed to the durable queue and not yet completed.
- redactionsinteger· uint64requiredmin 0
  Values rewritten as `[redacted:<kind>]` on capture, all sources.
investigationsobjectrequired
- open_sessionsinteger· uint64requiredmin 0
  Investigation sessions that are still open for conversation.
- runninginteger· uint64requiredmin 0
  First passes running right now.
repositoriesobject[]required
- existsbooleanrequired
  Whether the checkout is on this machine right now. False disables code access for its workers; grouping and evidence are unaffected.
- idstringrequired
- pathstringrequired
- workersstring[]required
sourcesobjectrequired
- logbooleanrequired
- tracebooleanrequired

sentinel::trace::get

Function

Read a trace from the live engine through the redactor, for an investigation comparing the frozen snapshot with what is there now. Answers null once the trace has left the engine's ring.

Request

include_spansboolean
Also return the flat span list, for a trace whose shape matters more than its tree.
trace_idstringrequired

Response

redactionsinteger· uint64requiredmin 0
Values rewritten before the response left this worker.
traceunknown
Null when the trace has already left the engine's ring: the frozen evidence is then all there is.

sentinel::ui-content

Function

Serve the sentinel worker's injected console UI assets (content function for its console:script / console:style / console:module triggers).

Request

pathstringrequired
The asset path from the trigger config (e.g. `state/page.js`).

Response

contentstringrequired
The asset source, verbatim.
content_typestringrequired
MIME type the console should serve the asset with.

Triggers2

sentinel::group-changed

Trigger

Fires when an error group is created, takes a new occurrence, or changes state. Config: { group_id?, service_name?, ops? } — ops narrows to created, occurrence or status. Occurrence events are coalesced per group over one second; re-read the group rather than trusting the payload.

Invocation

group_idstring
opsone of[]
one of (2)
Variant 1
valuestringenum created, status
Variant 2
valuestringenum occurrence
service_namestring

Return

group_idstringrequired
occurrence_countinteger· uint64requiredmin 0
opone ofrequired
one of (2)
Variant 1
valuestringenum created, status
Variant 2
valuestringenum occurrence
previous_statusany of
any of (2)
Variant 1
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed
Variant 2
valuenull
reasonany of
any of (2)
Variant 1
valuestringenum regression, ignore_expired, resolved, ignored, diagnosed, reopened, …
Variant 2
valuenull
statusone ofrequired
The lifecycle of a group. `resolved` and `ignored` are human decisions; `regressed` is the one the ingest makes on its own, and the reason the worker exists.
one of (3)
Variant 1
valuestringenum new, diagnosed, resolved, ignored
Variant 2
valuestringenum investigating
Variant 3
valuestringenum regressed

sentinel::investigation-changed

Trigger

Fires when an investigation is created, starts, or finishes. Config: { group_id? }.

Invocation

group_idstring
opsone of[]
one of (2)
Variant 1
valuestringenum created, status
Variant 2
valuestringenum occurrence
service_namestring

Return

valueunknown

Install

Functions25

Triggers2

Search the registry