CI must distinguish what the application did from what the runner, transport, test harness, and evidence publisher did. A green test runner followed by a broken wrapper is not a successful CI execution. An emulator that never became healthy is not an application assertion failure.
The Developer Workflow packages this boundary as:
schemas/ci-execution-receipt-v1.schema.json;scripts/evaluate_ci_execution.py;repository-template/ for adopting repositories.The evaluator is deterministic and local. It does not fetch CI logs, infer a root cause from an error string, call an external provider, or authorize merge or release.
Model a check as explicit phases rather than one wrapper exit code:
environment_preflight — required runtime, accelerator, service, device,
credential presence check, disk, port, or network boundary is available.
Do not record secret values.build — the candidate and any test artifact were created for the exact
revision.test — tests were discovered and executed; counts are internally
consistent.postprocess — reports, classification, cleanup needed for the result, and
local evidence transformations completed.evidence_publication — required artifacts and summaries were retained.Repositories select the phases required for each capability. A missing,
not_run, failed, or unavailable required phase prevents produced.
| Class | Meaning | Do not report as |
|---|---|---|
environment_preflight |
A required runner or environment prerequisite was absent or unusable | application failure |
build |
The candidate or test artifact could not be built | test failure |
test_transport |
The test target could not be reached, installed, started, or queried reliably | assertion failure |
test_harness |
Discovery, counting, orchestration, or the test framework was invalid | passing or failing product behavior |
application_assertion |
A test executed and an application assertion failed | infrastructure failure |
postprocess |
Tests may have completed, but required result processing failed | produced evidence |
evidence_publication |
Required artifacts or summaries were not retained | complete evidence |
unknown |
Available evidence cannot support a narrower class | an inferred root cause |
Classification is an attributable observation. When logs are ambiguous, use
unknown; do not guess from the final wrapper message.
An adopting repository creates an observation after its job phases complete:
{
"schema_version": "ai-safe2.ci-execution-observation.v1",
"receipt_id": "device-tests-pr-44@abc123",
"subject": {
"identifier": "pr-44",
"revision": "abc123",
"capability": "device-privacy-tests"
},
"runner": {
"provider": "github-actions",
"image": "ubuntu-24.04",
"operating_system": "Linux",
"architecture": "x86_64",
"environment_facts": {
"hardware_acceleration": true,
"target_api": 36
}
},
"required_phases": [
"environment_preflight",
"build",
"test",
"postprocess"
],
"phases": [
{
"name": "environment_preflight",
"status": "passed",
"exit_code": 0,
"failure_class": null,
"failure_reason": null
},
{
"name": "build",
"status": "passed",
"exit_code": 0,
"failure_class": null,
"failure_reason": null
},
{
"name": "test",
"status": "passed",
"exit_code": 0,
"failure_class": null,
"failure_reason": null
},
{
"name": "postprocess",
"status": "passed",
"exit_code": 0,
"failure_class": null,
"failure_reason": null
}
],
"test_results": {
"status": "passed",
"minimum_required": 1,
"discovered": 128,
"executed": 128,
"passed": 128,
"failed": 0,
"skipped": 0
},
"artifacts": [],
"limitations": []
}
Generate and gate on the receipt:
python scripts/evaluate_ci_execution.py \
--input ci-execution-observation.json \
--output ci-execution-receipt.json \
--require-produced
Without --require-produced, the command still writes a truthful failed or
unavailable receipt and exits successfully when the observation is valid.
Use --require-produced when the receipt represents a required merge gate.
For emulator or device jobs, run a small preflight before the full suite:
Keep multiline orchestration in a repository-owned, syntax-tested script. Action inputs and wrapper fields may execute each line independently even when the YAML value is syntactically valid.
Investigate the first unsupported boundary, not the final generic exit code:
Preserve the exact revision, runner image, safe environment facts, phase results, test counts, artifacts, and limitations. A receipt proves only what those observations support.
The receipt is an artifact for a deterministic-check evidence envelope:
produced may support the mapped capability for the same revision;failed maps to evidence execution failed;unavailable maps to evidence execution unavailable;This helper is not a general-purpose CI evidence bridge. Repositories still need bounded automation that collects their native results and maps the receipt to the appropriate capability.