ai-safe2-framework

AI SAFE² Challenge Lab

Falsification before claims

AI SAFE² Surface Method

Framework Home Cross-Pillar Governance AISM NEXUS Dashboard

Technology profiles and new validation candidates

Record challenge results in the Technology Contribution Profile with claim-level assurance. Keep C0-C5 claim maturity, E0-E5 assurance v1.0, technology contribution 0-5, and organizational AISM maturity separate; there is no automatic conversion. The validation guide and nine-example backlog frame new composed-capability, process-mimicking, side-effect, and policy-integrity tests. They do not modify frozen Challenge 001 or claim those experiments have run.

Why This Exists

AI governance controls should not receive credit because they sound appropriate, appear in a policy, or map to a standard. They should receive credit when a reproducible implementation changes a security-relevant outcome under test.

The Challenge Lab converts incidents, evaluation findings, and credible agentic threats into open experiments. Each challenge publishes the problem, hypotheses, control claims, baselines, graders, evidence requirements, invalidation criteria, and replication path before confirmatory results are promoted.

The framework is the subject of the test, not the source of its own proof.


Challenge Questions

Every challenge should be able to answer:

  1. Did the control prevent the prohibited state change?
  2. If prevention failed, did it contain the blast radius?
  3. Was the event detected soon enough to matter?
  4. Can an independent reviewer reconstruct what happened?
  5. Can the system recover to a known-good state?
  6. What legitimate work, latency, cost, and human effort did the control consume?
  7. Which enforcement plane and control/profile claims were actually exercised?

Challenge Index

ID Challenge Primary question Status
001 Anthropic Multi-Agent Turf War Can externally enforced authority stop destructive multi-agent conflict without blocking legitimate collaboration? Live study: design and pre-registration; offline CLI fixture available

Start with the executable Challenge CLI guide. The offline runner verifies fixture mechanics only. Its synthetic third-party example demonstrates translation, not independent evidence of control effectiveness.


Claim Maturity

Level Status Evidence
C0 Described Control language exists
C1 Implemented Machine-readable configuration and code exist
C2 Unit verified Positive and negative unit tests pass
C3 Scenario validated Control changes an adversarial scenario outcome
C4 Bypass tested Control survives replay, corruption, direct bypass, and fail-mode testing
C5 Independently replicated An unaffiliated operator reproduces the material result

Only C3 and above support the phrase validated in tested conditions. No maturity level establishes universal prevention.

Challenge maturity is separate from framework/profile conformance. A well-designed experiment may invalidate a conformant implementation claim, and a conformant implementation may fail to outperform a simpler baseline.


v3.1 Enforcement-Plane Scoping

AI SAFE² v3.1 distinguishes:

A challenge result must identify which plane was tested. Evidence from one plane does not automatically validate another.


Required Design for Every Challenge

Every challenge includes:

Material normative or grader changes after preregistration require a new preregistration version before confirmatory evidence is pooled.


Result Labels

An invalidated claim is a useful result. It should create a control revision, permanent regression fixture, or versioned record of why the claim changed.


Contributing

High-value contributions include:

The most important contribution is evidence that changes what the framework can honestly claim.


🔗 Navigation

Framework Home Challenge 001 Cross-Pillar Governance AISM NEXUS Contributing Security

AI SAFE² principle: If governance is not enforced at runtime, it is not governance.

AI SAFE² v3.1 · Cyber Strategy Institute