Graduated Validation Protocol

Graduated Validation Protocol

Overview

The evolutionary loop generates candidate improvements continuously. Not all of them should reach production. Any system that modifies itself in production needs a safety envelope. Ch7 provides it through the RPO spine and the Graduated Validation Protocol (GVP).

The RPO spine is three interlocking properties:

• Recursion — the improvement mechanisms are themselves part of the codebase the agent can analyze and improve. The self-improvement system applies to itself.
• Provenance — every change is cryptographically signed and committed to an immutable ledger. This is source control for thought: complete auditability and instant rollback for every model, prompt, or architectural change. lib.provenance_signature is the dev-time stand-in.
• Optimization — the GVP itself. Rather than a single approval gate that must be either permissive or strict, it routes each candidate to a risk tier and applies matching scrutiny.

The three tiers:

The second half is the entropy-collapse guard. In production the agent evolves continuously, hundreds of small improvements per week, each passing validation individually. The compounding problem is the accumulated weight of all changes: contradictory learnings, stale fixes for resolved issues, redundant nodes that slow retrieval. The Kepler architecture (OpenAI, 2026) solves this with a dual-store pattern: Knowledge is human-authored and version-controlled; Learnings are agent-generated and ephemeral. A daily garbage-collection traversal walks the Learnings subgraph and removes a node if any of three conditions holds: the issue was resolved and promoted to Knowledge, a newer higher-confidence learning contradicts it, or it has not been retrieved in 30 days.

When to Use

• Gating a continuous self-evolution loop: candidate changes must be validated before they reach live traffic (the chapter's full-evolutionary-loop example, the graduated_validation(candidate).passed branch)
• Routing a mixed stream of candidates (prompt tweaks, fine-tunes, new-tool integrations) to the right level of scrutiny automatically
• Enforcing "no safety regression" as a hard constraint on a fine-tune before it ships (Tier 2)
• Running the daily Learnings garbage-collection pass to prevent entropy collapse in a long-running agent
• Monitoring the Learnings-to-Knowledge promotion rate to tune criteria

Phrases: "should this change ship", "which validation tier", "canary the prompt update", "staging gauntlet", "airlock protocol", "graduated validation", "RPO spine", "entropy collapse", "garbage-collect learnings", "promotion rate".

When NOT to Use

• One-off manual deploys. A single human approval gate is enough; the tiered protocol pays off only when candidates arrive continuously.
• Generating the candidate. This skill validates a candidate; it does not diagnose, attribute, or intervene. Those are the execution-graph, semantic-backpropagation, and intervention skills.
• Hyperparameter optimization of retrieval infrastructure. That background optimization (chunk sizes, hop distances) is Ch8 production-hardening work; it runs alongside the loop, not inside this gate.
• As the ledger itself. provenance_signature is a content hash for the dev spike; production needs a real signing key and an append-only ledger.

Process

Rationalizations

Red Flags

• A fine-tune or new-tool change assigned to Tier 1. Tier 3 override was skipped; touches_safety / new_tool / novel flags are not being read.
• A staging pass with safety_regression True. The hard safety constraint was folded into the utility sum instead of gating on it.
• A Tier 3 GateResult with requires_human False. The airlock is not escalating; human review is being bypassed.
• Learnings count grows unbounded across days. The daily GC pass is not running, or TTL / promotion edges are never set. Entropy collapse follows.
• Promotion rate stuck below 10% with no flag. promotion_health is not wired into monitoring; valuable patterns are being discarded silently.
• CLI --help exits non-zero. SKILL.md / CLI mismatch; the multi-harness invariant is broken.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report 16/16 passed:
   • assign_tier routes prompt -> TIER1, fine_tune -> TIER2, touches_safety / new_tool / novel -> TIER3 (override)
   • canary passes on significant lift + no regression, fails on KPI regression and on non-significant p-value
   • staging fails when safety_regression is True even with high utility, passes net-positive over baseline with no regression
   • airlock is pending on None, passes only on "approve", stays requires_human
   • garbage_collect removes idle / promoted / contradicted, keeps a fresh high-confidence learning, honors the 30-day TTL boundary
   • promotion_health flags below 10%
2. Run the DevOps scenario. python cli.py scenario prompt-canary routes the CausalAttributionNode prompt refinement (stripe-python 3.2.1 -> 3.3.0 timeout 30s -> 10s cascade) into Tier 1 canary, shows lift with no regression, then promotes the resolution to Knowledge and garbage-collects the ephemeral Learnings.
3. Verify CLI help. python cli.py --help exits 0 and prints the SKILL.md description.

Security Posture

• Prompt injection. Candidate metadata, risk/reward reports, and Learning nodes may originate from agent-generated traces. lib.py treats every candidate field as untrusted data: it reads scalar metrics and boolean flags, never evaluates any field as code, and never interpolates a field into a shell or query. The airlock risk/reward report is truncated and JSON-encoded before display, so an injected instruction string is inert.
• Data exfiltration. No network calls anywhere in lib.py. Gate results and GC output are returned to the caller / printed to stdout; the caller owns downstream piping. provenance_signature is a local SHA-256 over the candidate content, computed in-process.
• Privilege escalation. No shell invocation, no eval/exec, no dynamic import, no file writes. The CLI reads only the explicit --path JSON files and writes nothing to disk. A candidate cannot escalate its own tier: the Tier 3 override in assign_tier is checked first, so a safety-touching or new-tool change cannot masquerade as a Tier 1 canary.

Composition

• Composes with the Anthropic agent-skills Reviewer pattern at the pattern layer and the Ghosh Workflow layer: it reviews a candidate change and returns a pass/fail verdict for an orchestration loop above it.
• Consumes candidates produced by the execution-graph, semantic-backpropagation, and intervention skills (Ch7). It is the graduated_validation(candidate).passed branch of the full evolutionary loop (the full-evolutionary-loop example).
• Composes with the RPO Provenance ledger: provenance_signature signs a candidate before the gate result is recorded, giving instant rollback.
• Pairs with the Kepler dual-store: the GC traversal runs on the same graph the execution-graph skill writes, distinguishing Knowledge from Learnings by provenance edges.
• Hands off to Ch8 production-hardening for the retrieval-infrastructure hyperparameter optimization that runs alongside the loop rather than inside this gate.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien) Chapter 7 — Self-Evolution and Evaluation, the sections "The RPO Spine and Graduated Validation Protocol" and "Persistent Self-Learning Without Entropy Collapse", plus the chapter's full-evolutionary-loop example. Key references named in the chapter: the RPO spine (Recursion / Provenance / Optimization, "source control for thought"); the three-tier Graduated Validation Protocol (canary release / staging gauntlet / airlock protocol) shown in Figure 7-3; the multi-objective utility U = w1accuracy + w2(1-cost) + w3*safety_score; and the Kepler dual-store garbage-collection architecture (OpenAI, 2026) with its 30-day TTL, daily GC pass, and 10% promotion-rate health threshold.