Intervention Selector

Intervention Selector

Overview

A diagnosis is only valuable if it leads to the right intervention. Different failure types call for different responses. A reasoning failure at a single node calls for a prompt update; a systemic knowledge gap calls for fine-tuning; a format violation calls for an architectural constraint. Applying the wrong fix wastes time at best and makes things worse at worst.

The diagnostic report already contains the failure type and target nodes, so mapping them to an intervention is a straightforward routing function (Ch7 the select_intervention routing example). The chapter Tip is explicit about why this is a function and not a human call: intervention selection should be deterministic and auditable, not a judgment call made differently by each on-call engineer.

The router applies four branches in strict order:

1. Insufficient context -> RETRIEVAL_FIX. The context was insufficient, so the fix lives upstream of the model in the retrieval pipeline. Flag the Knowledge Graph or retrieval gap.
2. FORMAT_VIOLATION -> STRUCTURAL_CONSTRAINT. The agent had the right knowledge and reasoning but failed to produce a machine-readable output. Attach an output schema to that node so the format error is impossible rather than less likely. Architectural change, not a model change, permanent fix for that component.
3. Localized REASONING failure -> PROMPT_REFINEMENT. REASONING failure, few low-InfoGain steps (len(low_infogain_steps) <= low_step_max), high knowledge index (knowledge_index > ki_floor). The agent has the capability; steer it at that node. Fast, reversible, low risk, the right first resort.
4. Everything else -> FINE_TUNE. A systemic knowledge gap, a recurring pattern of the same reasoning failure, or a persistent misalignment. Generate a curriculum via SEAL/TPT and retrain.

The second axis, the self-modification intensity hierarchy, orders the intervention types by cost and risk (Ch7): prompt tuning is the lightest intervention (fast, reversible, low risk), weight adaptation sits in the middle (slower, semi-reversible, moderate risk), and code modification is the heaviest (slowest, requires explicit rollback, highest risk). The router never emits CODE_MODIFICATION; it is the heaviest tier, reserved for explicit code-level self-modification loops (SICA) run in a sandbox with full rollback.

When to Use

• A Ch7 diagnostic report exists (Layer 1 context sufficiency, Layer 2 cognitive failure type, low-InfoGain steps, knowledge index) and you must pick the fix
• Closing a self-evolution loop: execution -> diagnosis -> feedback -> intervention -> validation. This is the intervention step
• Automating the fix decision so it is uniform across on-call engineers rather than a per-person judgment call
• Ranking a chosen intervention on the intensity hierarchy before it enters the Graduated Validation Protocol (lower-risk interventions enter at a lighter canary tier)

Phrases: "which intervention", "prompt fine-tune or constrain", "route the diagnosis", "select_intervention", "how do I fix this diagnosed failure", "intervention intensity".

When NOT to Use

• Producing the diagnosis. This skill consumes a report; it does not build one. The report comes from the Ch7 Layer 0/1/2 evaluation pipeline (Reasoning Shape Analysis, context sufficiency check). Feed a report in.
• Applying the fix. The router returns a routed intervention. SEAL/TPT generate the curriculum, Outlines attaches the schema, the workflow-graph editor updates the prompt. This skill decides; it does not execute.
• Emitting CODE_MODIFICATION. Code-level self-modification is the most dangerous form of self-evolution. It is never an automated router output; it belongs to a sandboxed SICA loop with an LLM overseer and full rollback.
• Cross-node coherence. The router handles a single node in isolation. For coherent evolution across the graph (preventing a good fix to one node from silently breaking a neighbor), use Semantic Backpropagation (Ch7), not this.

Process

Rationalizations

Red Flags

• Router output depends on dict-key order or run time. Selection must be a pure deterministic function of the report fields. Non-determinism breaks the auditability the Ch7 Tip requires.
• PROMPT_REFINEMENT chosen while knowledge_index is low. A low KI is a knowledge gap, not a reasoning slip. Branch 3 requires ki > ki_floor; if a low-KI report routes to a prompt fix, the sufficiency or KI field is wrong.
• FINE_TUNE chosen for a one-node, one-step reasoning failure. Heavyweight fix for a lightweight problem. Check low_infogain_steps count and KI; a localized failure should route to PROMPT_REFINEMENT.
• CODE_MODIFICATION emitted by the router. It never should be. The router emits only the four report-driven interventions; code modification is a sandboxed SICA-loop decision, not an automated routing output.
• intervention_intensity returns fewer than five keys. Downstream validation tiering reads tier and risk; a missing key breaks the Graduated Validation Protocol admission decision.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report all 16 gates pass: insufficient-context -> RETRIEVAL_FIX; FORMAT_VIOLATION -> STRUCTURAL_CONSTRAINT; REASONING/2-low-steps/ki-0.91 -> PROMPT_REFINEMENT; REASONING/4-low-steps -> FINE_TUNE; KNOWLEDGE/low-ki -> FINE_TUNE; the intensity hierarchy ordering prompt < fine-tune < code-modification by risk; and the DevOps scenario report resolving to PROMPT_REFINEMENT.
2. Run the DevOps scenario. python cli.py scenario devops-prediction feeds the Ch7 running-example report (stripe-python 3.2.1 -> 3.3.0, timeout 30s -> 10s, checkout-service -> order-service -> fulfillment-service, fictional AWS account 123456789012) and must resolve to PROMPT_REFINEMENT, the lightest intervention, because the failure was a reasoning pattern localized to one node, not a knowledge gap.
3. Verify CLI help. python cli.py --help exits 0 and prints the SKILL.md description.

Security Posture

• Prompt injection. The diagnostic report is trusted internal data emitted by your own evaluation pipeline. If any report field (diagnosis, target_nodes) can be influenced by untrusted user input that reached the Layer 2 diagnostic, treat those strings as untrusted and validate before the routed action is executed downstream. The router itself makes no decision from free-text fields: it branches only on sufficient (bool), failure_type (enum), low_infogain_steps (list length), and knowledge_index (float), which bounds the injection surface.
• Data exfiltration. lib.py makes no network calls and no shell calls. It reads the report dict passed in and returns an Intervention. The CLI reads the report from the explicit --path argument and prints to stdout; the caller owns downstream piping.
• Privilege escalation. No shell invocation, no concatenated input to a shell, no file writes anywhere. The CLI only reads the --path report file. The router decides; it never applies a fix, so it cannot itself trigger a fine-tune job, edit a prompt, or modify code. CODE_MODIFICATION is never an output, so the highest-risk intervention cannot be reached through this seam.

Composition

• Composes downstream with Semantic Backpropagation (Ch7): the router picks the intervention type for a node; semantic backpropagation generates the neighbor-aware feedback that fills a PROMPT_REFINEMENT, and guards against a good single-node fix silently breaking a neighbor.
• Composes downstream with SEAL and TPT (Ch7): a FINE_TUNE output is the entry point to SEAL/TPT curriculum generation and retraining.
• Composes downstream with constrained generation / Outlines (Ch6): a STRUCTURAL_CONSTRAINT output attaches an output schema to the target node.
• Feeds the Graduated Validation Protocol (Ch7): the intensity tier and risk of the chosen intervention set the canary tier it enters at (a prompt refinement enters at Tier 1; a code modification enters at the strictest).
• Consumes the diagnostic report from the Ch7 Layer 0/1/2 evaluation pipeline and the execution-graph primitive (self-evolution/execution-graph) that the diagnosis is built on.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien) Chapter 7 — Self-Evolution and Evaluation, the section "Choosing the Right Intervention: Prompt, Fine-Tune, or Constrain?" It ports the select_intervention routing example (select_intervention) exactly, including the chapter thresholds (two or fewer low-InfoGain steps for a prompt fix, a knowledge index above 0.8 for a reasoning-only diagnosis) exposed as tunable parameters per the accompanying Tip. The self-modification intensity hierarchy (prompt lightest, weight adaptation middle, code modification heaviest) is drawn from the chapter's SICA / self-modification-intensity discussion, and the DevOps running example (PROMPT_REFINEMENT for the CausalAttributionNode premature-closure failure) is the chapter's worked loop closure.