Four Layer Eval Cascade

Four-Layer Evaluation Cascade

Overview

The execution graph tells you what happened. This framework is the diagnostic engine that tells you why. It operates as a sequential filter, moving from the most general failure cause to the most specific, and it stops at the first layer that catches the failure. Each layer asks a progressively narrower question (Ch7 Figure 7-1):

• Layer 0 (hallucination gate) gates on grounding. A lightweight NLI-style classifier scores whether the answer is grounded in the retrieved premise. The threshold is 0.85. It catches 60-70% of hallucinations at less than 5% of the compute of a full LLM judge call. Scores between 0.5 and 0.85 escalate to the full pipeline (the SLM-LLM flywheel); scores below 0.5 hard-block.
• Layer 1 (context evaluator) asks: did the agent even possess the information it needed? A failure here is a knowledge-representation or retrieval failure, not a reasoning failure. The verdict is binary (sufficient or not) with a missing_information list (J1-style).
• Layer 2 (cognitive fault isolator) splits a cognitive failure into two mutually exclusive categories. A KNOWLEDGE failure means coherent reasoning over wrong facts (low Knowledge Index). A REASONING failure means the right facts, badly connected (near-zero or negative InfoGain steps).
• Layer 3 (TIR-Judge) verifies quantitative claims by executing code against the KG. Its reward is correctness * format * tool, multiplicative so a confidently wrong but well-formatted answer scores zero.

The output is a structured diagnostic report (the chapter's diagnostic-report example): it names the failure mode, locates it by node ID, and prescribes an intervention. That report is what drives the self-improvement engine.

The worked anchor is the DevOps premature-closure autopsy (the chapter's InfoGain-trace and diagnostic-report examples): sufficient context, Knowledge Index 0.91, InfoGain trace [0.34, 0.29, 0.22, 0.03, -0.01, 0.19], low-InfoGain steps [4, 5], fault at CausalAttributionNode, recommended intervention PROMPT_REFINEMENT.

When to Use

• Autopsy a failed agent execution when you need to name WHY, not just THAT, it failed
• Route the fix: is this a retrieval problem, a prompt problem, or a fine-tuning problem?
• Cheap prescreening: run Layer 0 on every response before spending judge compute on Layers 1-3 (the evaluation budget starts the inference budget)
• Feed the self-improvement engine — Semantic Backpropagation needs the report's fault_location and low_infogain_steps

Phrases: "cognitive autopsy", "why did the agent fail", "evaluation cascade", "hallucination gate", "context sufficiency", "knowledge vs reasoning failure", "TIR-Judge", "recommended intervention", "premature closure".

When NOT to Use

• One-shot single-call agents. There is no reasoning trace to isolate at Layer 2 and no multi-step execution to autopsy. Log the call instead.
• As a replacement for the execution graph. This cascade reads the graph; it does not build it. Build the execution-graph primitive first.
• As the production classifier stack. Every layer here is a deterministic dev-time stand-in. Swap Layer 0 for GLiClass/DeBERTa NLI, Layer 1 for a J1 judge, Layer 3 for a TIR-Judge CodeExecutor before production.
• For mechanistic circuit-level diagnosis. The chapter is explicit: for most failures, stop the diagnostic process at Layer 2 rather than produce spurious mechanistic explanations.

Process

Rationalizations

Red Flags

• Every response reaches Layer 3. Layer 0 is not screening. Check the grounding heuristic and the 0.85 threshold; the gate exists to keep 60-70% of failures out of the expensive layers.
• Layer 1 always returns sufficient. No required_claims are being passed, so insufficiency cannot be detected. The default sufficient verdict is optimistic by design; supply the claims a correct answer requires.
• A REASONING verdict routes to FINE_TUNE. Premature closure (a short low-InfoGain tail with high KI) should route to PROMPT_REFINEMENT. A REASONING failure sent to fine-tuning is treating a prompt problem as a weights problem.
• low_infogain_steps is empty on a known-bad trace. The infogain_floor is too low, or the trace was not captured per step. The premature-closure trace must surface steps [4, 5].
• Composite reward is nonzero on a wrong quantitative claim. The reward was made additive somewhere. A wrong value must zero the composite.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report all gates passed, including: multiplicative TIR zeroes on a wrong answer; the cascade stops at Layer 0 on an ungrounded answer; the cascade stops at Layer 1 on insufficient context; Layer 2 classifies REASONING vs KNOWLEDGE by Knowledge Index; low_infogain_steps computes to [4, 5] from the premature-closure trace; the devops-autopsy scenario recommends PROMPT_REFINEMENT.
2. Run the DevOps scenario. python cli.py scenario devops-autopsy rebuilds the chapter's premature-closure diagnostic report: stops at Layer 2, REASONING, fault at CausalAttributionNode, low steps [4, 5], recommended PROMPT_REFINEMENT.
3. Verify CLI help. python cli.py --help exits 0 and prints the SKILL.md description.

Security Posture

• Prompt injection. The answer, context_premise, and required_claims are treated as untrusted strings. lib.py only tokenizes and substring- matches them; it never evaluates, executes, or interpolates them into a shell or an LLM prompt. In production the Layer 3 seam runs generated Cypher or Python against the KG. That CodeExecutor must sandbox the generated code and parameterize queries. The # TODO(production) at the Layer 3 seam is the place to enforce it.
• Data exfiltration. No network calls in lib.py. The cascade returns the diagnostic report to the caller. CLI JSON is printed to stdout; downstream piping is the caller's responsibility.
• Privilege escalation. No shell invocation, no eval/exec, no file writes. cli.py reads exactly one file, the execution JSON named by the explicit --path flag. Everything else is pure in-memory computation.

Composition

• Reads the execution-graph primitive (self-evolution/execution-graph). The graph supplies the per-node input/output, the InfoGain trace, and the fault node ID this cascade autopsies.
• Feeds Semantic Backpropagation and the self-improvement engine. The report's fault_location, low_infogain_steps, and recommended_intervention are the inputs that scope a structured feedback signal to the responsible node.
• Composes with the Anthropic agent-skills Reviewer pattern at the pattern-taxonomy layer and the Ghosh Workflow layer: the four sequential gates are a review pipeline whose output is a routing decision, not a code edit.
• Extends via the dual-pathway model (Luo et al.): a pathway_classification field (Q-anchored vs A-anchored) can annotate a Layer 2 KNOWLEDGE failure to decide fine-tune (Q-anchored) vs better retrieval (A-anchored). Out of scope here; the report shape leaves room for it.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien), Chapter 7 — Self-Evolution and Evaluation, "The Multi-Layered Evaluation Framework: A Cognitive Autopsy" section, plus the DevOps cognitive-autopsy worked example. Chapter examples realized here: 7-2 (zero-shot hallucination pregate), 7-3 (J1 context sufficiency judge), 7-4 (diagnostic report structure), 7-6 (tool-integrated three-component reward), and 7-16 / 7-17 (the premature-closure InfoGain trace and its diagnostic report). Named references: GLiClass (Knowledgator, 2025), Meta J1, MICRO-ACT DECOMPOSE, TIR-Judge (Xu et al.), and the SLM-LLM flywheel (Microsoft Research, 2024). Production substrate: NLI classifier + J1 judge + CodeExecutor issuing Cypher against the KG.