Semantic Backprop Attributor

Semantic Backpropagation Attributor

Overview

Agents are graphs, not pipelines. The dangerous failure mode in a self-evolving system is not a bad update. It is a good update to one node that silently breaks another. In a deeply interconnected graph, improving a component in isolation causes "action at a distance" failures that are hard to trace. Semantic backpropagation is the mechanism that prevents this.

The idea adapts the chain rule. In numerical backpropagation, gradients flow backward through a computational graph, updating each parameter by how it contributed to the loss. Semantic backpropagation does the same in natural language: the gradient is a structured description of the required change, and it flows backward through the execution graph from the point of failure.

Neighbor-awareness is the decisive part. When generating feedback for node v based on what successor w needed, the feedback includes not just v's output and w's error but the outputs of ALL OTHER predecessors of w. That context is what makes the feedback precise.

The chapter's concrete example: an Extractor pulls "Revenue: $10M", a CurrencyConverter converts it to "EUR 9.5M", and a Validator (which also received a DateChecker's "Date: 2022") flags that the exchange rate was 0.9, not 0.95. Without neighbor context, feedback to the Extractor reads "your $10M led to a conversion error" and the Extractor might wrongly change its extraction. With neighbor context, the error is correctly assigned to the CurrencyConverter's rate lookup and the Extractor is left unchanged. the neighbor-aware feedback example shows the same shape for a DevOps CausalAttributionNode, with ChangelogRetrieval and KnowledgeGraphQuery as the neighbor predecessors.

Honesty note on the metaphor: "backpropagation" here is an analogy, not a mechanism. A numerical gradient is exact and deterministic; this skill's "gradient" is credit assignment produced by LLM judgment over the execution graph — structured, neighbor-aware, and far better than unstructured blame, but still a hypothesis about causality, not a derivative. Treat every attribution as a claim to verify (rerun the trace with the blamed node patched) before committing an intervention on it. What IS deterministic in this skill: the graph traversal, the neighbor-context assembly, and the routing of the verdict.

When to Use

• After a diagnostic report has localized a failing node and you need feedback that will not break the node's neighbors
• Multi-node agent graphs where a fix to one node could ripple ("action at a distance")
• Before any intervention (prompt refinement, SEAL curriculum, fine-tune) so the change is grounded in what every connected node needed
• Attributing a surfaced error to its true origin when the node that flagged it is not the node that caused it

Phrases: "semantic backpropagation", "neighbor-aware feedback", "which node caused this", "credit assignment", "textual gradient", "action at a distance", "leave the Extractor unchanged".

When NOT to Use

• Single-node or linear pipelines with no sibling predecessors: there is no neighbor context to add and no action-at-a-distance to prevent
• As the intervention itself: this decides WHERE and WHAT should change; SEAL / TPT / prompt refinement make the change stick (chapter, A Suite of Self-Improvement Frameworks)
• Before diagnosis: run the execution-graph and diagnostic layers first so you have a localized failing node to backpropagate from
• When neighbor outputs are untrusted external content: sanitize first (see Security Posture) before folding them into feedback

Process

Rationalizations

Red Flags

• Feedback sent to the node that surfaced the error rather than the node that caused it. Attribution collapsed to failure_node. Check that a predecessor carrying the wrong value is being implicated.
• neighbor_context is empty on a node with sibling predecessors. The successor was mis-identified or predecessors_of returned nothing. The feedback is now un-grounded and can trigger a wrong fix.
• neighbor_context includes the target node's own output. The exclusion in neighbor_context_for was bypassed. Neighbor evidence must be about the siblings, not the target.
• Attribution changes the Extractor in the currency case. Incorrect credit assignment. The DateChecker + Validator evidence exonerates the Extractor; the CurrencyConverter's rate lookup is the origin.
• Synthesized feedback names no neighbor evidence. An empty feedback_text produced a generic string. The default must weave in the neighbor outputs.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report 6/6: predecessors_of returns correct parents; neighbor_context_for excludes the target and includes the sibling predecessor; the currency case attributes to CurrencyConverter (not Extractor); the devops feedback contains the neighbor evidence (timeout 30s->10s present in input); SemanticFeedback.to_dict round-trips.
2. Run the currency scenario. python cli.py scenario currency shows attribution landing on CurrencyConverter with the Extractor left unchanged.
3. Run the devops scenario. python cli.py scenario devops-prediction emits the the neighbor-aware feedback example neighbor-aware feedback for the CausalAttributionNode.
4. Verify CLI help. python cli.py --help exits 0 and prints the SKILL.md description.

Security Posture

• Prompt injection. node_outputs and neighbor evidence are folded into the textual gradient. If any node output originates from untrusted external content (a scraped changelog, a retrieved document), sanitize it before it enters the feedback string. Treat neighbor outputs as untrusted until validated; a malicious changelog line could inject instructions into the synthesized feedback that a downstream prompt-update step would then apply.
• Data exfiltration. lib.py makes no network calls and no file writes. The CLI reads a caller-supplied --path JSON and prints results to stdout; the caller owns downstream piping. Nothing leaves the process.
• Privilege escalation. No shell invocation, no concatenated input to a shell, no file writes outside the given paths. Attribution is a deterministic numeric-evidence heuristic over in-memory dicts; the production swap to an LLM judge is a documented seam, not an ambient capability.

Composition

• Composes on top of the execution-graph primitive: the edges and node_outputs it operates over are the graph that primitive captures.
• Composes after the diagnostic report: attribution starts from the localized failing node the diagnosis produced.
• Feeds into the intervention frameworks (SEAL targeted curriculum, TPT, prompt refinement): the SemanticFeedback is their input. This skill decides where and what; they make the change stick.
• Generator pattern / Ghosh Workflow. It generates a structured feedback artifact by orchestrating graph traversal, attribution, and synthesis across several nodes, one workflow layer above a single primitive.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien) Ch7 — Semantic Backpropagation: Coherent Evolution Across the Graph, plus the neighbor-aware feedback example (Neighbor-aware semantic feedback for the causal attribution node). The chapter credits TextGrad for the foundational insight that textual feedback can serve as a gradient signal, and adapts the chain rule so a structured natural-language gradient flows backward through the execution graph with neighbor-aware context.