Xskill Self Improving Object

XSkill Self-Improving Graph Objects

Overview

The improvement mechanisms earlier in Ch7 (prompt refinement, SEAL data generation, fine-tuning) all modify the agent itself. Knowledge augmentation is lighter: it accumulates knowledge from past executions and retrieves it at inference time, touching neither the model nor its prompts. The motivating measurement from the chapter: on the Kaggle GameArena chess benchmark, 78% of Gemini-2.5-Flash losses were illegal moves, rule violations rather than weak strategy. The agent kept repeating the same category of mistake because it had no memory of past failures.

XSkill (Jiang et al., 2026) extracts two complementary knowledge types from trajectories:

• Experiences operate at the action level. Each execution node's input, action, and outcome becomes a candidate experience record. Experiences alone reduce tool errors from 29.9% to 16.3% (a 45% reduction).
• Skills operate at the task level. The path from the root query node to a successful resolution node becomes a candidate multistep skill. Together with experiences, the average success rate rises from 33.6% to 40.3%.

Cognee closes the remaining gap: XSkill's skills are static artifacts. Cognee treats a skill as a first-class graph node (the SkillNode example) with execution records, a success rate, and an amendment history. Its four-stage pipeline is add (parse SKILL.md, compute content hash) then cognify (extract trigger phrases and complexity) then search (route by which skill SUCCEEDS at similar tasks) then learn (log an observation per execution). When a skill degrades, amendify() rewrites it against the last 10 failures, validates the amendment against held-out records, and rolls back on failure.

When to Use

• After an agent has accumulated execution traces and you want it to stop repeating avoidable mistakes without retraining
• Environments that drift (a Kubernetes API change, a new CI/CD stage) where a static SKILL.md silently goes stale
• Routing among several overlapping skills where description similarity picks the wrong one and demonstrated success should decide
• Building the self-evolution loop on top of the execution-graph substrate

Phrases: "learn from past failures", "self-improving skill", "amendify", "route by success rate", "XSkill", "Cognee", "knowledge retirement", "dual-stream extraction".

When NOT to Use

• One-shot agents that never see similar tasks again. No trace stream, nothing to distill.
• The raw execution graph itself. That is the execution-graph skill; this skill consumes its output.
• As a substitute for evaluation. This distills and routes knowledge; it does not score answer quality (use the Ch7 evaluation layers for that).
• When the environment is fixed and the skill has a stable high success rate. amendify() will not fire and the machinery is dead weight.

Process

Rationalizations

Red Flags

• route() returns the higher-similarity skill instead of the higher-success one. The ranking key the whole design depends on has been swapped back to similarity.
• A skill's version keeps climbing but success_rate never recovers. amendify() is validating against the same failures it is fitting to; the held-out set is not actually held out.
• extract_experiences returns one record per node. The divergence filter is off; retrieval will be flooded with routine successes.
• extract_skills emits a skill from two executions. min_support is not being enforced; an anecdote is being sold as a pattern.
• Experiences never lose weight. Retirement is not running; the agent serves 90-day-old lessons against a changed environment.
• content_hash does not change after amendify(). The definition was not actually rewritten; change detection is broken.

Non-Negotiable Verification

1. Run the benchmark battery. python cli.py benchmark must report all gates passing: extract_experiences yields one experience per diverged node; extract_skills needs >= min_support; success_rate computes correctly; amendify() fires only below 0.6 and bumps version + changes content_hash (and rolls back on validation failure); route ranks by success_rate not similarity (canary vs general); flag_stale_skills flags a sub-60% skill; retire_experiences half-weights a >90-day experience.
2. Run the DevOps scenario. python cli.py scenario devops-skill shows the general deployment skill with higher description similarity losing the route to the canary-deployment skill on demonstrated success.
3. Verify CLI help. python cli.py --help exits 0 and prints this SKILL.md description.

Security Posture

• Prompt injection. Lessons, amendment text, and trigger phrases come from execution traces and skill definitions. When those originate from untrusted tool outputs or user-supplied definitions, treat every string field as untrusted until validated: a malicious error or context could bias an amendment or a lesson. The heuristic extractors here do no instruction following; the LLM seams marked # TODO(production) are where an author-verifier separation and input sanitization must be added.
• Data exfiltration. lib.py makes no network calls and performs no shell invocation. It reads only the paths passed on the CLI and prints results to stdout; the caller owns any downstream piping.
• Privilege escalation. No shell, no eval, no dynamic import of trace content, no file writes outside the explicit --path inputs. amendify() mutates only the in-memory skill definition; persistence is the caller's decision, and the content_hash makes any change auditable.

Composition

• Consumes the execution-graph skill (Ch7 the execution-graph example): the nodes and successful subgraphs it extracts from are execution-graph output.
• Composes with the Ch7 evaluation layers, which score the outcomes that become experience and skill success labels.
• Pairs with the intervention router: knowledge augmentation ("what the agent remembers from doing") sits beside prompt refinement ("what the agent is told to do") and fine-tuning ("what the agent knows how to do"); the router selects among them by failure type.

Source Attribution

Distilled from Agentic GraphRAG (O'Reilly, by Anthony Alcaraz and Sam Julien) Chapter 7 — Self-Evolution and Evaluation, the Inference-Time Knowledge Augmentation section (the KnowledgeAccumulator dual-stream extraction) and the Skills as Self-Improving Graph Objects section (Example 7-22, SkillNode / SkillGraph with amendify() and success-rate routing). Named sources: XSkill (Jiang et al., 2026) for dual-stream continual learning and the 29.9% to 16.3% / 33.6% to 40.3% figures; Cognee for the add-cognify-search-learn skill-as-graph-object pipeline and the 90-day / 60%-over-10 retirement rules.