Self Improving Agent

Self-Improving Agent

"An AI agent that learns from every interaction, accumulating patterns and insights to continuously improve its own capabilities." — Based on 2025 lifelong learning research

Overview

This is a universal self-improvement system that learns from ALL skill experiences, not just PRDs. It implements a complete feedback loop with:

• Multi-Memory Architecture: Semantic + Episodic + Working memory
• Self-Correction: Detects and fixes skill guidance errors
• Self-Validation: Periodically verifies skill accuracy
• Hooks Integration: Auto-triggers on skill events (before_start, after_complete, on_error)
• Evolution Markers: Traceable changes with source attribution

Research-Based Design

Based on 2025 research:

The Self-Improvement Loop

┌─────────────────────────────────────────────────────────────────┐
│                    UNIVERSAL SELF-IMPROVEMENT                    │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Skill Event → Extract Experience → Abstract Pattern → Update  │
│        │                  │                │         │          │
│        ▼                  ▼                ▼         ▼          │
│   ┌─────────────────────────────────────────────────────┐       │
│   │              MULTI-MEMORY SYSTEM                      │       │
│   ├─────────────────────────────────────────────────────┤       │
│   │  Semantic Memory   │  Episodic Memory  │ Working Memory │  │
│   │  (Patterns/Rules)  │  (Experiences)    │  (Current)     │  │
│   │  memory/semantic/  │  memory/episodic/ │  memory/working/│  │
│   └─────────────────────────────────────────────────────┘       │
│                                                                 │
│   ┌─────────────────────────────────────────────────────┐       │
│   │              FEEDBACK LOOP                            │       │
│   │  User Feedback → Confidence Update → Pattern Adapt   │       │
│   └─────────────────────────────────────────────────────┘       │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

When This Activates

Automatic Triggers (via hooks)

Manual Triggers

• User says "自我进化", "self-improve", "从经验中学习"
• User says "分析今天的经验", "总结教训"
• User asks to improve a specific skill

Evolution Priority Matrix

Trigger evolution when new reusable knowledge appears:

Multi-Memory Architecture

1. Semantic Memory (memory/semantic-patterns.json)

Stores abstract patterns and rules reusable across contexts:

{
  "patterns": {
    "pattern_id": {
      "id": "pat-2025-01-11-001",
      "name": "Pattern Name",
      "source": "user_feedback|implementation_review|retrospective",
      "confidence": 0.95,
      "applications": 5,
      "created": "2025-01-11",
      "category": "prd_structure|react_patterns|async_patterns|...",
      "pattern": "One-line summary",
      "problem": "What problem does this solve?",
      "solution": { ... },
      "quality_rules": [ ... ],
      "target_skills": [ ... ]
    }
  }
}

2. Episodic Memory (memory/episodic/)

Stores specific experiences and what happened:

memory/episodic/
├── 2025/
│   ├── 2025-01-11-prd-creation.json
│   ├── 2025-01-11-debug-session.json
│   └── 2025-01-12-refactoring.json

{
  "id": "ep-2025-01-11-001",
  "timestamp": "2025-01-11T10:30:00Z",
  "skill": "debugger",
  "situation": "User reported data not refreshing after form submission",
  "root_cause": "Empty callback in onRefresh prop",
  "solution": "Implement actual refresh logic in callback",
  "lesson": "Always verify callbacks are not empty functions",
  "related_pattern": "callback_verification",
  "user_feedback": {
    "rating": 8,
    "comments": "This was exactly the issue"
  }
}

3. Working Memory (memory/working/)

Stores current session context:

memory/working/
├── current_session.json   # Active session data
├── last_error.json        # Error context for self-correction
└── session_end.json       # Session end marker

Self-Improvement Process

Phase 1: Experience Extraction

After any skill completes, extract:

What happened:
  skill_used: {which skill}
  task: {what was being done}
  outcome: {success|partial|failure}

Key Insights:
  what_went_well: [what worked]
  what_went_wrong: [what didn't work]
  root_cause: {underlying issue if applicable}

User Feedback:
  rating: {1-10 if provided}
  comments: {specific feedback}

Phase 2: Pattern Abstraction

Convert experiences to reusable patterns:

Abstraction Rules:

If experience_repeats 3+ times:
  pattern_level: critical
  action: Add to skill's "Critical Mistakes" section

If solution_was_effective:
  pattern_level: best_practice
  action: Add to skill's "Best Practices" section

If user_rating >= 7:
  pattern_level: strength
  action: Reinforce this approach

If user_rating <= 4:
  pattern_level: weakness
  action: Add to "What to Avoid" section

Phase 3: Skill Updates

Update the appropriate skill files with evolution markers:

<!-- Evolution: 2025-01-12 | source: ep-2025-01-12-001 | skill: debugger -->

## Pattern Added (2025-01-12)

**Pattern**: Always verify callbacks are not empty functions

**Source**: Episode ep-2025-01-12-001

**Confidence**: 0.95

### Updated Checklist
- [ ] Verify all callbacks have implementations
- [ ] Test callback execution paths

Correction Markers (when fixing wrong guidance):

<!-- Correction: 2025-01-12 | was: "Use callback chain" | reason: caused stale refresh -->

## Corrected Guidance

Use direct state monitoring instead of callback chains:
```typescript
// ✅ Do: Direct state monitoring
const prevPendingCount = usePrevious(pendingCount);

### Phase 4: Memory Consolidation

1. **Update semantic memory** (`memory/semantic-patterns.json`)
2. **Store episodic memory** (`memory/episodic/YYYY-MM-DD-{skill}.json`)
3. **Update pattern confidence** based on applications/feedback
4. **Prune outdated patterns** (low confidence, no recent applications)

## Promotion Policy

Self-improvement has two separate jobs:

1. **Capture** facts, corrections, failed assumptions, and reusable patterns as memory or proposal artifacts.
2. **Promote** only validated patterns into `SKILL.md`, `AGENTS.md`, docs, or CLI behavior.

Default to capture-first. Promote a change only when one of these is true:

- The user explicitly asks to update a skill or repository instruction.
- The same pattern recurs across multiple episodes.
- A focused test or review proves the current guidance is wrong or incomplete.
- The change is low-risk documentation that preserves existing behavior and is clearly traceable.

Promotion targets:

| Artifact | Use For | Approval Level |
|----------|---------|----------------|
| `memory/episodic/*.json` | Raw episode facts and signals | Auto |
| `memory/semantic-patterns.json` | Candidate reusable patterns with confidence | Auto |
| `memory/proposals/*.md` | Proposed skill/doc/code changes with evidence | Auto |
| `SKILL.md` / `references/` | Validated workflow guidance | Ask first unless user requested editing |
| `AGENTS.md` / repo rules | Cross-repo behavior or hard constraints | Ask first |
| CLI/runtime code | Automation semantics | Require tests |

## Self-Correction (on_error hook)

Triggered when:
- Bash command returns non-zero exit code
- Tests fail after following skill guidance
- User reports the guidance produced incorrect results

**Process:**

```markdown
## Self-Correction Workflow

1. Detect Error
   - Capture error context from working/last_error.json
   - Identify which skill guidance was followed

2. Verify Root Cause
   - Was the skill guidance incorrect?
   - Was the guidance misinterpreted?
   - Was the guidance incomplete?

3. Create Proposal
   - Write a proposal with evidence, affected skill names, and expected behavior
   - Add correction marker text in the proposal, not directly in the skill yet
   - Update related patterns in semantic memory with low initial confidence

4. Validate Fix
   - Test the corrected guidance
   - Ask user to verify

5. Promote
   - Apply the skill/doc/code change after validation or explicit approval
   - Keep the source episode/proposal id in the change note

Example:

<!-- Correction: 2025-01-12 | was: "useMemo for claimable ids" | reason: stale data at click time -->

## Self-Correction: Click-Time Computation

**Issue**: Using useMemo for claimable IDs caused stale data
**Fix**: Compute at click time for always-fresh data
**Pattern**: click_time_vs_open_time_computation

Self-Validation

Use the validation template in references/appendix.md when reviewing updates.

Hooks Integration

Runtime Trigger Source

agent-playbook self-improve reads skill chaining from each skill's SKILL.md frontmatter:

metadata:
  hooks:
    after_complete:
      - trigger: self-improving-agent
        mode: background
        reason: "Extract patterns"

Treat metadata.hooks as the source of truth. Do not maintain a second hardcoded hook map in runtime code. This keeps skill behavior auditable and lets Skill Creator style reviews inspect the same file that the agent executes.

Wiring Hooks in Claude Code Settings

For Claude Code, install hooks through agent-playbook init --hooks when possible. If you need manual setup, add hook entries to Claude Code settings at the appropriate user or project scope.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash|Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/pre-tool.sh \"$TOOL_NAME\" \"$TOOL_INPUT\""
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/post-bash.sh \"$TOOL_OUTPUT\" \"$EXIT_CODE\""
          }
        ]
      }
    ],
    "Stop": [
      {
        "matcher": "",
        "hooks": [
          {
            "type": "command",
            "command": "bash ${SKILLS_DIR}/self-improving-agent/hooks/session-end.sh"
          }
        ]
      }
    ]
  }
}

Replace ${SKILLS_DIR} with your actual skills path.

Additional References

See references/appendix.md for memory structure, workflow diagrams, metrics, feedback templates, and research links.

Best Practices

DO

• ✅ Learn from EVERY skill interaction
• ✅ Extract patterns at the right abstraction level
• ✅ Update multiple related skills
• ✅ Track confidence and apply counts
• ✅ Ask for user feedback on improvements
• ✅ Use evolution/correction markers for traceability
• ✅ Validate guidance before applying broadly
• ✅ Write proposals before mutating durable skill guidance
• ✅ Keep hook routing in metadata.hooks

DON'T

• ❌ Over-generalize from single experiences
• ❌ Update skills without confidence tracking
• ❌ Ignore negative feedback
• ❌ Make changes that break existing functionality
• ❌ Create contradictory patterns
• ❌ Update skills without understanding context
• ❌ Silently promote self-improvement findings into repo rules
• ❌ Duplicate hook definitions in CLI code and skill frontmatter

Quick Start

After a high-signal skill workflow completes, this agent can:

1. Analyzes what happened
2. Extracts patterns and insights
3. Writes memory and proposal artifacts
4. Promotes validated improvements only when approval or evidence is sufficient
5. Reports summary to user

References

• SimpleMem: Efficient Lifelong Memory for LLM Agents
• A Survey on the Memory Mechanism of Large Language Model Agents
• Lifelong Learning of LLM based Agents
• Evo-Memory: DeepMind's Benchmark
• Let's Build a Self-Improving AI Agent
• OpenCrabs local self-improving agent
• ELL-StuLife experience-driven lifelong learning