Knowledge Crystallization Feedback Loop

Knowledge Crystallization Feedback Loop

Problem

AI agents that compete in multiple ML competitions generate enormous experience, but without systematic crystallization:

• Same mistakes repeat (e.g., submitting without format check — 5 times)
• Hard-won insights are lost between sessions
• Knowledge files grow to 100+ items with no organization
• Retrieval fails when needed (can't find relevant lesson for new competition)

The Crystallization Cycle

    ┌─────────────────────────────────────────┐
    │ 1. EXPERIMENT                            │
    │    Run competition / try approach        │
    └──────────────┬──────────────────────────┘
                   ▼
    ┌─────────────────────────────────────────┐
    │ 2. IDENTIFY OUTCOME                      │
    │    Success? Failure? Marginal? Dead end? │
    └──────────────┬──────────────────────────┘
                   ▼
    ┌─────────────────────────────────────────┐
    │ 3. EXTRACT PATTERN                       │
    │    What's the 1-sentence rule?           │
    │    What evidence supports it?            │
    │    When does it apply? When NOT?         │
    └──────────────┬──────────────────────────┘
                   ▼
    ┌─────────────────────────────────────────┐
    │ 4. CLASSIFY                              │
    │    feedback = "don't do X" (anti-pattern)│
    │    learned  = "do X for result Y"        │
    │    reference = "X is located at Y"       │
    └──────────────┬──────────────────────────┘
                   ▼
    ┌─────────────────────────────────────────┐
    │ 5. STORE WITH TRIGGERS                   │
    │    Frontmatter description = when to use │
    │    Tags = search keywords                │
    │    Links = [[related-skills]]            │
    └──────────────┬──────────────────────────┘
                   ▼
    ┌─────────────────────────────────────────┐
    │ 6. ACTIVATE ON MATCH                     │
    │    New task matches trigger → load skill │
    │    Verify still current → apply or update│
    └─────────────────────────────────────────┘

The 3-Layer Knowledge Architecture

L1: Core Capabilities (Monthly Update)

Basic ML competencies that rarely change:

• Cross-validation setup
• Feature engineering basics
• Model selection (GBDT vs NN vs ensemble)
• Submission format validation

L2: Domain Skills (Per-Competition Update)

Competition-specific patterns organized as SKILL.md files:

• autogluon-first — when to use AutoGluon
• trueskill-simulation-competition-strategy — simulation competitions
• kaggle-oof-lb-validation-protocol — validation methodology
• 31+ skills total, each with trigger conditions + evidence

L3: Cross-Domain Principles (Quarterly Review)

Abstract lessons that transfer across ALL competitions:

• work_smart_not_hard — external data ROI ~7× self-training
• local_optimum_trap — 3× <0.001 improvement → pivot
• controlled_variable — one change per experiment
• signal_dilution — correlated models > 0.97 = worse ensemble

What to Crystallize vs What to Forget

ALWAYS Crystallize

NEVER Crystallize (Forget Instead)

The Forgetting Protocol

Memory files become stale. Review quarterly:

For each memory file:
  1. Is the fact still true? (verify against current state)
  2. Was it superseded by a newer finding?
  3. Is the referenced file/path still valid?
  
  If NO to any → UPDATE or DELETE

Example: "task017 is best implemented with 1-node Conv (900 params)" — if a new technique reduces to 90 params, UPDATE the memory, don't create a new one.

Activation Triggers: Writing Good Descriptions

The description field in frontmatter is the ACTIVATION TRIGGER. It determines whether the skill is loaded for a new task.

Bad description (too vague):

description: "Tips for Kaggle competitions"

Good description (specific triggers):

description: |
  Use when: (1) Your OOF score improved but LB didn't, (2) You need to decide
  whether to submit based on OOF alone, (3) OOF/LB gap larger than 1%.
  Covers: 5 sources of OOF/LB gap, confirmation protocol, 3-strike rule.

AutoMem Integration (Graph + Vector Memory)

For agents with AutoMem (FalkorDB + Qdrant) deployed:

1. Store: key decisions → AutoMem graph with typed relationships
2. Recall: semantic query returns related decisions + alternatives considered
3. Multi-hop: "why PostgreSQL?" → finds the decision + alternatives + principle
4. Consolidation: merge duplicate memories, strengthen important ones

vs File-based memory:

Evidence

• 4+ months of Claude Code usage
• 20+ competitions crystallized into 31+ skills
• 109+ individual memory entries (AutoMem)
• Key validated patterns:
  • "0.6 × BEST_PUBLIC" rule (5+ competitions, mean ratio 0.99)
  • "3-strike rule" for OOF plateau (TPS May 2022, 27 configs converged)
  • "Re-submit = reset convergence" (PTCG, 8+ submissions tracked)
  • "Pipeline trim for code competitions" (ROGII, Pipeline A OOF 10.38)

Common Crystallization Failures