Analyzing Knowledge Gaps

Analyze Knowledge Gaps

Analyze Factory knowledge base to identify missing, shallow, or stale content

Systematically analyze the Factory's knowledge base against the topic taxonomy to identify gaps, shallow coverage, stale content, and cross-reference issues. Produces prioritized recommendations for knowledge extension.

Gap Types

Process

1. Review the task requirements.
2. Apply the skill's methodology.
3. Validate the output against the defined criteria.

Step 1: Run CLI Gap Analysis

Use the Factory CLI to get a structured gap report:

python cli/factory_cli.py --analyze-gaps

For scoped analysis:

# Analyze gaps for a specific blueprint
python cli/factory_cli.py --analyze-gaps --gap-scope blueprint --gap-filter python-fastapi

# Analyze gaps for a domain
python cli/factory_cli.py --analyze-gaps --gap-scope domain --gap-filter ai_development

# Analyze a specific topic
python cli/factory_cli.py --analyze-gaps --gap-scope topic --gap-filter constitutional_ai

Step 2: Interpret Results

The analyzer outputs a prioritized list:

Gap Analysis Results
====================

CRITICAL (Missing):
  - constitutional_ai (ai_development > safety_alignment)
    Required depth: 3, Current: 0
    Recommendation: Create {directories.knowledge}/constitutional-ai-patterns.json

HIGH (Shallow):
  - prompt_injection (ai_development > safety_alignment)
    Required depth: 2, Current: 1
    Source: security-patterns.json (1 mention)
    Recommendation: Expand with dedicated section

MEDIUM (Incomplete):
  - function_calling (ai_development > tool_use)
    Required depth: 2, Current: 1
    Missing: error handling, parallel execution

Step 3: Check Blueprint Coverage

For blueprint-specific analysis:

python cli/factory_cli.py --coverage-report ai-agent-development

Output shows:

• Topics covered by the blueprint's knowledge files
• Required topics from taxonomy that are missing
• Coverage percentage
• Specific recommendations

Step 4: Prioritize Extensions

Based on gap analysis, prioritize by:

1. Critical gaps - Block blueprint functionality
2. High gaps - Degrade blueprint quality
3. Blueprint alignment - Gaps in popular blueprints
4. User requests - Topics users have asked about

Using the Python API

For programmatic access:

from scripts.analysis.knowledge_gap_analyzer import KnowledgeGapAnalyzer, GapPriority
from pathlib import Path

# Initialize analyzer
factory_root = Path(".")
analyzer = KnowledgeGapAnalyzer(
    knowledge_dir=factory_root / "knowledge",
    taxonomy_dir=factory_root / "scripts" / "taxonomy"
)

# Run full analysis
result = analyzer.analyze("agent_taxonomy.json")

# Get gaps by priority
critical_gaps = [g for g in result.gaps if g.priority == GapPriority.CRITICAL]
high_gaps = [g for g in result.gaps if g.priority == GapPriority.HIGH]

# Show what's missing
for gap in critical_gaps:
    print(f"{gap.topic.name}: {gap.gap_type.value}")
    print(f"  Required depth: {gap.coverage.required_depth}")
    print(f"  Target file: {directories.knowledge}/{gap.topic.name.replace('_', '-')}-patterns.json")

Output Format

When reporting gaps to the user:

## Knowledge Gap Analysis

### Summary
- **Total topics analyzed**: 45
- **Adequate coverage**: 32 (71%)
- **Gaps identified**: 13

### Critical Gaps (Missing)

| Topic | Domain | Required Depth | Recommendation |
|-|--|-|-|
| constitutional_ai | ai_development | 3 | Create new knowledge file |
| prompt_caching | ai_development | 2 | Create new knowledge file |

### High Priority Gaps (Shallow)

| Topic | Current Depth | Required | Source File | Action |
|-||-|-|--|
| function_calling | 1 | 2 | mcp-patterns.json | Expand section |

### Recommendations

1. **Immediate**: Create `constitutional-ai-patterns.json` for safety alignment
2. **Short-term**: Expand `mcp-patterns.json` with function calling patterns
3. **Long-term**: Add cross-references between related knowledge files

Integration with extend-knowledge

After gap analysis, trigger the extend-knowledge skill:

Identified gap: constitutional_ai (priority: CRITICAL)

→ Invoke extend-knowledge skill with topic "constitutional_ai"
→ Use web search to gather current best practices
→ Create {directories.knowledge}/constitutional-ai-patterns.json
→ Update manifest and documentation

Fallback Procedures

Best Practices

• Run gap analysis before extending knowledge to prioritize what's most critical - focus on missing topics that block blueprint functionality first
• Use scoped analysis (--gap-scope) when investigating specific blueprints or domains rather than analyzing everything at once
• Prioritize gaps by impact: critical missing topics > high-priority shallow coverage > medium-priority stale content
• Cross-reference gap analysis with user requests and blueprint requirements to ensure alignment with actual needs
• Review gap analysis results periodically (monthly) to maintain knowledge base health and catch drift early
• After identifying gaps, immediately invoke extend-knowledge skill to fill critical gaps before they accumulate

References

• {directories.scripts}/analysis/knowledge_gap_analyzer.py - Core analyzer implementation
• {directories.scripts}/taxonomy/agent_taxonomy.json - Topic definitions and required depths
• {directories.knowledge}/manifest.json - Current knowledge file registry
• {directories.skills}/extend-knowledge/SKILL.md - Follow-up skill for gap filling

When to Use

This skill should be used when strict adherence to the defined process is required.

Prerequisites

• Basic understanding of the agent factory context.
• Access to the necessary tools and resources.