Pablodiegoo Data Pro Skill Context Optimizer

Context Optimizer

This skill transforms large, monolithic documents into a modular .agent/ folder structure optimized for AI agent context consumption. The goal is to minimize context window usage while maximizing information accessibility.

Quick Reference

Workflow

Phase 1: Analyze Source Document

Before splitting, understand the document's structure:

# Preview structure without splitting
head -100 <input_file> | grep -E "^#{1,3} "

Identify:

• Hierarchical depth: How many heading levels exist?
• Content density: Are sections long enough to justify separate files?
• Semantic groupings: Which sections belong together?

Phase 2: Decompose with Script

Use the bundled script to split the document:

python3 .agent/skills/context-optimizer/scripts/decompose.py <input_file> -o <output_dir> [options]

Arguments

Examples

# Split by ## (default)
python3 decompose.py project_spec.md -o .agent/temp_split

# Split by # (top-level only)
python3 decompose.py large_doc.md -o chunks -l 1

# Custom pattern (e.g., numbered sections)
python3 decompose.py report.md -r "^(\d+\.\s+.+)$" -o sections

Output: Creates numbered files (01_section_name.md, 02_...) plus 00_INDEX.md and 00_preamble.md.

Phase 3: Organize into .agent Structure

After decomposition, manually categorize each chunk:

.agent/
├── memory/                 # Persistent context (always loaded)
│   ├── user_global.md      # User preferences, patterns
│   ├── project_facts.md    # Tech stack, constraints, conventions
│   └── decisions.md        # ADRs, architectural decisions
│
├── workflows/              # Step-by-step procedures
│   ├── deploy.md           # Deployment process
│   ├── review.md           # Code review checklist
│   └── testing.md          # Testing procedures
│
├── tasks/                  # Active work items
│   ├── backlog.md          # Feature backlog
│   ├── current_sprint.md   # Active sprint items
│   └── implementation_plan.md  # Current implementation plan
│
├── references/             # On-demand documentation
│   ├── api_docs.md         # API specifications
│   ├── schema.md           # Database/data schemas
│   └── external_libs.md    # Third-party library docs
│
└── skills/                 # Reusable capabilities
    └── <skill-name>/
        └── SKILL.md

Phase 4: Optimize Each File

For each categorized file, apply these optimizations:

Memory Files (High Priority)

• Maximum size: ~500 lines (always loaded)
• Format: Bullet points, tables, concise rules
• Avoid: Long explanations, examples (move to references)

Workflow Files

• Format: Numbered steps with clear actions
• Include: // turbo annotations for auto-runnable steps
• Structure: Prerequisites → Steps → Verification

Task Files

• Format: Checkbox lists ([ ], [/], [x])
• Include: Priority, deadlines, dependencies
• Update: Mark items as in-progress/done during work

Reference Files

• Maximum size: Unlimited (loaded on-demand)
• Include: Table of contents for files > 100 lines
• Add: Grep patterns in SKILL.md for large files

Phase 5: Cleanup

Remove temporary files and validate structure:

# Remove decomposition output
rm -rf .agent/temp_split

# Validate structure (optional)
find .agent -name "*.md" -exec wc -l {} \; | sort -n

Decision Matrix

Use this matrix to decide where content belongs:

┌─────────────────────────────────────────────────────────────────┐
│                    Is it a PROCESS/HOW-TO?                      │
│                              │                                  │
│              ┌───────────────┴───────────────┐                  │
│              ▼ YES                           ▼ NO               │
│     ┌────────────────┐              ┌────────────────┐          │
│     │   workflows/   │              │ Is it ACTIVE   │          │
│     │                │              │ work to track? │          │
│     └────────────────┘              └───────┬────────┘          │
│                                     ┌───────┴───────┐           │
│                                     ▼ YES           ▼ NO        │
│                              ┌──────────┐   ┌──────────────┐    │
│                              │  tasks/  │   │ Is it a RULE │    │
│                              │          │   │ or FACT?     │    │
│                              └──────────┘   └──────┬───────┘    │
│                                             ┌──────┴──────┐     │
│                                             ▼ YES         ▼ NO  │
│                                      ┌──────────┐  ┌──────────┐ │
│                                      │ memory/  │  │references/│ │
│                                      └──────────┘  └──────────┘ │
└─────────────────────────────────────────────────────────────────┘

Best Practices

1. Memory files are expensive — Keep them under 500 lines total
2. Use references for large docs — They're loaded only when needed
3. One concept per file — Easier to update and search
4. Add TOC to large files — For files > 100 lines, include a table of contents
5. Use consistent naming — snake_case.md for all files
6. Delete empty directories — Don't keep placeholder folders

Phase 3: Semantic Grouping (Optimized)

Automatically categorize your chunks into .agent/ folders:

python3 .agent/skills/context-optimizer/scripts/group_sections.py <split_dir> --move

This script analyzes each chunk for keywords and structural markers to suggest whether it belongs in memory/, workflows/, tasks/, or references/.

Phase 4: Organize into .agent Structure

...

Related Skills

• skill-creator: For creating new skills from decomposed content
• documentation-mastery: For formatting the resulting markdown files

Converted and distributed by TomeVault — claim your Tome and manage your conversions.