Folder Analyzer Optimizer

Folder Analyzer & Optimizer

Skill generique de conversion documentaire. Transforme n'importe quel dossier de documents en knowledge base LLM-navigable : miroir Markdown individuel par document + index hierarchiques par dossier (LLM Wiki pattern, Karpathy).

Fonctionne pour tout domaine : ESG/B Corp, legal, M&A, audit, RH, due diligence, etc.

Pipeline hybride PyMuPDF/python-docx/pptx valide en production : 85-95% de reduction de tokens vs Read-tool-only.

Principe LLM Wiki (Karpathy)

Trois couches :

1. Sources brutes — documents originaux, jamais modifies
2. Wiki — pages Markdown individuelles (document.md) + index de dossier (_index.md)
3. Schema — CLAUDE.md du projet, configure les mappings metier

Navigation : lire _index.md racine → aller au sous-dossier pertinent → lire le document cible. Tous les fichiers sont visibles dans les index, meme ceux non parsables (images, .doc legacy) : rien n'est invisible dans la knowledge base.

Navigation intra-document via arbre

Pour les documents longs (≥ 50 pages) avec une structure exploitable (TOC native ou headings clairs), un arbre JSON sidecar est généré à côté du .md : document.tree.json. Cet arbre, inspiré de PageIndex (Vectify AI) mais porté en zero-token, contient les sections du document avec leurs plages de pages et un aperçu de 300 caractères par nœud.

L'agent navigue ainsi :

1. _parsed/_index.md racine → choisit le dossier
2. _parsed/dossier/_index.md → voit la section "Documents structurés" avec l'aperçu des chapitres des gros documents
3. Lit le .md ciblé sur la plage de pages identifiée — pas la totalité du document

Voir Step 6.5 pour les détails.

Deux modes d'invocation

Le mode incremental est le mode par defaut quand un _parse_manifest.json existe deja.

Domain mapping (optionnel)

Le skill accepte un domain_mapping : dictionnaire {nom_dossier: [tags]} injecte dans les index.

Exemple B Corp V7 :

DOMAIN_MAPPING = {
    "Gouvernance": ["PSG1-5", "FR1-3", "GACA1-2"],
    "Collaborateurs": ["FW1-2", "JEDI1-2", "HR1"],
    "Environnement": ["ESC2", "ESC5", "CA2", "CA3"],
    "Clients": ["PSG4"],
}

Si fourni, chaque _index.md affiche les tags correspondants. Si non fourni, les index sont generes sans enrichissement metier.

Output

{project_root}/
+-- _parsed/
    |-- _parse_manifest.json    # Manifest JSON: etat de parsing + tree_* fields
    |-- _inventory.md           # Inventaire tabullaire + table de navigation par dossier
    |-- _index.md               # ROOT INDEX: navigation vers tous les sous-dossiers
    |-- subfolder_A/
    |   |-- _index.md           # INDEX: liste tous les fichiers (parsés + non-parsés)
    |   |-- document1.md        # Miroir Markdown du document original
    |   |-- document1.tree.json # Arbre sidecar (si document1 >= 50p et structuré)
    |   +-- document2.md
    +-- subfolder_B/
        |-- _index.md
        +-- report.md

Step 0: Dependency check and install

Before any parsing, verify that the required Python packages are installed. If any are missing, ask the user for confirmation before installing.

import subprocess, sys, io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')  # Windows fix

REQUIRED = {
    "pymupdf": "pymupdf>=1.27.0",
    "docx": "python-docx>=1.1.0",
    "pptx": "python-pptx>=1.0.0",
}

missing = []
for import_name, pip_name in REQUIRED.items():
    try:
        __import__(import_name)
    except ImportError:
        missing.append(pip_name)

if missing:
    print(f"Missing dependencies: {', '.join(missing)}")
    # Ask user confirmation before installing
    subprocess.check_call([sys.executable, "-m", "pip", "install"] + missing)
    print("Dependencies installed successfully.")
else:
    print("All dependencies available.")

Package roles:

Step 1: File discovery and pre-analysis

1a. Identify the source folder

The source folder is either:

• Explicitly provided by the user: "Analyse le dossier X"
• The current project root (if the user says "Analyse ce dossier")
• Auto-discovered from CLAUDE.md or project structure

1b. Recursive file listing

List all files in the source folder recursively. For each file, classify by extension:

1c. PDF pre-analysis (PyMuPDF, ~0.3s per file)

For every PDF file, run PyMuPDF pre-analysis to count images and words per page:

import pymupdf

def preanalyze_pdf(file_path):
    doc = pymupdf.open(str(file_path))
    page_count = len(doc)
    total_images = 0
    total_words = 0
    page_details = []

    for page_num in range(page_count):
        page = doc[page_num]
        text = page.get_text("text")
        images = page.get_images()
        words = len(text.split())
        total_images += len(images)
        total_words += words
        page_details.append({
            "page": page_num + 1,
            "words": words,
            "images": len(images)
        })

    doc.close()

    avg_images = total_images / page_count if page_count > 0 else 0
    avg_words = total_words / page_count if page_count > 0 else 0

    if avg_words < 10:
        classification = "SCANNED"
    elif avg_images < 25 and total_images < 1500 and avg_words > 100:
        classification = "TEXT-HEAVY"
    else:
        classification = "MIXED"

    image_heavy_pages = [p["page"] for p in page_details
                         if p["images"] >= 3 and p["words"] < 150]

    return {
        "page_count": page_count,
        "total_images": total_images,
        "total_words": total_words,
        "avg_images_per_page": round(avg_images, 2),
        "avg_words_per_page": round(avg_words, 2),
        "classification": classification,
        "image_heavy_pages": image_heavy_pages
    }

Classification thresholds (optimized — validated on 126-file autoresearch dataset + 90-file scanned corpus):

Smart MIXED page selection: A page needs Read tool only if images >= 3 AND words < 150. Pages with enough text (>= 150 words) have annotations that describe the visual content. This reduces Read tool usage by 70-90% on large consulting decks (e.g., 437-page VDD: 43 pages instead of 437).

Step 2: Extraction routing

Based on classification from Step 1, route each file to the appropriate extractor.

Routing matrix (summary)

Decision rule: Route to SCANNED first (avg_words < 10), before attempting any free extractor. Running PyMuPDF text extraction on a scanned PDF wastes time and produces PARTIAL status with 0 words — a silent failure.

Detailed implementation, code, and edge cases: see references/pdf-routing.md.

Step 3: Manifest management

The manifest file _parsed/_parse_manifest.json is the single source of truth for parsing state. It tracks every file, its status, extractor used, classification, and cache metadata.

See examples/sample_manifest.json for a complete example.

Key manifest fields: source_path, output_path, mtime, parsed_at, page_count, word_count, status (OK/PARTIAL/FAILED/SKIP_*), extractor, routing_classification, preanalysis, error, tree_status, tree_path, tree_generated_at.

Save manifest after each file (crash resilience). Use mtime comparison for cache invalidation.

Detailed schema table, cache invalidation code, and crash resilience: see references/manifest-schema.md.

Step 4: Quality gates

After extraction, calculate average words per page and assign a quality status:

FAILED files do NOT block the pipeline. They are logged in the manifest and reported in the inventory. The user can manually review or provide alternative formats.

Mandatory vision fallback check (post-extraction)

After the initial extraction pass, always inspect the PARTIAL distribution before declaring success:

partials = [(k, v) for k, v in manifest["files"].items() if v.get("status") == "PARTIAL"]
near_zero = [p for p in partials if p[1].get("word_count", 0) <= 10]

if near_zero:
    pct = len(near_zero) / max(len(partials), 1) * 100
    print(f"WARNING: {len(near_zero)} PARTIAL files have word_count <= 10 ({pct:.0f}% of partials)")
    print("These are likely scanned/image-based PDFs. Applying Read tool vision fallback.")
    # → Re-route through Step 2a (SCANNED route) for all near_zero files

Alert triggers — investigate immediately if any of these are true after a full scan:

• More than 20% of PDFs are PARTIAL
• Any PARTIAL file has word_count <= 10 (near-zero = scanned, not partial)
• Full scan completed faster than 1 file/second on average (speed anomaly = no real content)
• Median word count across PARTIAL files < 20

Step 5: Inventory generation

Generate _parsed/_inventory.md — vue tabullaire complète regeneree a chaque run (full ou incremental).

Inventory format

Sections: Navigation LLM Wiki (table of folders → _index.md links with parsed/total counts + domain tags), Resume (counts by type: PDF/DOCX/PPTX/XLSX/images/.doc), Inventaire tabullaire (row per file: #, path, type, status, word count).

Header metadata: Source path, scan date, mode (Full scan / Incremental with N new/modified/deleted), duration.

Step 6: LLM Wiki — hierarchical folder indexes

Generer un _index.md pour CHAQUE dossier (y compris racine et sous-dossiers intermediaires). C'est la couche de navigation centrale de la knowledge base.

Principe

Chaque _index.md contient :

1. Les sous-dossiers directs avec liens vers leurs propres _index.md et comptage fichiers
2. Les fichiers directs du dossier, groupes en deux sections :
   • Parsés : lien vers le .md + metadata (pages, mots, statut)
   • Non parsés (référence) : nom + raison du skip — visibles, jamais omis
3. Les tags metier si domain_mapping fourni

Regle critique : visibilite totale

Tout fichier presente dans le dossier source doit apparaitre dans l'index, quel que soit son statut. Un fichier SKIP_IMAGE ou SKIP_LEGACY_FORMAT reste visible avec son label. L'objectif est qu'une personne (ou un LLM) lisant l'index ait une vue exhaustive du contenu du dossier source, y compris les elements non accessibles en texte.

Format _index.md

# [Dossier] — Index

**Standards/tags** : [tags metier si domain_mapping fourni]

*Index genere le [date]*

---

## Documents structurés
<!-- Section incluse UNIQUEMENT si au moins un document du dossier a un .tree.json -->

### [filename.pdf] — [doc complet]([filename.md]) · [arbre]([filename.tree.json])
*N p. | N,NNN mots | structure [pymupdf_toc|markdown_regex_headings|chunked]*

- **[Section niveau 1]** *(p. X-Y)* — [preview 80 caractères…]
  - [Section niveau 2] *(p. X-Y)*
  - [Section niveau 2] *(p. X-Y)*

---

## Sous-dossiers

- [SubA/](_parsed/SubA/_index.md) — N/M docs parsés [— Tags: X, Y]
- [SubB/](_parsed/SubB/_index.md) — N/M docs parsés

## Documents

### Parsés

- [rapport.pdf](_parsed/dossier/rapport.md) — 12p | 3,200 mots
- [contrat.docx](_parsed/dossier/contrat.md) — 850 mots *(extraction partielle)*

### Non parsés (référence)

> Ces fichiers existent dans le dossier source mais n'ont pas été parsés en texte.
> Ils sont listés ici pour que la knowledge base soit exhaustive.

- `photo_equipe.jpg` — **[IMAGE .JPG]** *(asset visuel — non parsé)*
- `ancien_contrat.doc` — **[LEGACY .DOC]** *(non parsé — convertir en .docx)*
- `export.eml` — **[FORMAT NON SUPPORTÉ .EML]**

Règles de rendu de la section "Documents structurés"

• Inclusion : section affichée seulement si au moins un fichier du dossier a un tree_status ∈ {OK, DEGRADED, FORCED_CHUNKED}.
• Niveau 1 : toujours montré, preview tronqué à 80 caractères inline (suffixe …).
• Niveau 2 : toujours montré (max_depth_in_index_md = 2 par défaut), sans preview.
• Niveau 3+ : non affichés dans _index.md. L'agent ouvre le .tree.json directement pour les détails fins.
• Cap visuel : au-delà de 6 nœuds niveau 1, n'afficher que les 6 premiers + *(+ N sections de niveau 1)*.
• Badges :
  • quality DEGRADED → suffixer le titre du doc par *(structure partielle)*
  • heading_extraction chunked → suffixer par *(découpage par plages de pages)*

L'implémentation Python utilise render_structured_documents_section() depuis scripts/tree_builder.py (cf. Step 6.5).

Implementation Python (key functions)

def write_folder_index(folder_rel, all_folders, folder_contents, manifest, output_root, domain_mapping=None):
    """Generates _index.md for one folder: domain tags header, Documents structurés
    (calls render_structured_documents_section() from scripts/tree_builder.py),
    Sous-dossiers list, then Parsés / Non parsés file lists."""

def format_file_entry(file_key, entry) -> str:
    """Markdown list item for a file. Parsed: linked with page/word count metadata.
    Skipped/Failed: backtick name + SKIP_IMAGE/SKIP_LEGACY_FORMAT/FAILED label."""

def get_domain_tags(folder_str, domain_mapping) -> list:
    """Tags for a folder path by matching path parts against domain_mapping keys."""

Incremental: regenerer tous les index

Les _index.md sont toujours regeneres en totalite a chaque run (full ou incremental), car l'ajout/suppression d'un fichier dans un dossier affecte son index. Cout negligeable (pur Python, zero tokens).

Step 6.5 : Tree builder (intra-document navigation)

Pour chaque document de ≥ 50 pages avec une structure exploitable (TOC native ou ≥ 5 headings de bonne qualité), génère un arbre JSON sidecar document.tree.json. Permet à l'agent de naviguer par section/plage de pages sans charger le .md complet.

Zero-token strict : aucun appel LLM. Toute la logique vit dans scripts/tree_builder.py.

Configuration (summary)

TREE_CONFIG = {
    "enabled": True,
    "tree_threshold_pages": 50,
    "min_headings": 5,
    "max_depth_in_index_md": 2,
    "preview_chars": 300,
    "force_chunked_tree": False,
    "chunk_pages": 10,
}

Cascade d'extraction (summary)

1. PyMuPDF TOC — haute fiabilité si l'outline PDF est embarqué
2. Markdown regex headings — regex ^(#{1,4})\s+(.+)$ sur le .md, page inférée via ## Page N
3. Chunked fallback — tranches de chunk_pages pages (opt-in via force_chunked_tree)

Status → action

• OK — arbre affiché normalement dans _index.md
• DEGRADED — badge *(structure partielle)*
• FORCED_CHUNKED — badge *(découpage par plages de pages)*
• SKIPPED_* — pas d'arbre, doc reste en ### Parsés classique

Detailed implementation, pipeline code, quality gate, tree.json format, and performance: see references/tree-builder.md.

Incremental mode (default when manifest exists)

On every invocation, compare folder state with manifest:

Lazy upgrade: entries without tree_status field auto-generate .tree.json on next run — zero tokens, zero re-parse.

Detailed change detection logic, tree behavior in incremental mode, workflow, and report format: see references/incremental-mode.md.

Error handling

Principle: Never let a single file failure block the entire pipeline. Log the error, continue, report at the end.

Full error table (13 entries including tree builder errors): see references/error-handling.md.

Validation rules

• Every file in the source folder MUST appear in the manifest (100% coverage)
• Every OK/PARTIAL file MUST have a corresponding .md in _parsed/
• The manifest MUST be saved after each file (crash resilient)
• FAILED files MUST have an error message in the manifest
• The inventory MUST be regenerated on every run (not stale)
• Source paths in manifest MUST be absolute paths
• Output paths in manifest MUST be relative to project root
• .md files MUST start with the metadata header comment
• Every file with tree_status ∈ {OK, DEGRADED, FORCED_CHUNKED} MUST have a corresponding .tree.json
• Every .tree.json MUST have a corresponding .md (1:1 coupling)
• The mtime of .tree.json MUST be ≥ mtime of the source file (otherwise: stale, regenerate next run)
• Every node MUST have start_page and end_page in [1, page_count]
• Every node MUST have end_page >= start_page
• Every node preview MUST be ≤ preview_chars configured
• Every node_id MUST be unique within the tree
• Every node level MUST be ≥ 1

Performance expectations

PyMuPDF text extraction handles 90% of PDFs in seconds (zero tokens). Smart MIXED routing reduces Read tool usage by 70-90%. Token budget: ~700 tokens/Read tool page; 88% reduction vs naive routing (137K tokens for a 126-file data room vs ~1.1M). Tree builder: ~120 ms, 0 tokens per eligible doc; ~2.5 s overhead for a 200-file data room with 20 eligible docs.