No Vendor Directory Scans

Vendor & Artifact Directory Exclusion Protocol

Overview

In modern software engineering, a typical repository containing 20 source files also contains 30,000+ third-party vendor files inside node_modules/, venv/, or Rust target/ directories.

When an autonomous agent runs an unconstrained recursive search or directory crawler without vendor exclusions, the tool attempts to index and return thousands of third-party package files.

Unconstrained vendor scans cause:

1. Catastrophic Context Overflow: A single list_dir or find on node_modules/ dumps 80,000+ tokens, instantly exhausting context windows.
2. Hallucinated Package Edits: Models mistakenly try to fix bugs by editing files inside node_modules/ or site-packages/ (which get wiped out on next npm install).
3. Execution Latency Stalls: Scanning 50,000 disk inodes takes 5 to 10 seconds of local I/O.

The Vendor & Artifact Directory Exclusion Protocol enforces strict, non-bypassable directory blacklists at the tool runner layer.

Unconstrained Directory Crawl vs. Vendor-Exclusion Guard

┌─────────────────────────────────────────────────────────────┐
│                 Directory Ingestion Comparison              │
│                                                             │
│  Unconstrained Recursive Crawl (32,000 Files / 90k Tokens): │
│  • `node_modules/lodash/package.json`                       │
│  • `node_modules/react/cjs/react.development.js`            │
│  • `venv/lib/python3.11/site-packages/pydantic/...`         │
│  • `.git/objects/4f/8a2b3...`                               │
│  ↳ 90,000 tokens billed, context window crashes!           │
│                                                             │
│  Vendor-Exclusion Guard (22 Files / 180 Tokens):            │
│  • `src/components/Button.tsx`                              │
│  • `src/services/auth.ts`                                   │
│  • `package.json`                                           │
│  ↳ 180 clean tokens (99.8% Reduction!), 100% active code    │
└─────────────────────────────────────────────────────────────┘

The Universal Vendor Exclusion Blacklist

Every agent tool (list_dir, grep_search, glob) must automatically filter out paths containing any of these segment names:

┌───────────────────────────────────────────────────────────────────────────┐
│ • JAVASCRIPT / TYPESCRIPT: `node_modules`, `.next`, `dist`, `build`, `.turbo`│
│ • PYTHON: `venv`, `.venv`, `__pycache__`, `.pytest_cache`, `site-packages`│
│ • RUST / GO: `target`, `vendor`                                           │
│ • JAVA / KOTLIN: `.gradle`, `build`, `target`, `.m2`                      │
│ • GIT & ENVIRONMENT: `.git`, `.idea`, `.vscode`, `.DS_Store`, `coverage`   │
└───────────────────────────────────────────────────────────────────────────┘

Production Python Directory Filter Middleware

from pathlib import Path
from typing import List, Set

BANNED_DIR_NAMES: Set[str] = {
    # Package managers & dependencies
    "node_modules", "venv", ".venv", "vendor", "site-packages",
    # Build artifacts & caches
    "dist", "build", "target", ".next", ".turbo", ".cache", "__pycache__",
    ".pytest_cache", ".mypy_cache", ".ruff_cache", "coverage",
    # Version control & IDEs
    ".git", ".svn", ".hg", ".idea", ".vscode", ".DS_Store"
}

def is_path_safe_from_vendor(path: Path, root: Path) -> bool:
    """Returns True if the path contains zero blacklisted vendor directory segments."""
    try:
        rel_parts = path.relative_to(root).parts
        return not any(part in BANNED_DIR_NAMES for part in rel_parts)
    except ValueError:
        return False

def safe_crawl_workspace(root: Path) -> List[str]:
    """Recursively lists repository files while strictly excluding vendor directories."""
    safe_files = []
    for p in root.rglob("*"):
        if p.is_file() and is_path_safe_from_vendor(p, root):
            safe_files.append(str(p.relative_to(root)).replace("\\", "/"))
    return safe_files

Benchmark Comparison

Listing repository contents for a modern Next.js + Python microservice:

Agent Operational Directive

MANDATORY: Filesystem exploration and search tools must automatically exclude node_modules, venv, dist, target, and .git by default. Never traverse into vendor dependency trees unless the user explicitly requests an audit of an external package.