Comment Stripping Preprocessor

Comment Stripping & License Filter Preprocessor

Overview

Open-source libraries and enterprise codebases are laden with verbose license headers (30-line Apache 2.0 / MIT notices), commented-out legacy code blocks, and obvious inline comments (// increment i by 1).

When an agent ingests full files for debugging or refactoring, these non-executable text lines consume 30% to 50% of the input context window, diluting the model's self-attention across legal boilerplate rather than algorithmic logic.

The Comment Stripping Preprocessor Protocol automatically sanitizes source code before context ingestion - stripping license headers and decorative comments while strictly preserving semantic type hints, security invariants, and critical FIXME/SAFETY tags.

Verbose Source File vs. Sanitized Code Stream

┌─────────────────────────────────────────────────────────────┐
│                 Comment Stripping Mechanics                 │
│                                                             │
│  Raw Ingested File (580 Tokens):                            │
│  /*                                                         │
│   * Copyright (c) 2024 Enterprise Corp.                     │
│   * Licensed under the Apache License, Version 2.0...       │
│   * [25 lines of legal boilerplate]                         │
│   */                                                        │
│  // TODO: remove this legacy function next sprint           │
│  // function oldHelper() { return null; }                   │
│  export function hashPassword(pwd: string): string {        │
│    // Hash the password using bcrypt with 10 salt rounds    │
│    return bcrypt.hashSync(pwd, 10);                         │
│  }                                                          │
│                                                             │
│  Sanitized Code Stream (110 Tokens - 81% Reduction):        │
│  export function hashPassword(pwd: string): string {        │
│    return bcrypt.hashSync(pwd, 10);                         │
│  }                                                          │
└─────────────────────────────────────────────────────────────┘

The Semantic Retention Standard

Not all comments should be stripped. The preprocessor must enforce these retention rules:

Production Python Preprocessor Script

import re

def strip_code_comments(source_code: str, language: str = "ts") -> str:
    """Strips license blocks, dead comments, and obvious filler while preserving compiler directives."""
    # 1. Strip multi-line block comments (preserving @ts-ignore / JSDoc param types if desired)
    def block_comment_replacer(match):
        comment = match.group(0)
        if any(tag in comment for tag in ["@ts-ignore", "@ts-expect-error", "SAFETY:", "INVARIANT:"]):
            return comment
        return ""

    if language in ["ts", "js", "java", "c", "cpp", "go", "rs"]:
        # Block comments: /* ... */
        source_code = re.sub(r"/\*[\s\S]*?\*/", block_comment_replacer, source_code)
        # Line comments: // ... (skip if contains directives)
        lines = source_code.splitlines()
        clean_lines = []
        for line in lines:
            stripped = line.strip()
            if stripped.startswith("//"):
                if any(tag in stripped for tag in ["@ts-", "TODO:", "FIXME:", "SAFETY:"]):
                    clean_lines.append(line)
            else:
                # Remove trailing inline comment if simple
                clean_lines.append(re.sub(r"\s*//(?![@/]).*$", "", line))
        return "\n".join(clean_lines)

    elif language == "py":
        # Python: strip license block at top, strip trailing inline comments
        lines = source_code.splitlines()
        clean_lines = []
        for line in lines:
            stripped = line.strip()
            if stripped.startswith("#"):
                if any(tag in stripped for tag in ["type:", "noqa", "TODO:", "FIXME:", "SAFETY:"]):
                    clean_lines.append(line)
            else:
                clean_lines.append(re.sub(r"\s*#(?![#]).*$", "", line))
        return "\n".join(clean_lines)

    return source_code

Token Reduction Benchmarks

Ingesting 15 production microservice files across diverse repositories:

Agent Operational Directive

MANDATORY: Context ingestion pipelines must sanitize license headers and decorative comments prior to injecting files into the LLM context window. Never alter the files on disk - perform comment stripping in-memory for context injection only.