Ansi Strip Logger

ANSI Escape Sequence Stripping & Output Normalization

Overview

When CLI tools and test runners (Jest, Pytest, Webpack, Docker, Cargo) execute in a terminal, they emit rich ANSI Escape Sequences (e.g. \x1b[32m, \x1b[0m, \033[2K) to render colors, bold text, spinners, and cursor movements.

LLM byte-pair tokenizers cannot parse escape codes semantically; instead, they fragment each escape string into 4 to 8 individual tokens (\*, *x1b*, *[*, *32*, *m). A simple 50-line test run can bloat from 500 clean tokens to over 2,500 noisy tokens, degrading model reasoning and wasting context budget.

The ANSI Stripping Protocol intercepts and sanitizes stdout/stderr streams before they are injected into agent context windows.

Token Fragmentation from ANSI Escape Codes

┌─────────────────────────────────────────────────────────────┐
│                 ANSI Token Fragmentation                    │
│                                                             │
│  Raw Terminal Output with Color Bytes:                      │
│  \x1b[32mPASS\x1b[0m \x1b[90msrc/auth.test.ts\x1b[0m       │
│  ↳ Tokenizer splits into: [\, x1b, [, 32, m, PASS, \, x1b, │
│    [, 0, m, \, x1b, [, 90, m, src, /, auth, ., test, ...]   │
│  ↳ Total: 38 Tokens for 1 Line!                             │
│                                                             │
│  Sanitized Plain Text Output:                               │
│  PASS src/auth.test.ts                                      │
│  ↳ Total: 5 Tokens (86.8% Reduction!)                       │
└─────────────────────────────────────────────────────────────┘

The Industrial ANSI Stripping Regex

To strip all standard 7-bit and 8-bit ANSI / VT100 escape sequences, cursor movements, and OSC operating system commands:

import re

# Comprehensive ECMA-48 / ANSI Escape Stripper
ANSI_REGEX = re.compile(
    r"""
    \x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])  # Standard CSI codes & SGR colors
    |\x1B\][0-9];[^\a\x1B]*[\a\x1B\\]     # OSC hyperlinks & title sequences
    |\r(?!\n)                              # Standalone carriage returns (spinners)
    """,
    re.VERBOSE
)

def strip_ansi(text: str) -> str:
    """Removes all ANSI color codes and control sequences from terminal output."""
    return ANSI_REGEX.sub("", text)

TypeScript / Node.js Agent Middleware

/**
 * Strips ANSI codes from command execution buffers before sending to LLM.
 */
const ANSI_REGEX = /[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g;

export function sanitizeTerminalOutput(rawOutput: string): string {
  return rawOutput
    .replace(ANSI_REGEX, '')
    .replace(/\r\n/g, '\n')
    .replace(/\r/g, '\n')
    .trim();
}

Native Environment Variable Suppressors

Before running CLI commands in agent subprocesses, pass these environment variables to disable ANSI output at the source:

# Set in agent execution runtime
export NO_COLOR=1            # Standard supported by 100+ modern CLI tools
export TERM=dumb             # Signals non-interactive terminal (disables spinners)
export FORCE_COLOR=0         # Disables chalk/color in Node.js
export CI=true               # Forces automated, non-interactive output

Example Subprocess Execution in Python:

import subprocess
import os

env = os.environ.copy()
env.update({
    "NO_COLOR": "1",
    "TERM": "dumb",
    "CI": "true",
    "FORCE_COLOR": "0"
})

result = subprocess.run(
    ["pytest", "--tb=short"],
    capture_output=True,
    text=True,
    env=env
)
clean_output = strip_ansi(result.stdout)

Benchmark Comparison

Agent Operational Directive

MANDATORY: Any agent tool that captures terminal stdout/stderr (run_command, exec_shell, bash) MUST pass outputs through an ANSI sanitizer before appending the result to the conversation transcript.