Head Tail Log Clipping

Head/Tail Log Clipping (Sandwich Truncation Protocol)

Overview

When running large test suites, Docker image builds, or package installations (npm test, cargo test, docker build), terminal commands frequently generate 500 to 5,000 lines of output.

In 99% of software failures, the critical diagnostic information is distributed in two places:

1. The Head (First 15-25 lines): Command invocation arguments, environment variables, compiler flags, and target module names.
2. The Tail (Last 30-50 lines): The fatal assertion failure, exact line number, exception stack trace, and final exit code summary.

The middle 90% of the log consists of redundant progress meters ("Passing test 1..140", "Downloading layer 4a3f..."). Ingesting unclipped logs burns 10,000+ tokens per failure turn.

The Head/Tail Log Clipping Protocol (also known as the Sandwich Truncation Protocol) extracts the Head and Tail while replacing the middle with a compact tombstone count.

3,000-Line Raw Log vs. Sandwich Truncation

┌─────────────────────────────────────────────────────────────┐
│                 Log Stream Ingestion Impact                 │
│                                                             │
│  Unclipped Terminal Log (3,000 Lines / 18,500 Tokens):      │
│  • Lines 1-20: `pytest tests/` (Environment info)           │
│  • Lines 21-2950: 2,930 lines of passing green dots / tests │
│  • Lines 2951-3000: AssertionError on line 42               │
│  ↳ 18,500 tokens billed, agent loses attention in middle    │
│                                                             │
│  Sandwich Truncated Log (60 Lines / 420 Tokens - 97.7% Cut):│
│  === HEAD (Lines 1-20) ===                                  │
│  pytest tests/auth/ -v --color=no                           │
│  [... 2,930 passing test lines truncated for efficiency ...]│
│  === TAIL (Lines 2960-3000) ===                             │
│  FAILED tests/auth/test_jwt.py::test_expiry - AssertionError │
│  ↳ 420 tokens billed, instant pinpoint fix on test_jwt.py   │
└─────────────────────────────────────────────────────────────┘

The Sandwich Truncation Parameters

For optimal token economy and diagnostic completeness:

┌───────────────────────────────────────────────────────────────────────────┐
│ • HEAD LINES ($H$): 20 lines (Command, Flags, Environment)                │
│ • MIDDLE TOMBSTONE: `[... <N> lines truncated for token efficiency ...]`  │
│ • TAIL LINES ($T$): 40 lines (Assertion, Traceback, Summary)             │
│ • TOTAL RETAINED: 60 lines max ($\le 500$ tokens)                         │
└───────────────────────────────────────────────────────────────────────────┘

Production Python Log Clipper Implementation

from typing import Optional

def clip_log_sandwich(
    raw_output: str,
    head_count: int = 20,
    tail_count: int = 40
) -> str:
    """Clips large terminal output into a token-efficient Head/Tail sandwich."""
    lines = raw_output.splitlines()
    total_lines = len(lines)
    
    # If log is already small, return untouched
    if total_lines <= (head_count + tail_count):
        return raw_output.strip()
        
    head_lines = lines[:head_count]
    tail_lines = lines[-tail_count:]
    omitted_count = total_lines - (head_count + tail_count)
    
    return (
        f"{chr(10).join(head_lines)}\n\n"
        f"--- [LOG TRUNCATED: {omitted_count} intermediate lines omitted for context efficiency] ---\n\n"
        f"{chr(10).join(tail_lines)}"
    )

Shell Script / CLI Pipeline Patterns

For agent terminal wrappers:

# Bash: Capture head and tail in one pipeline
pytest tests/ | (head -n 20; echo "[... intermediate logs elided ...]"; tail -n 40)

Benchmark Comparison

Running a full Jest test suite across 40 test suites (3,200 lines of output):

Agent Operational Directive

MANDATORY: Agent tool runners intercepting stdout from commands exceeding 100 lines MUST apply Head/Tail sandwich truncation ($H=20, T=40$) prior to returning the result into LLM context.