Stop Sequence Truncation

Stop Sequence Truncation Protocol (Sampler-Layer Cutoff)

Overview

Even when prompted to be concise ("Generate only the git commit message"), models frequently generate the requested deliverable on line 1, but then continue generating conversational explanations, alternative suggestions, and polite sign-offs on lines 2 through 6.

Standard post-processing requires regex to strip this trailing chatter, but the developer has already been billed for all generated output tokens.

Stop Sequences configure the LLM provider's sampler to immediately halt token generation the instant a specific character sequence or delimiter is emitted. This terminates generation at the GPU layer in 0 milliseconds, saving 100% of trailing token costs.

Post-Processed Trailing Chatter vs. Sampler Stop Cutoff

┌─────────────────────────────────────────────────────────────┐
│                 Stop Sequence Sampler Impact                │
│                                                             │
│  Unconfigured Stop Sampler (120 Output Tokens):             │
│  feat(auth): add redis token blocklist                      │
│                                                             │
│  This commit message explains that we added Redis to...     │
│  You can use `git commit -m` to apply it. Let me know...    │
│  ↳ 120 output tokens billed; regex strips text on client    │
│                                                             │
│  Sampler Stop Sequence (`stop: ["\n"]` - 8 Tokens):         │
│  feat(auth): add redis token blocklist                      │
│  ↳ Model hits `\n` $\rightarrow$ GPU terminates stream instantly! │
│  ↳ 8 tokens billed (93.3% Cost Reduction!)                  │
└─────────────────────────────────────────────────────────────┘

High-Yield Stop Sequence Archetypes

Production Python Implementation (OpenAI & Anthropic SDKs)

OpenAI Client:

from openai import OpenAI

client = OpenAI()

def generate_single_line_commit(git_diff: str) -> str:
    """Generates a 1-line commit message and halts GPU generation at newline."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Generate a 1-line conventional commit message for the git diff."
            },
            {"role": "user", "content": git_diff}
        ],
        temperature=0.0,
        # GPU sampler halts immediately on first newline
        stop=["\n"]
    )
    return response.choices[0].message.content.strip()

Anthropic Claude Client:

import anthropic

client = anthropic.Anthropic()

def extract_json_with_stop(raw_text: str) -> str:
    """Extracts JSON and halts immediately after closing brace."""
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[
            {"role": "user", "content": f"Extract user record as JSON:\n{raw_text}"}
        ],
        # Halts immediately upon closing markdown code block
        stop_sequences=["```\n\n", "\n\nUser:"]
    )
    return response.content[0].text

Benchmark Comparison

Running 500 single-value extraction and 1-line commit tasks:

Agent Operational Directive

MANDATORY: For 1-line extractions, commit messages, and isolated code blocks, API requests must include target stop sequences (stop: ["\n"] or stop: ["```\n"]) to terminate GPU sampling at the boundary.