LLM Pipeline

llm-pipeline

Wire multiple LLM calls into a reliable, controllable production pipeline. You chain steps where one call's validated output feeds the next, put a router in front of providers so an outage fails over instead of taking you down, and engineer the cross-cutting concerns: timeouts, bounded retries, fallbacks, caching, and cost caps.

Treat the LLM as an unreliable network dependency, not a local function call. Every rule below follows from that: providers have outages, rate limits, and latency tails, so no single provider is a single point of failure and no call is allowed to run unbounded.

Do you even need a pipeline?

This skill is the orchestration around calls. If you only have one call, you are in the wrong place.

A pipeline is a DAG you designed. The moment the model picks its own next step, it is an agent — go build that instead.

Design the chain as a typed DAG

Each step is a pure-ish function: (typed input) -> (typed output via structured output). Chaining small single-purpose steps beats one mega-prompt — reported ~20% output-quality gain — because each step is debuggable, cacheable, and retryable in isolation.

Rules:

• The structured output of step N is the input contract of step N+1. Validate it (Pydantic / JSON Schema) at the seam. A schema-valid object that fails validation here never poisons the next call.
• Keep steps small and single-purpose. "Extract entities" and "classify sentiment" are two steps, not one prompt doing both. Smaller steps route to cheaper models and cache better.
• Mark independent steps for parallel fan-out. If step B and step C both only need step A's output, run them concurrently — see ../parallel/SKILL.md. Sequential only where there is a real data dependency.
• Tag each step idempotent or side-effecting. Retries and replays must be safe; a step that writes to a DB or sends an email is not safe to blindly retry.

# Bad: free text flows between steps; step 2 silently mis-parses step 1
entities = client.responses.create(model="gpt-4o", input=f"Extract entities: {doc}").output_text
summary  = client.responses.create(model="gpt-4o", input=f"Summarize for {entities}").output_text

# Good: each step emits a validated object; the seam is a contract you assert on
from pydantic import BaseModel

class Entities(BaseModel):
    people: list[str]
    orgs: list[str]

step1 = client.responses.parse(model="gpt-4o", input=f"Extract: {doc}",
                               text_format=Entities, timeout=30)
ents: Entities = step1.output_parsed          # parse fails HERE, not downstream
summary = summarize(ents)                      # typed input, not a string blob

Put a gateway/router in front

Use a router so a failure on one deployment fails over to another, and so you can swap or load-balance models from config without touching call sites.

LiteLLM is the de-facto open-source LLM gateway (current stable line v1.83.3-stable). It exposes a unified OpenAI-format completion() across 100+ providers, with built-in retry/fallback, cost tracking, and budget management — usable as a Python SDK or as a Proxy Server.

Fallback semantics worth knowing: a request to an order=1 deployment that fails (connection error, 404, 429, ...) auto-tries order=2, then order=3; each order level gets its own num_retries before escalating; exhausting orders falls through to configured fallbacks. There are specialized buckets — content_policy_fallbacks (ContentPolicyViolationError), context_window_fallbacks (ContextWindowExceededError), and default_fallbacks.

from litellm import Router

router = Router(
    model_list=[
        {"model_name": "smart",
         "litellm_params": {"model": "anthropic/claude-sonnet-4-6", "timeout": 30}},
        {"model_name": "smart-backup",
         "litellm_params": {"model": "openai/gpt-4o", "timeout": 30}},
    ],
    fallbacks=[{"smart": ["smart-backup"]}],
    context_window_fallbacks=[{"smart": ["smart-backup"]}],
    num_retries=2,            # per order level, bounded
    timeout=30,               # hard cap, never unbounded
)
resp = router.completion(model="smart",
                         messages=[{"role": "user", "content": prompt}])

Build vs buy: raw SDK + a thin retry/timeout wrapper (fewest deps, fine for one provider) → LiteLLM SDK/proxy (widest provider coverage, fallbacks for free) → hosted gateway (Bifrost/Portkey, when you want it operated for you). Full config — all fallback buckets, redis cache params, budget/rate-limit settings, cost callbacks — is in references/litellm-router.md.

Reliability mechanics

• Always set a timeout. A 30–60s hard cap per call. Without it, one slow upstream hangs the request indefinitely.
• Bounded retries with exponential backoff + jitter. LiteLLM defaults: backoff from INITIAL_RETRY_DELAY 0.2s up to MAX_RETRY_DELAY 10s, with jitter to avoid thundering herds. Never while True.
• Only retry idempotent steps. Retrying a step that sent an email or charged a card double-fires. Gate retries behind the idempotency tag from the DAG design.
• Circuit-break and degrade gracefully. When a provider is down and fallbacks are exhausted, return a partial result, a cached result, or a cheaper-model result — never hang and never 500 the user if a degraded answer exists.

# Bad: naked call in a for-loop; no timeout, unbounded effect, no fallback
for _ in range(10000):
    try:
        return client.chat.completions.create(model="gpt-4o", messages=msgs)
    except Exception:
        continue          # hammers a down provider, blows the budget, may never exit

# Good: router does bounded retries + backoff + fallback; you degrade on exhaustion
try:
    return router.completion(model="smart", messages=msgs)   # timeout + num_retries set
except Exception:
    return cached_or_cheaper_answer(msgs)                     # graceful degradation

Caching: two distinct layers

The cheapest, fastest, most reliable call is the one you never made — so cache and tier before you tune prompts. Wording is the last lever, not the first.

Prefix cache is transparent and free to enable. OpenAI caches automatically at 50% off cached input tokens, no write penalty, no storage fee — first request full price, prefix hits half price. Anthropic is explicit (you mark cache breakpoints) and deeper: cache read = 0.1× base input (90% off), 5-minute write = 1.25× base, 1-hour write = 2× base, delivering ~90% cost and ~85% latency reduction on long stable prefixes. For both, put the stable content first (system prompt, instructions, fixed context) and the variable content last so the prefix matches.

Semantic cache matches an embedding-similar prior query and returns that prior response. The quality is dominated by the embedding model — a weak embedder produces false cache hits: a confidently wrong answer for a similar-but-different question (GPTCache is documented returning incorrect saved responses for similar prompts). So: strong embedder, tuned similarity threshold, and never on correctness-critical paths. Threshold tuning, embedding choice, TTL, and the multi-tier semantic → prefix → inference order are in references/caching-layers.md.

Cost & latency control

• Model-tier routing, cheap-first. Run the cheap tier, escalate only on low confidence or detected complexity. Anchor prices (Anthropic, 2026): Haiku 4.5 $1/$5 per M in/out, Sonnet 4.6 $3/$15, Opus 4.7 $5/$25 — flagship-for-everything is 5× the cost for zero quality gain on easy calls.
• Budget caps that ABORT, not just log. Per-request and per-tenant caps that kill a runaway loop. A logged-but-uncapped budget still lets a bug spend $10k overnight.
• Token and latency budgets per step, and stream output for perceived latency on user-facing calls.

The ledger — attribution, per-team dashboards, monthly reporting — is not this skill; that is ../cost-tracking/SKILL.md. This skill owns the controls (tiers, caching, caps that abort).

Observability hooks

Log per step, every call: model, tokens_in/out, cost, latency_ms, cache_hit, fallback_used, retry_count. These are exactly the fields you debug a production incident from ("why did p99 spike?" → fallback_used + retry_count). Wire them into the tracing backbone in ../observability/SKILL.md, and measure output quality with ../agent-eval/SKILL.md — logging is not evaluation.

Anti-patterns

Verify

Run scripts/verify.sh <file-or-dir> against your pipeline/gateway code. It is offline and read-only, and checks statically: every completion/chat call site has an explicit timeout, retries are bounded (no while True retry loops), at least one fallback is configured when a router/model_list is present, no hardcoded sk-/provider key literals (must be env-sourced), and any YAML/JSON config parses and lists ≥2 model entries so a fallback target exists. It prints PASS/FAIL per check and exits non-zero on any FAIL; an empty or clean target exits 0.