LLM Engineering Expert

LLM Engineering Expert

Building on a language model is engineering against a component that is non-deterministic, occasionally confident and wrong, and priced per token. The discipline is in the surrounding structure, not in the wording of prompts.

Core Concepts

The Model Is a Component, Not the System

A production LLM feature is a pipeline: input validation, context assembly, the model call, output parsing and validation, then a decision about what to do when any of that fails. Treating the model call as the whole feature is the root of most reliability problems.

Determinism Is Not Available

Even at temperature zero, output can vary across model versions, infrastructure and batching. Design for a distribution of outputs, not a fixed one: validate what comes back, and make the failure path as considered as the success path.

Context Is a Budget

Everything competes for the same window: system instructions, examples, retrieved documents, conversation history, the user's input, and room for the answer. Attention is not uniform across a long context — material at the beginning and end is used more reliably than material buried in the middle. Spend the budget deliberately.

Evaluation Precedes Iteration

Without a scored test set, prompt changes are superstition. Ten to fifty representative cases with expected properties are enough to start and will catch most regressions.

Prompt Design

Structure that holds up

Order matters: instructions first, reference material next, the specific request last. Ending with the request keeps it close to generation.

SYSTEM = """You extract structured invoice data.

Rules:
- Return only fields present in the document. Never infer a missing value.
- Amounts are decimal strings with two places, e.g. "1234.50".
- If the document is not an invoice, set "is_invoice" to false and stop.
- Dates are ISO 8601 (YYYY-MM-DD).
"""

USER = """<document>
{document_text}
</document>

Extract the invoice fields defined by the schema."""

What earns its place: an explicit role, hard constraints stated as rules, a defined behaviour for the out-of-scope case, and delimiters that separate untrusted content from instructions.

What does not: politeness, threats, "you are the world's best", and repeated emphasis. They consume budget and do not measurably improve output.

Few-shot examples

Two to five examples usually beat both zero and twenty. Choose them to cover the decision boundaries you care about — especially the case you keep getting wrong, and at least one negative or refusal case. Keep formatting identical across examples; inconsistency is learned as signal.

Give the model somewhere to think

For multi-step reasoning, let it work before it answers, then separate the reasoning from the result so parsing stays simple.

<thinking>Work through the calculation here.</thinking>
<answer>{"total": "1234.50"}</answer>

Reasoning-tuned models do this internally; adding your own scaffold on top of them can hurt. Test rather than assume.

Structured Output

Never parse prose you did not constrain. Use the provider's schema-constrained output where available, and validate regardless.

from pydantic import BaseModel, Field, ValidationError
from decimal import Decimal

class Invoice(BaseModel):
    is_invoice: bool
    invoice_number: str | None = None
    issued_on: str | None = Field(default=None, pattern=r"^\d{4}-\d{2}-\d{2}$")
    total: Decimal | None = None
    currency: str | None = Field(default=None, pattern=r"^[A-Z]{3}$")

def extract(document_text: str) -> Invoice:
    raw = call_model(SYSTEM, USER.format(document_text=document_text),
                     response_schema=Invoice.model_json_schema())
    try:
        return Invoice.model_validate_json(raw)
    except ValidationError as exc:
        # One repair attempt with the error, then fail closed.
        repaired = call_model(SYSTEM, REPAIR_PROMPT.format(raw=raw, errors=exc))
        return Invoice.model_validate_json(repaired)

Validation must be structural and semantic. A well-formed JSON object whose total does not equal the sum of its line items is still wrong; check the invariants your domain requires.

Evaluation

Build the set before tuning

CASES = [
    {"input": "...", "expect": {"is_invoice": True, "currency": "EUR"}},
    {"input": "...", "expect": {"is_invoice": False}},          # negative case
    {"input": "...", "expect": {"total": Decimal("0.00")}},     # edge case
]

def score(case) -> bool:
    got = extract(case["input"])
    return all(getattr(got, k) == v for k, v in case["expect"].items())

Run the set on every prompt or model change and record the score. A change that improves your favourite example and drops the aggregate is a regression.

Choosing a metric

Model-as-judge is useful and biased: it favours verbosity and its own style. Calibrate it against human labels on a sample before trusting it, and never judge with the same model that generated the output when you can avoid it.

Guardrails

• Validate the input. Length limits, content type, and a policy check before spending a call.
• Treat retrieved and user content as untrusted data. Delimit it, and never let it be read as instructions. Prompt injection is the central security problem of this component — see the agent-engineering-expert skill.
• Constrain the blast radius. A model that can only return a value is safe; one that can trigger an action needs the same authorisation checks as any other caller.
• Fail closed. When validation fails twice, return an error to the caller. Do not pass through unvalidated output.
• Log the decision inputs — model, version, prompt hash, token counts, latency, outcome — without logging customer content by default.

Cost and Latency

The levers, in order of typical effect:

1. Right-size the model. Route simple cases to a small model and escalate only on low confidence. Most workloads are dominated by easy cases.
2. Cache the stable prefix. Long system prompts and shared context are cacheable by most providers, cutting both cost and time to first token.
3. Cut the context. Retrieved chunks are the usual bloat; more context is not more accuracy past the point of relevance.
4. Cap the output. Output tokens usually cost several times input tokens.
5. Batch offline work. Asynchronous batch tiers are markedly cheaper when latency does not matter.
6. Stream when a human is waiting: it does not reduce cost, but it changes perceived latency more than any optimisation.

Measure cost per successful outcome, not per call. A cheap model that needs three attempts and a repair is not cheap.

Best Practices

• Version prompts like code. Store them in the repository, review changes, and record which version produced which output.
• Pin the model version. Provider defaults move; pin explicitly and upgrade deliberately with the eval set as the gate.
• Set a timeout and a retry budget with backoff and jitter on every call.
• Separate the system prompt from user content structurally, not by concatenation.
• Keep a human in the loop for anything irreversible.
• Write the failure path first. What the feature does when the model is unavailable, slow, or wrong is a product decision, not an afterthought.

Anti-Patterns

Prompt-tweaking without evaluation

Changing wording, checking one example, and shipping. Regressions land silently because nothing measured them.

Parsing unconstrained prose

Regexing a number out of a sentence works until the model phrases it differently. Constrain the output shape and validate it.

Stuffing the context

Sending everything available in the hope something helps. It raises cost and latency, and buries the relevant material.

Trusting a confident answer

Fluency is uncorrelated with accuracy. Anything factual needs grounding in a source or verification against one.

Retrying identically on a bad output

The same input with the same prompt tends to fail the same way. Change something — add the validation error, lower the temperature, escalate the model.

Treating model output as trusted input

Passing generated SQL, shell, paths or URLs to an executor without validation is remote code execution with extra steps.

Reference Documentation

• Prompt Patterns — reusable structures for extraction, classification, generation and refusal, with worked prompts
• Evaluation Harness — building a scored test set, regression gates in CI, model-as-judge calibration, and online monitoring

Resources

• Anthropic, Prompt engineering guide
• OpenAI, Structured outputs
• OWASP Top 10 for LLM Applications
• NIST, AI Risk Management Framework
• Liu et al., Lost in the Middle — position effects in long context