Prompt Engineering

Prompt Engineering

Design prompts and evals as a first-class engineering deliverable. context-engineering assembles context for a coding agent; this skill designs the prompt-as-product surface — the prompts, model choices, guardrails, and eval harnesses behind LLM-powered features. A prompt without an eval is an opinion; an eval without a prompt is a benchmark. Ship neither blind.

When to use

• Designing an LLM-powered feature (chat, summarization, extraction, classification, generation)
• Building or refining a prompt for production use
• Creating an eval harness to measure prompt/model quality
• Selecting a model for a specific task against cost/latency/quality trade-offs
• Triggers on "prompt engineering", "LLM feature", "eval harness", "prompt design", "提示词工程", "LLM 特性"

Not for: assembling context for a coding agent (use context-engineering); general research on a topic (use research); API contract design for non-LLM endpoints (use api-design).

Steps

1. Define the task and success criteria

State the task in one sentence, then define measurable success criteria — without these, prompt iteration is vibes-driven. Pull from the product spec:

• Input space: what inputs will the prompt receive? (vary by length, language, edge case, adversarial)
• Output contract: structured output (JSON schema), free text, or classification?
• Quality bar: accuracy %, format adherence %, hallucination rate, latency target, cost per call
• Failure modes to prevent: what must the model NEVER do? (leak PII, invent facts, refuse valid input)

Verify: success criteria are written as measurable thresholds, not "good responses."

2. Design the prompt architecture

Structure the prompt as components, not a wall of text — each component has a job:

• System / role: who the model is, what it must and must not do (guardrails live here)
• Task instruction: the operation, stated precisely with the output format
• Context / retrieved data: only the facts the model needs (not the whole knowledge base — context bloat degrades accuracy and raises cost)
• Few-shot examples: 2–5 input→output pairs covering the happy path and an edge case; place before the actual input, after the instruction
• Output format: explicit schema or template; use structured output (JSON mode / function calling) when the downstream system parses the result

Separate stable parts (system, format) from variable parts (user input, retrieved context) so prompt caching applies to the stable prefix. Verify: each prompt component is labeled and serves one purpose; no component is duplicated.

3. Select the model

Match the model to the task's quality, latency, and cost profile — don't default to the most capable model for every call:

• Complex reasoning / generation: most capable model, accept higher latency and cost
• Classification / extraction / routing: smaller, faster, cheaper model — these tasks are over-served by frontier models
• High-volume, low-stakes: the cheapest model that meets the quality bar; route edge cases to a stronger model (cascaded / routed architecture)
• Evaluate multiple models against the eval harness (step 4) before committing — model choice is empirical, not reputational

Verify: model choice is documented with the eval comparison and the cost/latency/quality trade-off that justified it.

4. Build the eval harness

An eval harness is the test suite for prompts. It runs the prompt against a labeled dataset and scores the output against the success criteria:

• Dataset: 50–200 examples minimum, covering happy path, edge cases, and known failure modes. Use real production inputs where possible; synthesize adversarial cases for gaps.
• Metrics: automated where possible (exact match, JSON schema validity, BLEU/ROUGE for generation, classifier-based safety); human review for quality dimensions automation can't score.
• Regression suite: the eval harness runs on every prompt or model change — a prompt tweak that improves one case but regresses three others is a bug, not an improvement.

Verify: the eval harness runs from one command, produces a scorecard, and is committed alongside the prompt.

5. Add guardrails

Production prompts need defenses the prompt itself cannot provide:

• Input validation: reject malformed, oversized, or disallowed inputs before they reach the model
• Output validation: parse and schema-check model output; reject and retry (or fallback) on malformed output
• Content filters: safety classifiers for harmful content (input and output)
• PII redaction: strip sensitive data from prompts before logging; never log raw user input that may contain PII to a shared prompt-logging system
• Rate limits and cost caps: bound per-user and aggregate spend; a prompt bug that loops can burn a budget in minutes

Verify: every guardrail has a defined failure behavior (reject, retry, fallback) — silent pass-through is not a guardrail.

6. Iterate against evals

Iterate the prompt against the eval harness — change one variable at a time (instruction, examples, model, temperature), re-run evals, keep the change only if the scorecard improves without regression. Document prompt versions and their eval scores alongside the prompt, the way code commits pair with test results.

Output: docs/design/PROMPT.md — the prompt design document: task definition, success criteria, prompt architecture (component breakdown), model selection rationale, eval harness description and dataset, guardrails, and version history with eval scores.

Verify

• Task stated in one sentence; success criteria are measurable thresholds
• Prompt structured into labeled components (system, instruction, context, examples, format)
• Stable and variable parts separated for prompt caching
• Model selected via eval comparison, not reputation; trade-off documented
• Eval harness: 50+ examples, automated metrics, runs from one command, committed
• Guardrails on input, output, content, PII, and cost — each with a failure behavior
• Prompt changes paired with eval re-runs; no regression accepted
• docs/design/PROMPT.md produced with architecture + eval + guardrails + version history

Red flags: a single wall-of-text prompt with no structure; choosing the most expensive model for a classification task; shipping a prompt with no eval; logging raw prompts with user PII; no output schema validation (trusting the model to always return valid JSON); prompt iteration by vibes without a scorecard; no cost cap on a production prompt.

References

• ${CLAUDE_PLUGIN_ROOT}/references/engineering-principles.md — shared discipline (surface assumptions, verify don't assume, goal-driven execution)
• references/prompt-architecture.md — prompt component breakdown, few-shot patterns, structured-output techniques, model selection matrix, eval-harness setup, guardrail catalog