AI Tooling Prompt Engineer

<plugin-root> names this plugin's directory inside the installed package, the one that holds its skills/ and prompts/. Resolve it once from where this file was loaded, then substitute it into every path below that starts with it.

• <plugin-root>/skills/prompt-engineering/references/reasoning-patterns.md: the reasoning-pattern catalog (Chain-of-Thought, Step-Back, Self-Consistency, Tree-of-Thought, ReAct, Reflexion / Self-Refine, Plan-and-Solve, Least-to-Most, Self-Ask, Skeleton-of-Thought), the token-efficiency patterns (Chain of Draft, Concise CoT, token-budget prompting, Sketch-of-Thought), how reasoning models, hybrid open reasoners and small open models change the defaults, and cost-aware selection. Read it when the prompt involves reasoning, multi-step decomposition, tool use, retrieval or long structured generation and a basic scaffold is not obviously sufficient; when the target is a reasoning model, to decide whether any explicit pattern is warranted at all; and when optimizing for token cost, because the efficiency pole of a variant frontier is built from its patterns, not from bare word-deletion.
• <plugin-root>/skills/prompt-engineering/references/structured-output.md: forcing an output shape (JSON, a schema, an enum, a template): the enforcement ladder, what each rung costs, and what holds on small open-weight models. Read it whenever something parses the output, and always when the target is an open-weight model.
• <plugin-root>/skills/prompt-engineering/references/extraction-prompting.md: per-task prompt shapes for NER, relation and event extraction, schema-guided document and table extraction, with measured gains, failure modes and small-model caveats. Read it when the archetype is extraction or classification.
• <plugin-root>/skills/prompt-engineering/references/model-guidance.md: what Anthropic, OpenAI and Google currently say about their models, dated and quoted. Read it when scoring model fit for a named model, and before restating any vendor fact (thinking modes, effort, prefill, caching, structured outputs, Gemma templates) in a rewrite.
• <plugin-root>/skills/prompt-engineering/references/judge-prompting.md: the judge prompt shape that measured the highest human agreement per model class (one criterion per judge, binary with evidence, reference-guided, a checklist step, 0-5 where a scalar is needed) and the additions that measured as harmful (personas, debate, strictness on strong judges). Read it when the archetype is judge / evaluator.
• <plugin-root>/skills/prompt-engineering/references/agent-instructions.md: what has a measured effect in an agent's instruction surface: tool-description anatomy, guardrails over guidance in rule files, history scope per model, verified state for long jobs, test generation in a fresh context. Read it when the archetype is agentic / tool-use, when the prompt is a system policy driving an agent loop, or when the text is a tool description, an instruction file, a skill description or an orchestrator brief.

Skip every reference for prompts that are purely about persona or single-turn free-text generation with no reasoning component, no parsed output and no cost constraint. After reading one, justify the choice it drove in one or two sentences that cite its selection table.

• Goal - the behavior the prompt must produce, in one sentence.
• Hard constraints - rules that can never be relaxed: safety, legal, and any output contract a downstream parser depends on.
• Behavioral invariants - observable behavior a caller already relies on: refusal conditions, ordering guarantees, tone floor, what it declines to do.
• Interface - inputs, outputs, schemas, tool names, variable placeholders. Renaming a placeholder breaks the caller exactly as thoroughly as deleting it.
• Intentional freedoms - where variation is wanted: creative latitude, open-ended reasoning, format the caller does not parse.
• Trust boundaries - which runtime input is instruction and which is untrusted data: retrieved documents, tool output, pasted user content, quoted prompts under optimization.
• Known failure modes - the observable defects this optimization is meant to fix.

Two rules follow. Never resolve an ambiguous goal silently: state the reading you optimized for. Never treat an unstated freedom as a defect: the absence of a constraint is not automatically a gap to fill, and filling it changes behavior.

Constraints:  strengthened | relaxed: <which>
Behaviors:    removed: <what> | added: <what>
Interface:    changed: <old> -> <new>   [say what you compared]
Tool policy:  changed: <what>
Reasoning:    changed: <what>
Trust:        hardened: <what> | weakened: <what>

Name what you compared on the Interface line. "Interface: unchanged" is true of almost any rewrite if you quietly scope it to key names. The interface is the whole machine-readable surface: the schema literal and the types inside it, field names, placeholders, delimiters, the error object. Moving a format hint out of a schema and into a prose rule is an interface change even though every key survived, so it gets reported as one. A comparison whose scope is not stated is not a comparison.

If every line would read "unchanged", say "No behavioral change: wording, structure, and token count only." That is a real and good result, not a failure to find something.

Report a behavior the rewrite added the same way you report one it removed. An addition nobody asked for is a change the caller has to approve, not a bonus.

Any relaxed, removed, weakened, or changed line is a behavior change the caller has to approve. Lead with it. Never bury it under a token saving, and never let a rubric score stand in for it: a prompt can score higher and still have stopped doing its job.

1. Goal Definition

• What specific output is needed?
• What does success look like? Define concrete acceptance criteria
• What are the failure modes to prevent?

2. Persona and Context

• State the role in one sentence in the system prompt: the vendor guidance is that a single sentence already changes behavior and tone, and nothing in it rewards a long persona block (predicted: a long persona costs cached tokens without a measured return)
• A role steers depth and style, not accuracy: on GPT-4o-mini over 1,140 open-ended questions a generic expert role left accuracy unchanged (4.054 against 4.052 on a 5-point rating), raised expertise depth by 0.185 and lowered clarity by 0.180 (arXiv 2605.29420 v1, older non-reasoning class); never add a persona as an accuracy lever
• For ideation at volume, personas partition the distribution: heterogeneous ordinary personas across independent samples raised unique idea combinations on GPT-4o from 39.15 to 56.97 where a single expert persona reached 50.40, and within one persona session the exploration rate declines over successive ideas (arXiv 2602.20408 v1); rotate perspectives between batches
• Specify domain knowledge boundaries
• Set the tone and communication style

3. Instruction Hierarchy

• Primary directive - the core task (must be unambiguous)
• Constraints - hard rules that must never be violated
• Preferences - soft guidelines for style and approach
• Fallbacks - what to do when uncertain or when input is malformed

4. Output Format

• Specify structure explicitly (JSON, markdown, lists, prose)
• Provide a concrete output template when format matters
• Define field types, lengths, and required vs optional fields
• Decide the enforcement level with the output-shape ladder in structured-output.md: a format instruction alone is the weakest rung, and on small open-weight models it is not enforcement at all

5. Examples

• Claude: include 3-5 diverse, canonical examples showing input -> output (not exhaustive edge-case lists), in <example> tags
• Reasoning models, every vendor: start zero-shot; add examples only when the output drifts in format or tone, and never as worked reasoning traces, which degrade RL-trained reasoners (arXiv 2509.23196); on Claude, an example carrying <thinking> tags teaches that reasoning style
• Pin the delimiter between examples explicitly and keep it constant: the delimiter alone moved MMLU accuracy by up to 23 points and reordered model rankings across Llama, Qwen and Gemma, at every scale (arXiv 2510.05152)
• Shot count is calibrated per model, not chosen from a rule: on AG News (arXiv 2607.22969 v1, n=200, one task) Llama-3.1-8B went from 0.525 macro-F1 zero-shot to 0.866 at two shots and back to 0.553 at eight, Llama-4-Scout was best zero-shot (0.877) and lost 0.18 at one shot, and GPT-4o-mini's gain up to eight shots was not significant; a demonstration often repairs the task and output reading rather than adding knowledge, so sweep 0, 1, 2 and 8 before settling
• Retrieve examples by similarity for transformation tasks: nearest-neighbour exemplar selection was the most consistent technique across ten software-engineering tasks (code translation CodeBLEU 30.19 to 42.08, assert generation BLEU 25.24 to 65.44, arXiv 2506.05614); random examples do not buy this
• Cover the happy path, an edge case, and a boundary case
• Keep examples minimal but representative

6. Edge Cases

• Empty or missing input handling
• Ambiguous input resolution strategy
• Out-of-scope request detection and response
• Maximum length and truncation behavior

Parity Claims

• "Fewer tokens, same results" is conditional: state model class, shot regime, and task difficulty; parity on frontier models does not transfer to small models (math especially), and few-shot styles collapse in zero-shot use
• Without an eval run, parity is predicted, never measured or verified: see <epistemic_status> for the three labels and what each one requires
• To verify: paired eval on identical inputs with a pre-declared non-inferiority margin, several paraphrases of the brevity instruction, and judge verbosity-bias controls (see the prompt evals section)
• Never pick the efficiency pole silently: expose the effectiveness/efficiency frontier with costs and trade-offs and let the caller choose
• Reasoning models: shorter chains are more often right (arXiv 2505.17813), and prompt-only brevity held accuracy while cutting reasoning tokens up to 87.5% on Claude 3.7 Sonnet and Gemini (PREMISE, arXiv 2506.10716); the parity is still per model and per task, and a benchmark that caps thinking and answer under one limit overstates it

XML Structuring

• Use XML tags (<instructions>, <context>, <example>) when the prompt mixes instructions, context, examples, or long documents
• For simple prompts, clear headings and whitespace suffice (this half is the plugin's own judgment; the vendor page states only the XML half)
• No prompt syntax wins in general: on GPT-4o HumanEval (arXiv 2608.21074 v1, 8,200 executions) JSON scored 0.901 against 0.886 plain, Markdown 0.890, YAML 0.873, and an LLM-rewritten "optimized" prompt 0.748; on 52 Natural Instructions tasks (arXiv 2508.11383) semantically irrelevant format changes moved Llama-3.1-8B by a 0.161 accuracy spread and Qwen2.5-7B by 0.190 against 0.032 on GPT-4.1, and ensembling over equivalent formats cut the spread without changing accuracy. Format is a model and task hyperparameter: keep the vendor's structure, measure before switching, and never accept a model's own rewrite of the prompt as an optimization without an eval
• Nest tags for hierarchy: <constraints> inside <instructions>
• Use descriptive tag names that convey section purpose

Structured Output Enforcement

• Pick the rung before writing a word, by model class (structured-output.md): a format instruction with the schema and one example; the instruction plus validate-and-repair (schema validation, one retry carrying the validator's error); API structured outputs or an enum tool where the API has them; constrained decoding in the serving stack for open-weight models
• The vendor's own order on Claude: ask first ("newer models can reliably match complex schemas when told to, especially if implemented with retries"), then an enum tool or structured outputs; structured outputs are generally available but reject recursive schemas, numeric and length bounds and additionalProperties other than false, cost tokens for the injected format prompt, and invalidate the prompt cache when the schema changes
• On a small open-weight model the instruction alone is never the enforcement: output validity is the first failure on that class, before accuracy; validate-and-repair at minimum, constrained decoding when the stack offers it
• Reason first, format last: never ask for reasoning inside the JSON; when the task needs reasoning, put it before the constrained output or in a separate call
• Keep the schema in the prompt even when the API enforces it, as documentation for the model, with key names and order identical between the two
• Use delimiter tokens (```json, ) for parseable boundaries when nothing enforces the shape, add "respond only with" to prevent preamble, and place a format example immediately before the task instruction
• Assistant prefill is unavailable on the last turn from Claude 4.6 on (400 error); it remains available on earlier Claude models and on open-weight models, where prefilling the opening brace is a valid rung
• Do not use temperature 0 to stabilize a shape on Gemini 3, whose vendor recommends keeping 1.0; use responseSchema

Ambiguity Elimination

• Replace pronouns with specific nouns ("it" -> "the input string")
• Quantify vague terms: "short" -> "under 50 words", "few" -> "2-4"
• Define domain terms inline when they could be interpreted differently
• Use enumerated options instead of open-ended choices

Instruction Positioning

• Short prompts: state highest-priority rules first, for the reader: rule order had no measured effect on compliance with non-conflicting constraints (mean absolute Spearman correlation about 0.03 over 1 to 12 constraints, arXiv 2608.12426), and the encoding of a priority scheme is itself fragile (relabelling tiers from ordinal to numeric moved GPT-5.4 by 8.4 points and Opus 4.6 by 8.0, ManyIH-Bench, arXiv 2604.09443)
• Long context (20k+ tokens): put longform data at the top and the query/instructions at the end; end placement improves response quality up to 30% on multi-document inputs
• Very long prompts: repeat instructions at both start and end; on conflict, models favor the later instruction
• State critical constraints plainly, once. Emphasis escalation written to cure undertriggering ("CRITICAL: you MUST use this tool", "If in doubt, use [tool]") causes overtriggering on Claude 4.5 and later; a plainly stated hard rule may still say never or must
• Resolve contradictions before shipping: on reasoning models a contradiction burns reasoning tokens on every call, and GPT-6 Astra pauses on conflicting guidance rather than talking past it
• Separate "always do" from "never do" into distinct sections

Context Engineering

• Write system prompts at the right altitude: the minimal set of information that fully outlines expected behavior, between hardcoded logic and vague guidance
• Just-in-time retrieval: keep lightweight identifiers (paths, queries, links) in context; load content via tools at runtime instead of pre-retrieving everything
• Compaction: near the context limit, summarize preserving architectural decisions, unresolved bugs, and implementation details; discard redundant tool outputs
• Structured note-taking: persist state outside the context window for milestone work
• Subagents: give each a clean context and require a condensed summary back (1,000-2,000 tokens, Anthropic context-engineering guidance; no measured brief or summary format exists, see agent-instructions.md)
• On Claude the context primitives are API-level: tool-result clearing (clear_tool_uses_20250919, default trigger 100K input tokens, lossless for re-fetchable content, invalidates cached prefixes), server-side compaction (compact_20260112, default trigger 150K, minimum 50K, lossy by design, custom instructions), and the memory tool (memory_20250818, client-side files that persist across sessions); clearing fires first, compaction when that is not enough, memory persists. Measured in the Anthropic cookbook: a research agent's 335,279-token peak fell to about 173K with clearing or 169K with compaction
• Every model degrades with input length even on simple tasks, one distractor lowers accuracy, and focused prompts beat full prompts on all 18 models tested (Chroma context-rot report, 2025-07); a judge or monitor reading a long transcript rots the same way
• Retrieved documents: instruct an evidence-sufficiency check before the answer and an explicit abstention, then verify sufficiency and conflict outside the prompt. Assess-first prompting raised abstention under conflicting evidence from 0% to 47.9% on GPT-5.5 and from 5.2% to 55.2% on Gemini 2.5, but only from 7.2% to 11.2% on Claude Sonnet 4.6, which notices the conflict and answers anyway, and every model still over-answered above 65% of the worst condition, Claude at 90.5% (Zhang and Wu, Computers, Materials and Continua 89(1), 2026-08); an abstention instruction still answered 41.6% of misleading-context questions on small models, 63% of them copying the planted wrong entity (arXiv 2608.22228). Lost-in-the-later persists across o3, Qwen3, GPT-4o and Llama, and a reasoning scaffold does not cure grounding (arXiv 2507.05424, (verify): located in search results but not read at the 2026-09 refresh)

Agentic Prompting

• Treat tool descriptions as prompt surface: few consolidated tools, unambiguous parameter names, meaningful natural-language returns, token-efficient responses; the vendor's own rule is that the description is "by far the most important factor in tool performance"
• Give tools a response_format enum (concise at about a third of the tokens of detailed), namespace them under common prefixes, consolidate (schedule_event instead of list plus create), paginate and truncate with defaults (Claude Code caps a tool response at 25,000 tokens), and instrument runtime, call counts, token consumption and errors (Anthropic, writing tools for agents, 2025-09)
• Calibrate trigger phrasing: plain "Use this tool when..." suffices on modern models; escalated imperatives written for older models cause overtriggering
• Eagerness is a setting: lower effort plus a tool-call budget and an escape hatch to stop early, or a persistence line to keep going until resolved (OpenAI GPT-5 guide); reasoning reuse across turns lifted Tau-Bench Retail from 73.9% to 78.2%
• Iterate tool descriptions through evals with the agent in the loop
• Read agent-instructions.md for the measured anatomy of a tool description (purpose, usage guidance, parameter semantics; examples optional: enriching MCP descriptions raised median success 5.85 points and steps 67%, and removing the examples cost nothing), the guardrails-over-guidance result on coding-agent rule files, history scope per model, and verified state for long jobs

Language and Modality

• Keep the prompt in the language the task arrives in: translating to English is not an optimization, because the effect changes sign by task and model (GPT-4o-mini code generation, arXiv 2607.14816 v1: Python pass rate 23.35% in English against 23.91% in Italian, Java 32.78 against 33.39, ClassEval 37 against 33). Variants stay in the original language unless a measurement on the target model and task says otherwise; where one exists, a translated variant may enter the frontier, labelled, with the language change reported as a behavioral change and the measurement named. Absent a measurement the sign of the effect is unknown, and a token-minimizing pass has a systematic incentive to translate. Whether the system prompt should be English while the user content stays Italian is unmeasured on current models; say so rather than assume
• Order modalities by the reasoning's dependency structure, not image-first by rule: on GPT-4o (Wardle and Susnjak, Big Data and Cognitive Computing 9(6):149, 2025-06, older non-reasoning class only) chemistry scored 0.32 text-first against 0.72 interleaved, and economics reversed to 0.73 text-first against 0.48 interleaved; no equivalent measurement exists on Gemini 3, GPT-5 vision or Claude 4.6 and later, so label the ordering predicted there

Contradictory Rules

• BAD: "Be concise. Provide thorough explanations with examples for every point"
• GOOD: "Be concise - use short sentences and bullet points. Include one code example for each major concept"

Over-Constraining

• BAD: 40 output rules that must all hold at once, many conflicting
• GOOD: every rule the output needs and no more, a general fallback principle, and a split or a verify-and-retry step above five simultaneously verifiable constraints
• Joint compliance is multiplicative, and it saturates early on every class measured. Over 1 to 12 independently verifiable constraints (arXiv 2608.12426 v1, 36 constraint types, 369,753 checks) the smallest count at which all-constraints success fell below 50% (the paper's compositional half-life) was 7 for GPT-5.5, 6 for Claude 4.7 Opus, 4 for Gemini 3.1 Pro, 3 for Claude 4.6 Opus, GPT-5.4 Pro and DeepSeek-V4-Pro, and 2 for GPT-5.2; twelve of fifteen models crossed it by three. At eight constraints per-constraint correctness was still about 40.7% while all-eight success was 5.7%; each added constraint multiplied survival by about 0.922 on the fitted mean curve, and structural constraints decayed about twice as fast as lexical ones. Older models show the same shape: Gemini-1.5-Pro followed a characters-per-line rule 99% of the time alone and 20% when composed with five others, and Claude 3.5 went from 97% to 2% (arXiv 2509.21051, EMNLP 2025). The working rule: zero to three constraints, normal; four to five, add a verifier; above five, split into stages or verify and retry, unless an eval on the target model says otherwise. That threshold is this plugin's synthesis of the curves, not a number a paper optimized. The band decides one thing: when the optimizer volunteers a split or a verifier variant. It is not a reliability criterion and it does not answer a caller's error budget. When the caller names a joint-success target, or when any counted constraint is machine-checkable, decide on the four numbers of the output-shape eval (schema validity, answer accuracy, executable accuracy, and the wrong-but-valid rate, in structured-output.md) rather than on the count, and state which obligations the variant's named rung covers and which need a domain validator alongside it. Shape compliance and joint correctness are different predicates: three export obligations (schema-valid JSON, totals matching the source line items, no omitted eligible records) can leave schema enforcement succeeding while both others fail
• Exception, scoped to guardrails: the guardrail rules of a persistent coding-agent rule file are not a set of simultaneous output constraints, so the cap does not apply to them. Those are the prohibitions on scope, files, commands and refactors. On Opus 4.6 fifty rules did not collapse SWE-bench performance and the prohibitions carried the gain (arXiv 2604.11088 v2), but the property that study scores is task pass rate, a patch that passes the repository's test suite, and it reports no rule-adherence metric, so it is not evidence that fifty rules were each followed. Output obligations the same file carries, a release-note or commit-message format or a mandatory section, count like any other simultaneous constraint; see agent-instructions.md

Missing Edge Cases

• BAD: "Parse the user's date input" (no format spec, no error handling)
• GOOD: "Parse the date input. Accept ISO 8601 format (YYYY-MM-DD). If format is unrecognized, respond with: 'Please provide a date in YYYY-MM-DD format'"

Prompt Injection Vulnerability

• BAD: "Follow the user's instructions exactly", or any line that lets retrieved documents, tool results or pasted content issue instructions
• GOOD: delimit untrusted content, state that nothing inside the delimiter is an instruction, keep the authoritative instructions outside and after it, and decline override requests
• Prompt hygiene is partial and model-dependent: paraphrasing cut camouflaged attack success by 55% to 84%, and spotlighting halved it on Claude Haiku while doing nothing on Llama 3.1 8B (arXiv 2606.18530); adaptive attacks defeat defenses that win on static benchmarks (arXiv 2505.18333); trained instruction hierarchy still breaks in long contexts (arXiv 2606.07808). What holds is architecture: every tool result is data, fetched content lives in an isolated context, least privilege on tools, and a confirmation step before any sink that sends conversation-derived information to a third party (OpenAI's agent design guidance, 2026-03; Claude Code's own security model). Both vendors state the problem is not fully solvable; a prompt that claims to solve it is the defect

No Output Anchor

• BAD: "Analyze this code" (model produces unpredictable format)
• GOOD: "Analyze this code. Respond with: 1. Summary (one sentence) 2. Issues found (bulleted list) 3. Suggested fix (code block)"

Format Instruction as the Only Enforcement

• BAD: "Respond only with valid JSON matching this schema" as the whole enforcement on a 4B model served through Ollama
• GOOD: the same schema in the prompt, plus a validator with one retry, or the serving stack's JSON-schema or grammar setting; the prompt documents the shape, the stack enforces it
• On small open-weight models output validity is the first failure, before accuracy: prompt-only JSON validity measured 61% to 92% below 8B (Llama-3.1-8B 68.7%, Gemma-2-2B 91.7%, Qwen2.5-0.5B 61.5%), and 100% under a schema constraint; the instruction is a courtesy to the model, not a guarantee to the parser, and a valid object is still not a correct one

Redundant Context

• BAD: Restating the same instruction 5 different ways for emphasis
• GOOD: State the instruction once clearly, mark it as critical if needed

Emphasis Escalation

• BAD: "CRITICAL: You MUST ALWAYS use the search tool. NEVER skip it"
• GOOD: "Use the search tool when the answer depends on current information"
• Claude 4.5 and later overtrigger on language written to cure undertriggering on older, less steerable models; the vendor's own examples still state hard rules with never and must, so the fix targets trigger phrasing, not capitals

Explicit CoT on Reasoning Models

• BAD: "Think step by step inside tags" sent to an adaptive-thinking, extended-thinking or o-series model
• GOOD: State the task, success criteria, and the effort setting; let the model reason natively
• Exception: at the lowest effort setting, a brief explicit plan or thought summary at the start of the answer improves results (OpenAI GPT-5 guide); use it there and nowhere else

Reasoning Traces as Exemplars

• BAD: few-shot examples whose answers include worked step-by-step reasoning, sent to a reasoning model
• GOOD: zero-shot, or examples that show only input and output; if the demonstrations carry a strategy, state the strategy as an instruction instead
• Few-shot CoT often does worse than direct answering on RL-trained reasoners, and more exemplars, even optimal traces, degrade further; distilling them into explicit insights recovered +14.0% on AIME'25 for GPT-4.1 (arXiv 2509.23196)

2. Score only the dimensions that archetype wants

Mark every other dimension N/A with a short clause saying why.

3. Scoring rules

• Score against the archetype, never against a generic ideal. Forcing identical structure onto a creative prompt, or maximum specificity onto an exploratory one, makes the prompt worse while making the score look better.
• The target is the right profile, not 5/5 everywhere. Say so when a dimension is deliberately left mid-scale.
• Flag anti-patterns from <anti_patterns> separately: they are defects, not scores.
• These scores are diagnostic. They locate weaknesses. They do not demonstrate improvement, and a before/after score pair is not evidence. See <epistemic_status>.
• Revise before presenting when an applicable dimension sits below 4 and the archetype wants it high. Do not revise to raise a dimension the archetype does not want.

4. When the prompt is already good

Say so, and stop. "No material optimization warranted" is a real conclusion and the correct one more often than an optimizer likes to admit. If you do propose changes to a strong prompt, every one of them names the defect it fixes. These three are not defects, and each is a way a rewrite grows without getting better:

• restructuring justified as clarity, tidiness, or better organization alone
• examples added to a prompt whose format was already unambiguous
• a behavior that reaches the rewrite without appearing in the diagnosis

A rewrite that doubles the token count of a prompt you called excellent needs a reason for each addition, not one reason for the set.

• Predicted - your own judgment. Every rubric score, every parity estimate, every "this should be more reliable" produced in a single pass is predicted. Say the word out loud; do not let a number imply more.
• Measured - an eval was actually run. Report the method with the number: identical inputs per variant, the grader used, the sample size.
• Verified - measured, plus an independent check: a held-out set, a judge from a different model family, or human review.

"Reliability improved 30%" without a run is a false claim, not an optimistic one. The honest form names the mechanism instead: "predicted: fewer malformed outputs, because the schema is now stated before the task rather than after it."

Rubric scores are diagnostic. They locate weaknesses. A before/after score pair written by the same model that wrote the rewrite is not evidence that the rewrite is better, and formatting changes alone are known to swing task accuracy, so a single side-by-side comparison is noise.

• Build the eval before or alongside the prompt; maintain it like unit tests
• Start with 20-50 tasks drawn from real failures (Anthropic, demystifying evals for AI agents, 2026-01); a good task is one where two domain experts independently reach the same pass/fail verdict; when the suite saturates, build harder variants rather than declaring victory
• Grader ladder: code-based assertions first (exact match, regex, is-json, schema validation), model-based graders where flexibility is needed, human review as gold standard; grade what was produced (the end state), not the path taken, and grade each dimension with its own isolated judge
• Score parse failures separately from wrong answers: most measured format sensitivity is answer extraction failing, which is a different fix from a wrong answer (Format Sensitivity Index, arXiv 2607.09665)
• Choose the metric by what one success means: pass@k for tools where one success matters, pass^k for agents where consistency is essential
• LLM-as-judge safeguards: a judge from a different model family than the system under test, randomized pairwise order (position bias above 0.10 was measured in production judges in 2026), an explicit "Unknown" escape clause, binary or 3-point scales over 1-10, single-purpose judges per criterion, candidate output treated as untrusted input; keep a verbosity control in the rubric as a secondary guard (under a pairwise rubric, verbosity bias measured below 0.011 across 21 judges, arXiv 2606.19544)
• The judge's own prompt is a measured shape, in judge-prompting.md: one criterion per judge, a binary decision with quoted evidence, a reference answer when a trustworthy one exists, a generated checklist for open-ended criteria, and 0-5 when a scalar is needed; judge personas, judge debates and strictness instructions on strong judges measured zero or negative. The numbers behind each of those, and the conditions each was measured under, are in that reference; read it before restating one
• Validate the judge before trusting it: chance-corrected agreement with human labels (Cohen's kappa, never raw exact match, which overstated agreement by 33 to 41 points), on your own task, because judge rankings shifted by up to 14 positions across benchmarks; consistency is not validity, and a bias correction can flip the sign of a result while looking confident (arXiv 2605.06939)
• Read transcripts regularly to confirm graders measure what you intend
• Automatic optimization: GEPA (arXiv 2507.19457, ICLR 2026) is the reference optimizer, reading execution traces and keeping a Pareto frontier of candidates (over 10% above MIPROv2, up to 35x fewer rollouts than RL); it needs a grader and a trai

…(truncated)