yanacuti1121 When to Use
- Use when: designing or iterating a system prompt
- Use when: model outputs are inconsistent, verbose, or off-target
- Use when: adding few-shot examples to improve output format
- Use when: managing multiple prompt versions across environments
- Do NOT use for: retrieval/RAG problems — those are a chunking/search issue
- Do NOT use for: model selection trade-offs — that is
model-routing-strategy
Prompt Anatomy
A complete prompt has these layers (in order):
1. Role framing — who the model is and its constraints
2. Task definition — what to do, output format
3. Context injection — user-specific data, retrieved chunks
4. Few-shot examples — 2–5 input/output pairs
5. Chain-of-thought — "think step by step" or explicit reasoning scaffold
6. Constraints — what NOT to do (negative space)
7. Output format — JSON schema, Markdown structure, word limit
Not every prompt needs all 7 — omit layers that add noise.
Core Techniques
Role framing
System: You are a senior software engineer reviewing pull requests.
You give precise, actionable feedback. You do not praise trivially.
You flag security issues before style issues.
- Be specific about expertise domain, not just "expert"
- Add behavioral constraints ("do not", "always", "never") — models follow negations well
- Keep role consistent — don't contradict it in user turns
Few-shot examples
# 2–5 examples optimal. More doesn't always help, can confuse.
Input: "Summarize this legal clause in plain English."
Output: "The contractor must deliver within 30 days or pay a 5% penalty."
Input: "Summarize this legal clause in plain English."
Output: [your actual example output here]
- Examples must cover the full output format, including edge cases
- Negative examples (what NOT to output) can be more powerful than positive ones
Chain-of-thought (CoT)
# Zero-shot CoT — works on reasoning tasks
"Think step by step before giving your final answer."
# Explicit scaffold — for complex multi-step tasks
"First, identify the problem type. Second, list relevant constraints.
Third, propose a solution. Finally, check for edge cases."
CoT significantly improves accuracy on math, logic, and multi-step tasks. Not needed for simple classification or extraction.
Constraint injection
Constraints:
- Output must be valid JSON matching this schema: {...}
- Do not include explanations outside the JSON object
- If information is missing, use null — do not hallucinate
- Maximum 3 items in the "recommendations" array
Explicit constraints reduce hallucination and format drift.
Output Format Control
JSON output
Output a JSON object with exactly these fields:
{
"summary": string (max 100 words),
"sentiment": "positive" | "negative" | "neutral",
"confidence": float (0.0–1.0)
}
Do not include any text outside the JSON object.
Markdown structure
Use this exact structure:
## Finding
[one paragraph]
## Impact
[one paragraph]
## Recommendation
[bulleted list, max 3 items]
Word/token budget
Answer in exactly 2–3 sentences. No more.
Models respect explicit word/sentence limits better than vague "be concise".
Prompt Versioning
Every prompt used in production must be versioned:
prompts/
review-pr/
v1.md ← initial
v2.md ← added CoT
v3.md ← added JSON constraint
current → v3 ← symlink or env var
Version commit message format:
prompt(review-pr): v2 — add chain-of-thought scaffold
Reason: v1 missed multi-file refactors (hit rate 0.61 → 0.84 with CoT)
Eval: tested on 50 PR samples, measured format compliance + accuracy
A/B Testing Prompts
Minimum viable prompt A/B test:
- Write variant A and variant B (change one variable only)
- Run both on same 50-item eval set
- Measure: format compliance, accuracy, latency, token cost
- Promote winner; document why loser lost
Never ship a prompt change without an eval baseline. "Felt better" is not evidence.
Common Failure Patterns
| Symptom | Cause | Fix |
|---|---|---|
| Model ignores format instruction | Instruction buried at end | Move format spec to top, before task |
| Outputs vary wildly | No examples | Add 2–3 few-shot examples |
| Model adds unwanted preamble | No negative constraint | Add "Do not explain — output only the result" |
| Model hedges excessively | Role too generic | Tighten role: "You are confident and direct" |
| Hallucination | No grounding | Add "Only use information from the provided context" |
| Instruction following degrades | Prompt too long (> 2K tokens) | Split into system + user; move examples to user |
Anti-Fake-Pass Rules
Before claiming a prompt is production-ready, you MUST show:
- Prompt tested on ≥ 20 representative inputs (not just 1 example)
- Output format validated (JSON parsed, Markdown structure checked)
- Version documented (v1, v2...) with reason for each change
- Failure cases identified and addressed (at least 2 edge cases tested)
- Token cost estimated for p50 / p95 input size
Reference: gates/anti-fake-pass-gate.md