zavelinski few-shot-curator
When zero-shot drifts on format or style, a few well-chosen examples steer it better than another paragraph of instructions. The trick is WHICH examples: relevant, diverse, correct.
Why this exists (evidence)
- In-context learning is strongly sensitive to which examples you pick: good example selection beats random examples, and relevant + diverse examples beat more-but-noisy ones (Liu et al., "What Makes Good In-Context Examples for GPT-3?", and the broader ICL literature). The win is from curation, not quantity.
- It targets the failure where the model "understands" the task but keeps producing the wrong shape/tone, and adding more prose instructions does not fix it.
When to use
- Output format/structure the model keeps getting subtly wrong (a specific JSON, a commit-message style, a tone).
- Extraction/classification where edge cases need to be shown, not described.
- NOT when zero-shot already nails it (examples just cost tokens), and not as a crutch for an unclear task (fix the instruction first).
The method
- Pick 2-5 examples, curated:
- Relevant: close to the actual inputs you expect.
- Diverse: cover the range and the edge cases (the tricky ones, not five near-duplicates).
- Correct: each example's output is exactly what you want (a wrong example teaches the wrong thing).
- Format-faithful: examples show the EXACT output shape you want back.
- Order intentionally: put the most representative/important last (recency in the prompt carries weight).
- Keep them lean: shortest examples that still demonstrate the pattern (compose with prompt-compression).
- Iterate: if it still drifts on a case, add an example of THAT case rather than more prose.
How to run it
- Maintain a small bank of golden input->output pairs for recurring task types; select the relevant few per task.
- For classification, include at least one example per class and per known confusable.
- Verify the examples are correct before using them (a bad few-shot example is worse than none).
Composes with
structured-output: examples + a schema = strong, consistent shape.prompt-compression: keep the examples minimal so they do not bloat context.self-consistency: if unsure which examples help, test a couple of sets and keep the one that yields consistent correct output.
Honest limits
- Examples cost context tokens every call; do not few-shot what zero-shot already does.
- Wrong/biased examples actively teach the wrong behavior; curate, do not grab the first samples.
- More examples is not better past a few; relevance and correctness beat count.