OmidZamani DSPy Optimizer Selection
Goal
Choose the smallest DSPy optimizer that matches the data, budget, and artifact being tuned. Establish a baseline before compiling anything.
Selection Matrix
| Need | Start with | Notes |
|---|---|---|
| Include a few labeled examples | dspy.LabeledFewShot |
Random labeled demos; useful as a baseline |
| About 10 examples | dspy.BootstrapFewShot |
Teacher-generated demos with metric filtering |
| 50+ examples and stronger demo search | dspy.BootstrapFewShotWithRandomSearch |
Searches multiple demo sets; alias: dspy.BootstrapRS |
| Per-input nearest demos | dspy.KNNFewShot |
Retrieves nearby examples before bootstrapping |
| Instruction-only hill climbing | dspy.COPRO |
Coordinate ascent over instructions |
| Instruction and demo search | dspy.MIPROv2 |
Bayesian search; install dspy[optuna] |
| Mini-batch introspective rules or demos | dspy.SIMBA |
Uses output variability and self-reflection |
| Rich textual feedback and trace reflection | dspy.GEPA |
Metric must accept five arguments |
| Distill prompts into model weights | dspy.BootstrapFinetune |
Requires a fine-tunable LM and set_lm() |
| Combine candidate programs | dspy.Ensemble |
Trades inference cost for robustness |
| Sequence prompt and weight optimization | dspy.BetterTogether |
Meta-optimizer for configurable optimizer chains |
Workflow
- Split data into train and validation sets.
- Evaluate the uncompiled program with dspy-evaluation-suite.
- Start with the least expensive optimizer that matches the need.
- Save the compiled program and compare it against the baseline.
- Escalate only when the measured gain justifies extra LM calls, fine-tuning, or inference cost.
Common Paths
Fast Demo Optimization
Use dspy-bootstrap-fewshot for the first optimization pass. Move to BootstrapFewShotWithRandomSearch when enough examples are available to search multiple demo sets.
Prompt Search
Use dspy-miprov2-optimizer for instruction and demonstration search. Install its optional dependency first:
pip install -U "dspy[optuna]>=3.2.1,<3.3"
Reflective Optimization
Use dspy-gepa-reflective when failures can be described with actionable text. Use dspy-simba-optimizer for a smaller mini-batch introspective loop with numeric metrics.
Prompt Plus Weight Optimization
Use dspy-better-together when a fine-tunable LM is available and prompt optimization alone has plateaued.
Best Practices
- Keep a held-out validation set.
- Track optimization cost and inference cost separately.
- Use reproducible seeds where supported.
- Avoid claiming one optimizer is universally best; compare measured results.
- Save intermediate candidates for expensive runs.
Official Documentation
- Optimizer guide: https://dspy.ai/learn/optimization/optimizers/
- Optimizer API index: https://dspy.ai/api/optimizers/
- DSPy releases: https://github.com/stanfordnlp/dspy/releases