Gepa

GEPA

GEPA is a reusable optimization workflow for bounded textual artifacts: prompts, rubrics, tool descriptions, extraction instructions, review checklists, or similar text whose quality can be measured on examples.

GEPA is not reflection. Reflection and Five Whys may notice a GEPA-shaped problem, but GEPA starts only after the target artifact, evaluator, dataset, budget, and adoption gate are concrete.

Contract

Trigger: Use GEPA when a prompt/rubric/tool description/extraction instruction keeps underperforming, the artifact is narrow enough to mutate safely, and better vs. worse can be measured on a trace set, labeled examples, or evaluator.

Requires:

1. Target artifact — exact file/path or prompt block plus frozen baseline text.
2. Evaluator — explicit score and diagnostic feedback per example. Numeric-only scores are not enough; the evaluator must return Actionable Side Information (ASI): what failed, where, and why.
3. Dataset / trace set — representative examples, with a holdout split when possible.
4. Budget / stopping rule — max metric calls, wall-clock, and minimum adoption threshold.
5. Adoption gate — optimized text lands as a proposed diff / PR / proposal; GEPA never auto-replaces a production prompt.

Produces: baseline-vs-candidate comparison, evaluator results, ASI failure analysis, known tradeoffs, and an adopt/reject recommendation.

Does not: optimize identity/persona/core values, replace reflection, repair missing instrumentation, invent a fake evaluator, or mutate code semantics hidden behind text.

Fit checklist

Use GEPA only when all are true:

• The target is one bounded textual artifact.
• The failure recurs or affects an important recurring path.
• A meaningful evaluator or labeled trace set exists now.
• The evaluator can explain failures with ASI, not just return a scalar.
• The artifact can be changed and reviewed independently of broad governance.
• A normal adoption gate can reject the candidate if it overfits or regresses.

If any item is false, do not run GEPA yet. File the prerequisite instead: build the dataset, write the evaluator, add instrumentation, or run Five Whys/reflection on the framing.

Anti-fit checklist

Do not use GEPA when any are true:

• The real problem is ambiguous goals, operator alignment, persona, or governance.
• The artifact is memory/core/*, a persona/value block, or a policy boundary.
• The first honest action is "define the evaluator" or "collect examples".
• The score is easy to Goodhart or cannot cite per-example failure reasons.
• The measured outcome depends mainly on non-text knobs GEPA cannot move.
• The proposed change would silently ship without human/operator review.

Rule of thumb: if a competent reviewer could not tell whether candidate text won for the right reasons, GEPA is premature.

Standard workflow

1. Freeze the task

Create or update a chainlink issue / spec that records:

• artifact path and exact baseline text or commit SHA
• dataset / trace paths
• evaluator command or scoring rubric
• metric priorities and failure modes the metric misses
• max_metric_calls / cost cap / wall-clock cap
• adoption threshold and review path

For codebase prompts, the adoption path is a normal feature branch and PR. For protected memory or prompt-template surfaces, use the protected-surface proposal workflow; do not edit live surfaces directly.

2. Verify evaluator quality before optimizing

Run the evaluator on the baseline and inspect examples. It must return ASI that is specific enough to guide rewrites, e.g.:

• source example id
• expected behavior / gold label
• actual extraction or decision
• false-positive / false-negative / compression / hallucination reason
• exact source text or trace span that supports the diagnosis

If the evaluator cannot produce this, stop. Improve the evaluator first.

3. Run a bounded optimization pass

Use the standalone gepa library rather than reimplementing the optimizer. It is not installed by default — it ships as the opt-in gepa extra (the optimizer is only needed while a pilot runs, so the base runtime stays lean):

• PyPI deployments: pip install 'mimir-agent[gepa]'
• workspace / uv deployments: uv sync --extra gepa

gepa has no langchain extra. Route its reflection LM (and the task model) through mimir's already-configured ChatModel via mimir.gepa_support rather than a separate LiteLLM / OpenAI key — reflection_lm takes a plain (prompt: str) -> str callable:

import gepa
from mimir.gepa_support import reflection_lm_from_config

result = gepa.optimize(
    seed_candidate={"instructions": baseline_prompt},
    trainset=train_examples,
    valset=holdout_examples,
    adapter=my_pilot_adapter,           # evaluator + ASI; see step 2 and chainlink #404
    reflection_lm=reflection_lm_from_config(),
    max_metric_calls=100,               # explicit cost cap; raise only after a useful pilot
)

Adapt the exact API to the installed gepa version, but keep the invariants: explicit baseline, explicit examples, ASI-rich evaluator, explicit metric-call cap, and no auto-application of result.best_candidate.

4. Compare on holdout and write the decision record

Record:

• baseline score and candidate score
• per-failure ASI themes
• examples improved, unchanged, and regressed
• whether gains hold outside the optimization slice
• any new failure mode introduced by the candidate
• recommendation: adopt, reject, or revise evaluator/dataset and rerun

Reject candidates that only improve the training slice, trade precision for unacceptable hallucinations, or win by exploiting evaluator blind spots.

5. Ship through the normal gate

If adopting:

• create a PR/proposal with the candidate text diff
• include the baseline-vs-candidate results in the body
• keep parser/schema/version bumps explicit when the artifact has one
• run the ordinary tests for that subsystem

Never write GEPA output directly into production prompts or protected memory.

Good first targets

• Commitment extraction prompt — bounded artifact in mimir/commitments/extractor.py; chainlink #137 already provides a 30-session corpus and precision/recall / over-compression metrics.
• Session-summary / unfinished extraction prompt — high impact, but only after a labeled corpus and evaluator exist.
• Tool descriptions — useful when there is a probe set with known desired tool choices; target one tool or one small family, not the whole catalog.
• Review skill prompt compliance — plausible after isolating text behavior from harness/workflow failures; success must include actual gh pr review submission.

Anti-target examples

• Weekly reflection as a whole — too broad and governance-heavy; reflection can recommend GEPA but should not become GEPA.
• Core identity / persona / values blocks — not local evaluator-backed prompt artifacts; narrow metrics would distort policy.
• "Be less sycophantic" globally — useful concern, bad GEPA target unless reduced to one artifact plus a labeled evaluation set.
• Recall/simhi bench prompt tweaks when scores are dominated by non-text knobs such as thresholds, triple toggles, and retrieval strategy.

Pilot template

Copy this into the chainlink issue or spec for a real run:

## GEPA pilot: <artifact>

- Target artifact: `<path>` / `<symbol>`
- Baseline commit/text: <sha or frozen block>
- Dataset / traces: `<path>`
- Evaluator command: `<command>`
- Primary metric: <precision/recall/etc.>
- ASI fields: <what diagnostic text must cite>
- Budget: max_metric_calls=<N>, wall-clock=<duration>
- Adoption threshold: <minimum holdout improvement and non-regression rules>
- Adoption gate: PR/proposal reviewed by <who/surface>
- Known blind spots: <metric limitations>

If you cannot fill this in, do not run GEPA yet.