Run Optimizations

Run a prompt optimization

opik-optimizer searches for a better prompt by scoring candidates against a dataset with a metric. It tunes the prompt and model params — not the retriever. Retrieval quality is a separate concern, watched via ContextRecall in run-evals.

Reference: run_optimization in optimization.py; promote in prompts.py; CLI cli.py optimize/promote.

When to refuse

Need a target example with an eval dataset to score against — optimization is meaningless without a metric over real data. If there's no dataset, route to add-dataset-items first.

Step 1 — Credentials gate

Needs OPIK_API_KEY + OPIK_WORKSPACE and the model provider key (e.g. ANTHROPIC_API_KEY). Dry-run first:

cd use-cases/f1_radio_rag
uv run f1rag optimize                   # DRY_RUN: prints what it would optimize
export ANTHROPIC_API_KEY=... OPIK_API_KEY=... OPIK_WORKSPACE=...
uv run f1rag optimize                   # real run: prints initial -> optimised score + run link

Step 2 — The metric is a plain callable

Optimizer metrics are not the metric classes directly — they're callables (dataset_item: dict, llm_output: str) -> float with __name__ set (the name shows in the Opik UI). Wrap an LLM-judge metric to score against something real:

from opik.evaluation.metrics import AnswerRelevance

def _relevance_metric(dataset_item: dict, llm_output: str) -> float:
    result = AnswerRelevance(model=JUDGE_MODEL).score(
        input=dataset_item["query"], output=llm_output, context=dataset_item["messages"],
    )
    return result.value

_relevance_metric.__name__ = "answer_relevance"

Step 3 — Build the prompt and run the optimizer

from opik_optimizer import ChatPrompt, MetaPromptOptimizer

prompt = ChatPrompt(name=PROMPT_NAME, system=SYSTEM_PROMPT, user=USER_TEMPLATE, model=GEN_MODEL)
optimizer = MetaPromptOptimizer(model=OPTIMIZER_MODEL, n_threads=4, skip_perfect_score=False)
result = optimizer.optimize_prompt(
    prompt=prompt, dataset=dataset, metric=_relevance_metric, max_trials=12, n_samples=8,
)

• ChatPrompt.user uses single-brace {field} placeholders that bind to dataset item fields.
• skip_perfect_score=False forces every round even when the baseline already scores high.
• max_trials / n_samples trade cost for thoroughness — keep them modest for demos.

Step 4 — Read the result

• result.initial_score → result.score — the improvement (or lack of it). Report both.
• result.get_run_link() — link to the run in Optimization Studio. Surface it.
• result.prompt.get_messages() — the optimised messages, for promotion.

Step 5 — Promote (optional, only when asked or score improved)

Saving versions the prompt in the Prompt Library; re-using the same name appends a version (upsert — no separate update call, no in-place edit).

client.create_chat_prompt(
    name=PROMPT_NAME,
    messages=result.prompt.get_messages(),
    change_description=f"optimised score {result.score:.3f}",
    tags=["optimised"],
)

Don't promote a regression. If result.score <= result.initial_score, report it and stop — ask before saving a worse prompt over a working one.

Verify

• Dry-run prints the prompt/dataset it would optimize before the real run.
• Real run prints initial -> optimised score and a run link; open it to inspect trials.
• After promote, the new version appears under the prompt name in the Opik Prompt Library UI.