Scaffold Example

Scaffold a new Opik example

This repo has two canonical shapes, each a real, runnable template under templates/:

• templates/use-case-template/ (default) — the f1_radio_rag shape: a uv project with a src/<pkg>/ package, ruff, a Typer CLI wired through the full Opik loop (traced app → dataset + test suite → evaluate/run_tests → optimizer → Prompt Library), data/ fixtures, 4-section README. For end-to-end demos.
• templates/script-template/ — the scripts/ shape: a single .py file with an argparse CLI, env-var credentials + DRY_RUN, ruff, and a 4-section README. For standalone utilities (trace cleanup, exports, dashboards).

Getting either right by hand is fiddly and easy to drift on. This skill copies the chosen template and renames it, so you start from something that already runs in DRY_RUN and passes ruff.

When to refuse

You need two things:

• A name (e.g. support_ticket_summarizer, trace_dumper). It becomes the folder, the Python module/package, the project name, and the default CLI command.
• A one-line description of what it does.

If either is missing, ask one short question. Don't invent a demo — an empty scaffold with a guessed purpose wastes the user's time.

Step 1 — Generate the project

Run the bundled generator. It copies the chosen template, renames its sentinel identity to the new name, and writes the description. It refuses to overwrite an existing folder.

# Use-case demo (default template, into use-cases/):
python .claude/skills/scaffold-example/scripts/scaffold.py <name> \
  --description "<one line>"

# Utility script (script template, into scripts/):
python .claude/skills/scaffold-example/scripts/scaffold.py <name> \
  --bucket scripts --template script-template --description "<one line>"

# options: --bucket use-cases (default; or scripts/guides/integrations),
#          --template use-case-template (default) | script-template, --command <cli-name>

Pick the template by shape: an end-to-end Opik demo → use-case-template; a standalone utility → script-template (with --bucket scripts). Naming derived from <name>: module + folder are snake_case, project name is kebab-case, the CLI command defaults to the kebab name (pass --command to override). The script prints the package/project/command it chose and a next-steps checklist.

Step 2 — Make it real

Script template: there's no src/ package — fill the TODO in the single <name>.py (the run() body) and the README's 4 sections, then skip to Step 3. The rest of this step is use-case-template only.

The scaffold runs as-is but is generic. Fill the parts marked TODO, in roughly this order:

• data/cases.json — replace the sample items with real eval cases for this use case (input, expected_output, context, assertions). The add-dataset-items and add-eval-suite-items skills cover the item/assertion shapes in depth.
• prompts.py — the SYSTEM_PROMPT and USER_TEMPLATE your task needs. The user template uses single-brace {field} placeholders binding to dataset item keys.
• app.py — the @opik.track-decorated run() is the deployed app. Replace its body with the real logic (retrieval, tool use, chains). It must return {"input", "output", "context"} so the metrics can score it.
• config.py — adjust model strings and the dataset/suite/prompt names if needed.
• README.md — fill the four sections (What this does / Prerequisites / Running it / How it works). This is required for every example (see AGENTS.md).
• pyproject.toml — add any extra dependencies the use case needs (e.g. a vector store).

Keep the repo conventions: credentials from env vars only, the DRY_RUN fallback intact, # WHY:-only comments.

Step 3 — Verify

From the new project directory:

uv sync
uv run <command> eval        # use-case: [DRY RUN] lists the dataset items + assertions
uv run <command> optimize    # use-case: [DRY RUN] line
uv run <command> --dry-run   # script: [DRY RUN] line
uv run ruff check .          # clean
bash run.sh                  # the exact dry-run smoke test CI runs (exits 0 without creds)

The scaffold includes a working run.sh at the project root — this is what the CI dry-run and live-run jobs execute. With credentials set (ANTHROPIC_API_KEY, OPIK_API_KEY, OPIK_WORKSPACE), the same commands talk to Claude + Opik and eval prints a pass rate + experiment URL. Running the eval and optimization workflows in depth is covered by the run-evals and run-optimizations skills.

What this skill does NOT do

• No git commit / PR. Leave the new files in the working tree for the user to review.
• No domain logic. It scaffolds; you (with the user) write the actual app, prompts, and data.