# Agent Observability Eval Pipeline > End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says "run the eval pipeline", "go from traces to evals", "bootstrap evals end to end", "classify then RCA then bootstrap", "build an eval set from scratch", "onboard me to datasets and experiments", "walk me through experiments", "I have an ml_app, now what", "Agent Observability onboarding", "guided experiment setup", "from traces to experiments", or wants a deterministic, narrated tour from production data through evalu - Skill: `gabrielmoreira/agent-observability-eval-pipeline` (Agent Skill) - Install (CLI): `npx skillmds@latest add gabrielmoreira/agent-observability-eval-pipeline` - Raw SKILL.md: https://api.skillmd.com/api/skills/gabrielmoreira/agent-observability-eval-pipeline/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: gabrielmoreira (https://skillmd.com/u/gabrielmoreira) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/gabrielmoreira/agent-observability-eval-pipeline --- ## Backend **Detection** — At the start of every invocation, before taking any action, determine which backend to use: 1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4. 2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__search_llmobs_spans` appears in your available tools. 3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in the sub-skill workflow sections. 4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available. 5. If pup responds → use **pup mode** throughout. Each sub-skill carries its own Tool Reference appendix with the full MCP→pup mapping. 6. If neither is available → stop and tell the user: > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs'`) or install pup." `--backend pup` is accepted anywhere in the invocation arguments. Strip it from args before passing to sub-skills, but carry the pup-mode decision forward — every sub-skill must also operate in pup mode for the entire pipeline run. **Sub-skill backend propagation**: The backend detected at startup applies to all sub-skills invoked across the six phases. Do not re-detect per phase. Announce once at startup: - MCP mode: "(Running in MCP mode — all features available.)" - pup mode: "(Running in pup mode — pup commands used throughout. All features available.)" **Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation. **Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-eval-pipeline[] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:agent-observability-eval-pipeline:start[] — ` instead (note the `:start` suffix). Example first call: `skill:agent-observability-eval-pipeline:start[3a9f1c2b] — Precheck: verify ml_app has traces in the last 7 days` --- # Agent Observability Eval Pipeline — Classify → RCA → Eval Bootstrap → Dataset → Experiment → Analyze A deterministic, six-phase guided pipeline for an already-instrumented `ml_app` owner. Each phase has the same envelope — a banner that names the entity being produced, an explanation of its purpose, the action (a sub-skill call or a small executable step), and a checkpoint. **You always know where you are.** ``` [Precheck] verify ml_app, project, backend, credentials, output dir ↓ [Phase 1: Classify ml_app traces] entity: ml_app, trace, span ↓ [Phase 2: Root cause analysis] entity: failure mode, root cause ↓ [Phase 3: Bootstrap evaluators] entity: evaluator, LLM judge ↓ (stop here with --stop-after eval-bootstrap ↓ for the classic eval-pipeline behavior) [Phase 4: Create + publish dataset] entity: dataset record, published dataset ↓ (executes: LLMObs.create_dataset) [Phase 5: Generate + run experiment] entity: experiment, task, evaluator, run ↓ (executes: python ) ↓ (in-phase review beat before run) [Phase 6: Analyze experiment] entity: metric, comparison, recommendation ``` This skill is **pure orchestration plus pedagogy** — no new analytical logic. The work happens inside the sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). What this skill adds is the deterministic envelope: every phase has the same shape, the same checkpoint contract, and the same entity-explanation banner — so the user gets a consistent, narrated experience regardless of how they phrased the original request. ## Usage ``` /agent-observability-eval-pipeline [--project-name ] [--timeframe ] [--trace-limit ] [--format py|ipynb] [--evaluator-style function|class|remote] [--offline-evaluators | --online-evaluators | --data-only] [--start-at classify|rca|eval-bootstrap|dataset|experiment|analyze] [--stop-after classify|rca|eval-bootstrap|dataset|experiment|analyze] [--classification-summary ] [--rca-report ] [--dataset-file ] [--dataset-name ] [--experiment-file ] [--experiment-id | --experiment-url ] [--app-root ] [--env-file ] [--output-dir ] [--backend pup] ``` Arguments: $ARGUMENTS ### Inputs | Input | Required | Default | Description | |-------|----------|---------|-------------| | `ml_app` | Yes | — | The instrumented LLM app to onboard / evaluate against. The precheck verifies it has recent traces. | | `--project-name` | No | derived from `pyproject.toml` / `setup.cfg` / `setup.py` / `package.json` / cwd (same order as `agent-observability-experiment-bootstrap`); falls back to `experiment-sdk-default` | The Datadog **project** the pipeline writes datasets and experiments into. The SDK lazily creates the project on first use via `LLMObs.enable(project_name=...)`. Surface this in the Precheck so the user can confirm before anything is created. | | `--timeframe` | No | `now-7d` | Lookback window for Phase 1 classification and Phase 4 dataset sampling. | | `--trace-limit` | No | `20` | Sampling cap for Phase 4. Phase 1 internally uses `min(20, --trace-limit)` for the classification sample. | | `--format` | No | `py` | Passed to `agent-observability-experiment-bootstrap` in Phase 5: `py` (script) or `ipynb` (Jupyter notebook). | | `--evaluator-style` | No | `function` | Passed to `agent-observability-eval-bootstrap` (Phase 3) and `agent-observability-experiment-bootstrap` (Phase 5): `function`, `class`, or `remote`. | | `--offline-evaluators` | No | on (default) | Phase 3: emit a Python SDK evaluator suite (BaseEvaluator / LLMJudge classes) that runs inside an experiment against a dataset. Maps internally to `agent-observability-eval-bootstrap` `sdk_code` mode. | | `--online-evaluators` | No | off | Phase 3: publish online LLM-judge evaluators directly to Datadog (created as disabled drafts; enable in the UI). Online evaluators run on production spans as they're emitted. Maps internally to `agent-observability-eval-bootstrap` `publish` mode (was `--publish`). | | `--data-only` | No | off | Phase 3: emit a local data blob only — no executable evaluator code or online publish. At Phase 3 entry the skill **prompts** the user to pick one of: (a) a `DatasetRecordRaw[]` JSON suitable for experiment use (maps internally to `agent-observability-eval-bootstrap --emit-dataset`), or (b) a framework-agnostic JSON evaluator spec for local analysis (maps internally to `agent-observability-eval-bootstrap` `data_only` mode). | | `--stop-after ` | No | `analyze` (run everything) | Stop after the named phase completes. `classify` = Phase 1 only. `rca` = through Phase 2. `eval-bootstrap` = through Phase 3 (matches the classic eval-pipeline). `dataset` = through Phase 4 (dataset created + published). `experiment` = through Phase 5 (experiment generated + run). `analyze` = all six phases (default). | | `--start-at ` | No | `classify` (start at the top) | Skip earlier phases and start at the named phase. Same vocabulary as `--stop-after`. The skill auto-loads any required prior-phase artifacts from `/state/` (see "State persistence and entry/exit" section). For phases that need an artifact the auto-load can't find, supply it via one of the override flags below. Combinable with `--stop-after` to run a contiguous slice of the pipeline. | | `--classification-summary ` | No | auto-loaded from `/state/01-classification.md` if `--start-at rca` or later | Override the Phase 1 output that Phase 2 consumes. Useful when the prior state file is missing or you want to point at a hand-edited version. | | `--rca-report ` | No | auto-loaded from `/state/02-rca-report.md` if `--start-at eval-bootstrap` or later | Override the Phase 2 output that Phase 3 consumes. | | `--dataset-file ` | No | auto-loaded from `/state/04-published-dataset.json`'s `dataset_file` field (or the most recent `/dataset__*.json`) | The local `DatasetRecordRaw[]` JSON. Used by Phase 4's publish sub-step when re-publishing without re-sampling. | | `--dataset-name ` | No | auto-loaded from `/state/04-published-dataset.json` if `--start-at experiment` | The name of a published Datadog dataset that Phase 5 wires the experiment to. | | `--experiment-file ` | No | auto-loaded from `/state/05-experiment-run.json`'s `experiment_file` field if `--start-at experiment` and the file already exists | The generated experiment file. When present, Phase 5 skips the codegen sub-step (5a) and goes straight to the review beat (5b) → run (5c). | | `--experiment-id ` | No | auto-loaded from `/state/05-experiment-run.json` if `--start-at analyze` | The Datadog experiment ID Phase 6 analyzes. Mutually exclusive with `--experiment-url`. | | `--experiment-url ` | No | auto-loaded as above | Alternative to `--experiment-id`. The skill parses the trailing UUID out of the URL. | | `--app-root` | No | resolved from cwd / `pyproject.toml` etc. | Restricts `agent-observability-experiment-bootstrap`'s task-function introspection to this directory tree. | | `--env-file` | No | none (auto-discovery walks standard locations) | Explicit `.env` path for credential loading. Surfaced in the Precheck and baked into the generated experiment as `ENV_FILE_OVERRIDE`. | | `--output-dir` | No | `./experiments` | Where the dataset JSON, publish script, and generated experiment file are written. | | `--backend` | No | auto-detect | `pup` forces pup mode regardless of MCP availability. | If `ml_app` is not provided, ask the user before proceeding. The three evaluator-output flags (`--offline-evaluators`, `--online-evaluators`, `--data-only`) are mutually exclusive — error out if more than one is set. If none are set, `--offline-evaluators` is the default. The legacy flag names `--publish` and `--sdk-code` are still accepted as aliases for backward compatibility but map to the new names in all output. --- ## Precheck Before Phase 1, run a single short verification pass — do **not** announce a "Phase" banner for this; it's plumbing. Output a one-block precheck summary, then move directly to Phase 1. 1. **Backend** — already detected at the top of this skill. Note the chosen backend in the precheck output so the user can confirm. 2. **ml_app has recent traces** — call `search_llmobs_spans(query="@ml_app:\"\"", root_spans_only=true, limit=1, from="")` (MCP) or the pup equivalent. If the result is empty, stop and tell the user the precheck failed — there is nothing to evaluate against — and suggest widening `--timeframe` or confirming the ml_app name. 3. **Resolve `project_name`** — if `--project-name ` was supplied, use it verbatim. Otherwise derive using the same resolution order as `agent-observability-experiment-bootstrap` (Workflow step 1): `pyproject.toml` → `setup.cfg` → `setup.py` → `package.json` → cwd basename (slugified). Final value is `experiment-`; fall back to `experiment-sdk-default` if nothing resolves and emit a warning telling the user to set `--project-name` explicitly. **Project creation semantics**: the project is created lazily by the Datadog SDK the first time `LLMObs.enable(project_name=...)` is called against the org (in Phase 4's publish script, and again in Phase 5's generated experiment). The user does not need to pre-create anything in the UI. Surface the chosen project name in the Precheck output so the user can override before Phase 4 if it isn't what they wanted. 4. **Ensure `--output-dir` and `/state/` exist** — `mkdir -p /state` via Bash. Cheap. The `state/` subdirectory is where phase outputs get persisted (see "State persistence and entry/exit" below). 5. **Resolve credentials.** Walk the discovery order below to find Datadog credentials before Phase 4 needs them — failing late at the publish step is bad UX. Read-only at this stage: do NOT write any new files. Do NOT print secret values to the user; only report which file was loaded and which keys were resolved. **Discovery order** (first hit per variable wins; shell env vars always override files): 1. **`--env-file `** override if supplied — always tried first. 2. **Current shell environment** (`os.environ`) — already-exported `DD_API_KEY` / `DD_APPLICATION_KEY` / `DD_APP_KEY` / `DD_SITE` take precedence over file values. If all required keys are already present, skip file loading entirely. 3. **`/.env`** — if the user previously ran the skill and dropped a `.env` next to past artifacts, prefer that. 4. **`/.env`** — the resolved `pyproject.toml` / `setup.cfg` / `setup.py` / `package.json` directory (or cwd if none). 5. **`/.env.local`** — git-ignored local override convention. 6. **`/.env`** — fallback if cwd differs from app-root. 7. **Parent walk**: from cwd, walk up directory by directory looking for `.env` until reaching `/` or the user's home directory. Stop at the first hit. 8. **`~/.datadog/credentials`** — Datadog's well-known per-user credentials file, if present. For each file checked, parse line-by-line: skip blanks / comment lines (`#`) / malformed lines (no `=`). Strip a leading `export ` if present (so `.envrc`-style files work). Split on the first `=`. Strip surrounding quotes on the value. Only set a variable that is not already in `os.environ` — never overwrite the shell. **Required keys**: `DD_API_KEY` AND (`DD_APPLICATION_KEY` OR `DD_APP_KEY`). `DD_SITE` is optional (defaults to `datadoghq.com`). Provider keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) are validated later in Phases 6/7 against the introspected task — not here. **If all required keys resolved** — record which file(s) were loaded and emit a single-line summary in the Precheck block. Continue. **If required keys NOT found after walking every location** — stop and prompt the user with a clear, actionable message: > "Datadog credentials were not found in your shell env or any discovered `.env` file. Two options before continuing: > - **Export in your shell**: `export DD_API_KEY=…` and `export DD_APPLICATION_KEY=…` (also `DD_SITE=…` if non-default), then re-invoke this skill. > - **Drop a `.env`** at `/.env` with `DD_API_KEY=…` and `DD_APPLICATION_KEY=…` on separate lines, then re-invoke. > > Make sure `.env` is in your `.gitignore` before committing." Do **not** offer to create the `.env` file for the user — secrets-on-disk decisions belong to the user, not the skill. Output the precheck summary, then start Phase 1: ``` ## Precheck - Backend: - ml_app `` has traces in : yes ( root spans found) - Project name: `` (created lazily on first LLMObs.enable() call in Phase 4) - Output dir: `` (created) - Credentials: (DD_API_KEY, DD_APPLICATION_KEY[, DD_SITE])" | "shell env + : keys resolved from both (shell overrode file for )" > - Stop-after: Starting Phase 1 of 6. ``` The exact list of keys in the credentials parenthetical reflects what was actually discovered (so the user can verify nothing surprising was loaded). Never print the values. --- ## Phase Template Every phase below uses this exact template. Do not deviate — the deterministic envelope is what makes the pipeline experience consistent across invocations. ``` ## Phase N of 6: **You are here.** Phase N of 6 — <one-line position summary>. **What this phase produces**: <Entity name> **What a <entity> is**: <2-3 sentence definition tailored to this phase> **Why it matters**: <1 sentence on why the user needs this> → Action: <invoke <sub-skill> | execute <script>> <full sub-skill output OR execution log reproduced here — do NOT summarize or truncate> <after the action completes successfully, Write the phase output to `<output-dir>/state/0N-<name>.{md,json}` per the State persistence contract — see "State persistence and entry/exit" section. This happens BEFORE the checkpoint so the file is on disk even if the user types `stop` next.> --- ### Checkpoint <N> <concise summary of what was produced, where it lives (path / Datadog URL), and any caveats> Before I continue to Phase N+1 (<next title>): - <2–3 phase-specific review prompts the user can answer> Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments. ``` **Never auto-advance.** Always pause at the checkpoint and wait for explicit user input. The whole point of this skill is determinism — that includes determinism over *when* the user moves on. If the current phase matches the value of `--stop-after`, replace the checkpoint prompt with a **Stop summary** (see "Stop-after handling" at the bottom of this file). --- ## Phase 1: Classify ml_app traces **Entity**: `ml_app`, `trace`, `span`. **Pedagogy banner** (use verbatim, adapted only to the actual ml_app name): > **What an ml_app is**: a logical LLM application — a name you tag spans with when instrumenting (`ml_app=<name>`). It groups all production traces and evaluator runs that belong to the same product surface. Every dataset, experiment, and evaluator you create later targets this scope. > > **What a trace is**: one end-to-end execution of your ml_app — typically the agent loop for a single user request, made up of one or more spans (LLM calls, tool calls, retrievals). > > **Why this phase matters**: before you root-cause failures (Phase 2), bootstrap evaluators (Phase 3), or curate a dataset (Phase 4), you want a quick read on what your app actually does in production and where its current failure modes are. Classification gives you that signal in one pass. **Trace pool preview (MANDATORY).** Before invoking the sub-skill, build and surface a Datadog Traces UI link that opens the **exact set of root spans Phase 4 will sample from**. This lets the user eyeball the pool, spot outliers, or adjust `--timeframe` before any classification work runs. The link must match the filter that `agent-observability-eval-bootstrap --emit-dataset` uses in Phase 4 (see `agent-observability/agent-observability-eval-bootstrap/SKILL.md` → Phase 3D → Sampling): `@ml_app:"<ml_app>" @status:ok`, root spans only. URL construction rules: 1. **Host**: prepend `app.` to `DD_SITE`. If `DD_SITE` is unset, default to `app.datadoghq.com`. - `datadoghq.com` → `app.datadoghq.com` - `datadoghq.eu` → `app.datadoghq.eu` - `us3.datadoghq.com` → `app.us3.datadoghq.com` - `us5.datadoghq.com` → `app.us5.datadoghq.com` - `ap1.datadoghq.com` → `app.ap1.datadoghq.com` - `ap2.datadoghq.com` → `app.ap2.datadoghq.com` - `datad0g.com` → `app.datad0g.com` (staging) 2. **Path**: `/llm/traces` (the Agent Observability Traces explorer). 3. **Query string**: URL-encode `@ml_app:"<ml_app>" @status:ok @parent_id:undefined` and bind to `query=`. 4. **Time window**: convert `--timeframe` to absolute epoch milliseconds (`now - <duration in ms>` for the start, `now` for the end) and bind to `start=` / `end=`. 5. **Optional**: append `&paused=true` so the UI does not live-stream the result on open. Surface the link immediately before the Action block: ``` **Trace pool being analyzed** ({timeframe}, filter `@ml_app:"<ml_app>" @status:ok`, root spans only): <full URL> Phase 1 will classify the first {min(20, trace-limit)} of these for orientation. Phase 4 will sample up to {trace-limit} from the same pool for the dataset. Click through if you want to see the pool before either runs, or adjust `--timeframe` and re-invoke if the window looks off. ``` **Action**: Follow the **`agent-observability-session-classify`** skill in **ml_app mode**, using: - `ml_app` = the provided ml_app - `timeframe` = the provided timeframe - `sample_limit` = `min(20, trace-limit)` — keep this fast; Phase 4 will do the bigger sample Run the complete ml_app mode workflow as defined in that skill (Steps M1 through M3). **Output the full classification output** (all compact per-unit blocks plus the final `# Session Classification Summary`) — do not summarize or truncate. Downstream Phase 2's RCA depends on the full text being in context. ### Checkpoint 1 After the `# Session Classification Summary` is output, present: ``` ## Phase 1 complete — you've seen what `<ml_app>` does in production [verdict distribution table from session-classify] [failure mode frequency table from session-classify] **Trace pool for Phase 4's dataset sample**: <full URL from the Trace pool preview above> (same filter / same timeframe — open it now if you want to scan for traces you'd rather exclude) Next up — Phase 2 will diagnose *why* the failing traces are failing. Before I continue: - Do these failure patterns look right? - Any traces you'd like to exclude from the dataset sample in Phase 4? (Paste trace IDs from the link above and I'll drop them.) - Any quality dimension you already know you want to measure later? Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments. ``` Wait for explicit user confirmation. If the user excludes specific traces, mark them as "excluded by user" — drop them from Phase 2's failure bucket and from Phase 4's sampling. Do NOT re-classify. --- ## Phase 2: Root cause analysis **Entity**: `failure mode`, `root cause`. **Pedagogy banner**: > **What a failure mode is**: a recurring pattern in *how* your app fails — e.g. "the model hallucinates citation URLs", "the agent forgets state across tool calls", "the retrieval returns irrelevant chunks". Each failure mode has one or more *root causes* (system prompt deficiency, tool gap, retrieval miss, etc.). > > **Why this phase matters**: an evaluator that scores generic "is this response good?" misses the specific things going wrong in your app. RCA lets Phase 3 propose evaluators that target your *actual* failure modes — sharper signal, fewer false alarms. **Action**: Follow the **`agent-observability-trace-rca`** skill. The `# Session Classification Summary` from Phase 1 is in context. The skill detects it automatically via its Phase 0 Step 0S check and enters the "from classifications" path — it extracts the failure bucket, presents the Classification Overview, and proceeds directly to Phase 2 (open coding) without running its own Phase 1 span search. Run the full workflow through Phase 6 (the compiled RCA report). **Output the full RCA report** — do not summarize. The full report must be in context for Phase 3's detection to work. ### Checkpoint 2 ``` ## Phase 2 complete — root causes identified [the Phase 6 RCA report is above] Next up — Phase 3 will bootstrap evaluators that target these failure modes. Before I continue: - Do these root causes look accurate? - Any failure modes to add, remove, or reframe? - Which root causes should the evaluators target? (Default: all of them.) Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments. ``` Wait for explicit user confirmation. If the user adjusts the taxonomy, incorporate the changes before continuing. --- ## Phase 3: Bootstrap evaluators **Entity**: `evaluator`, `LLM judge`. **Pedagogy banner**: > **What an evaluator is**: a function that grades one record's output. Returns `bool` / `float` / a structured `EvaluatorResult`. Two flavors: **code evaluators** (deterministic checks — JSON validity, regex match, length, custom Python) and **LLM-as-judge evaluators** (a model graded against a rubric — e.g. "is this response grounded in the retrieved documents?"). > > **Operating modes** (mutually exclusive; default is `--offline-evaluators`): > > - **`--offline-evaluators`** *(default)*: emit a Python SDK evaluator suite (BaseEvaluator / LLMJudge classes) that runs **inside an experiment against a dataset**. Use this when you'll run experiments locally. > - **`--online-evaluators`**: publish LLM-judge evaluators to Datadog as disabled drafts. They run **on production spans as they're emitted** once enabled. Use this when you want continuous evaluation in prod. > - **`--data-only`**: emit a local data blob only — no executable evaluator code or online publish. Prompts you to pick between a dataset for experiment use or a local analysis blob (see "Operating mode resolution" below). > > **Why this phase matters**: evaluators are the contract between "this output looks fine" and "this output meets our quality bar." The bootstrapped suite is grounded in the failure taxonomy from Phase 2 — sharper than generic evaluators you'd otherwise hand-write. **Action**: Follow the **`agent-observability-eval-bootstrap`** skill. The RCA report from Phase 2 is in context. The skill detects the `## Failure Taxonomy` heading automatically and enters its "from RCA" path in Phase 0. ### Operating mode resolution Before invoking the sub-skill, resolve the operating mode: 1. **`--offline-evaluators`** (default) → call `agent-observability-eval-bootstrap` with no mode flag (its `sdk_code` default). 2. **`--online-evaluators`** → call `agent-observability-eval-bootstrap --publish`. 3. **`--data-only`** → prompt the user via `AskUserQuestion`: > "You picked `--data-only` for Phase 3. What kind of local data blob do you want? > > - **Dataset for experiment use** — a `DatasetRecordRaw[]` JSON suitable for `LLMObs.create_dataset(records=...)`. Useful when you want to seed an experiment with the records this skill samples. (Internally calls `agent-observability-eval-bootstrap --emit-dataset <path>`.) > - **Local blob for analysis** — a framework-agnostic JSON evaluator spec describing what evaluators *would* be generated, without emitting Python code. Useful for inspecting evaluator coverage without running anything. (Internally calls `agent-observability-eval-bootstrap --data-only`.)" If the user picks "Dataset for experiment use" and the pipeline is running through Phase 4 (i.e. `--stop-after` is not `eval-bootstrap` or earlier), tell the user that Phase 4's sample step will be **skipped** in favor of this Phase 3 output — the dataset they produce here will be the one Phase 4 publishes. Pass `--evaluator-style` through unchanged. **The agent-observability-eval-bootstrap skill has its own mandatory proposal checkpoint** (the evaluator suite proposal before code generation). Honor it — do not skip or auto-confirm it. ### Checkpoint 3 ``` ## Phase 3 complete — evaluator suite ready - Mode: `<offline-evaluators | online-evaluators | data-only (dataset-for-experiment) | data-only (analysis-blob)>` - Output: `<path to .py / .json / dataset-record JSON / "published as drafts to Datadog">` - Evaluators emitted: <list of names> - Coverage: <one-liner: which failure-mode categories are now covered> Next up — Phase 4 will sample production traces into a dataset you can run experiments against (using these evaluators or the placeholders the experiment template ships with). If you only wanted evaluators (the classic eval-pipeline flow), this is the natural stopping point: re-invoke with `--stop-after eval-bootstrap` to formalize that as the exit. Before I continue: - Do the generated evaluators look right? - Any to drop or rename before they're referenced in the experiment? Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments. ``` Wait for explicit user confirmation. If `--stop-after eval-bootstrap` is set, this is where the pipeline ends — emit the Stop summary instead of the Checkpoint and exit. --- ## Phase 4: Create and publish dataset **Entity**: `dataset`, `dataset record`, published dataset (Datadog-side), `dataset_name`, version. **Pedagogy banner**: > **What a dataset is**: a named collection of records that an experiment runs against. Each record has `input_data` (what the task receives) and optionally `expected_output` (what you expect back). Datasets live in Datadog under your project and have a version — every push that changes records produces a new version. > > **What a record is**: a single `(input_data, expected_output)` pair, optionally with `metadata` and `tags`. One record = one experiment row. > > **What "publishing" means**: pushing the local records to Datadog so the dataset becomes addressable by name across all subsequent experiments. After publish, anyone in your org (with access) can pull it with `LLMObs.pull_dataset(dataset_name="…")` — including the experiment code we generate in Phase 5. > > **Why this phase matters**: experiments need a stable, addressable input set. Sampling production traces gives you a realistic starting dataset (the inputs your app actually sees), and publishing it under your project makes it the contract between dataset curation and the experiments that consume it. **This phase executes code on your machine** (the publish step writes to Datadog via `LLMObs.create_dataset(...)`). The dataset-create step is read-only; the publish step is the one that changes state. **Action — two sub-steps, no intermediate checkpoint:** **4a — Sample traces into a `DatasetRecordRaw[]` JSON.** Follow the **`agent-observability-eval-bootstrap`** skill in **`--emit-dataset` mode**: ``` /eval-bootstrap <ml_app> --timeframe <timeframe> --trace-limit <trace-limit> --emit-dataset <output-dir>/dataset_<ml_app>_<YYYYMMDD>.json ``` This mode samples root spans, extracts `(input_data, expected_output)` pairs, applies a PII scrub, and writes the JSON. **It does not propose or generate evaluators** — the dataset is the sole artifact. See `agent-observability/agent-observability-eval-bootstrap/SKILL.md` → Phase 3D for the full spec. If the user excluded specific traces in Checkpoint 1, pass that exclusion list along (sub-skill drops them during sampling — do NOT re-classify). Reproduce the sub-skill's `## Generated Dataset` summary verbatim. **Skip 4a if Phase 3 already produced a dataset.** When the user picked `--data-only` with the "Dataset for experiment use" sub-mode in Phase 3, the Phase 3 state file (`state/03-evaluators.json`) has `mode: data_only_dataset` and an `output_path` pointing at the dataset JSON. In that case, **skip sub-step 4a entirely** and feed the Phase 3 output to 4b's publish helper directly. Surface a one-line note: "Reusing dataset from Phase 3 (`<path>`); skipping fresh sampling." **4b — Publish to Datadog.** Immediately invoke the pre-shipped publish helper at `<this-skill-dir>/scripts/publish_dataset.py` via Bash. **Do not** inline the script content into this SKILL.md or re-write it from scratch — the helper is the source of truth for the publish flow (credential discovery, tag normalization, project creation, error handling). It accepts CLI args, so no placeholder substitution is needed. ```bash python <skill-dir>/scripts/publish_dataset.py \ --records <absolute path to JSON from 4a> \ --dataset-name <chosen dataset_name, default <ml_app>_seed_<YYYYMMDD>> \ --project-name <resolved project_name from Precheck> \ [--env-file <path>] # repeatable; takes precedence over auto-discovery ``` `<skill-dir>` resolves to wherever the skill is installed (e.g., `~/.claude/skills/agent-observability-eval-pipeline/`). The script prints either: - `Loaded credentials from: <file paths>` (if any `.env` files contributed values), then - `OK dataset_name=<name> record_count=<N> url=<url>` (on success), or - `ERROR: <message>` on stderr with a non-zero exit (auth, missing keys, ddtrace import failure, etc.). **Notes for the orchestrator:** - Before invoking the publish helper, do an import-availability precheck: `python -c "import ddtrace.llmobs"` via Bash. If it fails, stop and tell the user: > "`ddtrace` is not installed in the active Python environment. Run `pip install 'ddtrace>=4.7'` and re-invoke this skill (re-run from the top — Phases 1–3 outputs are idempotent)." - `LLMObs.enable(project_name=...)` inside the script is where the `--project-name` from the Precheck actually materializes — the Datadog project is created lazily on first call. - If the script prints a `WARNING:` line about tag normalization, surface it in Checkpoint 4 so the user knows their upstream dataset had malformed tags. - If the script prints `Loaded credentials from: ...`, include that file path in Checkpoint 4. - If the script exits non-zero with an auth error (401/403), surface the stderr and stop — do not retry. Tell the user the most likely cause is a stale `.env` value, and that `export DD_API_KEY=... DD_APPLICATION_KEY=...` in their shell takes precedence and can be used to override. - On success, capture the printed `dataset_name` and `url` and carry them into Phase 5. Write both to the Phase 4 state file (see State persistence section). **Why no intermediate checkpoint between 4a and 4b?** The local JSON is auto-extracted from traces and goes straight into `LLMObs.create_dataset(records=...)` — there is essentially no editable surface between the two steps in the common case. Users who want to inspect or edit the JSON before publish should run `eval-bootstrap --emit-dataset` standalone, edit the JSON, then re-enter this pipeline with `--start-at experiment --dataset-name <name>` once they've published manually. ### Checkpoint 4 ``` ## Phase 4 complete — dataset created and published - Local file: `<path>` (kept for inspection / re-publish) - Records emitted: <N> (skipped: <M> with no usable output) - PII redactions: <P> - Tag normalizations: <T> - Published as: `<dataset_name>` in project `<project_name>` <(created if it did not exist)> - Datadog UI: <url or "open Agent Observability → Datasets to confirm"> - Caveat: `expected_output` is the **current production behavior baseline**, not ground truth. Treat the dataset as a regression-style baseline before promoting it to a labelled gold set. Next up — Phase 5 will generate a Python experiment script that pulls `<dataset_name>` and runs your task code (auto-discovered) against it. Before I continue: - Confirm you can see the dataset in the Datadog UI (Agent Observability → Datasets → search `<dataset_name>`)? - Any second thoughts on the records (we can re-emit and re-publish before generating the experiment)? Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments. ``` Wait for confirmation. --- ## Phase 5: Generate and run experiment **Entity**: `experiment`, `task` function, `evaluator`, experiment run, `experiment.url`, metric stream. **Pedagogy banner**: > **What an experiment is**: a programmatic harness that, for each record in a dataset, calls a `task` function (your code under test — typically an LLM call), then runs one or more `evaluators` against the task's output. Datadog collects all those results into a single experiment view you can compare across runs. > > **What a task function is**: a Python callable that receives one record's `input_data` and a `config` dict, and returns whatever your app would have returned for that input. **The sub-skill introspects your project to find this function automatically** — no `# TODO(user)` placeholder unless nothing was found. > > **What an evaluator is**: covered in Phase 3 above. The generated experiment ships placeholder evaluators by default; if you ran Phase 3 with `--evaluator-style remote`, you can wire those names in here. > > **What `experiment.url` is**: the deep link to the run in the Datadog Experiments UI. Phase 6 uses this to analyze results. > > **Why this phase matters**: this is where your code actually executes against the dataset and produces measurements. Generation is the cheap part; running is what costs provider tokens and produces signal. **This phase executes code on your machine** (Python file is run end-to-end after an in-phase review beat). **Action — three sub-steps with an in-phase review beat between codegen and run:** **5a — Generate the experiment file.** Follow the **`agent-observability-experiment-bootstrap`** skill: ``` /agent-observability-experiment-bootstrap \ --dataset-name <dataset_name> \ --project-name <project_name> \ --format <format> \ --evaluator-style <evaluator-style> \ --app-root <app-root> \ --env-file <env-file if set> \ --output <output-dir>/experiment_<ml_app>_<YYYYMMDD>.<py|ipynb> ``` Reproduce the sub-skill's full output (including the generated SDK calls summary, the "Task function source" block, the credential discovery section, and the Next steps block) verbatim. Do not summarize. The "Task function source" block tells the user which `module:function` was auto-wired — that's load-bearing for the next sub-step. **5b — In-phase review beat (MANDATORY).** After the sub-skill output, **pause** with a brief inline prompt (this is NOT a full checkpoint with a new banner — it's a single-line review beat *inside* Phase 5): ``` ## Phase 5 — review the generated experiment before running - File: `<path>` - Wired to dataset: `<dataset_name>` (pulled at runtime via LLMObs.pull_dataset) - Task function source: <line lifted from the sub-skill output — module:function, or "placeholder fallback"> - Evaluators: <2–3 evaluator names> **Open the file and check three things before I run it:** 1. The wired `task_fn` (section 4) — confirm the sub-skill picked the right entry point. If it picked a helper or deprecated path, edit the import to point at the right function. If it fell back to a placeholder, replace it with a real call. 2. The placeholder evaluators (section 5) — these are starting points. If Phase 3 produced online evaluators via `--publish`, swap one of the placeholders for a `RemoteEvaluator(eval_name="...")`. 3. The `experiment.run(jobs=<N>)` parallelism (section 7) — defaults to 10; lower it if you're worried about rate limits. Type **`run`** to execute the file as-is, **`edit`** to pause here so you can edit and re-run this skill, or **`stop`** to exit cleanly. State for Phase 4 and earlier is preserved. ``` Wait for the user's reply. - If `run`: proceed to 5c. - If `edit`: tell the user the file is at `<path>` and they can re-invoke this skill with `--start-at experiment` once they're satisfied. End the run cleanly. The generated file path is written to the state file so `--start-at experiment` can resume codegen-free (just re-run sub-step 5c). - If `stop`: emit the Stop summary and end. State for completed phases (including the generated experiment file) is preserved on disk. - Anything else: treat as adjustment / question. Reason about it, answer, then re-show the review prompt. **5c — Execute the file.** - For `--format p …(truncated)