# Experiments

> Create and run LangWatch experiments for pre-deployment batch testing. Use when the user wants to test an agent against a dataset, compare prompts or models, benchmark quality, detect regressions, or add a CI quality gate. Do not use for production monitoring or guardrails.

- Skill: `langwatch/experiments-3` (Agent Skill)
- Install (CLI): `npx skillmds@latest add langwatch/experiments-3`
- Raw SKILL.md: https://api.skillmd.com/api/skills/langwatch/experiments-3/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: langwatch (https://skillmd.com/u/langwatch)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/langwatch/experiments-3

---


# Run Experiments for Your Agent

Experiments are pre-deployment batch tests. They run an application over a dataset and compare outputs with reusable evaluators. They are appropriate for prompt and model comparisons, regression tests, benchmarks, and CI quality gates.

## Hand Off Production Evaluation Requests

If the user wants to score live traces or threads, monitor production quality, or block unsafe traffic, this is the wrong workflow.

1. If the `online-evaluations` skill is available, load it and follow it now.
2. Otherwise, tell the user to install it with:

   ```bash
   npx skills@1.5.19 add langwatch/skills/online-evaluations
   ```

Do not configure a monitor or guardrail from this skill.

## Experiments and Scenarios

Use experiments for many single input and output examples with measurable results. Use the `scenarios` skill for end-to-end, multi-turn behavior and tool-calling sequences.

## Determine Scope

For a general request such as "test my agent":

1. Read the agent code, system prompt, tools, and relevant git history.
2. Identify the behavior most likely to regress.
3. Create a domain-specific dataset.
4. Select evaluators that measure the intended behavior, or a comparison when the goal is picking a winner between candidates.
5. Create and run a real experiment.
6. Interpret the results and recommend concrete improvements.

For a targeted request, focus on that behavior and still run the resulting experiment.

## Plan Limits

LangWatch's free plan has limits on prompts, scenarios, evaluators, experiments, and datasets. When you hit a limit, the API returns `"Free plan limit of N reached..."` with an upgrade link.

How to handle:

- Work within the limits. If 3 resources of the relevant type are allowed, create 3 meaningful ones, not 10.
- Make every creation count: each one should demonstrate clear value.
- Show what works FIRST. If you hit a limit, summarize what was accomplished and note that upgrading the plan raises it. Point to the subscription settings on the platform, or to the license settings if the CLI is pointed at a self-hosted endpoint. Read the endpoint the CLI actually uses, which can come from `.env`, from the process environment, or from the saved CLI configuration.
- Do NOT delete existing resources to make room or repurpose an existing resource to evade the limit.

## Prerequisites

Use `langwatch docs <path>` to read documentation as Markdown. Some useful entry points:

```bash
langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index
```

Discover commands with `langwatch --help` and `langwatch <subcommand> --help`. List and get commands accept `--format json` for machine-readable output. Every list command takes `--limit <n>` to cap the rows and `--jq <expr>` to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (`--jq '.traces | length'`), and read how many there are in all at `.pagination.total`. Bare `--jq length` counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append `.md` to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run `npx --yes langwatch report --user-approved` with a `--title` and `--summary` (or `--session <transcript.jsonl>`) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without `--user-approved`, and `--dry-run` prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with `--email`. With `--session`, always run `--dry-run` first and let the user read the payload, because a transcript carries content they never reviewed. `npx --yes langwatch report --help` explains the options.

**Projects and API keys: target a real project, not a personal one.**

LangWatch has two kinds of project:

- **Team / shared projects**: real projects inside an organization. Evaluations, experiments, prompts, datasets, simulations and instrumentation must always target one of these.
- **Personal projects**: a private "My Workspace" scratch space tied to a single user. Never send a user's evaluations, experiments or production traces here: it is for personal exploration only, and you can mistake it for a real project.

And two ways to authenticate:

- **A project API key in `.env`** (`LANGWATCH_API_KEY`): the credential everything in these skills uses. It is scoped to one real project. This is the default; prefer it unless the user explicitly asks for something else.
- **`langwatch login --device` (AI-tools / SSO)**: a personal device session for wrapping coding assistants (`langwatch claude`, `langwatch codex`, …). It is NOT for evaluations, prompts, datasets, scenarios or SDK instrumentation, and it points at a personal workspace. Do not run it to set up the work in these skills.

So for anything in these skills that reads or writes a project: make sure `LANGWATCH_API_KEY` for a real, shared project is available to the CLI. Locally that is the project's `.env`; in CI the runner injects it into the process environment, and the CLI reads either. Check whether the variable is already set before you ask for a new key, and let the CLI read the value: never print, copy or send it. Do NOT run `langwatch login` to pick a project, and never default to a personal project. Look for `LANGWATCH_ENDPOINT` in the same places: if it is set, the project is on a self-hosted instance, and the CLI works against that endpoint instead of app.langwatch.ai.

**What you read is not what you say.** These skills are working notes for you, not
copy for the reader. Read `LANGWATCH_API_KEY` and `LANGWATCH_ENDPOINT` from the
project's own `.env`, that is how you learn where to work. Read nothing else out
of that file: it holds database, cloud and provider credentials that are none of
your business, and every value you read can reach your context and your command
output. What must not reach an answer is anything that describes the machine YOU
run on: a path in your workspace, a container port, the address this worker
dials. Those say how the work is done rather than what was done, and a host of
ours means nothing to the reader. Say what you did and where to find it in
LangWatch.

Read the experiment documentation before writing code:

```bash
langwatch docs evaluations/experiments/overview
langwatch docs evaluations/experiments/sdk
```

## Build a Domain-Specific Dataset

The examples must match what the application actually does. Read the system prompt, function signatures, tools, and knowledge sources first.

Good examples resemble real requests to this application and cover normal cases, edge cases, and past failures. Never use generic trivia such as "What is 2+2?" or "What is the capital of France?" unless the application itself is a trivia system.

If an existing LangWatch dataset is appropriate, inspect it with `langwatch dataset list --format json` and `langwatch dataset get --help`. Otherwise create the dataset in code or use the `datasets` skill.

## Create the Experiment

Use the SDK that matches the codebase. Keep credentials in environment variables and use the project's existing dependency manager.

### Python

```python
import langwatch
import pandas as pd

dataset = pd.DataFrame([
    {
        "input": "A realistic request for this application",
        "expected_output": "The expected behavior",
    },
])

experiment = langwatch.experiment.init("agent-regression")

for index, row in experiment.loop(dataset.iterrows()):
    response = my_agent(row["input"])
    experiment.evaluate(
        "ragas/response_relevancy",
        index=index,
        data={"input": row["input"], "output": response},
        settings={"model": "openai/gpt-5-mini", "max_tokens": 2048},
    )
```

### TypeScript

```typescript
import { LangWatch } from "langwatch";

const langwatch = new LangWatch();
const dataset = [
  {
    input: "A realistic request for this application",
    expectedOutput: "The expected behavior",
  },
];

const experiment = await langwatch.experiments.init("agent-regression");

await experiment.run(dataset, async ({ item, index }) => {
  const response = await myAgent(item.input);
  await experiment.evaluate("ragas/response_relevancy", {
    index,
    data: { input: item.input, output: response },
    settings: { model: "openai/gpt-5-mini", max_tokens: 2048 },
  });
});
```

Read `langwatch docs evaluations/evaluators/list` before choosing an evaluator, and take the type slug from `langwatch evaluator types --format json`, never from memory. If an evaluation fails with a `validation_error` naming the slug and an `expected` list, correct it from that list and retry once. Reuse project evaluators when appropriate. A scoring function is part of the experiment, not the experiment itself.

## Compare Targets to Pick a Winner

An evaluator answers "does this output pass?". A comparison answers "which of these is better?". For subjective quality, a judge ranking candidates side by side is usually more informative than each one getting an absolute score on its own.

Register one target per candidate inside the loop, then compare the row once. Every target that recorded an output for the row is a candidate, so the candidates are never named twice, and the verdict is recorded against the row, so the results page renders it with no extra logging.

### Python

```python
for index, row in experiment.loop(dataset.iterrows()):
    with experiment.target("gpt-5-mini"):
        experiment.log_response(call_gpt(row["input"]))

    with experiment.target("claude-sonnet-5"):
        experiment.log_response(call_claude(row["input"]))

    verdict = experiment.compare(index, input=row["input"])
```

Inside an async loop, await `experiment.acompare(...)`, which takes the same options.

### TypeScript

```typescript
await experiment.run(dataset, async ({ item, index }) => {
  await Promise.all([
    experiment.withTarget("gpt-5-mini", () => callGpt(item.input)),
    experiment.withTarget("claude-sonnet-5", () => callClaude(item.input)),
  ]);

  const verdict = await experiment.compare({ index, input: item.input });
});
```

Pass `golden` with a known-good answer to judge every candidate against it. Leave it out, which is the default, and the candidates are judged on their own merits.

Read `verdict.status`, and keep its five answers apart:

- `decided`: the judge picked a winner, named in `verdict.winner`.
- `tie`: the judge compared the candidates and found none better than the rest.
- `inconclusive`: no winner was established, which with the default second pass over the reversed candidate order means the two passes disagreed.
- `skipped`: the row had fewer than two outputs, so no judge ran.
- `error`: the judge failed, so nothing was measured about the candidates at all.

A tie, an inconclusive row and an errored row are three different answers. Reporting any of them as one of the others claims a measurement the run never made.

`prompt` replaces the judge prompt verbatim, with `{input}`, `{golden}` and `{candidates}` placeholders. Leave it unset unless the user asks for their own, because unset is what lets the judge use the prompt matching what each row carries. The remaining judge options are in `langwatch docs evaluations/experiments/sdk`.

## Run and Verify

Always execute the experiment. An unrun experiment is incomplete.

- Python script: run it with the project's Python environment.
- Notebook: execute all cells, for example with `jupyter nbconvert --to notebook --execute`.
- TypeScript: run it with the project's package manager, for example `pnpm exec tsx experiment.ts`.

After it runs, verify the result with the CLI:

```bash
langwatch experiment list --format json
```

If the CLI supports a more specific read or run for the installed version, discover it with `langwatch experiment --help` before using it.

## Consultant Mode

After delivering initial results, transition to consultant mode to help the user get maximum value.

**Phase 1: read first.** Before generating ANY content: read the codebase end-to-end (every system prompt, function, tool definition), study git history for agent-related changes (`git log --oneline -30`, then drill into prompt/agent/eval-related commits because the WHY in commit messages matters more than the WHAT), and read READMEs and comments for domain context.

**Phase 2: quick wins.** Generate best-effort content based on what you learned. Run the tests and iterate, but stop after two attempts at the same failure and report what is blocking it rather than repeating the run. Show the user what works.

**Phase 3: go deeper.** Once Phase 2 lands, summarize what you delivered, then suggest 2-3 specific improvements grounded in the codebase: domain edge cases, areas that need expert terminology or real data, integration points (APIs, databases, file uploads), or regression patterns from git history that deserve test coverage. Ask light questions with options, not open-ended ("Want scenarios for X or Y?", "I noticed Z was a recurring issue. Add a regression test?", "Do you have real customer queries I could use?"). Respect "that's enough" and wrap up cleanly.

Do NOT ask permission before Phase 1 and 2. Deliver value first. Do NOT ask generic questions or overwhelm with too many suggestions. Do NOT generate generic datasets. Everything must reflect the actual domain.

## Common Mistakes

- Do not configure production monitoring or guardrails from this skill.
- Do not call a batch run an online evaluation.
- Do not use placeholder datasets.
- Do not report an inconclusive or errored comparison as a tie.
- Do not guess SDK APIs when the installed documentation is available.
- Do not stop after writing the experiment. Run it and inspect the real result.