Arize Experiment

Arize Experiment Skill

Concepts

• Experiment = a named evaluation run against a specific dataset version, containing one run per example
• Experiment Run = the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata
• Dataset = a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version
• Evaluation = a named metric attached to a run (e.g., correctness, relevance), with optional label, score, and explanation

The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs.

Prerequisites

Three things are needed: ax CLI, an API key (env var or profile), and a space ID. A project name is also needed but usually comes from the user's message.

Install ax

If ax is not installed, not on PATH, or below version 0.3.0, see ax-setup.md.

Verify environment

Run a quick check for credentials:

macOS/Linux (bash):

ax --version && echo "--- env ---" && if [ -n "$ARIZE_API_KEY" ]; then echo "ARIZE_API_KEY: (set)"; else echo "ARIZE_API_KEY: (not set)"; fi && echo "ARIZE_SPACE_ID: ${ARIZE_SPACE_ID:-(not set)}" && echo "--- profiles ---" && ax profiles show 2>&1

Windows (PowerShell):

ax --version; Write-Host "--- env ---"; Write-Host "ARIZE_API_KEY: $(if ($env:ARIZE_API_KEY) { '(set)' } else { '(not set)' })"; Write-Host "ARIZE_SPACE_ID: $env:ARIZE_SPACE_ID"; Write-Host "--- profiles ---"; ax profiles show 2>&1

Read the output and proceed immediately if either the env var or the profile has an API key. Only ask the user if both are missing. Resolve failures:

• No API key in env and no profile → AskQuestion: "Arize API key (https://app.arize.com/admin > API Keys)"
• Space ID unknown → run ax spaces list -o json to list all accessible spaces and pick the right one, or AskQuestion if the user prefers to provide it directly
• Project unclear → ask, or run ax projects list -o json --limit 100 and present as selectable options

Space ID and Project

Both are needed for most commands. Resolve each:

1. User provides it in the conversation -- note that space ID and project are resolved via the API key profile, not CLI flags.
2. Env var is set (ARIZE_SPACE_ID, ARIZE_DEFAULT_PROJECT) -- use silently.
3. If missing, AskQuestion once. Tell the user:
   • Run ax spaces list -o json to discover your space ID, or find it in the Arize URL: /spaces/{SPACE_ID}/...
   • Project is the project name as shown in the Arize UI.
   • For convenience, recommend setting env vars so they don't get asked again: export ARIZE_SPACE_ID="U3BhY2U6..." and export ARIZE_DEFAULT_PROJECT="my-project"

Prefer asking the user over searching or iterating through projects and API keys. If you get a 401 Unauthorized, tell the user their API key may not have access to that space and ask them to verify.

List Experiments: ax experiments list

Browse experiments, optionally filtered by dataset. Output goes to stdout.

ax experiments list
ax experiments list --dataset-id DATASET_ID --limit 20
ax experiments list --cursor CURSOR_TOKEN
ax experiments list -o json

Flags

Get Experiment: ax experiments get

Quick metadata lookup -- returns experiment name, linked dataset/version, and timestamps.

ax experiments get EXPERIMENT_ID
ax experiments get EXPERIMENT_ID -o json

Flags

Response fields

Export Experiment: ax experiments export

Download all runs to a file. By default uses the REST API; pass --all to use Arrow Flight for bulk transfer.

ax experiments export EXPERIMENT_ID
# -> experiment_abc123_20260305_141500/runs.json

ax experiments export EXPERIMENT_ID --all
ax experiments export EXPERIMENT_ID --output-dir ./results
ax experiments export EXPERIMENT_ID --stdout
ax experiments export EXPERIMENT_ID --stdout | jq '.[0]'

Flags

REST vs Flight (--all)

• REST (default): Lower friction -- no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page.
• Flight (--all): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port (flight.arize.com:443) which some corporate networks may block.

Agent auto-escalation rule: If a REST export returns exactly 500 runs, the result is likely truncated. Re-run with --all to get the full dataset.

Output is a JSON array of run objects:

[
  {
    "id": "run_001",
    "example_id": "ex_001",
    "output": "The answer is 4.",
    "evaluations": {
      "correctness": { "label": "correct", "score": 1.0 },
      "relevance": { "score": 0.95, "explanation": "Directly answers the question" }
    },
    "metadata": { "model": "gpt-4o", "latency_ms": 1234 }
  }
]

Create Experiment: ax experiments create

Create a new experiment with runs from a data file.

ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json
ax experiments create --name "claude-test" --dataset-id DATASET_ID --file runs.csv

Flags

Required columns in the runs file

Additional columns are passed through as additionalProperties on the run.

Delete Experiment: ax experiments delete

ax experiments delete EXPERIMENT_ID
ax experiments delete EXPERIMENT_ID --force   # skip confirmation prompt

Flags

Experiment Run Schema

Each run corresponds to one dataset example:

{
  "example_id": "required -- links to dataset example",
  "output": "required -- the model/system output for this example",
  "evaluations": {
    "metric_name": {
      "label": "optional string label (e.g., 'correct', 'incorrect')",
      "score": "optional numeric score (e.g., 0.95)",
      "explanation": "optional freeform text"
    }
  },
  "metadata": {
    "model": "gpt-4o",
    "temperature": 0.7,
    "latency_ms": 1234
  }
}

Evaluation fields

At least one of label, score, or explanation should be present per evaluation.

Workflows

Run an experiment against a dataset

1. Find or create a dataset:

   ax datasets list
   ax datasets export DATASET_ID --stdout | jq 'length'
   

2. Export the dataset examples:

   ax datasets export DATASET_ID
   

3. Process each example through your system, collecting outputs and evaluations
4. Build a runs file (JSON array) with example_id, output, and optional evaluations:

   [
     {"example_id": "ex_001", "output": "4", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}},
     {"example_id": "ex_002", "output": "Paris", "evaluations": {"correctness": {"label": "correct", "score": 1.0}}}
   ]
   

5. Create the experiment:

   ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json
   

6. Verify: ax experiments get EXPERIMENT_ID

Compare two experiments

1. Export both experiments:

   ax experiments export EXPERIMENT_ID_A --stdout > a.json
   ax experiments export EXPERIMENT_ID_B --stdout > b.json
   

2. Compare evaluation scores by example_id:

   # Average correctness score for experiment A
   jq '[.[] | .evaluations.correctness.score] | add / length' a.json
   
   # Same for experiment B
   jq '[.[] | .evaluations.correctness.score] | add / length' b.json
   

3. Find examples where results differ:

   jq -s '.[0] as $a | .[1][] | . as $run |
     {
       example_id: $run.example_id,
       b_score: $run.evaluations.correctness.score,
       a_score: ($a[] | select(.example_id == $run.example_id) | .evaluations.correctness.score)
     }' a.json b.json
   

4. Score distribution per evaluator (pass/fail/partial counts):

   # Count by label for experiment A
   jq '[.[] | .evaluations.correctness.label] | group_by(.) | map({label: .[0], count: length})' a.json
   

5. Find regressions (examples that passed in A but fail in B):

   jq -s '
     [.[0][] | select(.evaluations.correctness.label == "correct")] as $passed_a |
     [.[1][] | select(.evaluations.correctness.label != "correct") |
       select(.example_id as $id | $passed_a | any(.example_id == $id))
     ]
   ' a.json b.json
   

Statistical significance note: Score comparisons are most reliable with ≥ 30 examples per evaluator. With fewer examples, treat the delta as directional only — a 5% difference on n=10 may be noise. Report sample size alongside scores: jq 'length' a.json.

Download experiment results for analysis

1. ax experiments list --dataset-id DATASET_ID -- find experiments
2. ax experiments export EXPERIMENT_ID -- download to file
3. Parse: jq '.[] | {example_id, score: .evaluations.correctness.score}' experiment_*/runs.json

Pipe export to other tools

# Count runs
ax experiments export EXPERIMENT_ID --stdout | jq 'length'

# Extract all outputs
ax experiments export EXPERIMENT_ID --stdout | jq '.[].output'

# Get runs with low scores
ax experiments export EXPERIMENT_ID --stdout | jq '[.[] | select(.evaluations.correctness.score < 0.5)]'

# Convert to CSV
ax experiments export EXPERIMENT_ID --stdout | jq -r '.[] | [.example_id, .output, .evaluations.correctness.score] | @csv'

Related Skills

• arize-dataset: Create or export the dataset this experiment runs against → use arize-dataset first
• arize-prompt-optimization: Use experiment results to improve prompts → next step is arize-prompt-optimization
• arize-trace: Inspect individual span traces for failing experiment runs → use arize-trace
• arize-link: Generate clickable UI links to traces from experiment runs → use arize-link

Troubleshooting

Save Credentials for Future Use

At the end of the session, if the user manually provided any of the following during this conversation (via AskQuestion response, pasted text, or inline values) and those values were NOT already loaded from a saved profile or environment variable, offer to save them for future use.

Skip this entirely if:

• The API key was already loaded from an existing profile or ARIZE_API_KEY env var
• The space ID was already set via ARIZE_SPACE_ID env var
• The user only used base64 project IDs (no space ID was needed)

How to offer: Use AskQuestion: "Would you like to save your Arize credentials so you don't have to enter them next time?" with options "Yes, save them" / "No thanks".

If the user says yes:

1. API key — Check if ~/.arize/config.toml exists. If it does, read it and update the [auth] section. If not, create it with this minimal content:

   [profile]
   name = "default"
   
   [auth]
   api_key = "THE_API_KEY"
   
   [output]
   format = "table"
   

   Verify with: ax profiles show

2. Space ID — Persist the space ID as an environment variable:

   macOS/Linux — Detect the user's shell config file (~/.zshrc for zsh, ~/.bashrc for bash). Append:

   export ARIZE_SPACE_ID="THE_SPACE_ID"
   

   Tell the user to run source ~/.zshrc (or restart their terminal) for it to take effect.

   Windows (PowerShell) — Set a persistent user environment variable:

   [System.Environment]::SetEnvironmentVariable('ARIZE_SPACE_ID', 'THE_SPACE_ID', 'User')
   

   Tell the user to restart their terminal for it to take effect.