Arize Evaluator

Arize evaluator

Design evaluator prompts, create or update Arize evaluators, map task columns, run evaluations, and troubleshoot ax failures without fabricating results or reading local credential files.

When to invoke

• "Create an Arize LLM judge evaluator."
• "Run a hallucination eval on spans."
• "Score this experiment for correctness."
• "Fix evaluator column mapping."
• "Set up continuous monitoring with trigger-run."

Prerequisites and context

Proceed directly with the needed ax command; do not check versions, environment variables, or profiles up front. If an ax command fails, react to the error.

Security rule: never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys. If credentials are unavailable through those channels, ask the user.

SPACE, --space, and ARIZE_SPACE accept either a space name such as my-workspace or a base64 space ID such as U3BhY2U6...; find values with ax spaces list.

Evaluator model

Evaluators are versioned. Every prompt or model change creates a new immutable version, and the newest version is active.

Task model and granularity

For trace granularity, values are grouped by context.trace_id and comma-joined, with each value truncated to 100K characters. For session granularity, trace-level grouping happens first, then traces are ordered by start_time and grouped by attributes.session.id; session-level values are capped at 100K characters total. At session granularity, {conversation} renders as a JSON array of {input, output} turns from attributes.input.value / attributes.llm.input_messages and attributes.output.value / attributes.llm.output_messages. At span or trace granularity, {conversation} is resolved like any other mapped variable.

Multi-evaluator tasks may contain different granularities. Runtime uses the highest granularity, session > trace > span, and splits into one child run per evaluator. Per-evaluator query_filter in the task evaluators JSON narrows included spans, such as only tool-call spans within a session.

Template design rules

Limits

Never fabricate evaluation results. If a task fails, is cancelled, or produces no scores, report the failure and explain what happened. Do not perform a manual evaluation, invent quality scores, estimate percentages, or present agent analysis as Arize evaluation output. Recommend fixing the issue and retrying, trying the Arize UI, verifying credentials with ax ai-integrations list, or contacting https://arize.com/support.

Progressive disclosure and bundled resources

• references/evaluator-crud-workflows.md: GraphQL CRUD calls, project and experiment evaluator setup, trigger-run operations, task management, column mapping, continuous monitoring, and troubleshooting.
• references/ax-setup.md: ax install and version remediation.
• references/ax-profiles.md: profile creation and update workflow.

Related primitives

Output template

### Arize evaluator result

**Status:** created | updated | run complete | failed | blocked
**Space:** `<SPACE or ARIZE_SPACE>`
**Evaluator:** `<name/id/version>`
**Task:** `<task id or n/a>`
**Granularity:** span | trace | session

| Step | Command | Result |
| --- | --- | --- |
| <step> | `ax ...` | <output summary> |

### Column mappings
| Template variable | Data field |
| --- | --- |
| `{input}` | `<field path>` |
| `{output}` | `<field path>` |

### Evaluation results
- <actual Arize result, task status, or failure reason; never fabricated>

Quality gate

• The task proceeded with the needed ax command before speculative prechecks.
• SPACE / --space / ARIZE_SPACE was resolved by name or base64 ID.
• Credentials were checked only through ax profiles or ax ai-integrations; no .env files were read.
• Template labels exactly match --classification-choices.
• --invocation-params '{"temperature": 0}' is used unless a different temperature is justified.
• Column mappings connect every template variable to real span, trace, session, project, dataset, or experiment fields.
• Results reported are actual Arize outputs; failures are not converted into manual scores.

References

• Arize admin API keys
• Arize support