Arize Annotation

Arize annotation

Create and manage Arize annotation configs and queues, then transform human review labels into CLI queue annotations or Python SDK span updates.

When to invoke

• "Create an Arize annotation config for correctness labels."
• "Set up an annotation queue for human review."
• "Bulk annotate spans with update_annotations."
• "Annotate a queue record with a score and reviewer notes."
• "Troubleshoot ax annotation-configs or annotation-queues."

Prerequisites and context

• Proceed directly with the requested ax command. Do not check versions, environment variables, or profiles upfront.
• SPACE means a space name such as my-workspace or a base64 space ID such as U3BhY2U6...; --space and ARIZE_SPACE accept either. Find spaces with ax spaces list.
• Use ax profiles show only after an authorization or profile error.
• Never read .env files or search the filesystem for credentials. Use ax profiles for Arize credentials and ax ai-integrations for LLM provider keys.
• If the user needs an API key, direct them to https://app.arize.com/admin > API Keys.

Progressive disclosure and bundled resources

• references/ax-setup.md: use when ax is missing or the installed version is incompatible.
• references/ax-profiles.md: use when profile creation, update, or credential persistence is required.

Read these references only when the matching error or setup request occurs.

Annotation concepts

This skill covers ax annotation-configs, ax annotation-queues, CRUD operations, create/update flows, and bulk span writes with spans.update_annotations. Config types are categorical, continuous, and freeform; freeform captures open-ended text, continuous uses Min/Max score bounds, and categorical values may use {"label": str, "score": number} pairs. Common config names include Correctness and Helpfulness; use maximize when higher is better.

Always create or confirm the relevant annotation config before expecting labels to persist on spans, dataset examples, experiment-related records, or annotation queue items.

Annotation config commands

Create a categorical config:

ax annotation-configs create \
  --name "Correctness" \
  --space SPACE \
  --type categorical \
  --value correct \
  --value incorrect \
  --optimization-direction maximize

Create a continuous config:

ax annotation-configs create \
  --name "Quality Score" \
  --space SPACE \
  --type continuous \
  --min-score 0 \
  --max-score 10 \
  --optimization-direction maximize

Create a freeform config:

ax annotation-configs create \
  --name "Reviewer Notes" \
  --space SPACE \
  --type freeform

Deletion is irreversible. Any annotation queue associations to the deleted config are also removed in the product; queues may remain and need association repair in the Arize UI.

Annotation queue commands

Create a queue with at least one --annotation-config-id:

ax annotation-queues create \
  --name "Correctness Review" \
  --space SPACE \
  --annotation-config-id CONFIG_ID \
  --annotator-email reviewer@example.com \
  --instructions "Label each response as correct or incorrect." \
  --assignment-method all

Use --assignment-method random instead of all when each item should go to one reviewer. Repeat --annotation-config-id and --annotator-email to attach multiple configs or reviewers.

List flags fully replace existing values when provided. Pass all desired values, not only the new ones:

ax annotation-queues update NAME_OR_ID --space SPACE \
  --annotation-config-id CONFIG_ID_A \
  --annotation-config-id CONFIG_ID_B

Submit annotations to queue records. Annotations are upserted by config name; supply at least one of --score, --label, or --text:

ax annotation-queues annotate-record NAME_OR_ID RECORD_ID \
  --annotation-name "Correctness" \
  --label "correct" \
  --space SPACE

ax annotation-queues annotate-record NAME_OR_ID RECORD_ID \
  --annotation-name "Quality Score" \
  --score 8.5 \
  --text "Response was accurate but slightly verbose." \
  --space SPACE

Python SDK span annotations

Use the Python SDK to bulk-apply annotations to project spans when labels already exist in a review export or external labeling tool.

import os
import pandas as pd
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])

annotations_df = pd.DataFrame([
    {
        "context.span_id": "span_001",
        "annotation.Correctness.label": "correct",
        "annotation.Correctness.updated_by": "reviewer@example.com",
    },
    {
        "context.span_id": "span_002",
        "annotation.Correctness.label": "incorrect",
        "annotation.Correctness.updated_by": "reviewer@example.com",
    },
])

response = client.spans.update_annotations(
    space_id=os.environ["ARIZE_SPACE"],
    project_name="your-project",
    dataframe=annotations_df,
    validate=True,
)

DataFrame schema:

Annotations apply only to spans within 31 days prior to submission.

Troubleshooting

Related primitives

Gotchas

• List updates fully replace values: queue --annotation-config-id and --annotator-email updates must include every desired value, not just the delta.
• Annotation configs are the label schema: create configs before applying labels to spans, queues, dataset examples, or experiment records.
• and/or surfaces are real: human annotations may apply through the UI and/or the SDK depending on the target surface.

Output template

## Arize annotation result

**Status:** completed | blocked | needs input
**Space:** <SPACE or ARIZE_SPACE value used>
**Target:** annotation config | annotation queue | queue record | project spans

| Action | Command or API | Result |
| --- | --- | --- |
| <action> | `<ax command or ArizeClient.spans.update_annotations>` | <created/updated/listed/deleted/annotated> |

### Identifiers
- Config: <NAME_OR_ID or CONFIG_ID>
- Queue: <NAME_OR_ID>
- Record: <RECORD_ID>
- Project: <project_name>

### Validation
- <command or SDK validation>: pass/fail with evidence

Quality gate

• The command uses --space SPACE or the SDK uses ARIZE_SPACE where required.
• Annotation configs exist before queue creation or span annotation.
• Queue updates that pass list flags include the full desired replacement list.
• annotate-record supplies at least one of --score, --label, or --text.
• SDK span annotation uses context.span_id and at least one annotation.<name>.label or annotation.<name>.score column.
• No .env file or filesystem credential search was performed.
• Setup/profile references were read only when the matching error or task required them.