Arize Phoenix

Arize Phoenix

Phoenix is an open-source AI observability platform built on OpenTelemetry that helps developers understand, debug, and improve AI applications. It provides comprehensive tracing, evaluation, prompt engineering, and experimentation capabilities for LLM-based systems. Phoenix captures detailed execution information from AI applications, measures output quality with evaluators, enables systematic prompt iteration, and supports data-driven experimentation to optimize AI performance.

When to Use This Skill

• Debugging AI application failures by inspecting LLM calls, tool executions, and retrieval operations
• Measuring and improving AI output quality using LLM-based or code-based evaluators
• Iterating on prompts using real production examples and testing variations systematically
• Comparing different versions of AI applications (prompts, models, architectures) using experiments
• Monitoring LLM costs, token usage, latency, and error rates in production
• Building datasets from production traces for evaluation and fine-tuning
• Tracking multi-turn conversations and maintaining context across interactions
• Optimizing RAG systems by analyzing retrieval quality and document relevance
• Evaluating agent performance including tool call accuracy and actionability
• Managing prompt versions and deploying them across different environments

Capabilities

Agents can leverage Phoenix to:

• Trace AI application execution with detailed visibility into LLM calls, tool executions, retrieval operations, embeddings, and prompt templates
• Evaluate output quality using pre-built or custom evaluators with LLM-as-a-judge or code-based evaluation logic
• Annotate traces with human feedback, scores, labels, and quality signals for continuous improvement
• Experiment systematically by comparing different versions of applications using datasets and evaluators
• Monitor performance metrics including latency, token usage, costs, and error rates across projects
• Iterate on prompts using the playground, span replay, and dataset-based testing
• Organize traces into projects and sessions for better management and analysis
• Integrate with 20+ AI frameworks and LLM providers via OpenTelemetry instrumentation

Skills

Tracing

• Capture traces via OpenTelemetry (OTLP) protocol with automatic instrumentation for major frameworks
• View execution flow showing every LLM call, tool execution, retrieval operation, embedding generation, and response generation
• Inspect LLM parameters including temperature, system prompts, function calls, and invocation parameters
• Analyze retrieval operations with document scores, order, and embedding text for RAG systems
• Track token usage with detailed breakdowns by token type (input/output) and model
• Monitor latency at trace, span, and component levels with quantile analysis
• Organize with projects to separate traces by environment, application, team, or use case
• Group with sessions to track multi-turn conversations and maintain context across interactions
• Add metadata to traces with custom attributes, tags, and structured data for filtering and analysis
• Annotate traces with scores, labels, human feedback, and LLM evaluations for quality measurement
• Export and import traces for backup, migration, or analysis in external tools
• Track costs with automatic calculation based on token usage and model pricing

Evaluation

• Run LLM-as-a-judge evaluations using any LLM provider (OpenAI, Anthropic, Gemini, custom endpoints) to assess output quality
• Build custom evaluators with Python or TypeScript using custom prompts, scoring logic, and evaluation criteria
• Use pre-built evaluators for common tasks including faithfulness, relevance, toxicity, summarization, agent evaluation, and RAG quality
• Write code-based evaluators for deterministic checks like exact match, regex patterns, or custom Python/TypeScript logic
• Execute evaluations at scale with automatic concurrency, rate limit handling, error management, and batching via executors
• Map complex inputs using input schemas and mappings to transform nested data structures for evaluators
• View evaluator traces with complete transparency into prompts, model reasoning, scores, and execution metadata
• Run batch evaluations on traces, datasets, or custom data sources with automatic retry and error handling
• Integrate evaluations into workflows by running evals on production traces or test datasets

Datasets & Experiments

• Create datasets from traces, code, CSV files, or manually curated examples with inputs and optional reference outputs
• Build golden datasets with reference outputs (ground truth) for objective evaluation using code-based evaluators
• Version datasets with automatic tracking of inserts, updates, and deletes for reproducibility
• Run experiments by executing task functions against datasets with evaluators to compare different versions
• Compare experiments side-by-side in the UI to see performance differences, score distributions, and individual example results
• Use repetitions to run experiments multiple times for statistical confidence and account for LLM variability
• Organize with splits to separate datasets into train/test/validation splits for proper evaluation workflows
• Export datasets in JSONL or CSV formats for fine-tuning, analysis, or sharing
• View experiment results in the Phoenix UI with task function traces, scores per example, and aggregate performance metrics

Prompt Engineering

• Manage prompts with versioning, storage, and deployment across different environments
• Test prompts interactively in the Prompt Playground with various models, parameters, and tools
• Replay LLM spans from production traces in the playground to debug failures and test improvements
• Test at scale by running prompts against datasets to evaluate performance systematically
• Compare prompt versions side-by-side to see which performs better on your data
• Optimize automatically using automated prompt optimization features
• Sync prompts via SDK to keep prompts in sync across applications and environments programmatically
• Tag prompts for deployment control across development, staging, and production environments
• Track prompt changes with version history, author information, and timestamps

Projects & Organization

• Create projects to organize traces by environment (development, staging, production), application, or team
• Set up sessions to track multi-turn conversations with chatbot-like UI showing conversation history
• View metrics dashboards with pre-defined metrics including latency, errors, token usage, costs, and model performance
• Filter and search traces by metadata, attributes, annotations, or custom tags
• Configure data retention policies to control how long trace and evaluation data is stored

API & Programmatic Access

• Use Python SDK (arize-phoenix-client, arize-phoenix-evals, arize-phoenix-otel) for programmatic access
• Use TypeScript SDK (arizeai-phoenix-client, arizeai-phoenix-evals, arizeai-phoenix-otel) for JavaScript/TypeScript applications
• Access REST API for annotations, datasets, experiments, traces, spans, prompts, projects, and users
• Instrument manually using OpenTelemetry decorators, wrappers, or direct OpenInference SDKs
• Generate API keys for programmatic access with role-based permissions

Authentication & Security

• Configure RBAC with role-based access control for user permissions and project access
• Set up authentication including SSO and user management for self-hosted instances
• Manage API keys for secure programmatic access to Phoenix APIs and SDKs
• Control data privacy with self-hosting options for VPC deployment or local execution

Workflows

Workflow 1: Instrument and Trace an AI Application

1. Choose integration - Select appropriate Phoenix integration for your framework (LangChain, LlamaIndex, OpenAI, etc.)
2. Install package - Install Phoenix client and OpenTelemetry packages for your language (Python or TypeScript)
3. Configure endpoint - Set Phoenix endpoint URL and optionally configure project name and session tracking
4. Instrument application - Add auto-instrumentation or manual instrumentation to capture LLM calls, tool executions, and retrievals
5. View traces - Open Phoenix UI to see execution flow, latency, token usage, and detailed span information
6. Add annotations - Add scores, labels, or human feedback to traces for quality measurement

Workflow 2: Evaluate AI Output Quality

1. Choose evaluator type - Select LLM-as-a-judge for subjective quality or code-based for objective checks
2. Configure LLM provider - Set up evaluator LLM (OpenAI, Anthropic, Gemini, or custom endpoint)
3. Define evaluation logic - Use pre-built evaluator or create custom evaluator with prompts/scoring logic
4. Run evaluation - Execute evaluator on traces, datasets, or custom data with automatic batching and concurrency
5. Review results - View evaluator traces, scores, explanations, and labels in Phoenix UI
6. Iterate - Adjust evaluator prompts or logic based on results and human feedback

Workflow 3: Run Experiments to Compare Versions

1. Create dataset - Build dataset with inputs and optional reference outputs from traces, code, or CSV
2. Define task function - Create Python function that wraps your AI application logic and returns outputs
3. Select evaluators - Choose code-based evaluators for ground truth comparison or LLM-as-a-judge for subjective quality
4. Run experiment - Execute task function against dataset with evaluators to generate scores
5. Compare results - View experiment results in UI with aggregate metrics, score distributions, and per-example analysis
6. Iterate - Make changes to prompts, models, or architecture and run new experiment to compare performance

Workflow 4: Optimize Prompts with Playground

1. Identify prompt - Find prompt in traces or load existing prompt from prompt management
2. Open playground - Load prompt into Prompt Playground with current parameters and tools
3. Test variations - Modify prompt text, model parameters, tools, or response format and test with real inputs
4. View traces - All playground runs are automatically recorded as traces for analysis
5. Test at scale - Run prompt variations against dataset examples to evaluate performance systematically
6. Save and deploy - Save best-performing prompt version, tag for environment, and deploy via SDK

Workflow 5: Debug Production Issues

1. Identify problematic trace - Search or filter traces to find failed or low-quality executions
2. Inspect execution flow - View detailed span information including LLM calls, tool executions, and retrievals
3. Replay span - Load problematic LLM span into Prompt Playground to test fixes
4. Test improvements - Modify prompts, parameters, or tools in playground and compare outputs
5. Add to dataset - Add problematic examples to dataset for future testing
6. Run experiment - Test improved version against dataset to verify fix before deployment

Integrations

LLM Providers

OpenAI, Anthropic, Amazon Bedrock, Google (Gemini), Groq, MistralAI, VertexAI, LiteLLM, OpenRouter, Together, Vercel AI

Python Frameworks

AG2, Agno, AutoGen, BeeAI, CrewAI, DSPy, Google ADK, Graphite, Guardrails AI, Haystack, Hugging Face smolagents, Instructor, LlamaIndex, LangChain, LangGraph, MCP, NVIDIA, Portkey, Pydantic AI

TypeScript Frameworks

BeeAI, LangChain.js, Mastra, MCP, Vercel AI SDK

Java Frameworks

LangChain4j, Spring AI, Arconia

Platforms

Dify, Flowise, LangFlow, Prompt Flow

Vector Databases

MongoDB, OpenSearch, Pinecone, Qdrant, Weaviate, Zilliz/Milvus, Couchbase

Evaluation Integrations

Cleanlab, Ragas, UQLM

Observability Protocols

OpenTelemetry (OTLP), OpenInference

Developer Tools

Claude Code, Cursor, Phoenix MCP Server

Cloud Platforms

AWS (CloudFormation), Kubernetes (Helm), Docker, Railway

Context

OpenTelemetry: Phoenix tracing is built on OpenTelemetry (OTLP), an industry-standard observability protocol. This means instrumentation code written for Phoenix can be reused with other observability platforms, avoiding vendor lock-in.

OpenInference: Phoenix uses OpenInference instrumentation, an extension of OpenTelemetry specifically designed for AI/LLM applications. OpenInference adds semantic conventions for LLM spans, retrieval operations, and embeddings.

Traces and Spans: A trace represents the complete execution path of a request through an AI application. Spans are individual units of work within a trace (e.g., a single LLM call, tool execution, or retrieval operation). Spans can be nested to show hierarchical execution flow.

Projects: Projects provide organizational structure for traces, allowing separation by environment, application, or team. Each project has its own metrics dashboard and data isolation.

Sessions: Sessions group related traces into conversational threads, enabling tracking of multi-turn conversations with context maintained across interactions.

Evaluators: Evaluators measure the quality of AI outputs. LLM-based evaluators use LLMs as judges to assess subjective quality. Code-based evaluators use deterministic logic for objective checks. All evaluators return scores with optional labels, explanations, and metadata.

Datasets: Datasets are collections of examples with inputs and optional reference outputs. Golden datasets contain reference outputs (ground truth) for objective evaluation. Datasets are versioned automatically.

Experiments: Experiments run task functions (wrapped AI application logic) against datasets with evaluators to systematically compare different versions. Experiments track scores per example and aggregate metrics.

Prompts: In Phoenix, a prompt includes the prompt template, invocation parameters (temperature, etc.), tools, and response format. Prompts are versioned and can be tagged for deployment across environments.

Executors: Executors handle evaluation execution with automatic concurrency, rate limit management, error handling, and batching. They can achieve up to 20x speedup compared to direct API calls.

Self-Hosting: Phoenix can be self-hosted on Docker, Kubernetes, AWS, Railway, or locally. Self-hosted instances support authentication, email configuration, and data retention policies.

For additional documentation: https://arize.com/docs/phoenix/llms.txt