arize-ai
- 50 skills
- 0 followers
- 8 repo stars
- last month last updated
- ▌ Check Models 2 · arize-ai bundleFind and update out-of-date OpenAI and Anthropic model references (in docs, MDX, notebooks, and code) to the latest size-equivalent models, and apply the code changes each new generation requires (e.g. max_tokens → max_completion_tokens for GPT-5). Use when asked to "check the models", "update model versions", "are these models current", "migrate the models in the docs/tutorials", or before publishing content that names a model.
- ▌ Check Models · arize-ai bundleFind and update out-of-date OpenAI and Anthropic model references (in docs, MDX, notebooks, and code) to the latest size-equivalent models, and apply the code changes each new generation requires (e.g. max_tokens → max_completion_tokens for GPT-5). Use when asked to "check the models", "update model versions", "are these models current", "migrate the models in the docs/tutorials", or before publishing content that names a model.
- ▌
- ▌ Phoenix Tracing · arize-ai bundleOpenInference semantic conventions and instrumentation for Phoenix AI observability. Use when implementing LLM tracing, creating custom spans, or deploying to production.
- ▌ Debug Setup · arize-aiUse when validating Claude Code telemetry setup for this collector, especially to confirm Claude emits OTLP data to a local file before testing the configured upstream destination.
- ▌ Debug Live Collector · arize-aiUse when testing an already-running Claude collector by sending a live marker trace to its configured OTLP endpoint without starting a second collector.
- ▌ Context Graph Apply · arize-aiINVOKE THIS SKILL when translating a context-graph-mining report into an experiment variant for the procurement-agent. Reads the report's proposed diffs and writes a variant config bundle to `experiments/variants/<id>/` (manifest.yaml + system_prompt.txt + optional vendors.json + departments.json). Does NOT modify procurement-agent source — runtime parameterization means the variant is purely additive config. Use after `context-graph-mining` produces a report; output feeds `procurement-experiment run-experiment`.
- ▌ Context Graph Mining · arize-aiINVOKE THIS SKILL when mining the procurement-agent Arize project for patterns in agent decisions vs human overrides, building a context graph, and proposing updates to procurement-agent rules. Use after a batch of reviews has been captured (process.run + override.run spans with annotations) to surface (a) where the agent's recommendations systematically diverge from reviewer overrides, (b) tribal knowledge embedded in reviewer reasoning that should become explicit rules, and (c) concrete code-change proposals against the procurement-agent codebase.
- ▌ Arize Datasets · arize-aiManage datasets in Arize AI using the ax CLI. Use when users want to list datasets, get dataset details, create new datasets, delete datasets, export dataset data, or work with dataset examples. Triggers on "list datasets", "create dataset", "ax datasets", "export dataset", "delete dataset", or any request about managing Arize datasets via CLI.
- ▌ Arize Projects · arize-aiManage projects in Arize AI using the ax CLI. Use when users want to list projects, get project details, create new projects, delete projects, or organize work within Arize spaces. Triggers on "list projects", "create project", "ax projects", "delete project", or any request about managing Arize projects via CLI.
- ▌ Setup Claude Code Tracing · arize-aiSet up and configure Arize tracing for Claude Code sessions or Agent SDK applications. Use when users want to set up tracing, configure Arize AX or Phoenix, create a new Arize project, get an API key, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up tracing", "configure Arize", "configure Phoenix", "enable tracing", "setup-claude-code-tracing", "create Arize project", "get Arize API key", "agent sdk tracing", or any request about connecting Claude Code or the Agent SDK to Arize or Phoenix for observability.
- ▌ Code Review · arize-aiUse before opening a pull request to this repo. Reviews the working diff for correctness bugs and repo conventions, runs the test/lint gates, and can post findings directly on the PR. Triggers on "review my changes", "review this PR", "pre-PR review".
- ▌ Phoenix CLI · arize-ai bundleDebug LLM applications using the Phoenix CLI. Fetch traces, analyze errors, structure trace review with open coding and axial coding, inspect datasets, review experiments, query annotation configs, and use the GraphQL API. Use whenever the user is analyzing traces or spans, investigating LLM/agent failures, deciding what to do after instrumenting an app, building failure taxonomies, choosing what evals to write, or asking "what's going wrong", "what kinds of mistakes", or "where do I focus" — even without naming a technique.
- ▌ Manage Omp Tracing · arize-aiSet up and configure Arize tracing for Oh My Pi (omp) terminal coding sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for Oh My Pi, enable/disable omp tracing, or troubleshoot tracing issues. Triggers on "set up omp tracing", "configure Arize for Oh My Pi", "configure Phoenix for omp", "enable omp tracing", "setup-omp-tracing", or any request about connecting omp / Oh My Pi to Arize or Phoenix for observability.
- ▌ Manage Kiro Tracing · arize-aiSet up and configure Arize tracing for Kiro CLI sessions. Use when users want to set up Kiro tracing, configure Arize AX or Phoenix for Kiro, enable/disable tracing, choose or set a default traced agent, or troubleshoot Kiro tracing issues. Triggers on "set up kiro tracing", "configure Arize for Kiro", "configure Phoenix for Kiro", "enable kiro tracing", "setup-kiro-tracing", "kiro agent tracing", or any request about connecting Kiro CLI to Arize or Phoenix for observability.
- ▌ Manage Codex Tracing · arize-aiSet up and configure Arize tracing for OpenAI Codex CLI sessions. Use when users want to set up Codex tracing, configure Arize AX or Phoenix for Codex, enable/disable tracing, or troubleshoot Codex tracing issues. Triggers on "set up codex tracing", "configure Arize for Codex", "configure Phoenix for Codex", "enable codex tracing", "setup-codex-tracing", or any request about connecting Codex to Arize or Phoenix for observability.
- ▌ Manage Cursor Tracing · arize-aiSet up and configure Arize tracing for Cursor IDE sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for Cursor, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up cursor tracing", "configure Arize for Cursor", "configure Phoenix for Cursor", "enable cursor tracing", "setup-cursor-tracing", or any request about connecting Cursor to Arize or Phoenix for observability.
- ▌ Manage Gemini Tracing · arize-aiSet up and configure Arize tracing for Gemini CLI sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for Gemini, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up gemini tracing", "configure Arize for Gemini", "configure Phoenix for Gemini", "enable gemini tracing", "setup-gemini-tracing", or any request about connecting Gemini CLI to Arize or Phoenix for observability.
- ▌ Manage Copilot Tracing · arize-aiSet up and configure Arize tracing for GitHub Copilot sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for Copilot, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up copilot tracing", "configure Arize for Copilot", "configure Phoenix for Copilot", "enable copilot tracing", "setup-copilot-tracing", or any request about connecting GitHub Copilot to Arize or Phoenix for observability.
- ▌ Manage Opencode Tracing · arize-aiSet up and configure Arize tracing for opencode terminal coding sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for opencode, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up opencode tracing", "configure Arize for opencode", "configure Phoenix for opencode", "enable opencode tracing", "setup-opencode-tracing", or any request about connecting opencode to Arize or Phoenix for observability.
- ▌ Manage Antigravity Tracing · arize-aiSet up and configure Arize tracing for Google Antigravity CLI/IDE sessions. Use when users want to set up tracing, configure Arize AX or Phoenix for Antigravity, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up antigravity tracing", "configure Arize for Antigravity", "configure Phoenix for Antigravity", "enable antigravity tracing", "setup-antigravity-tracing", or any request about connecting the Antigravity CLI/IDE to Arize or Phoenix for observability.
- ▌ Manage Claude Code Tracing · arize-aiSet up and configure Arize tracing for Claude Code sessions or Agent SDK applications. Use when users want to set up tracing, configure Arize AX or Phoenix, create a new Arize project, get an API key, enable/disable tracing, or troubleshoot tracing issues. Triggers on "set up tracing", "configure Arize", "configure Phoenix", "enable tracing", "setup-claude-code-tracing", "create Arize project", "get Arize API key", "agent sdk tracing", or any request about connecting Claude Code or the Agent SDK to Arize or Phoenix for observability.
- ▌ Rosetta Aaaj · arize-aiAttach Agent-as-a-Judge (AaaJ / harness) evaluators to any Arize AX Wonder Toys project and trigger a one-time backfill. Reads scoring instructions from evals/aaaj/*.md and creates evaluators + tasks via GraphQL (REST and the ax CLI cannot create harness evals). Use when the user asks to add AaaJ, agent-as-a-judge, harness evals, or to score AX traces without the UI. AX-only; not part of rosetta-test e2e. Default PROJECT_DIR is ax/langchain-py; any ax/<framework> with .env.local works.
- ▌ Rosetta Test · arize-aiEnd-to-end test a Rosetta Stone framework × platform combination. Provisions a fresh isolated project on Arize AX or Phoenix, generates 25 synthetic traces, runs the 6 evals, verifies every trace was evaluated, then deletes the project. Trigger when the user asks to "test the <framework> <platform> project", "run e2e on <framework> <platform>", "verify <framework> works on <platform>", or any similar phrasing. Frameworks and platforms are discovered from the directory layout — no hardcoded list, so this works for any new framework added under ax/ or phoenix/.
- ▌ Rosetta Test Evals · arize-aiRun the 6 Wonder Toys evals against a Rosetta Stone test project. Phoenix path runs the built-in `npm run evals` script. AX path ensures the stable space-level rosetta-e2e-* evaluators exist (creating only missing ones from the templates in evals/README.md), then creates and triggers a per-run eval task scoped to the project. Part of the rosetta-test e2e flow; can also be invoked standalone.
- ▌ Rosetta Test Setup · arize-aiProvision a fresh Rosetta Stone test project on Arize AX or Phoenix. Validates the target framework directory and credentials, mints a unique project name, writes a sibling .env.test-local overlay so the real .env.local is untouched, and pre-creates the AX project (Phoenix auto-creates on first trace). Part of the rosetta-test e2e flow; can also be invoked standalone.
- ▌ Rosetta Test Traces · arize-aiRun the 25 synthetic Wonder Toys requests against a Rosetta Stone framework to generate traces in an isolated Arize AX or Phoenix project. Loads the .env.test-local overlay so the project name override takes effect without touching .env.local. Part of the rosetta-test e2e flow; can also be invoked standalone if setup already ran.
- ▌ Rosetta Test Verify · arize-aiVerify a Rosetta Stone test project has 25 traces and every trace has every expected eval annotation. Presence-only check — eval scores are not inspected. Reports a per-trace coverage matrix and pass/fail. Part of the rosetta-test e2e flow; can also be invoked standalone after evals have completed.
- ▌ Rosetta Demo Capture · arize-aiRecord a full Wonder Toys demo by running a canned 3-turn conversation (search dragons → buy plushie → ship), then opening Arize AX in Safari and screenshotting the session view plus every trace in it. Use when the user asks to "capture a demo", "record screenshots of an Arize session", "demo the agent flow", or any similar phrasing. macOS only — uses AppleScript and `screencapture`.
- ▌ Rosetta Test Cleanup · arize-aiTear down a Rosetta Stone test project — deletes the platform project on Arize AX or Phoenix, removes the .env.test-local overlay, and kills any leftover dev server / ChromaDB processes started by the run. Idempotent and safe to call multiple times. Use the --keep flag at the orchestrator level to skip this entirely. Part of the rosetta-test e2e flow; can also be invoked standalone to clean up after a crashed run.
- ▌ Rosetta Add Framework · arize-aiAdd a new agent framework to the Rosetta Stone repo — researches the framework, builds all three observability tiers (no-observability, phoenix, ax), tests each, runs a Playwright smoke against the UI, updates README + TODO, and raises a PR. Trigger when the user asks to "add the <framework> framework", "implement <framework>", "wire up <framework>", or similar. The framework must be one of the Arize-supported agent frameworks (see TODO below).
- ▌ Rosetta Pr Screenshots · arize-aiCapture AX trace UI, Phoenix trace UI, and Wonder Toys app UI screenshots for a framework's PR, then upload as GitHub release assets and embed in the PR body. Called automatically by rosetta-add-framework-docs as part of new-framework PRs; can also be invoked standalone to retrofit existing PRs. Cross-platform — uses Playwright end-to-end.
- ▌ Rosetta Add Framework Docs · arize-aiFinalise a newly-added framework — updates the README's supported-frameworks table, directory tree, and per-framework "what differs" section, marks off the framework in the orchestrator skill's embedded TODO, commits per tier, and raises a PR. Part of the rosetta-add-framework flow.
- ▌ Rosetta Add Framework Discover · arize-aiRefresh the list of agent frameworks supported by Arize tracing and diff against what's already in the repo. Pulls live data from https://arize.com/docs/llms.txt and produces a clean to-do list. Part of the rosetta-add-framework flow; can also be invoked standalone to answer "what frameworks are left to add?"
- ▌ Rosetta Add Framework Tier Test · arize-aiTest a freshly-built tier for a new framework — boots the backend, smoke-tests the chat endpoint, runs synthetic requests, and (for phoenix) runs the eval harness. Verifies traces land in the right project. Part of the rosetta-add-framework flow; can be invoked standalone after a build to validate.
- ▌ Rosetta Add Framework Playwright · arize-ai bundleRun a public-flow Playwright smoke test against a freshly-built framework tier's Next.js frontend. Covers home page rendering and product browsing. The Playwright project (package.json, config, tests) lives inside this skill directory and is checked into the repo. Part of the rosetta-add-framework flow.
- ▌ Rosetta Add Framework Tier Build · arize-aiBuild a single tier (no-observability, phoenix, or ax) for a new framework. Clones the closest existing tier, swaps in framework-specific agent.py / tools.py / requirements.txt, and (for observability tiers) adds tracing.py + main.py wiring + eval-harness scripts. Part of the rosetta-add-framework flow; can be invoked standalone to rebuild a single tier from scratch.
- ▌ Arize Link · arize-ai bundleGenerates deep links to the Arize UI for projects, traces, spans, sessions, datasets, labeling queues, evaluators, and annotation configs. Discovers organization and project IDs with the ax CLI and produces clickable URLs for sharing Arize resources with team members. Use when the user wants to link to or open a project, trace, span, session, dataset, evaluator, or annotation config in the Arize UI.
- ▌ Arize Admin · arize-ai bundleManages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML/SSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys.
- ▌ Arize Trace · arize-ai bundleDownloads, exports, and inspects existing Arize traces and spans to understand what an LLM app is doing or debug runtime issues. Covers exporting traces by ID, spans by ID, sessions by ID, and root-cause investigation using the ax CLI. Use when the user wants to look at existing trace data, see what their LLM app is doing, export traces, download spans, investigate errors, or analyze behavior regressions.
- ▌ Arize Dataset · arize-ai bundleCreates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set.
- ▌ Arize Prompts · arize-ai bundleINVOKE THIS SKILL for Arize Prompt Hub and `ax prompts` workflows: author or import templates and save (Workflows A–B), label/promote (C), or list/get/edit/delete/duplicate (D). Use when the user mentions ax prompts, Prompt Hub, creating/editing/saving a prompt, `{variable}` placeholders, or production/staging labels. For improving prompt text using traces or eval scores, use arize-prompt-optimization. For running experiments, use arize-experiment.
- ▌ Arize Evaluator · arize-ai bundleHandles LLM-as-judge and code evaluator workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
- ▌ Arize Annotation · arize-ai bundleCreates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review.
- ▌ Arize Experiment · arize-ai bundleCreates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.
- ▌ Arize Span Routing · arize-ai bundleUse when one Python service must send each agent's, tenant's, team's, or request's spans to its correct Arize space and project using application metadata. Covers dynamic OpenTelemetry routing for custom agent builders and multi-tenant applications, including register_with_routing, set_routing_context, multi-space tracing, and custom span routing.
- ▌ Arize Instrumentation · arize-ai bundleAdds Arize AX tracing to an LLM application for the first time. Detects the stack, routes to the single matching integration doc, wires auto-instrumentation after user confirmation, and verifies traces land. Use when the user wants to instrument their app, add tracing from scratch, set up LLM observability, integrate OpenTelemetry or openinference, or get started with Arize tracing.
- ▌ Arize Prompt Optimization · arize-ai bundleOptimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.
- ▌ Arize Instrumentation Health · arize-ai bundleAudits instrumentation health of existing Arize traces. Runs deterministic checks over a bounded span sample (orphaned/uncategorized/duplicate spans, flat structure, blank root I/O, unset status, missing token counts or children) and returns a ranked report. Use when the user asks why traces look empty/flat/broken, wants to verify instrumentation is healthy, find instrumentation issues, or why evals or token/cost dashboards show n/a or zero. To debug app behavior or errors, use arize-trace.
- ▌ Arize AI Provider Integration · arize-ai bundleCreates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize.