How do I install the Observe AI Production skill?

Run npx skillmds@latest add hiteshbandhu/observe-ai-production in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

Is Observe AI Production safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Observe AI Production?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Observe AI Production free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Observe AI Production?

hiteshbandhu (@hiteshbandhu) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Observe AI Production by hiteshbandhu

Name: Observe AI Production
Author: hiteshbandhu

Observe AI Production

Observes and improves production LLM apps with Arize Phoenix—tracing, layered agent evals, prompt learning loops, and PM/engineering eval pipelines. Use when the user mentions Arize, Phoenix, OpenInference, agent router evals, prompt optimization from traces, or shipping AI with production observability.

hiteshbandhu Updated Sep 17, 2026

File contents

Observe AI production (Arize)

Action playbook from six Arize @ AI Engineer talks. Do not summarize talks — pick a workflow and execute it.

Supporting files: workflows.md · source-index.md

Optional: ./skill-outputs/observe-ai-production/

Step 0 — Pick workflow

What is the user trying to do?
├─ Layered agent evals (router, skills, convergence)     → agent-evals
├─ Hands-on Phoenix agent eval workshop                  → phoenix-workshop
├─ Prompt learning from production traces                → prompt-learning
├─ PM-facing eval frameworks and release gates           → pm-evals
└─ Org-scale eval pipelines (CI, versioning)             → eval-pipelines

See workflows.md. Related: run-llm-evals for cross-vendor eval theory.

Install

cp -r skills/observe-ai-production ~/.claude/skills/
cp -r skills/observe-ai-production ~/.cursor/skills/
cp -r skills/observe-ai-production ~/.codex/skills/

Source: playlists/arize-ai-engineer/.

Cross-cutting rules

Rule	Source
Eval router decisions, not only final answers	[src-001 @ 0:08:21]
Trace every tool/LLM call for agent eval substrate	[src-002 @ 0:06:18]
Calibrate LLM judges on human labels before automation	[src-002 @ 0:89:37]
Close loop: observability → dataset → prompt patch	[src-003 @ 0:20:00]
Version datasets/scorers like production code	[src-005 @ 0:10:00]

Output

Name workflow; save artifacts to ./skill-outputs/observe-ai-production/ when requested; do not auto-commit.

---name: observe-ai-productiondescription: Observes and improves production LLM apps with Arize Phoenix—tracing, layered agent evals, prompt learning loops, and PM/engineering eval pipelines. Use when the user mentions Arize, Phoenix, OpenInference, agent router evals, prompt optimization from traces, or shipping AI with production observability.---# Observe AI production (Arize)Action playbook from six Arize @ AI Engineer talks. **Do not summarize talks** — pick a workflow and execute it.**Supporting files:** [workflows.md](workflows.md) · [source-index.md](source-index.md)Optional: `./skill-outputs/observe-ai-production/`---## Step 0 — Pick workflow```What is the user trying to do?├─ Layered agent evals (router, skills, convergence) → agent-evals├─ Hands-on Phoenix agent eval workshop → phoenix-workshop├─ Prompt learning from production traces → prompt-learning├─ PM-facing eval frameworks and release gates → pm-evals└─ Org-scale eval pipelines (CI, versioning) → eval-pipelines```See [workflows.md](workflows.md). Related: `run-llm-evals` for cross-vendor eval theory.---## Install```bashcp -r skills/observe-ai-production ~/.claude/skills/cp -r skills/observe-ai-production ~/.cursor/skills/cp -r skills/observe-ai-production ~/.codex/skills/```Source: `playlists/arize-ai-engineer/`.---## Cross-cutting rules| Rule | Source ||------|--------|| Eval router decisions, not only final answers | [src-001 @ 0:08:21] || Trace every tool/LLM call for agent eval substrate | [src-002 @ 0:06:18] || Calibrate LLM judges on human labels before automation | [src-002 @ 0:89:37] || Close loop: observability → dataset → prompt patch | [src-003 @ 0:20:00] || Version datasets/scorers like production code | [src-005 @ 0:10:00] |---## OutputName workflow; save artifacts to `./skill-outputs/observe-ai-production/` when requested; do not auto-commit.

Frequently asked questions