# Rosetta Test

> End-to-end test a Rosetta Stone framework × platform combination. Provisions a fresh isolated project on Arize AX or Phoenix, generates 25 synthetic traces, runs the 6 evals, verifies every trace was evaluated, then deletes the project. Trigger when the user asks to "test the <framework> <platform> project", "run e2e on <framework> <platform>", "verify <framework> works on <platform>", or any similar phrasing. Frameworks and platforms are discovered from the directory layout — no hardcoded list, so this works for any new framework added under ax/ or phoenix/.

- Skill: `arize-ai/rosetta-test` (Agent Skill)
- Install (CLI): `npx skillmds@latest add arize-ai/rosetta-test`
- Raw SKILL.md: https://api.skillmd.com/api/skills/arize-ai/rosetta-test/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: arize-ai (https://skillmd.com/u/arize-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/arize-ai/rosetta-test

---


# Rosetta Stone End-to-End Test

Orchestrates a full lifecycle test of one framework × platform combination from this repo.

## Inputs

Parse from the user's invocation:

- `<framework>` — e.g. `mastra`, `langchain-py`, `microsoft-agent-py`, `vercel-ai-sdk`, or any new framework directory
- `<platform>` — `ax` or `phoenix`
- `--keep` (optional) — skip the cleanup phase so the project remains for inspection

The skill is **generic**: it never hardcodes the framework list. New frameworks added to `ax/` or `phoenix/` work automatically as long as they follow the existing convention.

## Convention contract

A `<platform>/<framework>/` directory is testable if:

1. It exists.
2. Its `package.json` defines `synthetic-requests` as an npm script.
3. For `phoenix` only: `package.json` also defines `evals`.

Validate this before doing anything else. Abort early with a clear error if violated — list the directories that *are* available under that platform so the user can correct typos.

## Phases

Execute in order. Each phase has a dedicated skill file with the detailed steps — read it and follow.

1. **Setup** — `.claude/skills/rosetta-test-setup/SKILL.md`
   Validates the target dir + credentials, mints a unique project name, writes a `.env.test-local` overlay, pre-creates the AX project (Phoenix auto-creates on first trace).

2. **Traces** — `.claude/skills/rosetta-test-traces/SKILL.md`
   Runs `npm run synthetic-requests` with the env overlay loaded.

3. **Evals** — `.claude/skills/rosetta-test-evals/SKILL.md`
   Phoenix: runs `npm run evals`.
   AX: ensures shared `rosetta-e2e-*` evaluators exist (creates only the missing ones), then creates and triggers a per-run task scoped to the new project.

4. **Verify** — `.claude/skills/rosetta-test-verify/SKILL.md`
   Asserts 25 root traces exist and every trace has every applicable eval annotation. Reports pass/fail with details. Eval score values are not checked — only presence.

5. **Cleanup** — `.claude/skills/rosetta-test-cleanup/SKILL.md`
   Always runs (success or failure), **unless** the user passed `--keep`. Deletes the platform project and removes `.env.test-local`. If a prior phase aborted, still attempt cleanup.

## State to thread through phases

Setup emits these and every later phase needs them:

- `PROJECT_NAME` — the unique name (`rosetta-e2e-<framework>-<YYYYMMDDHHMM>-<rand4>`)
- `PROJECT_DIR` — absolute path to `<platform>/<framework>/`
- `PLATFORM` — `ax` or `phoenix`
- `ENV_OVERLAY` — absolute path to `<PROJECT_DIR>/.env.test-local`
- For AX: `ARIZE_SPACE_ID` (resolved from `.env.local`)

Carry these forward when invoking subsequent phases. Print them at the start of the run so the user can manually resume if something breaks.

## Failure handling

- Setup failure → no cleanup needed (nothing was created).
- Traces / Evals / Verify failure → still run cleanup unless `--keep`.
- Verify failure with `--keep` → leave the project so the user can inspect it in the UI.

Report verify results before invoking cleanup so the user sees the truth even if deletion succeeds.

## Output

End-of-run summary:

```
Rosetta E2E: <framework> on <platform>
  Project:  <PROJECT_NAME>
  Traces:   <count> / 25
  Evals:    <coverage report>
  Result:   PASS | FAIL
  Cleanup:  deleted | kept (--keep)
```