Maestro Jev

maestro-jev — goal-driven UI verification with Jev

Jev decides, Maestro executes, code owns the loop. One Jev request per step (goal_reached Noul + next_action Choice over code-enumerated candidates + speculative input_value Choice); each action runs as a Maestro flow. The driver talks to a persistent maestro mcp session (JSON-RPC over stdio): inspect_screen ≈0.2s and run ≈1s after a one-time driver warm-up, versus 10-18s per operation when the CLI is spawned per step.

Works as a skill in pi, Claude Code, and Codex: ./install.sh links it into ~/.pi/agent/skills, ~/.claude/skills, and ~/.codex/skills.

Two layers — pick the right one

The scripts drive Maestro through its own persistent MCP session (falling back to the CLI only if maestro mcp is unavailable). Your agent's MCP tool names may carry a server prefix (pi: maestro_run; Claude Code: mcp__maestro__run; Codex: run under the maestro server) — the scripts do not depend on them.

Prerequisites

• maestro CLI on PATH and a booted simulator/emulator (or chromium).
• A TypeSafe key in TYPESAFE_API_KEY (or JEV_API_KEY, or ~/.jev-router.env). Never hardcode it. Create one at https://console.typesafe.ai/keys.
• Node 20+.
• Check everything: node scripts/doctor.mjs

Fast path: verify a goal

cd <this skill directory>
node scripts/verify.mjs \
  --device <device-id-from-maestro_list_devices> \
  --app-id com.example.app \
  --goal "在登录页切换到验证码登录" \
  --criterion "页面显示验证码登录表单（出现验证码输入框或获取验证码控件）" \
  --emit-flow out/code-login.yaml

--criterion defaults to --goal; make the criterion something visible on screen. Values for typing are passed by name and never sent to Jev:

--inputs '{"email":"a@b.com","password":{"value":"wrong","desc":"a wrong password to test the error path"}}'

Output: the verdict JSON on stdout, human summary on stderr. Exit code 0 iff PASS.

The JSON also carries plan (source + instructions), segments (per-instruction verdict), failed_segment, goal_probability, witness (the element that proves the criterion), per-step steps with commands and effects, usage (Jev tokens), and artifacts (trace run.json, screenshot, emitted flow).

Latency: the first device operation warms up the iOS driver (~10-20s), then each step costs roughly 0.2s inspect + ~1s Jev + ~1s action — a typical verification runs in 30-60s including warm-up. --no-mcp falls back to spawning the CLI per step (10-18s per operation) for debugging or older Maestro builds. launchApp counts as the warm-up operation. Budgets: --max-steps 16 (total across all instructions), --max-seconds 300, --min-action-confidence 0.35 (below it the loop stops instead of guessing).

Compound goals

A goal with several sequential instructions runs as one session with a shared step budget: each instruction gets fresh completion evidence, and the run is PASS only if the final screen also satisfies the original acceptance criterion.

Split it yourself — no extra credentials, and the reliable choice when you are already an LLM:

node scripts/verify.mjs --device <id> --app-id com.example.app \
  --goal "切换到验证码登录并输入手机号" \
  --criterion "验证码登录表单显示且输入框中有 13800138000" \
  --steps '["切换到验证码登录", "在手机号输入框中输入 13800138000"]' \
  --inputs '{"phone":"13800138000"}'

Let maestro-jev split it (mirrors computer-use-jev; needs a splitter key):

MAESTRO_JEV_PROVIDER=openai MAESTRO_JEV_MODEL=gpt-4o-mini OPENAI_API_KEY=... \
  node scripts/verify.mjs --device <id> --goal "<compound goal>" --plan

--plan first asks Jev whether the goal is compound (Noul); single goals cost one extra cheap Jev call and need no provider. With a provider, the split happens once, outside the loop. Instructions are strings, or {goal, criterion} objects when an instruction needs a different acceptance criterion. All of it is recorded in the trace (plan, segments).

One-shot check and ad-hoc judgments

# Is the currently visible screen the expected one? (no actions taken)
node scripts/check.mjs --device <id> --criterion "显示验证码登录表单"

# Any TypeSafe question, using this skill's client
echo '{"id":{"type":"noul","instructions":"Is the button enabled?"}}' > /tmp/q.json
node scripts/judge.mjs --state /tmp/screen.json --questions /tmp/q.json

With Maestro MCP

The verification driver runs its own persistent maestro mcp session (that is why steps are fast). Separately, the agent's MCP tools remain for interactive work:

1. Your agent's Maestro MCP list_devices tool → pick a device_id.
2. Interactive exploration: inspect_screen / take_screenshot.
3. Let Jev judge MCP data: save the hierarchy JSON, then node scripts/judge.mjs --state <file> --questions <file>.
4. After verify returns PASS, open the saved screenshot (read the image) when the verification has consequences; PASS is a judgment, not a proof.
5. When verify returns UNCERTAIN/FAIL, use take_screenshot and inspect_screen to decide whether it is a bug or a test problem.

Emitting a regression flow

--emit-flow out/flow.yaml replays the successful action trace as a deterministic Maestro flow ending in an assertVisible on Jev's witness element (or assertWithAI on the criterion when no single witness exists). Validate and run it with maestro check-syntax out/flow.yaml / maestro test out/flow.yaml — or maestro_run through MCP. Review the header: the run only starts with launchApp when --app-id is set.

Tuning the judgments

All question text and thresholds live in lib/questions.mjs; the catalog with rationale is in references/questions.md. Thresholds: GOAL_YES = 0.8, GOAL_NO = 0.2, MIN_ACTION_CONFIDENCE = 0.35. Offline tests (no device, no key): node test/smoke.mjs.

Safety and secrets

• Input values are never sent to Jev — only key names and descriptions. Local artifacts (trace run.json, emitted flow) do contain the typed values, so treat the artifacts directory as sensitive when typing real credentials.
• Without --inputs the loop can only tap, scroll, and navigate — pass inputs when the goal requires typing.
• The action set is tap / type / scroll / back / wait / hideKeyboard / launchApp. No destructive commands are generated.
• One run per device: a lock file in the temp dir (delete it or use --no-lock if a run died). Artifacts land in ./.maestro-jev/.
• --relaunch --clear-state wipes app data — don't use it when login state matters.
• Android: inputText does not support Unicode characters; WebViews may expose a sparse hierarchy, in which case expect UNCERTAIN and verify visually.